<br><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div>Date: Mon, 23 Jul 2012 12:16:20 +0200</div><div>From: Andreas Kurz </div><div><br></div><div>On 07/23/2012 07:06 AM, David Barchas wrote:</div><blockquote type="cite"><div><div>Hello.</div><div><br></div><div>I have been working on this for 3 days now, and must be so stressed out</div><div>that I am being blinded to what is probably an obvious cause of this. In</div><div>a word, HELP.</div><div><br></div><div>I am trying specifically to utilize ocf:heartbeat:IPaddr2, but this</div><div>issue seems to occur with any of the ocf:heartbeat agents. I will just</div><div>focus on IPaddr2 for purposes of figuring this out, but it happens</div><div>exactly the same with any of the default agents. However, I can</div><div>successfully use ocf:linbit:drbd for example. it seems to be limited to</div><div>the RAs that are installed along with coro/pace in the resource-agents</div><div>package.</div></div></blockquote><div><br></div><div>What are the exact package versions you have installed?</div><div><br></div><div>pacemaker*</div><div>resource-agents</div><div>cluster-glue*</div><div><br></div></span></blockquote><div>bah, all the info i provide and miss that.</div><div><div>clusterlib-3.0.12.1-32.el6.x86_64</div><div>cluster-glue-1.0.5-6.el6.x86_64</div><div>cluster-glue-libs-1.0.5-6.el6.x86_64</div></div><div><div>pacemaker-cli-1.1.7-6.el6.x86_64</div><div>pacemaker-libs-1.1.7-6.el6.x86_64</div><div>pacemaker-cluster-libs-1.1.7-6.el6.x86_64</div><div>pacemaker-1.1.7-6.el6.x86_64</div></div><div>resource-agents-3.9.2-12.el6.x86_64</div><div><br></div><div>my full rpm -qa just in case its helpful http://pastebin.com/d2y7Sii4 </div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div></div><div><br></div><blockquote type="cite"><div><div><br></div><div>I am using CentOS 6.3, fully updated (though this happens in 6.2 with no</div><div>updates as well). Install pacemaker/coro from default repo. I have</div><div>stripped everything down to figure this out in vmware and just install</div><div>centos, update it, install pace/coro (no drbd for this discussion),</div><div>configure coro, and then start it. pacemaker starts up fine (or at least</div><div>I think its fine). I can set quorum ignore for example from crm. (crm</div><div>configure property no-quorum-policy="ignore")</div><div><br></div><div>here is the process list</div><div>root 1447 0.3 0.6 556080 6636 ? Ssl 21:09 0:00 corosync</div><div>499 1453 0.0 0.5 88720 5556 ? S 21:09 0:00 \_</div><div>/usr/libexec/pacemaker/cib</div><div>root 1454 0.0 0.3 86968 3488 ? S 21:09 0:00 \_</div><div>/usr/libexec/pacemaker/stonithd</div><div>root 1455 0.0 0.2 76188 2492 ? S 21:09 0:00 \_</div><div>/usr/lib64/heartbeat/lrmd</div><div>499 1456 0.0 0.3 91160 3432 ? S 21:09 0:00 \_</div><div>/usr/libexec/pacemaker/attrd</div><div>499 1457 0.0 0.3 87440 3824 ? S 21:09 0:00 \_</div><div>/usr/libexec/pacemaker/pengine</div><div>499 1458 0.0 0.3 91312 3884 ? S 21:09 0:00 \_</div><div>/usr/libexec/pacemaker/crmd</div></div></blockquote><div><br></div><div>so you are using plugin version 0 to start Pacemaker .... That would</div><div>explain why /etc/init.d/pacemaker is unable to start ... it is already</div><div>started by Corosync.</div></span></blockquote><div>i mostly included that info "just in case" and because its confusing to me that I can't start pacemaker even from fresh install before configuring or starting corosync. </div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><br></div><blockquote type="cite"><div><div><br></div><div>499 is hacluster btw.</div><div><br></div><div>***BUT***</div><div><br></div><div>When I run as root the following:</div><div># crm ra meta ocf:heartbeat:IPaddr2</div><div><br></div><div>I get this response:</div><div>lrmadmin[1484]: 2012/07/22_13:28:23 ERROR:</div><div>lrm_get_rsc_type_metadata(578): got a return code HA_FAIL from a reply</div><div>message of rmetadata with function get_ret_from_msg.</div><div>ERROR: ocf:heartbeat:IPaddr2: could not parse meta-data: </div><div><br></div><div>And this is in /var/log/messages:</div><div>Jul 22 16:35:14 MST lrmd: [48093]: ERROR: get_resource_meta: pclose</div><div>failed: Resource temporarily unavailable</div><div>Jul 22 16:35:14 MST lrmd: [48093]: WARN: on_msg_get_metadata: empty</div><div>metadata for ocf::heartbeat::IPaddr2.</div><div>Jul 22 16:35:14 MST lrmd: [48093]: WARN: G_SIG_dispatch: Dispatch</div><div>function for SIGCHLD was delayed 200 ms (> 100 ms) before being called</div><div>(GSource: 0x187df10)</div><div>Jul 22 16:35:14 MST lrmd: [48093]: info: G_SIG_dispatch: started at</div><div>429616889 should have started at 429616869</div><div>Jul 22 16:35:14 MST lrmadmin: [48254]: ERROR:</div><div>lrm_get_rsc_type_metadata(578): got a return code HA_FAIL from a reply</div><div>message of rmetadata with function get_ret_from_msg.</div><div><br></div><div>I am using crm ra meta as a way to test, but crm will not accept my</div><div>trying to add the resource as a primitive either.</div><div><br></div><div>In my research, I have found that often it's permissions. So just to</div><div>rule that out i set my entire system to 777 permissions. no joy.</div><div><br></div><div>Another suggestion i find often has been to set OCF_ROOT (export</div><div>OCF_ROOT=/usr/lib/ocf) and then do</div><div>/usr/lib/ocf/resource.d/heartbeat/IPaddr2 meta-data.</div><div>That produces the desired output. But does not work before i export. </div><div>And CRM still does not accept my meta request </div><div><br></div><div>Another suggestion i find is to make sure that shellfuncs exists in the</div><div>agents folder. the soft links exist</div><div>lrwxrwxrwx. 1 root root 32 Jul 22 04:08 .ocf-binaries -></div><div>../../lib/heartbeat/ocf-binaries</div><div>lrwxrwxrwx. 1 root root 35 Jul 22 04:08 .ocf-directories -></div><div>../../lib/heartbeat/ocf-directories</div><div>lrwxrwxrwx. 1 root root 35 Jul 22 04:08 .ocf-returncodes -></div><div>../../lib/heartbeat/ocf-returncodes</div><div>lrwxrwxrwx. 1 root root 34 Jul 22 04:08 .ocf-shellfuncs -></div><div>../../lib/heartbeat/ocf-shellfuncs</div><div><br></div><div>And just to make sure I did un-hidden soft links as well with no joy.</div></div></blockquote><div><br></div><div>Strange, that errors are typically related to wrong paths for</div><div>initialization of environment and helper functions:</div><div><br></div><div># Initialization:</div><div><br></div><div>: ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat}</div><div>. ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs</div><div><br></div><div>DRBD agent has an extra failback check, that may be the reason that it</div><div>still works ...</div><div><br></div><div># Resource-agents have moved their ocf-shellfuncs file around.</div><div># There are supposed to be symlinks or wrapper files in the old location,</div><div># pointing to the new one, but people seem to get it wrong all the time.</div><div># Try several locations.</div><div><br></div><div>if test -n "${OCF_FUNCTIONS_DIR}" ; then</div><div> if test -e "${OCF_FUNCTIONS_DIR}/ocf-shellfuncs" ; then</div><div> . "${OCF_FUNCTIONS_DIR}/ocf-shellfuncs"</div><div> elif test -e "${OCF_FUNCTIONS_DIR}/.ocf-shellfuncs" ; then</div><div> . "${OCF_FUNCTIONS_DIR}/.ocf-shellfuncs"</div><div> fi</div><div>else</div><div> if test -e "${OCF_ROOT}/lib/heartbeat/ocf-shellfuncs" ; then</div><div> . "${OCF_ROOT}/lib/heartbeat/ocf-shellfuncs"</div><div> elif test -e "${OCF_ROOT}/resource.d/heartbeat/.ocf-shellfuncs"; then</div><div> . "${OCF_ROOT}/resource.d/heartbeat/.ocf-shellfuncs"</div><div> fi</div><div>fi</div><div><br></div></span></blockquote><div>I noticed this as well, and I tried updating the IPaddr2 agent to use the directory code from DRBD (what you have above) with no success either.</div><div>Though, pace was already running. I assume it doesn't load all the agents into ram and never read them again. Instead executing them when needed. So no caching issue.</div><div>i am going to try that again though because it really does sound like it could fix it. Though not explain why its busted in the first place. right now i'll take a hack though if it works. Pretty sure it won't though.</div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div></div><div><br></div><div>Regards,</div><div>Andreas</div></span></blockquote><div>thanks for the help. greatly appreciated.
</div>