<div dir="ltr">FYI - I've opened a ticket on the RH bugzilla (<a href="https://bugzilla.redhat.com/show_bug.cgi?id=966150">https://bugzilla.redhat.com/show_bug.cgi?id=966150</a>) against the fence_agents component.<br>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, May 22, 2013 at 12:00 PM, John McCabe <span dir="ltr"><<a href="mailto:john@johnmccabe.net" target="_blank">john@johnmccabe.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">No joy with ipport sadly<div><br></div><div><nvpair id="st-rhevm-instance_attributes-ipport" name="ipport" value="443"/><br>
</div><div><nvpair id="st-rhevm-instance_attributes-shell_timeout" name="shell_timeout" value="10"/> <br>
</div><div><br></div><div>Can you share the changes you made to fence_rhevm for the API change? I've got what *should* be the latest packages from the HA channel on both systems.</div></div><div class="HOEnZb"><div class="h5">
<div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, May 22, 2013 at 11:34 AM, Andrew Beekhof <span dir="ltr"><<a href="mailto:andrew@beekhof.net" target="_blank">andrew@beekhof.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div><br>
On 22/05/2013, at 7:31 PM, John McCabe <<a href="mailto:john@johnmccabe.net" target="_blank">john@johnmccabe.net</a>> wrote:<br>
<br>
> Hi,<br>
> I've been trying to get fence_rhevm (fence-agents-3.1.5-25.el6_4.2.x86_64) working within pacemaker (pacemaker-1.1.8-7.el6.x86_64) but am unable to get it to work as intended, using fence_rhevm on the command line works as expected, as does stonith_admin but from within pacemaker (triggered by deliberately killing corosync on the node to be fenced):<br>
><br>
> May 21 22:21:32 defiant corosync[1245]: [TOTEM ] A processor failed, forming new configuration.<br>
> May 21 22:21:34 defiant corosync[1245]: [QUORUM] Members[1]: 1<br>
> May 21 22:21:34 defiant corosync[1245]: [TOTEM ] A processor joined or left the membership and a new membership was formed.<br>
> May 21 22:21:34 defiant kernel: dlm: closing connection to node 2<br>
> May 21 22:21:34 defiant corosync[1245]: [CPG ] chosen downlist: sender r(0) ip(10.10.25.152) ; members(old:2 left:1)<br>
> May 21 22:21:34 defiant corosync[1245]: [MAIN ] Completed service synchronization, ready to provide service.<br>
> May 21 22:21:34 defiant crmd[1749]: notice: crm_update_peer_state: cman_event_callback: Node enterprise[2] - state is now lost<br>
> May 21 22:21:34 defiant crmd[1749]: warning: match_down_event: No match for shutdown action on enterprise<br>
> May 21 22:21:34 defiant crmd[1749]: notice: peer_update_callback: Stonith/shutdown of enterprise not matched<br>
> May 21 22:21:34 defiant crmd[1749]: notice: do_state_transition: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=check_join_state ]<br>
> May 21 22:21:34 defiant fenced[1302]: fencing node enterprise<br>
> May 21 22:21:34 defiant logger: fence_pcmk[2219]: Requesting Pacemaker fence enterprise (reset)<br>
> May 21 22:21:34 defiant stonith_admin[2220]: notice: crm_log_args: Invoked: stonith_admin --reboot enterprise --tolerance 5s<br>
> May 21 22:21:35 defiant attrd[1747]: notice: attrd_local_callback: Sending full refresh (origin=crmd)<br>
> May 21 22:21:35 defiant attrd[1747]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<br>
> May 21 22:21:36 defiant pengine[1748]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
> May 21 22:21:36 defiant pengine[1748]: notice: process_pe_message: Calculated Transition 64: /var/lib/pacemaker/pengine/pe-input-60.bz2<br>
> May 21 22:21:36 defiant crmd[1749]: notice: run_graph: Transition 64 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-60.bz2): Complete<br>
> May 21 22:21:36 defiant crmd[1749]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br>
> May 21 22:21:44 defiant logger: fence_pcmk[2219]: Call to fence enterprise (reset) failed with rc=255<br>
> May 21 22:21:45 defiant fenced[1302]: fence enterprise dev 0.0 agent fence_pcmk result: error from agent<br>
> May 21 22:21:45 defiant fenced[1302]: fence enterprise failed<br>
> May 21 22:21:48 defiant fenced[1302]: fencing node enterprise<br>
> May 21 22:21:48 defiant logger: fence_pcmk[2239]: Requesting Pacemaker fence enterprise (reset)<br>
> May 21 22:21:48 defiant stonith_admin[2240]: notice: crm_log_args: Invoked: stonith_admin --reboot enterprise --tolerance 5s<br>
> May 21 22:21:58 defiant logger: fence_pcmk[2239]: Call to fence enterprise (reset) failed with rc=255<br>
> May 21 22:21:58 defiant fenced[1302]: fence enterprise dev 0.0 agent fence_pcmk result: error from agent<br>
> May 21 22:21:58 defiant fenced[1302]: fence enterprise failed<br>
> May 21 22:22:01 defiant fenced[1302]: fencing node enterprise<br>
><br>
> and with corosync.log showing "warning: match_down_event: No match for shutdown action on enterprise", "notice: peer_update_callback: Stonith/shutdown of enterprise not matched"<br>
><br>
> May 21 22:21:32 corosync [TOTEM ] A processor failed, forming new configuration.<br>
> May 21 22:21:34 corosync [QUORUM] Members[1]: 1<br>
> May 21 22:21:34 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.<br>
> May 21 22:21:34 [1749] defiant crmd: info: cman_event_callback: Membership 296: quorum retained<br>
> May 21 22:21:34 [1744] defiant cib: info: pcmk_cpg_membership: Left[5.0] cib.2<br>
> May 21 22:21:34 [1744] defiant cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node enterprise[2] - corosync-cpg is now offline<br>
> May 21 22:21:34 [1744] defiant cib: info: pcmk_cpg_membership: Member[5.0] cib.1<br>
> May 21 22:21:34 [1745] defiant stonith-ng: info: pcmk_cpg_membership: Left[5.0] stonith-ng.2<br>
> May 21 22:21:34 [1745] defiant stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node enterprise[2] - corosync-cpg is now offline<br>
> May 21 22:21:34 corosync [CPG ] chosen downlist: sender r(0) ip(10.10.25.152) ; members(old:2 left:1)<br>
> May 21 22:21:34 corosync [MAIN ] Completed service synchronization, ready to provide service.<br>
> May 21 22:21:34 [1745] defiant stonith-ng: info: pcmk_cpg_membership: Member[5.0] stonith-ng.1<br>
> May 21 22:21:34 [1749] defiant crmd: notice: crm_update_peer_state: cman_event_callback: Node enterprise[2] - state is now lost<br>
> May 21 22:21:34 [1749] defiant crmd: info: peer_update_callback: enterprise is now lost (was member)<br>
> May 21 22:21:34 [1744] defiant cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/150, version=0.22.3): OK (rc=0)<br>
> May 21 22:21:34 [1749] defiant crmd: info: pcmk_cpg_membership: Left[5.0] crmd.2<br>
> May 21 22:21:34 [1749] defiant crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node enterprise[2] - corosync-cpg is now offline<br>
> May 21 22:21:34 [1749] defiant crmd: info: peer_update_callback: Client enterprise/peer now has status [offline] (DC=true)<br>
> May 21 22:21:34 [1749] defiant crmd: warning: match_down_event: No match for shutdown action on enterprise<br>
> May 21 22:21:34 [1749] defiant crmd: notice: peer_update_callback: Stonith/shutdown of enterprise not matched<br>
> May 21 22:21:34 [1749] defiant crmd: info: crm_update_peer_expected: peer_update_callback: Node enterprise[2] - expected state is now down<br>
> May 21 22:21:34 [1749] defiant crmd: info: abort_transition_graph: peer_update_callback:211 - Triggered transition abort (complete=1) : Node failure<br>
> May 21 22:21:34 [1749] defiant crmd: info: pcmk_cpg_membership: Member[5.0] crmd.1<br>
> May 21 22:21:34 [1749] defiant crmd: notice: do_state_transition: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=check_join_state ]<br>
> May 21 22:21:34 [1749] defiant crmd: info: abort_transition_graph: do_te_invoke:163 - Triggered transition abort (complete=1) : Peer Halt<br>
> May 21 22:21:34 [1749] defiant crmd: info: join_make_offer: Making join offers based on membership 296<br>
> May 21 22:21:34 [1749] defiant crmd: info: do_dc_join_offer_all: join-7: Waiting on 1 outstanding join acks<br>
> May 21 22:21:34 [1749] defiant crmd: info: update_dc: Set DC to defiant (3.0.7)<br>
> May 21 22:21:34 [1749] defiant crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]<br>
> May 21 22:21:34 [1749] defiant crmd: info: do_dc_join_finalize: join-7: Syncing the CIB from defiant to the rest of the cluster<br>
> May 21 22:21:34 [1744] defiant cib: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/154, version=0.22.5): OK (rc=0)<br>
> May 21 22:21:34 [1744] defiant cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/155, version=0.22.6): OK (rc=0)<br>
> May 21 22:21:34 [1749] defiant crmd: info: stonith_action_create: Initiating action metadata for agent fence_rhevm (target=(null))<br>
> May 21 22:21:35 [1749] defiant crmd: info: do_dc_join_ack: join-7: Updating node state to member for defiant<br>
> May 21 22:21:35 [1749] defiant crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='defiant']/lrm<br>
> May 21 22:21:35 [1744] defiant cib: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='defiant']/lrm (origin=local/crmd/156, version=0.22.7): OK (rc=0)<br>
> May 21 22:21:35 [1749] defiant crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]<br>
> May 21 22:21:35 [1749] defiant crmd: info: abort_transition_graph: do_te_invoke:156 - Triggered transition abort (complete=1) : Peer Cancelled<br>
> May 21 22:21:35 [1747] defiant attrd: notice: attrd_local_callback: Sending full refresh (origin=crmd)<br>
> May 21 22:21:35 [1747] defiant attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<br>
> May 21 22:21:35 [1744] defiant cib: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/158, version=0.22.9): OK (rc=0)<br>
> May 21 22:21:35 [1744] defiant cib: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/160, version=0.22.11): OK (rc=0)<br>
> May 21 22:21:36 [1748] defiant pengine: info: unpack_config: Startup probes: enabled<br>
> May 21 22:21:36 [1748] defiant pengine: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
> May 21 22:21:36 [1748] defiant pengine: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0<br>
> May 21 22:21:36 [1748] defiant pengine: info: unpack_domains: Unpacking domains<br>
> May 21 22:21:36 [1748] defiant pengine: info: determine_online_status_fencing: Node defiant is active<br>
> May 21 22:21:36 [1748] defiant pengine: info: determine_online_status: Node defiant is online<br>
> May 21 22:21:36 [1748] defiant pengine: info: native_print: st-rhevm (stonith:fence_rhevm): Started defiant<br>
> May 21 22:21:36 [1748] defiant pengine: info: LogActions: Leave st-rhevm (Started defiant)<br>
> May 21 22:21:36 [1748] defiant pengine: notice: process_pe_message: Calculated Transition 64: /var/lib/pacemaker/pengine/pe-input-60.bz2<br>
> May 21 22:21:36 [1749] defiant crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<br>
> May 21 22:21:36 [1749] defiant crmd: info: do_te_invoke: Processing graph 64 (ref=pe_calc-dc-1369171296-118) derived from /var/lib/pacemaker/pengine/pe-input-60.bz2<br>
> May 21 22:21:36 [1749] defiant crmd: notice: run_graph: Transition 64 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-60.bz2): Complete<br>
> May 21 22:21:36 [1749] defiant crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br>
><br>
><br>
> I can get the node enterprise to fence as expected from the command line with:<br>
><br>
> stonith_admin --reboot enterprise --tolerance 5s<br>
><br>
> fence_rhevm -o reboot -a <hypervisor ip> -l <user>@<domain> -p <password> -n enterprise -z<br>
><br>
> My config is as follows:<br>
><br>
> cluster.conf -----------------------------------<br>
><br>
> <?xml version="1.0"?><br>
> <cluster config_version="1" name="cluster"><br>
> <logging debug="off"/><br>
> <clusternodes><br>
> <clusternode name="defiant" nodeid="1"><br>
> <fence><br>
> <method name="pcmk-redirect"><br>
> <device name="pcmk" port="defiant"/><br>
> </method><br>
> </fence><br>
> </clusternode><br>
> <clusternode name="enterprise" nodeid="2"><br>
> <fence><br>
> <method name="pcmk-redirect"><br>
> <device name="pcmk" port="enterprise"/><br>
> </method><br>
> </fence><br>
> </clusternode><br>
> </clusternodes><br>
> <fencedevices><br>
> <fencedevice name="pcmk" agent="fence_pcmk"/><br>
> </fencedevices><br>
> <cman two_node="1" expected_votes="1"><br>
> </cman><br>
> </cluster><br>
><br>
> pacemaker cib ---------------------------------<br>
><br>
> Stonith device created with:<br>
><br>
> pcs stonith create st-rhevm fence_rhevm login="<user>@<domain>" passwd="<password>" ssl=1 ipaddr="<hypervisor ip>" verbose=1 debug="/tmp/debug.log"<br>
><br>
><br>
> <cib epoch="18" num_updates="88" admin_epoch="0" validate-with="pacemaker-1.2" update-origin="defiant" update-client="cibadmin" cib-last-written="Tue May 21 07:55:31 2013" crm_feature_set="3.0.7" have-quorum="1" dc-uuid="defiant"><br>
> <configuration><br>
> <crm_config><br>
> <cluster_property_set id="cib-bootstrap-options"><br>
> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.8-7.el6-394e906"/><br>
> <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="cman"/><br>
> <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/><br>
> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="true"/><br>
> </cluster_property_set><br>
> </crm_config><br>
> <nodes><br>
> <node id="defiant" uname="defiant"/><br>
> <node id="enterprise" uname="enterprise"/><br>
> </nodes><br>
> <resources><br>
> <primitive class="stonith" id="st-rhevm" type="fence_rhevm"><br>
> <instance_attributes id="st-rhevm-instance_attributes"><br>
> <nvpair id="st-rhevm-instance_attributes-login" name="login" value="<user>@<domain>"/><br>
> <nvpair id="st-rhevm-instance_attributes-passwd" name="passwd" value="<password>"/><br>
> <nvpair id="st-rhevm-instance_attributes-debug" name="debug" value="/tmp/debug.log"/><br>
> <nvpair id="st-rhevm-instance_attributes-ssl" name="ssl" value="1"/><br>
> <nvpair id="st-rhevm-instance_attributes-verbose" name="verbose" value="1"/><br>
> <nvpair id="st-rhevm-instance_attributes-ipaddr" name="ipaddr" value="<hypervisor ip>"/><br>
> </instance_attributes><br>
> </primitive><br>
<br>
</div></div>Mine is:<br>
<br>
<primitive id="Fencing" class="stonith" type="fence_rhevm"><br>
<instance_attributes id="Fencing-params"><br>
<nvpair id="Fencing-ipport" name="ipport" value="443"/><br>
<nvpair id="Fencing-shell_timeout" name="shell_timeout" value="10"/><br>
<nvpair id="Fencing-passwd" name="passwd" value="{pass}"/><br>
<nvpair id="Fencing-ipaddr" name="ipaddr" value="{ip}"/><br>
<nvpair id="Fencing-ssl" name="ssl" value="1"/><br>
<nvpair id="Fencing-login" name="login" value="{user}@{domain}"/><br>
</instance_attributes><br>
<operations><br>
<op id="Fencing-monitor-120s" interval="120s" name="monitor" timeout="120s"/><br>
<op id="Fencing-stop-0" interval="0" name="stop" timeout="60s"/><br>
<op id="Fencing-start-0" interval="0" name="start" timeout="60s"/><br>
</operations><br>
</primitive><br>
<br>
Maybe ipport is important?<br>
Also, there was a RHEVM API change recently, I had to update the fence_rhevm agent before it would work again.<br>
<div><div><br>
> </resources><br>
> <constraints/><br>
> </configuration><br>
> <status><br>
> <node_state id="defiant" uname="defiant" in_ccm="true" crmd="online" crm-debug-origin="do_state_transition" join="member" expected="member"><br>
> <transient_attributes id="defiant"><br>
> <instance_attributes id="status-defiant"><br>
> <nvpair id="status-defiant-probe_complete" name="probe_complete" value="true"/><br>
> </instance_attributes><br>
> </transient_attributes><br>
> <lrm id="defiant"><br>
> <lrm_resources><br>
> <lrm_resource id="st-rhevm" type="fence_rhevm" class="stonith"><br>
> <lrm_rsc_op id="st-rhevm_last_0" operation_key="st-rhevm_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.7" transition-key="2:1:0:1e7972e8-6f9a-4325-b9c3-3d7e2950d996" transition-magic="0:0;2:1:0:1e7972e8-6f9a-4325-b9c3-3d7e2950d996" call-id="14" rc-code="0" op-status="0" interval="0" last-run="1369119332" last-rc-change="0" exec-time="232" queue-time="0" op-digest="3bc7e1ce413fe37998a289f77f85d159"/><br>
> </lrm_resource><br>
> </lrm_resources><br>
> </lrm><br>
> </node_state><br>
> <node_state id="enterprise" uname="enterprise" in_ccm="true" crmd="online" crm-debug-origin="do_update_resource" join="member" expected="member"><br>
> <lrm id="enterprise"><br>
> <lrm_resources><br>
> <lrm_resource id="st-rhevm" type="fence_rhevm" class="stonith"><br>
> <lrm_rsc_op id="st-rhevm_last_0" operation_key="st-rhevm_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.7" transition-key="5:59:7:8170c498-f66b-4974-b3c0-c17eb45ba5cb" transition-magic="0:7;5:59:7:8170c498-f66b-4974-b3c0-c17eb45ba5cb" call-id="5" rc-code="7" op-status="0" interval="0" last-run="1369170800" last-rc-change="0" exec-time="4" queue-time="0" op-digest="3bc7e1ce413fe37998a289f77f85d159"/><br>
> </lrm_resource><br>
> </lrm_resources><br>
> </lrm><br>
> <transient_attributes id="enterprise"><br>
> <instance_attributes id="status-enterprise"><br>
> <nvpair id="status-enterprise-probe_complete" name="probe_complete" value="true"/><br>
> </instance_attributes><br>
> </transient_attributes><br>
> </node_state><br>
> </status><br>
> </cib><br>
><br>
><br>
> The debug log output from fence_rhevm doesn't appear to show pacemaker trying to request the reboot, only a vms command sent to the hypervisor which responds with xml listing the VMs.<br>
><br>
> I can't quite see why its failing? Are you aware of any issues with fence_rhevm (fence-agents-3.1.5-25.el6_4.2.x86_64) not working with pacemaker (pacemaker-1.1.8-7.el6.x86_64) on RHEL6.4?<br>
><br>
> All the best,<br>
> /John<br>
</div></div>> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>