<div dir="ltr">Hi Dejan,<div><br></div><div>Thanks a lot for your input!</div><div><br></div><div>I cherry picked this commit & this solves the problem. I will raise a ubuntu launchpad bug for them to pull this correction in trusty stable. Thanks!</div><div><br></div><div>--Shyam</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 15, 2015 at 1:21 PM, Dejan Muhamedagic <span dir="ltr"><<a href="mailto:dejanmm@fastmail.fm" target="_blank">dejanmm@fastmail.fm</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<div><div class="h5"><br>
On Wed, Oct 14, 2015 at 10:31:40PM +0530, Shyam wrote:<br>
> Hi all,<br>
><br>
> We were previously using Pacemaker 1.1.6 on Ubuntu Precise. We are shifting<br>
> to Ubuntu Trusty & it has Pacemaker 1.1.10.<br>
><br>
> We have our own STONITH resource & with the pacemaker upgrade this is<br>
> broken. The problem that I am hitting is not something new & is exactly<br>
> that was reported in<br>
> <a href="http://clusterlabs.org/pipermail/users/2015-June/000686.html" rel="noreferrer" target="_blank">http://clusterlabs.org/pipermail/users/2015-June/000686.html</a><br>
><br>
> Basically Hearbeat + Pacemaker stack suffers this problem (even with the<br>
> github mainline code) & since a lot of people are using corosync I think<br>
> the problem is not felt with.<br>
><br>
> The basic problem is this. Between Pacemaker 1.1.6 to 1.1.10, the following<br>
> change was done<br>
><br>
> fencing/remote.c<br>
><br>
> static void remote_op_done(remote_fencing_op_t * op, xmlNode * data, int<br>
> rc, int dup)<br>
><br>
> if (dup == FALSE && safe_str_neq(subt, "broadcast")) {<br>
> /* Defer notification until the bcast message arrives */<br>
> bcast_result_to_peers(op, rc);<br>
> goto remote_op_done_cleanup;<br>
><br>
> remote_op_done() previously used to send a direct reply. Now it was changed<br>
> that it does a broadcast of the result & this broadcast is supposed to come<br>
> back to the same fencing agent upon which finishes the operation. However<br>
> this broadcast that is being sent by stonithd doesnt reach back itself.<br>
> This causes eventually crmd to timeout on stonith_async_timeout_handler()<br>
> to be triggered & keep retrying STONITH. so essentially STONITH is broken.<br>
<br>
</div></div>I can vaguelly recall some stonith/heartbeat issue which got<br>
fixed by Lars:<br>
<br>
commit cc34288a7b2276aa238546f4aa92fa79b8cbcf88<br>
Author: Lars Ellenberg <<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>><br>
Date: Tue Dec 9 10:48:33 2014 +0100<br>
<br>
Sounds like the same problem.<br>
<br>
Thanks,<br>
<br>
Dejan<br>
<div><div class="h5"><br>
<br>
> I tried setting up bcast in /etc/ha.d/<a href="http://ha.cf" rel="noreferrer" target="_blank">ha.cf</a>, but it doesnt help. To confirm<br>
> the issue, I removed the above flow for bcast_result_to_peers() & then<br>
> STONITH works perfectly fine. Any pointers/help appreciated on how to<br>
> resolve this issue in the right way?<br>
><br>
><br>
> From logs<br>
><br>
> CRMD issues STONITH reboot to stonith-ng<br>
><br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG: Dumping message with<br>
> 19 fields<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[0] :<br>
> [__name__=stonith_command]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[1] :<br>
> [__name__=stonith_command]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[2] : [t=stonith-ng]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[3] :<br>
> [st_async_id=617dce02-057f-42b8-9df2-33bc4e64780d]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[4] : [st_op=st_fence]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[5] : [st_callid=2]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[6] : [st_callopt=0]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[7] :<br>
> [st_remote_op=617dce02-057f-42b8-9df2-33bc4e64780d]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[8] : [st_target=node1]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[9] :<br>
> [st_device_action=reboot]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[10] : [st_origin=node0]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[11] :<br>
> [st_clientid=346fea6b-a55d-4873-b630-14287d5bc71e]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[12] :<br>
> [st_clientname=crmd.14483]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[13] : [st_timeout=90]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[14] : [st_mode=smart]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[15] : [dest=node0]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[16] : [oseq=2]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[17] :<br>
> [from_id=stonith-ng]<br>
> Oct 14 14:57:01 node0 heartbeat: [14463]: debug: MSG[18] :<br>
> [to_id=stonith-ng]<br>
><br>
><br>
> stonith-ng successfully finishes the STONITH<br>
> Oct 14 14:57:03 node0 stonith-ng[14481]: notice: log_operation: Operation<br>
> 'reboot' [14657] (call 2 from crmd.14483) for host 'node1' with device<br>
> 'Z-FENCE:0' returned: 0 (OK)<br>
> Oct 14 14:57:03 node0 stonith-ng[14481]: debug:<br>
> stonith_send_async_reply: Directed reply to node0<br>
> Oct 14 14:57:03 node0 stonith-ng[14481]: debug: send_ha_message:<br>
> outbound <st-reply st_origin="node0" t="stonith-ng" st_op="st_fence"<br>
> st_device_id="Z-FENCE:0"<br>
> st_remote_op="617dce02-057f-42b8-9df2-33bc4e64780d"<br>
> st_clientid="346fea6b-a55d-4873-b630-14287d5bc71e"<br>
> st_clientname="crmd.14483" st_target="node1" st_device_action="reboot"<br>
> st_callid="2" st_callopt="0" st_rc="0" st_output="Performing: stonith -t<br>
> external/zstonith -T reset node1\nsuccess: node1 0\n"/><br>
><br>
><br>
> & stonith-ng sends a self message for st-reply<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG: Dumping message with<br>
> 18 fields<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[0] :<br>
> [__name__=st-reply]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[1] : [st_origin=node0]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[2] : [t=stonith-ng]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[3] : [st_op=st_fence]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[4] :<br>
> [st_device_id=Z-FENCE:0]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[5] :<br>
> [st_remote_op=617dce02-057f-42b8-9df2-33bc4e64780d]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[6] :<br>
> [st_clientid=346fea6b-a55d-4873-b630-14287d5bc71e]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[7] :<br>
> [st_clientname=crmd.14483]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[8] : [st_target=node1]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[9] :<br>
> [st_device_action=reboot]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[10] : [st_callid=2]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[11] : [st_callopt=0]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[12] : [st_rc=0]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[13] :<br>
> [st_output=Performing: stonith -t external/zstonith -T reset<br>
> node1#012success: node1 0#012]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[14] : [dest=node0]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[15] : [oseq=3]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[16] :<br>
> [from_id=stonith-ng]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[17] :<br>
> [to_id=stonith-ng]<br>
><br>
><br>
> As part of this stonith-ng does a broadcast<br>
> Oct 14 14:57:03 node0 stonith-ng[14481]: debug: stonith_command:<br>
> Processing st_fence reply 0 from node0 ( 0)<br>
> Oct 14 14:57:03 node0 stonith-ng[14481]: debug: remote_op_done:<br>
> Broadcasting result<br>
> Oct 14 14:57:03 node0 stonith-ng[14481]: debug: send_ha_message:<br>
> outbound <st-reply t="st_notify" subt="broadcast" st_op="st_notify"<br>
> count="1"><br>
> Oct 14 14:57:03 node0 stonith-ng[14481]: debug: send_ha_message:<br>
> outbound <st_calldata><br>
> Oct 14 14:57:03 node0 stonith-ng[14481]: debug: send_ha_message:<br>
> outbound <st_notify_fence state="2" st_rc="0" st_target="node1"<br>
> st_device_action="reboot" st_delegate="node0"<br>
> st_remote_op="617dce02-057f-42b8-9df2-33bc4e64780d" st_origin="node0"<br>
> st_clientid="346fea6b-a55d-4873-b630-14287d5bc71e"<br>
> st_clientname="crmd.14483"/><br>
> Oct 14 14:57:03 node0 stonith-ng[14481]: debug: send_ha_message:<br>
> outbound </st_calldata><br>
> Oct 14 14:57:03 node0 stonith-ng[14481]: debug: send_ha_message:<br>
> outbound </st-reply><br>
><br>
><br>
> that is realized at hearbeat<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG: Dumping message with<br>
> 8 fields<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[0] :<br>
> [__name__=st-reply]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[1] : [t=st_notify]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[2] : [subt=broadcast]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[3] : [st_op=st_notify]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[4] : [count=1]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[5] :<br>
> [st_calldata=<st_calldata><st_notify_fence state="2" st_rc="0"<br>
> st_target="node1" st_device_action="reboot" st_delegate="node0"<br>
> st_remote_op="617dce02-057f-42b8-9df2-33bc4e64780d" st_origin="node0"<br>
> st_clientid="346fea6b-a55d-4873-b630-14287d5bc71e"<br>
> st_clientname="crmd.14483"/></st_calldata>]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[6] :<br>
> [from_id=stonith-ng]<br>
> Oct 14 14:57:03 node0 heartbeat: [14463]: debug: MSG[7] : [to_id=stonith-ng]<br>
><br>
> but then on nothing really happens.<br>
><br>
> eventually crmd gives up on timeout<br>
> Oct 14 14:59:48 node0 crmd[14483]: error: stonith_async_timeout_handler:<br>
> Async call 2 timed out after 168000ms<br>
><br>
> Thanks.<br>
><br>
> --Shyam<br>
<br>
</div></div>> _______________________________________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> <a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div><br></div>