<div dir="ltr"><div><div><div>Hi all,</div><div><br></div><div>I've faced a problem when a node is not actually rebooted in case a resource fails to stop on it.</div><div>A fence agent is a self-written. And it works in case of network outage and all other cases.</div><div>I went through all the logs on both nodes and I couldn't understand why node-0 is not actually rebooted.</div><div>I would be appreciated for some help here.</div><div>Bellow are two brief (most interesting from my point) snips from the logs.</div><div>I also attached log files and screenshot of "crm_mon" on node-1.</div><div> </div><div><br></div><div>The problem:</div><div>-------------------</div><div>"stop" action for "sm1dh" fails on "node-0", and "node-0" is not actually rebooted by "node-1".</div></div><div><br></div><div><br></div><div>The setup:</div><div>---------------</div><div><div>There are two nodes: node-0,node-1</div><div><br></div><div><br></div><div>Fence agents are configured with:</div><div>-------------------------------------</div><div>crm configure primitive STONITH_node-1 stonith:fence_avid_sbb_hw</div><div>crm configure primitive STONITH_node-0 stonith:fence_avid_sbb_hw \</div><div><span class="" style="white-space:pre"> </span>params delay="10"</div><div><br></div><div>crm configure location dont_run_STONITH_node-1_on_node-1 STONITH_node-1 -inf: node-1</div><div>crm configure location dont_run_STONITH_node-0_on_node-0 STONITH_node-0 -inf: node-0</div></div><div><br></div><div><br></div><div>Few lines from the vim /var/log/cluster/corosync.log on "node-0":<br></div><div>-------------------------------------------------------------------------------------------</div><div>Feb 10 19:29:40 [3204] isis-seth943f pengine: info: native_print: sm1dh (ocf::avid:diskHelper): FAILED node-0<br></div><div>...</div><div>Feb 10 19:29:40 [3201] isis-seth943f stonithd: notice: handle_request: <span class="" style="white-space:pre"> </span>Client crmd.3205.09022f74 wants to fence (reboot) 'node-0' with device '(any)'<br></div><div><div>Feb 10 19:29:40 [3201] isis-seth943f stonithd: notice: initiate_remote_stonith_op: <span class="" style="white-space:pre"> </span>Initiating remote operation reboot for node-0: 51063a89-0df0-4dd7-8f22-667ca5db05f0 (0)</div><div>Feb 10 19:29:41 [3201] isis-seth943f stonithd: info: process_remote_stonith_query: <span class="" style="white-space:pre"> </span>Query result 2 of 2 from node-1 for node-0/reboot (1 devices) 51063a89-0df0-4dd7-8f22-667ca5db05f0</div></div><div>...<br></div><div>Feb 10 19:29:51 [3205] isis-seth943f crmd: crit: tengine_stonith_notify: <span class="" style="white-space:pre"> </span>We were alegedly just fenced by node-1 for node-0!<br></div><div>...<br></div><div><div>Feb 10 19:29:51 [3198] isis-seth943f pacemakerd: error: pcmk_child_exit: <span class="" style="white-space:pre"> </span>Child process crmd (3205) exited: Network is down (100)</div><div>Feb 10 19:29:51 [3198] isis-seth943f pacemakerd: warning: pcmk_child_exit: <span class="" style="white-space:pre"> </span>Pacemaker child process crmd no longer wishes to be respawned. Shutting ourselves down</div></div><div><br></div><div><br></div><div>Few lines from the vim /var/log/cluster/corosync.log on "node-1":<br></div><div>-------------------------------------------------------------------------------------------</div><div>Feb 10 19:28:15 [3184] isis-seth944b stonithd: notice: log_operation: <span class="" style="white-space:pre"> </span>Operation 'reboot' [4596] (call 2 from crmd.3205) for host 'node-0' with device 'STONITH_node-0' returned: 0 (OK)</div><div>Feb 10 19:28:15 [3184] isis-seth944b stonithd: warning: get_xpath_object: <span class="" style="white-space:pre"> </span>No match for //@st_delegate in /st-reply</div><div>Feb 10 19:28:15 [3184] isis-seth944b stonithd: notice: remote_op_done: <span class="" style="white-space:pre"> </span>Operation reboot of node-0 by node-1 for crmd.3205@node-0.51063a89: OK</div><div>Feb 10 19:28:15 [3188] isis-seth944b crmd: notice: tengine_stonith_notify: <span class="" style="white-space:pre"> </span>Peer node-0 was terminated (reboot) by node-1 for node-0: OK (ref=51063a89-0df0-4dd7-8f22-667ca5db05f0) by client crmd.3205</div></div><div><br></div><div><br></div><div>Time difference between the nodes (sorry for that):</div><div>------------------------------------------------------------------------</div><div>node-0: t</div><div>node-1: t - 97 seconds</div><div><br></div><br clear="all"><div><div class="gmail_signature"><div dir="ltr">Thank you,<div>Kostya</div></div></div></div>
</div>