<div dir="ltr">Hi Andrew,<div><br></div><div><span style="font-size:12.8000001907349px">>> Recent versions allow this depending on what the configured fencing devices report.</span><br></div><div><span style="font-size:12.8000001907349px">So the device should be able/configured/report that it can STONITH the node on which it is running?</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">>> </span><span style="font-size:12.8000001907349px">You left out “but the devices reports that it did”. Your fencing agent needs to report the truth</span></div><div><span style="font-size:12.8000001907349px">Yes, that was a hole in the configuration - I didn't specify </span><span style="font-size:12.8000001907349px">pcmk_host_list="node-1" and pcmk_host_check="static-list".</span></div><div><span style="font-size:12.8000001907349px">But the "</span><span style="font-size:12.8000001907349px">safety check</span><span style="font-size:12.8000001907349px">" that you mentioned before worked perfect, so I didn't notice my mistake in the configuration.</span></div><div><br></div><div>Now I see that I shouldn't rely on the "<span style="font-size:12.8000001907349px">safety check" and should have a proper configuration for STONITH.</span></div><div><span style="font-size:12.8000001907349px">The thing is -</span><span style="font-size:12.8000001907349px"> I am trying to understand the way I should modify my config.</span></div><div><span style="font-size:12.8000001907349px">Using two-node cluster solutions it is possible to have only one node running, and my current STONITH agents, with "</span><span style="font-size:12.8000001907349px">pcmk_host_list" and "pcmk_host_check",</span><span style="font-size:12.8000001907349px"> won't work.</span></div><div><span style="font-size:12.8000001907349px">Even more, it will lead to the situation when a started cluster couldn't find a device to reboot itself (say "stop" failed) remains working while must be STONITHed.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">Could you help me to find the best way to "self-stonithing"?</span></div><div><span style="font-size:12.8000001907349px">Will it be sufficient to create another stonith agent with will issue "reboot -f"?</span></div><div><span style="font-size:12.8000001907349px"><br></span></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr">Thank you,<div>Kostya</div></div></div></div>
<br><div class="gmail_quote">On Mon, Aug 17, 2015 at 1:15 AM, Andrew Beekhof <span dir="ltr"><<a href="mailto:andrew@beekhof.net" target="_blank">andrew@beekhof.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
> On 13 Aug 2015, at 9:39 pm, Kostiantyn Ponomarenko <<a href="mailto:konstantin.ponomarenko@gmail.com">konstantin.ponomarenko@gmail.com</a>> wrote:<br>
><br>
> Hi,<br>
><br>
> Brief description of the STONITH problem:<br>
><br>
> I see two different behaviors with two different STONITH configurations. If Pacemaker cannot find a device that can STONITH a problematic node, the node remains up and running. Which is bad, because it must be STONITHed.<br>
> As opposite to it, if Pacemaker finds a device that, it thinks, can STONITH a problematic node, even if the device actually cannot,<br>
<br>
</span>You left out “but the devices reports that it did”. Your fencing agent needs to report the truth.<br>
<span class=""><br>
> Pacemaker goes down after STONITH returns false positive. The Pacemaker shutdowns itself right after STONITH.<br>
> Is it the expected behavior?<br>
<br>
</span>Yes, its a safety check:<br>
<div class="HOEnZb"><div class="h5"><br>
Aug 11 16:09:53 [9009] A6-4U24-402-T crmd: crit: tengine_stonith_notify: We were alegedly just fenced by node-0 for node-0!<br>
<br>
<br>
> Do I need to configure a two more STONITH agents for just rebooting nodes on which they are running (e.g. with # reboot -f)?<br>
><br>
><br>
><br>
> +-------------------------<br>
> + Set-up:<br>
> +-------------------------<br>
> - two node cluster (node-0 and node-1);<br>
> - two fencing (STONITH) agents are configured (STONITH_node-0 and STONITH_node-1).<br>
> - "STONITH_node-0" runs only on "node-1" // this fencing agent can only fence node-0<br>
> - "STONITH_node-1" runs only on "node-0" // this fencing agent can only fence node-1<br>
><br>
> +-------------------------<br>
> + Environment:<br>
> +-------------------------<br>
> - one node - "node-0" - is up and running;<br>
> - one STONITH agent - "STONITH_node-1" - is up and running<br>
><br>
> +-------------------------<br>
> + Test case:<br>
> +-------------------------<br>
> Simulate error of stopping a resource.<br>
> 1. start cluster<br>
> 2. change a RA's script to return "$OCF_ERR_GENERIC" from "Stop" function.<br>
> 3. stop the resource by "# crm resource stop <resource>"<br>
><br>
> +-------------------------<br>
> + Actual behavior:<br>
> +-------------------------<br>
><br>
> CASE 1:<br>
> STONITH is configured with:<br>
> # crm configure primitive STONITH_node-1 stonith:fence_sbb_hw \<br>
> params pcmk_host_list="node-1" pcmk_host_check="static-list"<br>
><br>
> After issuing a "stop" command:<br>
> - the resource changes its state to "FAILED"<br>
> - Pacemaker remains working<br>
><br>
> See below LOG_snippet_1 section.<br>
><br>
><br>
> CASE 2:<br>
> STONITH is configured with:<br>
> # crm configure primitive STONITH_node-1 stonith:fence_sbb_hw<br>
><br>
> After issuing a "stop" command:<br>
> - the resource changes its state to "FAILED"<br>
> - Pacemaker stops working<br>
><br>
> See below LOG_snippet_2 section.<br>
><br>
><br>
> +-------------------------<br>
> + LOG_snippet_1:<br>
> +-------------------------<br>
> Aug 12 16:42:47 [39206] A6-4U24-402-T stonithd: notice: handle_request: Client crmd.39210.fa40430f wants to fence (reboot) 'node-0' with device '(any)'<br>
> Aug 12 16:42:47 [39206] A6-4U24-402-T stonithd: notice: initiate_remote_stonith_op: Initiating remote operation reboot for node-0: 18cc29db-b7e4-4994-85f1-df891f091a0d (0)<br>
> ....<br>
> Aug 12 16:42:47 [39206] A6-4U24-402-T stonithd: notice: can_fence_host_with_device: STONITH_node-1 can not fence (reboot) node-0: static-list<br>
> ....<br>
> Aug 12 16:42:47 [39206] A6-4U24-402-T stonithd: notice: stonith_choose_peer: Couldn't find anyone to fence node-0 with <any><br>
> Aug 12 16:42:47 [39206] A6-4U24-402-T stonithd: info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node node-0 for crmd.39210.18cc29db<br>
> Aug 12 16:42:47 [39206] A6-4U24-402-T stonithd: info: call_remote_stonith: None of the 1 peers have devices capable of terminating node-0 for crmd.39210 (0)<br>
> ....<br>
> Aug 12 16:42:47 [39206] A6-4U24-402-T stonithd: warning: get_xpath_object: No match for //@st_delegate in /st-reply<br>
> Aug 12 16:42:47 [39206] A6-4U24-402-T stonithd: error: remote_op_done: Operation reboot of node-0 by node-0 for crmd.39210@node-0.18cc29db: No such device<br>
> ....<br>
> Aug 12 16:42:47 [39210] A6-4U24-402-T crmd: notice: tengine_stonith_callback: Stonith operation 3/23:16:0:0856a484-6b69-4280-b93f-1af9a6a542ee: No such device (-19)<br>
> Aug 12 16:42:47 [39210] A6-4U24-402-T crmd: notice: tengine_stonith_callback: Stonith operation 3 for node-0 failed (No such device): aborting transition.<br>
> Aug 12 16:42:47 [39210] A6-4U24-402-T crmd: info: abort_transition_graph: Transition aborted: Stonith failed (source=tengine_stonith_callback:697, 0)<br>
> Aug 12 16:42:47 [39210] A6-4U24-402-T crmd: notice: tengine_stonith_notify: Peer node-0 was not terminated (reboot) by node-0 for node-0: No such device<br>
><br>
><br>
> +-------------------------<br>
> + LOG_snippet_2:<br>
> +-------------------------<br>
> Aug 11 16:09:42 [9005] A6-4U24-402-T stonithd: notice: handle_request: Client crmd.9009.cabd2154 wants to fence (reboot) 'node-0' with device '(any)'<br>
> Aug 11 16:09:42 [9005] A6-4U24-402-T stonithd: notice: initiate_remote_stonith_op: Initiating remote operation reboot for node-0: 3b06d3ce-b100-46d7-874e-96f10348d9e4 (0)<br>
> ....<br>
> Aug 11 16:09:42 [9005] A6-4U24-402-T stonithd: notice: can_fence_host_with_device: STONITH_node-1 can fence (reboot) node-0: none<br>
> ....<br>
> Aug 11 16:09:42 [9005] A6-4U24-402-T stonithd: info: call_remote_stonith: Total remote op timeout set to 60 for fencing of node node-0 for crmd.9009.3b06d3ce<br>
> Aug 11 16:09:42 [9005] A6-4U24-402-T stonithd: info: call_remote_stonith: Requesting that node-0 perform op reboot node-0 for crmd.9009 (72s)<br>
> ....<br>
> Aug 11 16:09:42 [9005] A6-4U24-402-T stonithd: notice: can_fence_host_with_device: STONITH_node-1 can fence (reboot) node-0: none<br>
> Aug 11 16:09:42 [9005] A6-4U24-402-T stonithd: info: stonith_fence_get_devices_cb: Found 1 matching devices for 'node-0'<br>
> ....<br>
> Aug 11 16:09:53 [9005] A6-4U24-402-T stonithd: notice: log_operation: Operation 'reboot' [25511] (call 3 from crmd.9009) for host 'node-0' with device 'STONITH_node-1' returned: 0 (OK)<br>
> Aug 11 16:09:53 [9005] A6-4U24-402-T stonithd: warning: get_xpath_object: No match for //@st_delegate in /st-reply<br>
> Aug 11 16:09:53 [9005] A6-4U24-402-T stonithd: notice: remote_op_done: Operation reboot of node-0 by node-0 for crmd.9009@node-0.3b06d3ce: OK<br>
> ....<br>
> Aug 11 16:09:53 [9009] A6-4U24-402-T crmd: notice: tengine_stonith_callback: Stonith operation 3/23:115:0:70ac834e-5b67-4ca6-9080-c98d2b59e2ee: OK (0)<br>
> Aug 11 16:09:53 [9009] A6-4U24-402-T crmd: info: crm_update_peer_join: crmd_peer_down: Node node-0[1] - join-2 phase 4 -> 0<br>
> Aug 11 16:09:53 [9009] A6-4U24-402-T crmd: info: crm_update_peer_expected: crmd_peer_down: Node node-0[1] - expected state is now down (was member)<br>
> ....<br>
> Aug 11 16:09:53 [9009] A6-4U24-402-T crmd: crit: tengine_stonith_notify: We were alegedly just fenced by node-0 for node-0!<br>
> ....<br>
> Aug 11 16:09:53 [9002] A6-4U24-402-T pacemakerd: error: pcmk_child_exit: Child process crmd (9009) exited: Network is down (100)<br>
> ....<br>
> Aug 11 16:09:53 [9002] A6-4U24-402-T pacemakerd: warning: pcmk_child_exit: Pacemaker child process crmd no longer wishes to be respawned. Shutting ourselves down.<br>
> ....<br>
> Aug 11 16:09:53 [9002] A6-4U24-402-T pacemakerd: notice: pcmk_shutdown_worker: Shuting down Pacemaker<br>
><br>
><br>
> Thank you,<br>
> Kostya<br>
</div></div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> <a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>