<div dir="ltr"><span style="font-size:12.8px">>>>> Hi,</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> On my two-node active/passive setup, I configured fencing via</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> fence_vmware_soap. I configured pcmk_delay=0 on both nodes so I</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> expected</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> that both nodes will be stonithed simultaenously.</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> On my test scenario, Node1 has ClusterIP resource. When I</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> disconnect</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> service/corosync link physically, Node1 was fenced and Node2 keeps</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> alive</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> given pcmk_delay=0 on both nodes.</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Can you explain the behavior above?</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> #node1 could not connect to ESX because links were disconnected. As</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> the</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> #most obvious explanation.</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> #You have logs, you are the only one who can answer this question</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> with</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> #some certainty. Others can only guess.</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> Oops, my bad. I forgot to tell. I have two interfaces on each virtual</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> machine (nodes). second interface was used for ESX links, so fence</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> can be executed even though corosync links were disconnected. Looking</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> forward to your response. Thanks</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> #Having no fence delay means a death match (each node killing the other)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> #is possible, but it doesn't guarantee that it will happen. Some of the</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> #time, one node will detect the outage and fence the other one before</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> #the other one can react.</span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>> #It's basically an Old West shoot-out -- they may reach for their guns</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> #at the same time, but one may be quicker.</span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>> #As Andrei suggested, the logs from both nodes could give you a timeline</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> #of what happened when.</span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Hi andrei, kindly see below logs. Based on time of logs, Node1 should have</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> fenced first Node2, but in actual test/scenario, Node1 was fenced/shutdown</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> by Node2.</span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">> Node1 tried to fence but failed. It could be connectivity, it could be</span><br style="font-size:12.8px"><span style="font-size:12.8px">> credentials.</span><br style="font-size:12.8px"><span style="font-size:12.8px">></span><div><br></div><div>Maybe this is the reason but it's still weird, I run so many tests and I conclude all of them have pattern, the Node that was physically disconnected is the one that gets fenced. It's not random.<br><br>See diagram on this link: <a href="https://drive.google.com/open?id=1pbJef_wJdQelJSv1L72c4H6NAvUqV_p-">https://drive.google.com/open?id=1pbJef_wJdQelJSv1L72c4H6NAvUqV_p-</a><br><br>And also based on my test, if Node1 gets fenced, after reboot, it doesn't automatically run the cluster. Different from what happens on Node2, even after reboot, it automatically run/join the cluster.</div><div><br></div><div><br style="font-size:12.8px"><span style="font-size:12.8px">>> Is it possible to have a 2-Node active/passive setup in pacemaker/corosync</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> that the node that gets disconnected/interface down is the only one that</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> gets fenced?</span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">> If you could determine which node was disconnected you would not need</span><br style="font-size:12.8px"><span style="font-size:12.8px">> any fencing at all.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">#True but there is still good reason taking connection into account.</span><br style="font-size:12.8px"><span style="font-size:12.8px">#Of course the foreseen survivor can't know that his peer got</span><br style="font-size:12.8px"><span style="font-size:12.8px">#disconnected directly.</span><br style="font-size:12.8px"><span style="font-size:12.8px">#But what you can do is that if you see that you are disconnected</span><br style="font-size:12.8px"><span style="font-size:12.8px">#yourself (e.g. ping-connection to routers, test-access to some</span><br style="font-size:12.8px"><span style="font-size:12.8px">#web-servers, ...) you can decide to shoot with a delay or not</span><br style="font-size:12.8px"><span style="font-size:12.8px">#shoot at all because starting services locally would anyway</span><br style="font-size:12.8px"><span style="font-size:12.8px">#be no good.</span><br style="font-size:12.8px"><span style="font-size:12.8px">#That is the basic idea behind fence_heuristics_ping fence-agent.</span><br style="font-size:12.8px"><span style="font-size:12.8px">#There was some discussion just recently about approaches</span><br style="font-size:12.8px"><span style="font-size:12.8px">#like that on the list.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">#Regards,</span><br style="font-size:12.8px"><span style="font-size:12.8px">#</span><span style="font-size:12.8px">Klaus</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">fence_heuristics_ping seems not available on my Rhel7 version. I do wonder if it is deprecated.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"><br></span><div><span style="font-size:12.8px">?</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Thanks guys</span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>> *LOGS from Node2:*</span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:27 ArcosRhel2 corosync[1048]: [TOTEM ] A processor failed,</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> forming new configuration.</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ...</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:30 ArcosRhel2 pengine[1083]: warning: Node ArcosRhel1 will be</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> fenced because the node is no longer part of the cluster</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ...</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:50 ArcosRhel2 stonith-ng[1080]: notice: Operation 'reboot'</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> [2323] (call 2 from crmd.1084) for host 'ArcosRhel1' with device 'Fence1'</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> returned: 0 (OK)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:50 ArcosRhel2 stonith-ng[1080]: notice: Operation reboot of</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> ArcosRhel1 by ArcosRhel2 for crmd.1084@ArcosRhel2.0426e6e1: OK</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:50 ArcosRhel2 crmd[1084]: notice: Stonith operation</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> 2/12:0:0:f9418e1f-1f13-4033-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">9eaa-aec705f807ef: OK (0)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:50 ArcosRhel2 crmd[1084]: notice: Peer ArcosRhel1 was</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> terminated (reboot) by ArcosRhel2 for ArcosRhel2: OK</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ...</span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>> *LOGS from NODE1*</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:26 ArcoSRhel1 corosync[1464]: [TOTEM ] A processor failed,</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> forming new configuration....</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:28 ArcoSRhel1 pengine[1476]: warning: Node ArcosRhel2 will be</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> fenced because the node is no longer part of the cluster</span><br style="font-size:12.8px"><span style="font-size:12.8px">> ...</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:28 ArcoSRhel1 stonith-ng[1473]: warning: Mapping action='off'</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> to pcmk_reboot_action='off'</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:28 ArcoSRhel1 stonith-ng[1473]: notice: Fence1 can not fence</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> (reboot) ArcosRhel2: static-list</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:28 ArcoSRhel1 stonith-ng[1473]: notice: fence2 can fence</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> (reboot) ArcosRhel2: static-list</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:28 ArcoSRhel1 stonith-ng[1473]: notice: Fence1 can not fence</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> (reboot) ArcosRhel2: static-list</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:28 ArcoSRhel1 stonith-ng[1473]: notice: fence2 can fence</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> (reboot) ArcosRhel2: static-list</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:46 ArcoSRhel1 fence_vmware_soap: Unable to connect/login to</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> fencing device</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:46 ArcoSRhel1 stonith-ng[1473]: warning:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> fence_vmware_soap[7157] stderr: [ Unable to connect/login to fencing device</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:46 ArcoSRhel1 stonith-ng[1473]: warning:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> fence_vmware_soap[7157] stderr: [ ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> Jul 17 13:33:46 ArcoSRhel1 stonith-ng[1473]: warning:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>> fence_vmware_soap[7157] stderr: [ ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> See my config below:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> [root@ArcosRhel2 cluster]# pcs config</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Cluster Name: ARCOSCLUSTER</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Corosync Nodes:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ArcosRhel1 ArcosRhel2</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Pacemaker Nodes:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ArcosRhel1 ArcosRhel2</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Resources:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ?Attributes: cidr_netmask=32 ip=172.16.10.243</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ?Operations: monitor interval=30s (ClusterIP-monitor-interval-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">30s)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ? ? ? ? ? ? ?start interval=0s timeout=20s (ClusterIP-start-</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> interval-0s)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ? ? ? ? ? ? ?stop interval=0s timeout=20s (ClusterIP-stop-</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> interval-0s)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Stonith Devices:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? Resource: Fence1 (class=stonith type=fence_vmware_soap)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ?Attributes: action=off ipaddr=172.16.10.151 login=admin</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> passwd=123pass</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> pcmk_host_list=ArcosRhel1 pcmk_monitor_timeout=60s</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> port=ArcosRhel1(Joniel)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ssl_insecure=1 pcmk_delay_max=0s</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ?Operations: monitor interval=60s (Fence1-monitor-interval-60s)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? Resource: fence2 (class=stonith type=fence_vmware_soap)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ?Attributes: action=off ipaddr=172.16.10.152 login=admin</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> passwd=123pass</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> pcmk_delay_max=0s pcmk_host_list=ArcosRhel2</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>> pcmk_monitor_timeout=60s</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> port=ArcosRhel2(Ben) ssl_insecure=1</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ?Operations: monitor interval=60s (fence2-monitor-interval-60s)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Fencing Levels:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Location Constraints:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ?Resource: Fence1</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ? ?Enabled on: ArcosRhel2 (score:INFINITY)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> (id:location-Fence1-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">ArcosRhel2-INFINITY)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ?Resource: fence2</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ? ?Enabled on: ArcosRhel1 (score:INFINITY)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> (id:location-fence2-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">ArcosRhel1-INFINITY)</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Ordering Constraints:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Colocation Constraints:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Ticket Constraints:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Alerts:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? No alerts defined</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Resources Defaults:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? No defaults set</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Operations Defaults:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? No defaults set</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Cluster Properties:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? cluster-infrastructure: corosync</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? cluster-name: ARCOSCLUSTER</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? dc-version: 1.1.16-12.el7-94ff4df</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? have-watchdog: false</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? last-lrm-refresh: 1531810841</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? stonith-enabled: true</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>></span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> Quorum:</span><br style="font-size:12.8px"><span style="font-size:12.8px">>>>> ? ?Options:</span><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 18, 2018 at 8:00 PM, <span dir="ltr"><<a href="mailto:users-request@clusterlabs.org" target="_blank">users-request@clusterlabs.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send Users mailing list submissions to<br>
<a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:users-request@clusterlabs.org">users-request@clusterlabs.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:users-owner@clusterlabs.org">users-owner@clusterlabs.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Users digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Weird Fencing Behavior (Andrei Borzenkov)<br>
2. Re: Weird Fencing Behavior (Klaus Wenninger)<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Wed, 18 Jul 2018 07:22:25 +0300<br>
From: Andrei Borzenkov <<a href="mailto:arvidjaar@gmail.com">arvidjaar@gmail.com</a>><br>
To: <a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a><br>
Subject: Re: [ClusterLabs] Weird Fencing Behavior<br>
Message-ID: <<a href="mailto:a58c2151-2519-46c0-209c-8f19cd0c7646@gmail.com">a58c2151-2519-46c0-209c-<wbr>8f19cd0c7646@gmail.com</a>><br>
Content-Type: text/plain; charset=utf-8<br>
<br>
18.07.2018 04:21, Confidential Company ?????:<br>
>>> Hi,<br>
>>><br>
>>> On my two-node active/passive setup, I configured fencing via<br>
>>> fence_vmware_soap. I configured pcmk_delay=0 on both nodes so I<br>
>> expected<br>
>>> that both nodes will be stonithed simultaenously.<br>
>>><br>
>>> On my test scenario, Node1 has ClusterIP resource. When I<br>
>> disconnect<br>
>>> service/corosync link physically, Node1 was fenced and Node2 keeps<br>
>> alive<br>
>>> given pcmk_delay=0 on both nodes.<br>
>>><br>
>>> Can you explain the behavior above?<br>
>>><br>
>><br>
>> #node1 could not connect to ESX because links were disconnected. As<br>
>> the<br>
>> #most obvious explanation.<br>
>><br>
>> #You have logs, you are the only one who can answer this question<br>
>> with<br>
>> #some certainty. Others can only guess.<br>
>><br>
>><br>
>> Oops, my bad. I forgot to tell. I have two interfaces on each virtual<br>
>> machine (nodes). second interface was used for ESX links, so fence<br>
>> can be executed even though corosync links were disconnected. Looking<br>
>> forward to your response. Thanks<br>
> <br>
> #Having no fence delay means a death match (each node killing the other)<br>
> #is possible, but it doesn't guarantee that it will happen. Some of the<br>
> #time, one node will detect the outage and fence the other one before<br>
> #the other one can react.<br>
> <br>
> #It's basically an Old West shoot-out -- they may reach for their guns<br>
> #at the same time, but one may be quicker.<br>
> <br>
> #As Andrei suggested, the logs from both nodes could give you a timeline<br>
> #of what happened when.<br>
> <br>
> <br>
> Hi andrei, kindly see below logs. Based on time of logs, Node1 should have<br>
> fenced first Node2, but in actual test/scenario, Node1 was fenced/shutdown<br>
> by Node2.<br>
> <br>
<br>
Node1 tried to fence but failed. It could be connectivity, it could be<br>
credentials.<br>
<br>
> Is it possible to have a 2-Node active/passive setup in pacemaker/corosync<br>
> that the node that gets disconnected/interface down is the only one that<br>
> gets fenced?<br>
> <br>
<br>
If you could determine which node was disconnected you would not need<br>
any fencing at all.<br>
<br>
> Thanks guys<br>
> <br>
> *LOGS from Node2:*<br>
> <br>
> Jul 17 13:33:27 ArcosRhel2 corosync[1048]: [TOTEM ] A processor failed,<br>
> forming new configuration.<br>
...<br>
> Jul 17 13:33:30 ArcosRhel2 pengine[1083]: warning: Node ArcosRhel1 will be<br>
> fenced because the node is no longer part of the cluster<br>
...<br>
> Jul 17 13:33:50 ArcosRhel2 stonith-ng[1080]: notice: Operation 'reboot'<br>
> [2323] (call 2 from crmd.1084) for host 'ArcosRhel1' with device 'Fence1'<br>
> returned: 0 (OK)<br>
> Jul 17 13:33:50 ArcosRhel2 stonith-ng[1080]: notice: Operation reboot of<br>
> ArcosRhel1 by ArcosRhel2 for crmd.1084@ArcosRhel2.0426e6e1: OK<br>
> Jul 17 13:33:50 ArcosRhel2 crmd[1084]: notice: Stonith operation<br>
> 2/12:0:0:f9418e1f-1f13-4033-<wbr>9eaa-aec705f807ef: OK (0)<br>
> Jul 17 13:33:50 ArcosRhel2 crmd[1084]: notice: Peer ArcosRhel1 was<br>
> terminated (reboot) by ArcosRhel2 for ArcosRhel2: OK<br>
...<br>
> <br>
> <br>
> <br>
> *LOGS from NODE1*<br>
> Jul 17 13:33:26 ArcoSRhel1 corosync[1464]: [TOTEM ] A processor failed,<br>
> forming new configuration....<br>
> Jul 17 13:33:28 ArcoSRhel1 pengine[1476]: warning: Node ArcosRhel2 will be<br>
> fenced because the node is no longer part of the cluster<br>
...<br>
> Jul 17 13:33:28 ArcoSRhel1 stonith-ng[1473]: warning: Mapping action='off'<br>
> to pcmk_reboot_action='off'<br>
> Jul 17 13:33:28 ArcoSRhel1 stonith-ng[1473]: notice: Fence1 can not fence<br>
> (reboot) ArcosRhel2: static-list<br>
> Jul 17 13:33:28 ArcoSRhel1 stonith-ng[1473]: notice: fence2 can fence<br>
> (reboot) ArcosRhel2: static-list<br>
> Jul 17 13:33:28 ArcoSRhel1 stonith-ng[1473]: notice: Fence1 can not fence<br>
> (reboot) ArcosRhel2: static-list<br>
> Jul 17 13:33:28 ArcoSRhel1 stonith-ng[1473]: notice: fence2 can fence<br>
> (reboot) ArcosRhel2: static-list<br>
> Jul 17 13:33:46 ArcoSRhel1 fence_vmware_soap: Unable to connect/login to<br>
> fencing device<br>
> Jul 17 13:33:46 ArcoSRhel1 stonith-ng[1473]: warning:<br>
> fence_vmware_soap[7157] stderr: [ Unable to connect/login to fencing device<br>
> ]<br>
> Jul 17 13:33:46 ArcoSRhel1 stonith-ng[1473]: warning:<br>
> fence_vmware_soap[7157] stderr: [ ]<br>
> Jul 17 13:33:46 ArcoSRhel1 stonith-ng[1473]: warning:<br>
> fence_vmware_soap[7157] stderr: [ ]<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
>>> See my config below:<br>
>>><br>
>>> [root@ArcosRhel2 cluster]# pcs config<br>
>>> Cluster Name: ARCOSCLUSTER<br>
>>> Corosync Nodes:<br>
>>> ? ArcosRhel1 ArcosRhel2<br>
>>> Pacemaker Nodes:<br>
>>> ? ArcosRhel1 ArcosRhel2<br>
>>><br>
>>> Resources:<br>
>>> ? Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)<br>
>>> ? ?Attributes: cidr_netmask=32 ip=172.16.10.243<br>
>>> ? ?Operations: monitor interval=30s (ClusterIP-monitor-interval-<wbr>30s)<br>
>>> ? ? ? ? ? ? ? ?start interval=0s timeout=20s (ClusterIP-start-<br>
>> interval-0s)<br>
>>> ? ? ? ? ? ? ? ?stop interval=0s timeout=20s (ClusterIP-stop-<br>
>> interval-0s)<br>
>>><br>
>>> Stonith Devices:<br>
>>> ? Resource: Fence1 (class=stonith type=fence_vmware_soap)<br>
>>> ? ?Attributes: action=off ipaddr=172.16.10.151 login=admin<br>
>> passwd=123pass<br>
>>> pcmk_host_list=ArcosRhel1 pcmk_monitor_timeout=60s<br>
>> port=ArcosRhel1(Joniel)<br>
>>> ssl_insecure=1 pcmk_delay_max=0s<br>
>>> ? ?Operations: monitor interval=60s (Fence1-monitor-interval-60s)<br>
>>> ? Resource: fence2 (class=stonith type=fence_vmware_soap)<br>
>>> ? ?Attributes: action=off ipaddr=172.16.10.152 login=admin<br>
>> passwd=123pass<br>
>>> pcmk_delay_max=0s pcmk_host_list=ArcosRhel2<br>
>> pcmk_monitor_timeout=60s<br>
>>> port=ArcosRhel2(Ben) ssl_insecure=1<br>
>>> ? ?Operations: monitor interval=60s (fence2-monitor-interval-60s)<br>
>>> Fencing Levels:<br>
>>><br>
>>> Location Constraints:<br>
>>> ? ?Resource: Fence1<br>
>>> ? ? ?Enabled on: ArcosRhel2 (score:INFINITY)<br>
>>> (id:location-Fence1-<wbr>ArcosRhel2-INFINITY)<br>
>>> ? ?Resource: fence2<br>
>>> ? ? ?Enabled on: ArcosRhel1 (score:INFINITY)<br>
>>> (id:location-fence2-<wbr>ArcosRhel1-INFINITY)<br>
>>> Ordering Constraints:<br>
>>> Colocation Constraints:<br>
>>> Ticket Constraints:<br>
>>><br>
>>> Alerts:<br>
>>> ? No alerts defined<br>
>>><br>
>>> Resources Defaults:<br>
>>> ? No defaults set<br>
>>> Operations Defaults:<br>
>>> ? No defaults set<br>
>>><br>
>>> Cluster Properties:<br>
>>> ? cluster-infrastructure: corosync<br>
>>> ? cluster-name: ARCOSCLUSTER<br>
>>> ? dc-version: 1.1.16-12.el7-94ff4df<br>
>>> ? have-watchdog: false<br>
>>> ? last-lrm-refresh: 1531810841<br>
>>> ? stonith-enabled: true<br>
>>><br>
>>> Quorum:<br>
>>> ? ?Options:<br></blockquote></div></div></div></div></div>