<div dir="ltr">What I'm having trouble understanding is why dlm flattens the remaining "running" node when the already fenced node is shutdown... I'm having trouble understanding how power fencing would cause dlm to behave any differently than just shutting down the fenced node. <div><br></div><div>-Pat</div></div><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Sep 6, 2018 at 2:35 PM Vladislav Bogdanov <<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="gmail_quote">On September 6, 2018 2:36:03 PM UTC, Patrick Whitney <<a href="mailto:pwhitney@luminoso.com" target="_blank">pwhitney@luminoso.com</a>> wrote:<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div dir="ltr"><div dir="ltr"><div><div>Good Morning Everyone,</div><div><br></div><div>I'm hoping someone with more experience with corosync and pacemaker can see what I am doing wrong. </div><div><div><br></div></div><div>I've got a test setup of 2 nodes, with dlm and clvm setup as clones, and using fence_scsi as my fencing agent. </div><div><br></div><div>I've got it to the point where the cluster is up, and reports it is happy. I then began testing fencing. When issuing 'pcs stonith fence' it appears to work; that is, the scsi reservation is pulled and the output of 'pcs status' looks sane, and I'm able to access resources on the un-fenced node. </div><div><br></div><div>Things go awry when I shutdown (init 0) the fenced node... my unfenced node decides to fence itself (which looks like it was initiated by dlm due to an abandoned lockspace). </div><div><br></div><div>I suspect this is due to misconfiguration, since I'm new to the toolset, but I'm not quite sure what I need to change. </div><div><br></div><div>Any and all input is appreciated! </div><div><br></div><div>Below is a chronology of events; my corosync config and cib.xml; command output; and annotated logs. </div><div><br></div><div>Again, any hints, suggestions, wild guesses, or premonitions are welcomed -- I'm stuck! Please let me know if there is additional information which would be helpful. </div><div><br></div><div>Many thanks,</div><div>-Patrick W. </div><div><br></div><div>Sep 6 08:54:14 -- Cluster is up and running; UI reports everything</div><div> healthy. </div><div><br></div><div>Sep 6 08:55:44 -- 'pcs stonith fence' called against node 1 (coro-test-1); </div><div> UI reports everything as expected -- that is, resources show only running on unfenced node and they're available. </div><div> Oddly, although the UI says dlm is stopped on fenced node, the dlm_controld is still running. </div><div><br></div><div>Sep 6 09:03:38 -- node 1 is shutdown, and node 2 falls to pieces.</div><div> - First, corosync sees lost member -- seems like this is appropriate, to me. </div><div> - Next, dlm_controld calls to fence everything</div><div> - stonith-ng tries to fence node 1 (but its already fenced!)</div><div> - dlm closes connection to "node 2" (does dlm "nodes" map to cluster nodes? I'm not sure they do)</div><div> - clvmd dlm lockspace is now abandoned; cluster attempts to fence the remaining node </div><div> (But can't because scsi_fence doesn't work like that). </div><div><br></div><div>***</div><div>****** -- Configuration -- </div><div>***</div><div><div><div>root@coro-test-2:~# pcs --version</div><div>0.9.149</div><div>root@coro-test-2:~# pacemakerd --version</div><div>Pacemaker 1.1.14</div><div>Written by Andrew Beekhof</div><div>root@coro-test-2:~# corosync -v</div><div>Corosync Cluster Engine, version '2.3.5'</div><div>Copyright (c) 2006-2009 Red Hat, Inc.</div><div>root@coro-test-2:~#</div></div><div><br></div></div><div>#</div><div>## COROSYNC</div><div>#</div><div>root@coro-test-1:~# more /etc/corosync/corosync.conf</div><div>totem {</div><div> version: 2</div><div> secauth: off</div><div> cluster_name: my_first_cluster</div><div> transport: udpu</div><div>}</div><div><br></div><div>nodelist {</div><div> node {</div><div> ring0_addr: coro-test-1</div><div> nodeid: 1</div><div> }</div><div><br></div><div> node {</div><div> ring0_addr: coro-test-2</div><div> nodeid: 2</div><div> }</div><div>}</div><div><br></div><div>quorum {</div><div> provider: corosync_votequorum</div><div> two_node: 1</div><div>}</div><div><br></div><div>logging {</div><div> to_logfile: yes</div><div> logfile: /var/log/corosync/corosync.log</div><div> to_syslog: yes</div><div>}</div><div><br></div><div>##</div><div>### PACEMAKER CIB.XML</div><div>##</div><div>root@coro-test-1:/var/lib/pacemaker/cib# cat cib.xml</div><div><cib crm_feature_set="3.0.10" validate-with="pacemaker-2.4" epoch="30" num_updates="0" admin_epoch="0" cib-last-written="Thu Sep 6 10:12:46 2018" update-origin="coro-test-2" update-client="cibadmin" update-user="root" have-quorum="1" dc-uuid="1"></div><div> <configuration></div><div> <crm_config></div><div> <cluster_property_set id="cib-bootstrap-options"></div><div> <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/></div><div> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.14-70404b0"/></div><div> <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/></div><div> <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="my_first_cluster"/></div><div> <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/></div><div> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="true"/></div><div> <nvpair id="cib-bootstrap-options-stonith-action" name="stonith-action" value="off"/></div><div> </cluster_property_set></div><div> </crm_config></div><div> <nodes></div><div> <node id="1" uname="coro-test-1"/></div><div> <node id="2" uname="coro-test-2"/></div><div> </nodes></div><div> <resources></div><div> <clone id="dlm-clone"></div><div> <primitive class="ocf" id="dlm" provider="pacemaker" type="controld"></div><div> <instance_attributes id="dlm-instance_attributes"/></div><div> <operations></div><div> <op id="dlm-start-interval-0s" interval="0s" name="start" timeout="90"/></div><div> <op id="dlm-stop-interval-0s" interval="0s" name="stop" timeout="100"/></div><div> <op id="dlm-monitor-interval-30s" interval="30s" name="monitor" on-fail="fence"/></div><div> </operations></div><div> </primitive></div><div> <meta_attributes id="dlm-clone-meta_attributes"></div><div> <nvpair id="dlm-clone-meta_attributes-interleave" name="interleave" value="true"/></div><div> <nvpair id="dlm-clone-meta_attributes-ordered" name="ordered" value="true"/></div><div> </meta_attributes></div><div> </clone></div><div> <clone id="clvmd-clone"></div><div> <primitive class="ocf" id="clvmd" provider="heartbeat" type="clvm"></div><div> <instance_attributes id="clvmd-instance_attributes"/></div><div> <operations></div><div> <op id="clvmd-start-interval-0s" interval="0s" name="start" timeout="90"/></div><div> <op id="clvmd-stop-interval-0s" interval="0s" name="stop" timeout="90"/></div><div> <op id="clvmd-monitor-interval-30s" interval="30s" name="monitor" on-fail="fence"/></div><div> </operations></div><div> </primitive></div><div> <meta_attributes id="clvmd-clone-meta_attributes"></div><div> <nvpair id="clvmd-clone-meta_attributes-interleave" name="interleave" value="true"/></div><div> <nvpair id="clvmd-clone-meta_attributes-ordered" name="ordered" value="true"/></div><div> </meta_attributes></div><div> </clone></div><div> <primitive class="stonith" id="scsi-shooter" type="fence_scsi"></div><div> <instance_attributes id="scsi-shooter-instance_attributes"></div><div> <nvpair id="scsi-shooter-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="coro-test-1 coro-test-2"/></div><div> <nvpair id="scsi-shooter-instance_attributes-pcmk_monitor_action" name="pcmk_monitor_action" value="metadata"/></div><div> <nvpair id="scsi-shooter-instance_attributes-pcmk_reboot_action" name="pcmk_reboot_action" value="off"/></div><div> <nvpair id="scsi-shooter-instance_attributes-pcmk_delay_max" name="pcmk_delay_max" value="15"/></div><div> <nvpair id="scsi-shooter-instance_attributes-devices" name="devices" value="/dev/sdb1"/></div><div> </instance_attributes></div><div> <meta_attributes id="scsi-shooter-meta_attributes"></div><div> <nvpair id="scsi-shooter-meta_attributes-provides" name="provides" value="unfencing"/></div><div> </meta_attributes></div><div> <operations></div><div> <op id="scsi-shooter-monitor-interval-60s" interval="60s" name="monitor"/></div><div> </operations></div><div> </primitive></div><div> </resources></div><div> <constraints></div><div> <rsc_order first="dlm-clone" first-action="start" id="order-dlm-clone-clvmd-clone-mandatory" then="clvmd-clone" then-action="start"/></div><div> <rsc_colocation id="colocation-clvmd-clone-dlm-clone-INFINITY" rsc="clvmd-clone" score="INFINITY" with-rsc="dlm-clone"/></div><div> </constraints></div><div> </configuration></div><div></cib></div><div><br></div><div>***</div><div>****** -- Console Output -- Comments begin with "#"</div><div>***</div><div><br></div><div>#</div><div>##</div><div>### -- Moments before testing starts (Node 2)--</div><div>##</div><div>#</div><div>root@coro-test-2:~# date; pcs status</div><div>Thu Sep 6 08:54:14 EDT 2018</div><div>Cluster name: my_first_cluster</div><div>Last updated: Thu Sep 6 08:54:14 2018<span style="white-space:pre-wrap"> </span>Last change: Wed Sep 5 16:28:53 2018 by root via cibadmin on coro-test-2</div><div>Stack: corosync</div><div>Current DC: coro-test-1 (version 1.1.14-70404b0) - partition with quorum</div><div>2 nodes and 5 resources configured</div><div><br></div><div>Online: [ coro-test-1 coro-test-2 ]</div><div><br></div><div>Full list of resources:</div><div><br></div><div> Clone Set: dlm-clone [dlm]</div><div> Started: [ coro-test-1 coro-test-2 ]</div><div> Clone Set: clvmd-clone [clvmd]</div><div> Started: [ coro-test-1 coro-test-2 ]</div><div> scsi-shooter<span style="white-space:pre-wrap"> </span>(stonith:fence_scsi):<span style="white-space:pre-wrap"> </span>Started coro-test-1</div><div><br></div><div>PCSD Status:</div><div> coro-test-1: Online</div><div> coro-test-2: Online</div><div><br></div><div>Daemon Status:</div><div> corosync: active/enabled</div><div> pacemaker: active/enabled</div><div> pcsd: active/enabled</div><div>root@coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1</div><div> PR generation=0x41, 2 registered reservation keys follow:</div><div> 0x3ff20000</div><div> 0x3ff20001</div><div>root@coro-test-2:~# dlm_tool status</div><div>cluster nodeid 2 quorate 1 ring seq 496 496</div><div>daemon now 58996 fence_pid 0</div><div>node 1 M add 92 rem 0 fail 0 fence 0 at 0 0</div><div>node 2 M add 92 rem 0 fail 0 fence 0 at 0 0</div><div>root@coro-test-2:~# dlm_tool ls</div><div>dlm lockspaces</div><div>name clvmd</div><div>id 0x4104eefa</div><div>flags 0x00000000</div><div>change member 2 joined 1 remove 0 failed 0 seq 1,1</div><div>members 1 2</div><div><br></div><div><br></div><div>#</div><div>##</div><div>### -- Node 1 is fenced -- </div><div>##</div><div>#</div><div>root@coro-test-2:~# date; pcs stonith fence coro-test-1</div><div>Thu Sep 6 08:55:44 EDT 2018</div><div>Node: coro-test-1 fenced</div><div>root@coro-test-2:~# pcs status</div><div>Cluster name: my_first_cluster</div><div>Last updated: Thu Sep 6 08:56:40 2018<span style="white-space:pre-wrap"> </span>Last change: Wed Sep 5 16:28:53 2018 by root via cibadmin on coro-test-2</div><div>Stack: corosync</div><div>Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum</div><div>2 nodes and 5 resources configured</div><div><br></div><div>Node coro-test-1: pending</div><div>Online: [ coro-test-2 ]</div><div><br></div><div>Full list of resources:</div><div><br></div><div> Clone Set: dlm-clone [dlm]</div><div> Started: [ coro-test-2 ]</div><div> Stopped: [ coro-test-1 ]</div><div> Clone Set: clvmd-clone [clvmd]</div><div> Started: [ coro-test-2 ]</div><div> Stopped: [ coro-test-1 ]</div><div> scsi-shooter<span style="white-space:pre-wrap"> </span>(stonith:fence_scsi):<span style="white-space:pre-wrap"> </span>Stopped</div><div><br></div><div>PCSD Status:</div><div> coro-test-1: Online</div><div> coro-test-2: Online</div><div><br></div><div>Daemon Status:</div><div> corosync: active/enabled</div><div> pacemaker: active/enabled</div><div> pcsd: active/enabled</div><div>root@coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1</div><div> PR generation=0x42, 1 registered reservation key follows:</div><div> 0x3ff20001</div><div>root@coro-test-2:~# dlm_tool ls</div><div>dlm lockspaces</div><div>name clvmd</div><div>id 0x4104eefa</div><div>flags 0x00000000</div><div>change member 2 joined 1 remove 0 failed 0 seq 1,1</div><div>members 1 2</div><div><br></div><div>root@coro-test-2:~# dlm_tool status</div><div>cluster nodeid 2 quorate 1 ring seq 496 496</div><div>daemon now 59166 fence_pid 0</div><div>node 1 M add 92 rem 0 fail 0 fence 0 at 0 0</div><div>node 2 M add 92 rem 0 fail 0 fence 0 at 0 0</div><div>root@coro-test-2:~#</div><div><br></div><div>#</div><div>## NOTE: DLM still running on other node, coro-test-1, at this point.</div><div>#</div><div>root@coro-test-1:~# ps -aef | grep dlm</div><div>root 1633 1 0 Sep05 ? 00:00:00 dlm_controld -s 0</div><div>root 1752 2 0 Sep05 ? 00:00:00 [dlm_scand]</div><div>root 1754 2 0 Sep05 ? 00:00:00 [dlm_recv]</div><div>root 1755 2 0 Sep05 ? 00:00:00 [dlm_send]</div><div>root 1756 2 0 Sep05 ? 00:00:00 [dlm_recoverd]</div><div>root 29673 1850 0 08:58 pts/0 00:00:00 grep --color=auto dlm</div><div>root@coro-test-1:~#</div><div><br></div><div><br></div><div>#</div><div>##</div><div>### -- node 2 output after node 1 is shutdown</div><div>##</div><div>#</div><div>#### NODE 1 SHUTDOWN ####</div><div>root@coro-test-1:~# pcs status; date; init 0</div><div>Error: cluster is not currently running on this node</div><div>Thu Sep 6 09:03:38 EDT 2018</div><div><br></div><div>#### NODE 2 UI OUTPUT ####</div><div>root@coro-test-2:~# pcs status</div><div>Cluster name: my_first_cluster</div><div>Last updated: Thu Sep 6 09:04:16 2018<span style="white-space:pre-wrap"> </span>Last change: Wed Sep 5 16:28:53 2018 by root via cibadmin on coro-test-2</div><div>Stack: corosync</div><div>Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum</div><div>2 nodes and 5 resources configured</div><div><br></div><div>Node coro-test-2: UNCLEAN (online)</div><div>OFFLINE: [ coro-test-1 ]</div><div><br></div><div>Full list of resources:</div><div><br></div><div> Clone Set: dlm-clone [dlm]</div><div> dlm<span style="white-space:pre-wrap"> </span>(ocf::pacemaker:controld):<span style="white-space:pre-wrap"> </span>FAILED coro-test-2</div><div> Stopped: [ coro-test-1 ]</div><div> Clone Set: clvmd-clone [clvmd]</div><div> Started: [ coro-test-2 ]</div><div> Stopped: [ coro-test-1 ]</div><div> scsi-shooter<span style="white-space:pre-wrap"> </span>(stonith:fence_scsi):<span style="white-space:pre-wrap"> </span>Started coro-test-2</div><div><br></div><div>Failed Actions:</div><div>* dlm_monitor_30000 on coro-test-2 'unknown error' (1): call=17, status=complete, exitreason='none',</div><div> last-rc-change='Thu Sep 6 09:04:09 2018', queued=0ms, exec=0ms</div><div><br></div><div><br></div><div>PCSD Status:</div><div> coro-test-1: Offline</div><div> coro-test-2: Online</div><div><br></div><div>Daemon Status:</div><div> corosync: active/enabled</div><div> pacemaker: active/enabled</div><div> pcsd: active/enabled</div><div>root@coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1</div><div> PR generation=0x42, 1 registered reservation key follows:</div><div> 0x3ff20001</div><div>root@coro-test-2:~# dlm_tool status</div><div>cannot connect to dlm_controld</div><div>root@coro-test-2:~# dlm_tool ls</div><div>root@coro-test-2:~#</div><div><br></div><div>***</div><div>****** -- Syslog Output -- Comments begin with "#"</div><div>***</div><div>#</div><div>###</div><div>##### NODE 2 FENCES NODE 1 (pcs stonith fence coro-test-1)</div><div>###</div><div>#</div><div><br></div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: fence_scsi[27468] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-1' ]</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: fence_scsi[27468] stderr: [ ]</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation 'reboot' [27468] (call 2 from stonith_admin.27448) for host 'coro-test-1' with device 'scsi-shooter' returned: 0 (OK)</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation reboot of coro-test-1 by coro-test-2 for stonith_admin.27448@coro-test-2.5886f6b5: OK</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: new_event_notification (1541-27448-12): Broken pipe (32)</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback ]</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=election_timeout_popped ]</div><div>Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was member)</div><div>Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Removing coro-test-1/1 from the membership list</div><div>Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Purged 1 peers with id=1 and/or uname=coro-test-1 from the membership cache</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was member)</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Removing coro-test-1/1 from the membership list</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Purged 1 peers with id=1 and/or uname=coro-test-1 from the membership cache</div><div>Sep 6 08:55:55 coro-test-2 cib[1540]: notice: crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was member)</div><div>Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Removing coro-test-1/1 from the membership list</div><div>Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Purged 1 peers with id=1 and/or uname=coro-test-1 from the membership cache</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: warning: FSA: Input I_ELECTION_DC from do_election_check() received in state S_INTEGRATION</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action dlm:0_monitor_0 on coro-test-1 is unrunnable (pending)</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action clvmd:0_monitor_0 on coro-test-1 is unrunnable (pending)</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action scsi-shooter_monitor_0 on coro-test-1 is unrunnable (pending)</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Unfencing coro-test-1: node discovery</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Start scsi-shooter#011(coro-test-2 - blocked)</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: error: Operation on of coro-test-1 by <no-one> for crmd.1545@coro-test-2.b06d9d39: No such device</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition 0 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-107.bz2): Complete</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Calculated Transition 0: /var/lib/pacemaker/pengine/pe-input-107.bz2</div><div>Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPREQUEST of 10.0.2.5 on enp0s3 to 10.0.2.3 port 67 (xid=0x1c247c88)</div><div>Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPACK of 10.0.2.5 from 10.0.2.3</div><div>Sep 6 08:58:22 coro-test-2 dhclient[1070]: bound to 10.0.2.5 -- renewal in 466 seconds.</div><div>Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPREQUEST of 192.168.57.4 on enp0s8 to 192.168.57.2 port 67 (xid=0x5dfb57d7)</div><div>Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPACK of 192.168.57.4 from 192.168.57.2</div><div>Sep 6 08:59:20 coro-test-2 dhclient[860]: bound to 192.168.57.4 -- renewal in 599 seconds.</div><div><br></div><div>#</div><div>###</div><div>##### NODE 1 IS SHUTDOWN</div><div>###</div><div>#</div><div><br></div><div>##</div><div>### Cluster appears to see the loss of node 1</div><div>##</div><div>Sep 6 09:03:38 coro-test-2 corosync[1505]: notice [TOTEM ] A new membership (<a href="http://10.0.2.5:500" target="_blank">10.0.2.5:500</a>) was formed. Members left: 1</div><div>Sep 6 09:03:38 coro-test-2 corosync[1505]: [TOTEM ] A new membership (<a href="http://10.0.2.5:500" target="_blank">10.0.2.5:500</a>) was formed. Members left: 1</div><div>Sep 6 09:03:38 coro-test-2 pacemakerd[1537]: notice: crm_reap_unseen_nodes: Node coro-test-1[1] - state is now lost (was member)</div><div>Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 fence request 1 pid 28247 procdown time 1536239018 fence_all dlm_stonith</div><div>Sep 6 09:03:38 coro-test-2 crmd[1545]: notice: crm_reap_unseen_nodes: Node coro-test-1[1] - state is now lost (was member)</div><div>Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 tell corosync to remove nodeid 1 from cluster</div><div><br></div><div>##</div><div>### DLM appears to close off connection to NODE 1</div><div>##</div><div>Sep 6 09:03:38 coro-test-2 kernel: [59552.844406] dlm: closing connection to node 1</div><div>Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Found 2 entries for 1/(null): 0 in progress, 1 completed</div><div>Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Node 1/(null) last kicked at: 1536238555</div><div><br></div><div>##</div><div>### Stonith attempts to fence node 1 (again; it was already fenced)</div><div>##</div><div>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28248] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-1' ]</div><div>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28248] stderr: [ ]</div><div>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation 'reboot' [28248] (call 2 from stonith-api.28247) for host 'coro-test-1' with device 'scsi-shooter' returned: 0 (OK)</div><div>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation reboot of coro-test-1 by coro-test-2 for stonith-api.28247@coro-test-2.2365d094: OK</div><div>Sep 6 09:03:42 coro-test-2 stonith-api[28247]: stonith_api_kick: Node 1/(null) kicked: reboot</div><div>Sep 6 09:03:42 coro-test-2 crmd[1545]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]</div><div>Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div><br></div><div>##</div><div>### I begin to get confused here: Why is dlm closing the connection to the node 2 (itself??)</div><div>##</div><div>Sep 6 09:03:42 coro-test-2 kernel: [59556.684699] dlm: closing connection to node 2</div><div><br></div><div>##</div><div>### LOCKSPACE GOES ABANDONED!! why? WHY????</div><div>##</div><div>Sep 6 09:03:42 coro-test-2 dlm_controld[1657]: 59554 abandoned lockspace clvmd</div><div><br></div><div>##</div><div>### Alas, my cluster begins to topple itself</div><div>##</div><div>Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Start scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Calculated Transition 1: /var/lib/pacemaker/pengine/pe-input-108.bz2</div><div>Sep 6 09:03:42 coro-test-2 kernel: [59556.693578] dlm: dlm user daemon left 1 lockspaces</div><div>Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Operation scsi-shooter_start_0: ok (node=coro-test-2, call=20, rc=0, cib-update=56, confirmed=true)</div><div>Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Transition 1 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-108.bz2): Complete</div><div>Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div><div>Sep 6 09:04:05 coro-test-2 controld(dlm)[28324]: ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing</div><div>Sep 6 09:04:07 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:07 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:09 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation 'reboot' [28357] (call 2 from stonith_admin.controld.28343) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation reboot of coro-test-2 by coro-test-2 for stonith_admin.controld.28343@coro-test-2.fd726e0b: Generic Pacemaker error</div><div>Sep 6 09:04:09 coro-test-2 crmd[1545]: notice: coro-test-2-dlm_monitor_30000:17 [ Command failed: Generic Pacemaker error\n ]</div><div>Sep 6 09:04:09 coro-test-2 crmd[1545]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated Transition 2: /var/lib/pacemaker/pengine/pe-warn-68.bz2</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated Transition 3: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:04:13 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:13 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:14 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28383] (call 3 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.171aafb6: Generic Pacemaker error</div><div>Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition 3 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Calculated Transition 4: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:04:19 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:19 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:20 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28445] (call 4 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.0b907f76: Generic Pacemaker error</div><div>Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition 4 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Calculated Transition 5: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:04:24 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:24 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:25 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28490] (call 5 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.559102cb: Generic Pacemaker error</div><div>Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition 5 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Calculated Transition 6: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:04:39 coro-test-2 controld(dlm)[28520]: ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing</div><div>Sep 6 09:04:39 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:39 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:40 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28553] (call 6 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.58604e64: Generic Pacemaker error</div><div>Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition 6 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Calculated Transition 7: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:04:49 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:49 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ]</div><div>Sep 6 09:04:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:50 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation 'reboot' [28581] (call 2 from stonith_admin.controld.28539) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation reboot of coro-test-2 by coro-test-2 for stonith_admin.controld.28539@coro-test-2.29f17c2f: Generic Pacemaker error</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: new_event_notification (1541-28539-12): Broken pipe (32)</div><div>Sep 6 09:04:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:51 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:52 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28607] (call 7 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.1c099fde: Generic Pacemaker error</div><div>Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition 7 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Calculated Transition 8: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:05:01 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:01 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:02 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28656] (call 8 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.41c7a339: Generic Pacemaker error</div><div>Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition 8 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Calculated Transition 9: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:05:16 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:16 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ]</div><div>Sep 6 09:05:17 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:17 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28682] (call 9 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.16fd9d3b: Generic Pacemaker error</div><div>Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition 9 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Calculated Transition 10: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:05:21 coro-test-2 controld(dlm)[28695]: ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing</div><div>Sep 6 09:05:29 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:29 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28764] (call 10 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.c6433fd3: Generic Pacemaker error</div><div>Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition 10 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Calculated Transition 11: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:31 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation 'reboot' [28790] (call 2 from stonith_admin.controld.28714) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation reboot of coro-test-2 by coro-test-2 for stonith_admin.controld.28714@coro-test-2.ffe64359: Generic Pacemaker error</div><div>Sep 6 09:05:36 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:36 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:37 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28816] (call 11 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.7dfe97e8: Generic Pacemaker error</div><div>Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition 11 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Calculated Transition 12: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:05:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:50 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:51 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28842] (call 12 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.fb005423: Generic Pacemaker error</div><div>Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition 12 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Calculated Transition 13: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:06:01 coro-test-2 controld(dlm)[28874]: ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing</div><div>Sep 6 09:06:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:06:02 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28907] (call 13 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.e8cc12be: Generic Pacemaker error</div><div>Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition 13 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div><div>Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ]</div><div>Sep 6 09:06:04 </div></div></div></div></div></blockquote></div></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-1013368680270866034gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Patrick Whitney<div>DevOps Engineer -- Tools</div></div></div>