<html><head></head><body><div class="gmail_quote">On September 6, 2018 2:36:03 PM UTC, Patrick Whitney <pwhitney@luminoso.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div dir="ltr"><div dir="ltr"><div dir="ltr"><div><div>Good Morning Everyone,</div><div><br></div><div>I'm hoping someone with more experience with corosync and pacemaker can see what I am doing wrong. </div><div><div><br></div></div><div>I've got a test setup of 2 nodes, with dlm and clvm setup as clones, and using fence_scsi as my fencing agent. </div><div><br></div><div>I've got it to the point where the cluster is up, and reports it is happy. I then began testing fencing. When issuing 'pcs stonith fence' it appears to work; that is, the scsi reservation is pulled and the output of 'pcs status' looks sane, and I'm able to access resources on the un-fenced node. </div><div><br></div><div>Things go awry when I shutdown (init 0) the fenced node... my unfenced node decides to fence itself (which looks like it was initiated by dlm due to an abandoned lockspace). </div><div><br></div><div>I suspect this is due to misconfiguration, since I'm new to the toolset, but I'm not quite sure what I need to change. </div><div><br></div><div>Any and all input is appreciated! </div><div><br></div><div>Below is a chronology of events; my corosync config and cib.xml; command output; and annotated logs. </div><div><br></div><div>Again, any hints, suggestions, wild guesses, or premonitions are welcomed -- I'm stuck! Please let me know if there is additional information which would be helpful. </div><div><br></div><div>Many thanks,</div><div>-Patrick W. </div><div><br></div><div>Sep 6 08:54:14 -- Cluster is up and running; UI reports everything</div><div> healthy. </div><div><br></div><div>Sep 6 08:55:44 -- 'pcs stonith fence' called against node 1 (coro-test-1); </div><div> UI reports everything as expected -- that is, resources show only running on unfenced node and they're available. </div><div> Oddly, although the UI says dlm is stopped on fenced node, the dlm_controld is still running. </div><div><br></div><div>Sep 6 09:03:38 -- node 1 is shutdown, and node 2 falls to pieces.</div><div> - First, corosync sees lost member -- seems like this is appropriate, to me. </div><div> - Next, dlm_controld calls to fence everything</div><div> - stonith-ng tries to fence node 1 (but its already fenced!)</div><div> - dlm closes connection to "node 2" (does dlm "nodes" map to cluster nodes? I'm not sure they do)</div><div> - clvmd dlm lockspace is now abandoned; cluster attempts to fence the remaining node </div><div> (But can't because scsi_fence doesn't work like that). </div><div><br></div><div>***</div><div>****** -- Configuration -- </div><div>***</div><div><div><div>root@coro-test-2:~# pcs --version</div><div>0.9.149</div><div>root@coro-test-2:~# pacemakerd --version</div><div>Pacemaker 1.1.14</div><div>Written by Andrew Beekhof</div><div>root@coro-test-2:~# corosync -v</div><div>Corosync Cluster Engine, version '2.3.5'</div><div>Copyright (c) 2006-2009 Red Hat, Inc.</div><div>root@coro-test-2:~#</div></div><div><br></div></div><div>#</div><div>## COROSYNC</div><div>#</div><div>root@coro-test-1:~# more /etc/corosync/corosync.conf</div><div>totem {</div><div> version: 2</div><div> secauth: off</div><div> cluster_name: my_first_cluster</div><div> transport: udpu</div><div>}</div><div><br></div><div>nodelist {</div><div> node {</div><div> ring0_addr: coro-test-1</div><div> nodeid: 1</div><div> }</div><div><br></div><div> node {</div><div> ring0_addr: coro-test-2</div><div> nodeid: 2</div><div> }</div><div>}</div><div><br></div><div>quorum {</div><div> provider: corosync_votequorum</div><div> two_node: 1</div><div>}</div><div><br></div><div>logging {</div><div> to_logfile: yes</div><div> logfile: /var/log/corosync/corosync.log</div><div> to_syslog: yes</div><div>}</div><div><br></div><div>##</div><div>### PACEMAKER CIB.XML</div><div>##</div><div>root@coro-test-1:/var/lib/pacemaker/cib# cat cib.xml</div><div><cib crm_feature_set="3.0.10" validate-with="pacemaker-2.4" epoch="30" num_updates="0" admin_epoch="0" cib-last-written="Thu Sep 6 10:12:46 2018" update-origin="coro-test-2" update-client="cibadmin" update-user="root" have-quorum="1" dc-uuid="1"></div><div> <configuration></div><div> <crm_config></div><div> <cluster_property_set id="cib-bootstrap-options"></div><div> <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/></div><div> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.14-70404b0"/></div><div> <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/></div><div> <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="my_first_cluster"/></div><div> <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/></div><div> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="true"/></div><div> <nvpair id="cib-bootstrap-options-stonith-action" name="stonith-action" value="off"/></div><div> </cluster_property_set></div><div> </crm_config></div><div> <nodes></div><div> <node id="1" uname="coro-test-1"/></div><div> <node id="2" uname="coro-test-2"/></div><div> </nodes></div><div> <resources></div><div> <clone id="dlm-clone"></div><div> <primitive class="ocf" id="dlm" provider="pacemaker" type="controld"></div><div> <instance_attributes id="dlm-instance_attributes"/></div><div> <operations></div><div> <op id="dlm-start-interval-0s" interval="0s" name="start" timeout="90"/></div><div> <op id="dlm-stop-interval-0s" interval="0s" name="stop" timeout="100"/></div><div> <op id="dlm-monitor-interval-30s" interval="30s" name="monitor" on-fail="fence"/></div><div> </operations></div><div> </primitive></div><div> <meta_attributes id="dlm-clone-meta_attributes"></div><div> <nvpair id="dlm-clone-meta_attributes-interleave" name="interleave" value="true"/></div><div> <nvpair id="dlm-clone-meta_attributes-ordered" name="ordered" value="true"/></div><div> </meta_attributes></div><div> </clone></div><div> <clone id="clvmd-clone"></div><div> <primitive class="ocf" id="clvmd" provider="heartbeat" type="clvm"></div><div> <instance_attributes id="clvmd-instance_attributes"/></div><div> <operations></div><div> <op id="clvmd-start-interval-0s" interval="0s" name="start" timeout="90"/></div><div> <op id="clvmd-stop-interval-0s" interval="0s" name="stop" timeout="90"/></div><div> <op id="clvmd-monitor-interval-30s" interval="30s" name="monitor" on-fail="fence"/></div><div> </operations></div><div> </primitive></div><div> <meta_attributes id="clvmd-clone-meta_attributes"></div><div> <nvpair id="clvmd-clone-meta_attributes-interleave" name="interleave" value="true"/></div><div> <nvpair id="clvmd-clone-meta_attributes-ordered" name="ordered" value="true"/></div><div> </meta_attributes></div><div> </clone></div><div> <primitive class="stonith" id="scsi-shooter" type="fence_scsi"></div><div> <instance_attributes id="scsi-shooter-instance_attributes"></div><div> <nvpair id="scsi-shooter-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="coro-test-1 coro-test-2"/></div><div> <nvpair id="scsi-shooter-instance_attributes-pcmk_monitor_action" name="pcmk_monitor_action" value="metadata"/></div><div> <nvpair id="scsi-shooter-instance_attributes-pcmk_reboot_action" name="pcmk_reboot_action" value="off"/></div><div> <nvpair id="scsi-shooter-instance_attributes-pcmk_delay_max" name="pcmk_delay_max" value="15"/></div><div> <nvpair id="scsi-shooter-instance_attributes-devices" name="devices" value="/dev/sdb1"/></div><div> </instance_attributes></div><div> <meta_attributes id="scsi-shooter-meta_attributes"></div><div> <nvpair id="scsi-shooter-meta_attributes-provides" name="provides" value="unfencing"/></div><div> </meta_attributes></div><div> <operations></div><div> <op id="scsi-shooter-monitor-interval-60s" interval="60s" name="monitor"/></div><div> </operations></div><div> </primitive></div><div> </resources></div><div> <constraints></div><div> <rsc_order first="dlm-clone" first-action="start" id="order-dlm-clone-clvmd-clone-mandatory" then="clvmd-clone" then-action="start"/></div><div> <rsc_colocation id="colocation-clvmd-clone-dlm-clone-INFINITY" rsc="clvmd-clone" score="INFINITY" with-rsc="dlm-clone"/></div><div> </constraints></div><div> </configuration></div><div></cib></div><div><br></div><div>***</div><div>****** -- Console Output -- Comments begin with "#"</div><div>***</div><div><br></div><div>#</div><div>##</div><div>### -- Moments before testing starts (Node 2)--</div><div>##</div><div>#</div><div>root@coro-test-2:~# date; pcs status</div><div>Thu Sep 6 08:54:14 EDT 2018</div><div>Cluster name: my_first_cluster</div><div>Last updated: Thu Sep 6 08:54:14 2018<span style="white-space:pre"> </span>Last change: Wed Sep 5 16:28:53 2018 by root via cibadmin on coro-test-2</div><div>Stack: corosync</div><div>Current DC: coro-test-1 (version 1.1.14-70404b0) - partition with quorum</div><div>2 nodes and 5 resources configured</div><div><br></div><div>Online: [ coro-test-1 coro-test-2 ]</div><div><br></div><div>Full list of resources:</div><div><br></div><div> Clone Set: dlm-clone [dlm]</div><div> Started: [ coro-test-1 coro-test-2 ]</div><div> Clone Set: clvmd-clone [clvmd]</div><div> Started: [ coro-test-1 coro-test-2 ]</div><div> scsi-shooter<span style="white-space:pre"> </span>(stonith:fence_scsi):<span style="white-space:pre"> </span>Started coro-test-1</div><div><br></div><div>PCSD Status:</div><div> coro-test-1: Online</div><div> coro-test-2: Online</div><div><br></div><div>Daemon Status:</div><div> corosync: active/enabled</div><div> pacemaker: active/enabled</div><div> pcsd: active/enabled</div><div>root@coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1</div><div> PR generation=0x41, 2 registered reservation keys follow:</div><div> 0x3ff20000</div><div> 0x3ff20001</div><div>root@coro-test-2:~# dlm_tool status</div><div>cluster nodeid 2 quorate 1 ring seq 496 496</div><div>daemon now 58996 fence_pid 0</div><div>node 1 M add 92 rem 0 fail 0 fence 0 at 0 0</div><div>node 2 M add 92 rem 0 fail 0 fence 0 at 0 0</div><div>root@coro-test-2:~# dlm_tool ls</div><div>dlm lockspaces</div><div>name clvmd</div><div>id 0x4104eefa</div><div>flags 0x00000000</div><div>change member 2 joined 1 remove 0 failed 0 seq 1,1</div><div>members 1 2</div><div><br></div><div><br></div><div>#</div><div>##</div><div>### -- Node 1 is fenced -- </div><div>##</div><div>#</div><div>root@coro-test-2:~# date; pcs stonith fence coro-test-1</div><div>Thu Sep 6 08:55:44 EDT 2018</div><div>Node: coro-test-1 fenced</div><div>root@coro-test-2:~# pcs status</div><div>Cluster name: my_first_cluster</div><div>Last updated: Thu Sep 6 08:56:40 2018<span style="white-space:pre"> </span>Last change: Wed Sep 5 16:28:53 2018 by root via cibadmin on coro-test-2</div><div>Stack: corosync</div><div>Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum</div><div>2 nodes and 5 resources configured</div><div><br></div><div>Node coro-test-1: pending</div><div>Online: [ coro-test-2 ]</div><div><br></div><div>Full list of resources:</div><div><br></div><div> Clone Set: dlm-clone [dlm]</div><div> Started: [ coro-test-2 ]</div><div> Stopped: [ coro-test-1 ]</div><div> Clone Set: clvmd-clone [clvmd]</div><div> Started: [ coro-test-2 ]</div><div> Stopped: [ coro-test-1 ]</div><div> scsi-shooter<span style="white-space:pre"> </span>(stonith:fence_scsi):<span style="white-space:pre"> </span>Stopped</div><div><br></div><div>PCSD Status:</div><div> coro-test-1: Online</div><div> coro-test-2: Online</div><div><br></div><div>Daemon Status:</div><div> corosync: active/enabled</div><div> pacemaker: active/enabled</div><div> pcsd: active/enabled</div><div>root@coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1</div><div> PR generation=0x42, 1 registered reservation key follows:</div><div> 0x3ff20001</div><div>root@coro-test-2:~# dlm_tool ls</div><div>dlm lockspaces</div><div>name clvmd</div><div>id 0x4104eefa</div><div>flags 0x00000000</div><div>change member 2 joined 1 remove 0 failed 0 seq 1,1</div><div>members 1 2</div><div><br></div><div>root@coro-test-2:~# dlm_tool status</div><div>cluster nodeid 2 quorate 1 ring seq 496 496</div><div>daemon now 59166 fence_pid 0</div><div>node 1 M add 92 rem 0 fail 0 fence 0 at 0 0</div><div>node 2 M add 92 rem 0 fail 0 fence 0 at 0 0</div><div>root@coro-test-2:~#</div><div><br></div><div>#</div><div>## NOTE: DLM still running on other node, coro-test-1, at this point.</div><div>#</div><div>root@coro-test-1:~# ps -aef | grep dlm</div><div>root 1633 1 0 Sep05 ? 00:00:00 dlm_controld -s 0</div><div>root 1752 2 0 Sep05 ? 00:00:00 [dlm_scand]</div><div>root 1754 2 0 Sep05 ? 00:00:00 [dlm_recv]</div><div>root 1755 2 0 Sep05 ? 00:00:00 [dlm_send]</div><div>root 1756 2 0 Sep05 ? 00:00:00 [dlm_recoverd]</div><div>root 29673 1850 0 08:58 pts/0 00:00:00 grep --color=auto dlm</div><div>root@coro-test-1:~#</div><div><br></div><div><br></div><div>#</div><div>##</div><div>### -- node 2 output after node 1 is shutdown</div><div>##</div><div>#</div><div>#### NODE 1 SHUTDOWN ####</div><div>root@coro-test-1:~# pcs status; date; init 0</div><div>Error: cluster is not currently running on this node</div><div>Thu Sep 6 09:03:38 EDT 2018</div><div><br></div><div>#### NODE 2 UI OUTPUT ####</div><div>root@coro-test-2:~# pcs status</div><div>Cluster name: my_first_cluster</div><div>Last updated: Thu Sep 6 09:04:16 2018<span style="white-space:pre"> </span>Last change: Wed Sep 5 16:28:53 2018 by root via cibadmin on coro-test-2</div><div>Stack: corosync</div><div>Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum</div><div>2 nodes and 5 resources configured</div><div><br></div><div>Node coro-test-2: UNCLEAN (online)</div><div>OFFLINE: [ coro-test-1 ]</div><div><br></div><div>Full list of resources:</div><div><br></div><div> Clone Set: dlm-clone [dlm]</div><div> dlm<span style="white-space:pre"> </span>(ocf::pacemaker:controld):<span style="white-space:pre"> </span>FAILED coro-test-2</div><div> Stopped: [ coro-test-1 ]</div><div> Clone Set: clvmd-clone [clvmd]</div><div> Started: [ coro-test-2 ]</div><div> Stopped: [ coro-test-1 ]</div><div> scsi-shooter<span style="white-space:pre"> </span>(stonith:fence_scsi):<span style="white-space:pre"> </span>Started coro-test-2</div><div><br></div><div>Failed Actions:</div><div>* dlm_monitor_30000 on coro-test-2 'unknown error' (1): call=17, status=complete, exitreason='none',</div><div> last-rc-change='Thu Sep 6 09:04:09 2018', queued=0ms, exec=0ms</div><div><br></div><div><br></div><div>PCSD Status:</div><div> coro-test-1: Offline</div><div> coro-test-2: Online</div><div><br></div><div>Daemon Status:</div><div> corosync: active/enabled</div><div> pacemaker: active/enabled</div><div> pcsd: active/enabled</div><div>root@coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1</div><div> PR generation=0x42, 1 registered reservation key follows:</div><div> 0x3ff20001</div><div>root@coro-test-2:~# dlm_tool status</div><div>cannot connect to dlm_controld</div><div>root@coro-test-2:~# dlm_tool ls</div><div>root@coro-test-2:~#</div><div><br></div><div>***</div><div>****** -- Syslog Output -- Comments begin with "#"</div><div>***</div><div>#</div><div>###</div><div>##### NODE 2 FENCES NODE 1 (pcs stonith fence coro-test-1)</div><div>###</div><div>#</div><div><br></div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: fence_scsi[27468] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-1' ]</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: fence_scsi[27468] stderr: [ ]</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation 'reboot' [27468] (call 2 from stonith_admin.27448) for host 'coro-test-1' with device 'scsi-shooter' returned: 0 (OK)</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation reboot of coro-test-1 by coro-test-2 for stonith_admin.27448@coro-test-2.5886f6b5: OK</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: new_event_notification (1541-27448-12): Broken pipe (32)</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback ]</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=election_timeout_popped ]</div><div>Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was member)</div><div>Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Removing coro-test-1/1 from the membership list</div><div>Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Purged 1 peers with id=1 and/or uname=coro-test-1 from the membership cache</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was member)</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Removing coro-test-1/1 from the membership list</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Purged 1 peers with id=1 and/or uname=coro-test-1 from the membership cache</div><div>Sep 6 08:55:55 coro-test-2 cib[1540]: notice: crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was member)</div><div>Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Removing coro-test-1/1 from the membership list</div><div>Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Purged 1 peers with id=1 and/or uname=coro-test-1 from the membership cache</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: warning: FSA: Input I_ELECTION_DC from do_election_check() received in state S_INTEGRATION</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action dlm:0_monitor_0 on coro-test-1 is unrunnable (pending)</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action clvmd:0_monitor_0 on coro-test-1 is unrunnable (pending)</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action scsi-shooter_monitor_0 on coro-test-1 is unrunnable (pending)</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Unfencing coro-test-1: node discovery</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Start scsi-shooter#011(coro-test-2 - blocked)</div><div>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: error: Operation on of coro-test-1 by <no-one> for crmd.1545@coro-test-2.b06d9d39: No such device</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition 0 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-107.bz2): Complete</div><div>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div><div>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Calculated Transition 0: /var/lib/pacemaker/pengine/pe-input-107.bz2</div><div>Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPREQUEST of 10.0.2.5 on enp0s3 to 10.0.2.3 port 67 (xid=0x1c247c88)</div><div>Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPACK of 10.0.2.5 from 10.0.2.3</div><div>Sep 6 08:58:22 coro-test-2 dhclient[1070]: bound to 10.0.2.5 -- renewal in 466 seconds.</div><div>Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPREQUEST of 192.168.57.4 on enp0s8 to 192.168.57.2 port 67 (xid=0x5dfb57d7)</div><div>Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPACK of 192.168.57.4 from 192.168.57.2</div><div>Sep 6 08:59:20 coro-test-2 dhclient[860]: bound to 192.168.57.4 -- renewal in 599 seconds.</div><div><br></div><div>#</div><div>###</div><div>##### NODE 1 IS SHUTDOWN</div><div>###</div><div>#</div><div><br></div><div>##</div><div>### Cluster appears to see the loss of node 1</div><div>##</div><div>Sep 6 09:03:38 coro-test-2 corosync[1505]: notice [TOTEM ] A new membership (<a href="http://10.0.2.5:500">10.0.2.5:500</a>) was formed. Members left: 1</div><div>Sep 6 09:03:38 coro-test-2 corosync[1505]: [TOTEM ] A new membership (<a href="http://10.0.2.5:500">10.0.2.5:500</a>) was formed. Members left: 1</div><div>Sep 6 09:03:38 coro-test-2 pacemakerd[1537]: notice: crm_reap_unseen_nodes: Node coro-test-1[1] - state is now lost (was member)</div><div>Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 fence request 1 pid 28247 procdown time 1536239018 fence_all dlm_stonith</div><div>Sep 6 09:03:38 coro-test-2 crmd[1545]: notice: crm_reap_unseen_nodes: Node coro-test-1[1] - state is now lost (was member)</div><div>Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 tell corosync to remove nodeid 1 from cluster</div><div><br></div><div>##</div><div>### DLM appears to close off connection to NODE 1</div><div>##</div><div>Sep 6 09:03:38 coro-test-2 kernel: [59552.844406] dlm: closing connection to node 1</div><div>Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Found 2 entries for 1/(null): 0 in progress, 1 completed</div><div>Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Node 1/(null) last kicked at: 1536238555</div><div><br></div><div>##</div><div>### Stonith attempts to fence node 1 (again; it was already fenced)</div><div>##</div><div>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28248] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-1' ]</div><div>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28248] stderr: [ ]</div><div>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation 'reboot' [28248] (call 2 from stonith-api.28247) for host 'coro-test-1' with device 'scsi-shooter' returned: 0 (OK)</div><div>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation reboot of coro-test-1 by coro-test-2 for stonith-api.28247@coro-test-2.2365d094: OK</div><div>Sep 6 09:03:42 coro-test-2 stonith-api[28247]: stonith_api_kick: Node 1/(null) kicked: reboot</div><div>Sep 6 09:03:42 coro-test-2 crmd[1545]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]</div><div>Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div><br></div><div>##</div><div>### I begin to get confused here: Why is dlm closing the connection to the node 2 (itself??)</div><div>##</div><div>Sep 6 09:03:42 coro-test-2 kernel: [59556.684699] dlm: closing connection to node 2</div><div><br></div><div>##</div><div>### LOCKSPACE GOES ABANDONED!! why? WHY????</div><div>##</div><div>Sep 6 09:03:42 coro-test-2 dlm_controld[1657]: 59554 abandoned lockspace clvmd</div><div><br></div><div>##</div><div>### Alas, my cluster begins to topple itself</div><div>##</div><div>Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Start scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Calculated Transition 1: /var/lib/pacemaker/pengine/pe-input-108.bz2</div><div>Sep 6 09:03:42 coro-test-2 kernel: [59556.693578] dlm: dlm user daemon left 1 lockspaces</div><div>Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Operation scsi-shooter_start_0: ok (node=coro-test-2, call=20, rc=0, cib-update=56, confirmed=true)</div><div>Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Transition 1 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-108.bz2): Complete</div><div>Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div><div>Sep 6 09:04:05 coro-test-2 controld(dlm)[28324]: ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing</div><div>Sep 6 09:04:07 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:07 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:09 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] stderr: [ ]</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation 'reboot' [28357] (call 2 from stonith_admin.controld.28343) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation reboot of coro-test-2 by coro-test-2 for stonith_admin.controld.28343@coro-test-2.fd726e0b: Generic Pacemaker error</div><div>Sep 6 09:04:09 coro-test-2 crmd[1545]: notice: coro-test-2-dlm_monitor_30000:17 [ Command failed: Generic Pacemaker error\n ]</div><div>Sep 6 09:04:09 coro-test-2 crmd[1545]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated Transition 2: /var/lib/pacemaker/pengine/pe-warn-68.bz2</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated Transition 3: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:04:13 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:13 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:14 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] stderr: [ ]</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28383] (call 3 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.171aafb6: Generic Pacemaker error</div><div>Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition 3 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Calculated Transition 4: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:04:19 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:19 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:20 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] stderr: [ ]</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28445] (call 4 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.0b907f76: Generic Pacemaker error</div><div>Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition 4 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Calculated Transition 5: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:04:24 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:24 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:25 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] stderr: [ ]</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28490] (call 5 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.559102cb: Generic Pacemaker error</div><div>Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition 5 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Calculated Transition 6: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:04:39 coro-test-2 controld(dlm)[28520]: ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing</div><div>Sep 6 09:04:39 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:39 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:40 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] stderr: [ ]</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28553] (call 6 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.58604e64: Generic Pacemaker error</div><div>Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition 6 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Calculated Transition 7: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:04:49 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:49 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] stderr: [ ]</div><div>Sep 6 09:04:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:50 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation 'reboot' [28581] (call 2 from stonith_admin.controld.28539) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation reboot of coro-test-2 by coro-test-2 for stonith_admin.controld.28539@coro-test-2.29f17c2f: Generic Pacemaker error</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: new_event_notification (1541-28539-12): Broken pipe (32)</div><div>Sep 6 09:04:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:51 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:04:52 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] stderr: [ ]</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28607] (call 7 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.1c099fde: Generic Pacemaker error</div><div>Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition 7 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Calculated Transition 8: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:05:01 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:01 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:02 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] stderr: [ ]</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28656] (call 8 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.41c7a339: Generic Pacemaker error</div><div>Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition 8 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Calculated Transition 9: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:05:16 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:16 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] stderr: [ ]</div><div>Sep 6 09:05:17 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:17 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] stderr: [ ]</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28682] (call 9 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.16fd9d3b: Generic Pacemaker error</div><div>Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition 9 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Calculated Transition 10: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:05:21 coro-test-2 controld(dlm)[28695]: ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing</div><div>Sep 6 09:05:29 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:29 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28764] (call 10 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.c6433fd3: Generic Pacemaker error</div><div>Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition 10 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Calculated Transition 11: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:31 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] stderr: [ ]</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation 'reboot' [28790] (call 2 from stonith_admin.controld.28714) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation reboot of coro-test-2 by coro-test-2 for stonith_admin.controld.28714@coro-test-2.ffe64359: Generic Pacemaker error</div><div>Sep 6 09:05:36 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:36 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:37 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] stderr: [ ]</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28816] (call 11 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.7dfe97e8: Generic Pacemaker error</div><div>Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition 11 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Calculated Transition 12: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:05:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:50 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:05:51 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] stderr: [ ]</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28842] (call 12 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.fb005423: Generic Pacemaker error</div><div>Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition 12 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: On loss of CCM Quorum: Ignore</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Processing failed op monitor for dlm:0 on coro-test-2: unknown error (1)</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Node coro-test-2 will be fenced because of resource failure(s)</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 with instance of dlm-clone</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 with instance of dlm-clone</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Scheduling Node coro-test-2 for STONITH</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop of failed resource dlm:0 is implicit after coro-test-2 is fenced</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop dlm:0#011(coro-test-2)</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop clvmd:0#011(coro-test-2)</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop scsi-shooter#011(coro-test-2)</div><div>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Calculated Transition 13: /var/lib/pacemaker/pengine/pe-warn-69.bz2</div><div>Sep 6 09:06:01 coro-test-2 controld(dlm)[28874]: ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing</div><div>Sep 6 09:06:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:06:02 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation 'off' [28907] (call 13 from crmd.1545) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation off of coro-test-2 by coro-test-2 for crmd.1545@coro-test-2.e8cc12be: Generic Pacemaker error</div><div>Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition aborted: Stonith failed (source=tengine_stonith_callback:748, 0)</div><div>Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition 13 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete</div><div>Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]</div><div>Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] stderr: [ ]</div><div>Sep 6 09:06:04 coro-test-2 fence_scsi: Failed: keys cannot be same. You can not fence yourself.</div><div>Sep 6 09:06:04 coro-test-2 fence_scsi: Please use '-h' for usage</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28935] stderr: [ WARNING:root:Parse error: Ignoring unknown option 'port=coro-test-2' ]</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28935] stderr: [ ]</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28935] stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28935] stderr: [ ]</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28935] stderr: [ Failed: keys cannot be same. You can not fence yourself. ]</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28935] stderr: [ ]</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28935] stderr: [ ERROR:root:Please use '-h' for usage ]</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28935] stderr: [ ]</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28935] stderr: [ Please use '-h' for usage ]</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28935] stderr: [ ]</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: error: Operation 'reboot' [28935] (call 2 from stonith_admin.controld.28893) for host 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)</div><div>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: error: Operation reboot of coro-test-2 by coro-test-2 for stonith_admin.controld.28893@coro-test-2.185da33b: Generic Pacemaker error</div><div>Sep 6 09:06:08 coro-test-2 dhclient[1070]: DHCPREQUEST of 10.0.2.5 on enp0s3 to 10.0.2.3 port 67 (xid=0x1c247c88)</div><div>Sep 6 09:06:08 coro-test-2 dhclient[1070]: DHCPACK of 10.0.2.5 from 10.0.2.3</div><div>Sep 6 09:06:08 coro-test-2 dhclient[1070]: bound to 10.0.2.5 -- renewal in 571 seconds.</div><div><br></div><div><br></div></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Patrick Whitney<div>DevOps Engineer -- Tools</div></div></div></div></div></div>
</blockquote></div><br clear="all">Hi<br>I believe fabric fencing is not enough for dlm. You definitely need power fencing here so dlm kernel counterpart is reset properly.<br></body></html>