[ClusterLabs] Antw: Re: 2 node cluster dlm/clvm trouble

Ulrich Windl Ulrich.Windl at rz.uni-regensburg.de
Tue Sep 11 10:00:30 EDT 2018


Hi!

I think the simple answer is that DLM wants to be shut down orderly, just as cLVM and OCFS2 do.

Regards,
Ulrich

>>> Patrick Whitney <pwhitney at luminoso.com> schrieb am 11.09.2018 um 15:02 in
Nachricht
<CAE0zLk9OnX99vE0o9Xj+XtKLnP+uMUQFU6=2ygdNq4z_uxjaaA at mail.gmail.com>:
> What I'm having trouble understanding is why dlm flattens the remaining
> "running" node when the already fenced node is shutdown...  I'm having
> trouble understanding how power fencing would cause dlm to behave any
> differently than just shutting down the fenced node.
> 
> -Pat
> 
> 
> On Thu, Sep 6, 2018 at 2:35 PM Vladislav Bogdanov <bubble at hoster-ok.com>
> wrote:
> 
>> On September 6, 2018 2:36:03 PM UTC, Patrick Whitney <
>> pwhitney at luminoso.com> wrote:
>>>
>>> Good Morning Everyone,
>>>
>>> I'm hoping someone with more experience with corosync and pacemaker can
>>> see what I am doing wrong.
>>>
>>> I've got a test setup of 2 nodes, with dlm and clvm setup as clones, and
>>> using fence_scsi as my fencing agent.
>>>
>>> I've got it to the point where the cluster is up, and reports it is
>>> happy.  I then began testing fencing.   When issuing 'pcs stonith fence' it
>>> appears to work; that is, the scsi reservation is pulled and the output of
>>> 'pcs status' looks sane, and I'm able to access resources on the un-fenced
>>> node.
>>>
>>> Things go awry when I shutdown (init 0) the fenced node... my unfenced
>>> node decides to fence itself (which looks like it was initiated by dlm due
>>> to an abandoned lockspace).
>>>
>>> I suspect this is due to misconfiguration, since I'm new to the toolset,
>>> but I'm not quite sure what I need to change.
>>>
>>> Any and all input is appreciated!
>>>
>>> Below is a chronology of events; my corosync config and cib.xml; command
>>> output; and annotated logs.
>>>
>>> Again, any hints, suggestions, wild guesses, or premonitions are welcomed
>>> -- I'm stuck!   Please let me know if there is additional information which
>>> would be helpful.
>>>
>>> Many thanks,
>>> -Patrick W.
>>>
>>> Sep  6 08:54:14  -- Cluster is up and running; UI reports everything
>>>                                healthy.
>>>
>>> Sep  6 08:55:44  -- 'pcs stonith fence' called against node 1
>>> (coro-test-1);
>>>                                UI reports everything as expected -- that
>>> is, resources show only running on unfenced node and they're available.
>>>                                Oddly, although the UI says dlm is stopped
>>> on fenced node, the dlm_controld is still running.
>>>
>>> Sep  6 09:03:38  -- node 1 is shutdown, and node 2 falls to pieces.
>>>                                - First, corosync sees lost member --
>>> seems like this is appropriate, to me.
>>>                                - Next, dlm_controld calls to fence
>>> everything
>>>                                - stonith-ng tries to fence node 1 (but
>>> its already fenced!)
>>>                                - dlm closes connection to "node 2" (does
>>> dlm "nodes" map to cluster nodes? I'm not sure they do)
>>>                                - clvmd dlm lockspace is now abandoned;
>>> cluster attempts to fence the remaining node
>>>                                  (But can't because scsi_fence doesn't
>>> work like that).
>>>
>>> ***
>>> ******   -- Configuration --
>>> ***
>>> root at coro-test-2:~# pcs --version
>>> 0.9.149
>>> root at coro-test-2:~# pacemakerd --version
>>> Pacemaker 1.1.14
>>> Written by Andrew Beekhof
>>> root at coro-test-2:~# corosync -v
>>> Corosync Cluster Engine, version '2.3.5'
>>> Copyright (c) 2006-2009 Red Hat, Inc.
>>> root at coro-test-2:~#
>>>
>>> #
>>> ##  COROSYNC
>>> #
>>> root at coro-test-1:~# more /etc/corosync/corosync.conf
>>> totem {
>>>     version: 2
>>>     secauth: off
>>>     cluster_name: my_first_cluster
>>>     transport: udpu
>>> }
>>>
>>> nodelist {
>>>     node {
>>>         ring0_addr: coro-test-1
>>>         nodeid: 1
>>>     }
>>>
>>>     node {
>>>         ring0_addr: coro-test-2
>>>         nodeid: 2
>>>     }
>>> }
>>>
>>> quorum {
>>>     provider: corosync_votequorum
>>>     two_node: 1
>>> }
>>>
>>> logging {
>>>     to_logfile: yes
>>>     logfile: /var/log/corosync/corosync.log
>>>     to_syslog: yes
>>> }
>>>
>>> ##
>>> ###  PACEMAKER CIB.XML
>>> ##
>>> root at coro-test-1:/var/lib/pacemaker/cib# cat cib.xml
>>> <cib crm_feature_set="3.0.10" validate-with="pacemaker-2.4" epoch="30"
>>> num_updates="0" admin_epoch="0" cib-last-written="Thu Sep  6 10:12:46 2018"
>>> update-origin="coro-test-2" update-client="cibadmin" update-user="root"
>>> have-quorum="1" dc-uuid="1">
>>>   <configuration>
>>>     <crm_config>
>>>       <cluster_property_set id="cib-bootstrap-options">
>>>         <nvpair id="cib-bootstrap-options-have-watchdog"
>>> name="have-watchdog" value="false"/>
>>>         <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
>>> value="1.1.14-70404b0"/>
>>>         <nvpair id="cib-bootstrap-options-cluster-infrastructure"
>>> name="cluster-infrastructure" value="corosync"/>
>>>         <nvpair id="cib-bootstrap-options-cluster-name"
>>> name="cluster-name" value="my_first_cluster"/>
>>>         <nvpair id="cib-bootstrap-options-no-quorum-policy"
>>> name="no-quorum-policy" value="ignore"/>
>>>         <nvpair id="cib-bootstrap-options-stonith-enabled"
>>> name="stonith-enabled" value="true"/>
>>>         <nvpair id="cib-bootstrap-options-stonith-action"
>>> name="stonith-action" value="off"/>
>>>       </cluster_property_set>
>>>     </crm_config>
>>>     <nodes>
>>>       <node id="1" uname="coro-test-1"/>
>>>       <node id="2" uname="coro-test-2"/>
>>>     </nodes>
>>>     <resources>
>>>       <clone id="dlm-clone">
>>>         <primitive class="ocf" id="dlm" provider="pacemaker"
>>> type="controld">
>>>           <instance_attributes id="dlm-instance_attributes"/>
>>>           <operations>
>>>             <op id="dlm-start-interval-0s" interval="0s" name="start"
>>> timeout="90"/>
>>>             <op id="dlm-stop-interval-0s" interval="0s" name="stop"
>>> timeout="100"/>
>>>             <op id="dlm-monitor-interval-30s" interval="30s"
>>> name="monitor" on-fail="fence"/>
>>>           </operations>
>>>         </primitive>
>>>         <meta_attributes id="dlm-clone-meta_attributes">
>>>           <nvpair id="dlm-clone-meta_attributes-interleave"
>>> name="interleave" value="true"/>
>>>           <nvpair id="dlm-clone-meta_attributes-ordered" name="ordered"
>>> value="true"/>
>>>         </meta_attributes>
>>>       </clone>
>>>       <clone id="clvmd-clone">
>>>         <primitive class="ocf" id="clvmd" provider="heartbeat"
>>> type="clvm">
>>>           <instance_attributes id="clvmd-instance_attributes"/>
>>>           <operations>
>>>             <op id="clvmd-start-interval-0s" interval="0s" name="start"
>>> timeout="90"/>
>>>             <op id="clvmd-stop-interval-0s" interval="0s" name="stop"
>>> timeout="90"/>
>>>             <op id="clvmd-monitor-interval-30s" interval="30s"
>>> name="monitor" on-fail="fence"/>
>>>           </operations>
>>>         </primitive>
>>>         <meta_attributes id="clvmd-clone-meta_attributes">
>>>           <nvpair id="clvmd-clone-meta_attributes-interleave"
>>> name="interleave" value="true"/>
>>>           <nvpair id="clvmd-clone-meta_attributes-ordered" name="ordered"
>>> value="true"/>
>>>         </meta_attributes>
>>>       </clone>
>>>       <primitive class="stonith" id="scsi-shooter" type="fence_scsi">
>>>         <instance_attributes id="scsi-shooter-instance_attributes">
>>>           <nvpair id="scsi-shooter-instance_attributes-pcmk_host_list"
>>> name="pcmk_host_list" value="coro-test-1 coro-test-2"/>
>>>           <nvpair
>>> id="scsi-shooter-instance_attributes-pcmk_monitor_action"
>>> name="pcmk_monitor_action" value="metadata"/>
>>>           <nvpair
>>> id="scsi-shooter-instance_attributes-pcmk_reboot_action"
>>> name="pcmk_reboot_action" value="off"/>
>>>           <nvpair id="scsi-shooter-instance_attributes-pcmk_delay_max"
>>> name="pcmk_delay_max" value="15"/>
>>>           <nvpair id="scsi-shooter-instance_attributes-devices"
>>> name="devices" value="/dev/sdb1"/>
>>>         </instance_attributes>
>>>         <meta_attributes id="scsi-shooter-meta_attributes">
>>>           <nvpair id="scsi-shooter-meta_attributes-provides"
>>> name="provides" value="unfencing"/>
>>>         </meta_attributes>
>>>         <operations>
>>>           <op id="scsi-shooter-monitor-interval-60s" interval="60s"
>>> name="monitor"/>
>>>         </operations>
>>>       </primitive>
>>>     </resources>
>>>     <constraints>
>>>       <rsc_order first="dlm-clone" first-action="start"
>>> id="order-dlm-clone-clvmd-clone-mandatory" then="clvmd-clone"
>>> then-action="start"/>
>>>       <rsc_colocation id="colocation-clvmd-clone-dlm-clone-INFINITY"
>>> rsc="clvmd-clone" score="INFINITY" with-rsc="dlm-clone"/>
>>>     </constraints>
>>>   </configuration>
>>> </cib>
>>>
>>> ***
>>> ******   -- Console Output -- Comments begin with "#"
>>> ***
>>>
>>> #
>>> ##
>>> ###  -- Moments before testing starts (Node 2)--
>>> ##
>>> #
>>> root at coro-test-2:~# date; pcs status
>>> Thu Sep  6 08:54:14 EDT 2018
>>> Cluster name: my_first_cluster
>>> Last updated: Thu Sep  6 08:54:14 2018 Last change: Wed Sep  5 16:28:53
>>> 2018 by root via cibadmin on coro-test-2
>>> Stack: corosync
>>> Current DC: coro-test-1 (version 1.1.14-70404b0) - partition with quorum
>>> 2 nodes and 5 resources configured
>>>
>>> Online: [ coro-test-1 coro-test-2 ]
>>>
>>> Full list of resources:
>>>
>>>  Clone Set: dlm-clone [dlm]
>>>      Started: [ coro-test-1 coro-test-2 ]
>>>  Clone Set: clvmd-clone [clvmd]
>>>      Started: [ coro-test-1 coro-test-2 ]
>>>  scsi-shooter (stonith:fence_scsi): Started coro-test-1
>>>
>>> PCSD Status:
>>>   coro-test-1: Online
>>>   coro-test-2: Online
>>>
>>> Daemon Status:
>>>   corosync: active/enabled
>>>   pacemaker: active/enabled
>>>   pcsd: active/enabled
>>> root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
>>>   PR generation=0x41, 2 registered reservation keys follow:
>>>     0x3ff20000
>>>     0x3ff20001
>>> root at coro-test-2:~# dlm_tool status
>>> cluster nodeid 2 quorate 1 ring seq 496 496
>>> daemon now 58996 fence_pid 0
>>> node 1 M add 92 rem 0 fail 0 fence 0 at 0 0
>>> node 2 M add 92 rem 0 fail 0 fence 0 at 0 0
>>> root at coro-test-2:~# dlm_tool ls
>>> dlm lockspaces
>>> name          clvmd
>>> id            0x4104eefa
>>> flags         0x00000000
>>> change        member 2 joined 1 remove 0 failed 0 seq 1,1
>>> members       1 2
>>>
>>>
>>> #
>>> ##
>>> ###  -- Node 1 is fenced --
>>> ##
>>> #
>>> root at coro-test-2:~# date; pcs stonith fence coro-test-1
>>> Thu Sep  6 08:55:44 EDT 2018
>>> Node: coro-test-1 fenced
>>> root at coro-test-2:~# pcs status
>>> Cluster name: my_first_cluster
>>> Last updated: Thu Sep  6 08:56:40 2018 Last change: Wed Sep  5 16:28:53
>>> 2018 by root via cibadmin on coro-test-2
>>> Stack: corosync
>>> Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum
>>> 2 nodes and 5 resources configured
>>>
>>> Node coro-test-1: pending
>>> Online: [ coro-test-2 ]
>>>
>>> Full list of resources:
>>>
>>>  Clone Set: dlm-clone [dlm]
>>>      Started: [ coro-test-2 ]
>>>      Stopped: [ coro-test-1 ]
>>>  Clone Set: clvmd-clone [clvmd]
>>>      Started: [ coro-test-2 ]
>>>      Stopped: [ coro-test-1 ]
>>>  scsi-shooter (stonith:fence_scsi): Stopped
>>>
>>> PCSD Status:
>>>   coro-test-1: Online
>>>   coro-test-2: Online
>>>
>>> Daemon Status:
>>>   corosync: active/enabled
>>>   pacemaker: active/enabled
>>>   pcsd: active/enabled
>>> root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
>>>   PR generation=0x42, 1 registered reservation key follows:
>>>     0x3ff20001
>>> root at coro-test-2:~# dlm_tool ls
>>> dlm lockspaces
>>> name          clvmd
>>> id            0x4104eefa
>>> flags         0x00000000
>>> change        member 2 joined 1 remove 0 failed 0 seq 1,1
>>> members       1 2
>>>
>>> root at coro-test-2:~# dlm_tool status
>>> cluster nodeid 2 quorate 1 ring seq 496 496
>>> daemon now 59166 fence_pid 0
>>> node 1 M add 92 rem 0 fail 0 fence 0 at 0 0
>>> node 2 M add 92 rem 0 fail 0 fence 0 at 0 0
>>> root at coro-test-2:~#
>>>
>>> #
>>> ## NOTE:  DLM still running on other node, coro-test-1, at this point.
>>> #
>>> root at coro-test-1:~# ps -aef | grep dlm
>>> root      1633     1  0 Sep05 ?        00:00:00 dlm_controld -s 0
>>> root      1752     2  0 Sep05 ?        00:00:00 [dlm_scand]
>>> root      1754     2  0 Sep05 ?        00:00:00 [dlm_recv]
>>> root      1755     2  0 Sep05 ?        00:00:00 [dlm_send]
>>> root      1756     2  0 Sep05 ?        00:00:00 [dlm_recoverd]
>>> root     29673  1850  0 08:58 pts/0    00:00:00 grep --color=auto dlm
>>> root at coro-test-1:~#
>>>
>>>
>>> #
>>> ##
>>> ###  -- node 2 output after node 1 is shutdown
>>> ##
>>> #
>>> #### NODE 1 SHUTDOWN ####
>>> root at coro-test-1:~# pcs status; date; init 0
>>> Error: cluster is not currently running on this node
>>> Thu Sep  6 09:03:38 EDT 2018
>>>
>>> #### NODE 2 UI OUTPUT ####
>>> root at coro-test-2:~# pcs status
>>> Cluster name: my_first_cluster
>>> Last updated: Thu Sep  6 09:04:16 2018 Last change: Wed Sep  5 16:28:53
>>> 2018 by root via cibadmin on coro-test-2
>>> Stack: corosync
>>> Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum
>>> 2 nodes and 5 resources configured
>>>
>>> Node coro-test-2: UNCLEAN (online)
>>> OFFLINE: [ coro-test-1 ]
>>>
>>> Full list of resources:
>>>
>>>  Clone Set: dlm-clone [dlm]
>>>      dlm (ocf::pacemaker:controld): FAILED coro-test-2
>>>      Stopped: [ coro-test-1 ]
>>>  Clone Set: clvmd-clone [clvmd]
>>>      Started: [ coro-test-2 ]
>>>      Stopped: [ coro-test-1 ]
>>>  scsi-shooter (stonith:fence_scsi): Started coro-test-2
>>>
>>> Failed Actions:
>>> * dlm_monitor_30000 on coro-test-2 'unknown error' (1): call=17,
>>> status=complete, exitreason='none',
>>>     last-rc-change='Thu Sep  6 09:04:09 2018', queued=0ms, exec=0ms
>>>
>>>
>>> PCSD Status:
>>>   coro-test-1: Offline
>>>   coro-test-2: Online
>>>
>>> Daemon Status:
>>>   corosync: active/enabled
>>>   pacemaker: active/enabled
>>>   pcsd: active/enabled
>>> root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
>>>   PR generation=0x42, 1 registered reservation key follows:
>>>     0x3ff20001
>>> root at coro-test-2:~# dlm_tool status
>>> cannot connect to dlm_controld
>>> root at coro-test-2:~# dlm_tool ls
>>> root at coro-test-2:~#
>>>
>>> ***
>>> ******   -- Syslog Output -- Comments begin with "#"
>>> ***
>>> #
>>> ###
>>> #####  NODE 2 FENCES NODE 1 (pcs stonith fence coro-test-1)
>>> ###
>>> #
>>>
>>> Sep  6 08:55:55 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[27468]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-1' ]
>>> Sep  6 08:55:55 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[27468]
>>> stderr: [  ]
>>> Sep  6 08:55:55 coro-test-2 stonith-ng[1541]:   notice: Operation
>>> 'reboot' [27468] (call 2 from stonith_admin.27448) for host 'coro-test-1'
>>> with device 'scsi-shooter' returned: 0 (OK)
>>> Sep  6 08:55:55 coro-test-2 stonith-ng[1541]:   notice: Operation reboot
>>> of coro-test-1 by coro-test-2 for stonith_admin.27448 at coro-test-2.5886f6b5:
>>> OK
>>> Sep  6 08:55:55 coro-test-2 stonith-ng[1541]:  warning:
>>> new_event_notification (1541-27448-12): Broken pipe (32)
>>> Sep  6 08:55:55 coro-test-2 crmd[1545]:   notice: State transition
>>> S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK
>>> origin=peer_update_callback ]
>>> Sep  6 08:55:55 coro-test-2 crmd[1545]:   notice: State transition
>>> S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED
>>> origin=election_timeout_popped ]
>>> Sep  6 08:55:55 coro-test-2 attrd[1543]:   notice: crm_update_peer_proc:
>>> Node coro-test-1[1] - state is now lost (was member)
>>> Sep  6 08:55:55 coro-test-2 attrd[1543]:   notice: Removing coro-test-1/1
>>> from the membership list
>>> Sep  6 08:55:55 coro-test-2 attrd[1543]:   notice: Purged 1 peers with
>>> id=1 and/or uname=coro-test-1 from the membership cache
>>> Sep  6 08:55:55 coro-test-2 stonith-ng[1541]:   notice:
>>> crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was member)
>>> Sep  6 08:55:55 coro-test-2 stonith-ng[1541]:   notice: Removing
>>> coro-test-1/1 from the membership list
>>> Sep  6 08:55:55 coro-test-2 stonith-ng[1541]:   notice: Purged 1 peers
>>> with id=1 and/or uname=coro-test-1 from the membership cache
>>> Sep  6 08:55:55 coro-test-2 cib[1540]:   notice: crm_update_peer_proc:
>>> Node coro-test-1[1] - state is now lost (was member)
>>> Sep  6 08:55:55 coro-test-2 cib[1540]:   notice: Removing coro-test-1/1
>>> from the membership list
>>> Sep  6 08:55:55 coro-test-2 cib[1540]:   notice: Purged 1 peers with id=1
>>> and/or uname=coro-test-1 from the membership cache
>>> Sep  6 08:55:55 coro-test-2 crmd[1545]:  warning: FSA: Input
>>> I_ELECTION_DC from do_election_check() received in state S_INTEGRATION
>>> Sep  6 08:55:55 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 08:55:55 coro-test-2 pengine[1544]:  warning: Action
>>> dlm:0_monitor_0 on coro-test-1 is unrunnable (pending)
>>> Sep  6 08:55:55 coro-test-2 pengine[1544]:  warning: Action
>>> clvmd:0_monitor_0 on coro-test-1 is unrunnable (pending)
>>> Sep  6 08:55:55 coro-test-2 pengine[1544]:  warning: Action
>>> scsi-shooter_monitor_0 on coro-test-1 is unrunnable (pending)
>>> Sep  6 08:55:55 coro-test-2 pengine[1544]:   notice: Unfencing
>>> coro-test-1: node discovery
>>> Sep  6 08:55:55 coro-test-2 pengine[1544]:   notice: Start
>>>  scsi-shooter#011(coro-test-2 - blocked)
>>> Sep  6 08:55:55 coro-test-2 stonith-ng[1541]:    error: Operation on of
>>> coro-test-1 by <no-one> for crmd.1545 at coro-test-2.b06d9d39: No such
>>> device
>>> Sep  6 08:55:55 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 08:55:55 coro-test-2 crmd[1545]:   notice: Transition 0
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=2,
>>> Source=/var/lib/pacemaker/pengine/pe-input-107.bz2): Complete
>>> Sep  6 08:55:55 coro-test-2 crmd[1545]:   notice: State transition
>>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>>> origin=notify_crmd ]
>>> Sep  6 08:55:55 coro-test-2 pengine[1544]:   notice: Calculated
>>> Transition 0: /var/lib/pacemaker/pengine/pe-input-107.bz2
>>> Sep  6 08:58:22 coro-test-2 dhclient[1070]: DHCPREQUEST of 10.0.2.5 on
>>> enp0s3 to 10.0.2.3 port 67 (xid=0x1c247c88)
>>> Sep  6 08:58:22 coro-test-2 dhclient[1070]: DHCPACK of 10.0.2.5 from
>>> 10.0.2.3
>>> Sep  6 08:58:22 coro-test-2 dhclient[1070]: bound to 10.0.2.5 -- renewal
>>> in 466 seconds.
>>> Sep  6 08:59:20 coro-test-2 dhclient[860]: DHCPREQUEST of 192.168.57.4 on
>>> enp0s8 to 192.168.57.2 port 67 (xid=0x5dfb57d7)
>>> Sep  6 08:59:20 coro-test-2 dhclient[860]: DHCPACK of 192.168.57.4 from
>>> 192.168.57.2
>>> Sep  6 08:59:20 coro-test-2 dhclient[860]: bound to 192.168.57.4 --
>>> renewal in 599 seconds.
>>>
>>> #
>>> ###
>>> #####   NODE 1 IS SHUTDOWN
>>> ###
>>> #
>>>
>>> ##
>>> ###     Cluster appears to see the loss of node 1
>>> ##
>>> Sep  6 09:03:38 coro-test-2 corosync[1505]: notice  [TOTEM ] A new
>>> membership (10.0.2.5:500) was formed. Members left: 1
>>> Sep  6 09:03:38 coro-test-2 corosync[1505]:  [TOTEM ] A new membership (
>>> 10.0.2.5:500) was formed. Members left: 1
>>> Sep  6 09:03:38 coro-test-2 pacemakerd[1537]:   notice:
>>> crm_reap_unseen_nodes: Node coro-test-1[1] - state is now lost (was member)
>>> Sep  6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 fence request 1 pid
>>> 28247 procdown time 1536239018 fence_all dlm_stonith
>>> Sep  6 09:03:38 coro-test-2 crmd[1545]:   notice: crm_reap_unseen_nodes:
>>> Node coro-test-1[1] - state is now lost (was member)
>>> Sep  6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 tell corosync to
>>> remove nodeid 1 from cluster
>>>
>>> ##
>>> ###     DLM appears to close off connection to NODE 1
>>> ##
>>> Sep  6 09:03:38 coro-test-2 kernel: [59552.844406] dlm: closing
>>> connection to node 1
>>> Sep  6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Found 2
>>> entries for 1/(null): 0 in progress, 1 completed
>>> Sep  6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Node 1/(null)
>>> last kicked at: 1536238555
>>>
>>> ##
>>> ###     Stonith attempts to fence node 1 (again; it was already fenced)
>>> ##
>>> Sep  6 09:03:42 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28248]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-1' ]
>>> Sep  6 09:03:42 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28248]
>>> stderr: [  ]
>>> Sep  6 09:03:42 coro-test-2 stonith-ng[1541]:   notice: Operation
>>> 'reboot' [28248] (call 2 from stonith-api.28247) for host 'coro-test-1'
>>> with device 'scsi-shooter' returned: 0 (OK)
>>> Sep  6 09:03:42 coro-test-2 stonith-ng[1541]:   notice: Operation reboot
>>> of coro-test-1 by coro-test-2 for stonith-api.28247 at coro-test-2.2365d094:
>>> OK
>>> Sep  6 09:03:42 coro-test-2 stonith-api[28247]: stonith_api_kick: Node
>>> 1/(null) kicked: reboot
>>> Sep  6 09:03:42 coro-test-2 crmd[1545]:   notice: State transition S_IDLE
>>> -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
>>> origin=abort_transition_graph ]
>>> Sep  6 09:03:42 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>>
>>> ##
>>> ###     I begin to get confused here:  Why is dlm closing the connection
>>> to the node 2 (itself??)
>>> ##
>>> Sep  6 09:03:42 coro-test-2 kernel: [59556.684699] dlm: closing
>>> connection to node 2
>>>
>>> ##
>>> ###     LOCKSPACE GOES ABANDONED!!  why?  WHY????
>>> ##
>>> Sep  6 09:03:42 coro-test-2 dlm_controld[1657]: 59554 abandoned lockspace
>>> clvmd
>>>
>>> ##
>>> ###     Alas, my cluster begins to topple itself
>>> ##
>>> Sep  6 09:03:42 coro-test-2 pengine[1544]:   notice: Start
>>>  scsi-shooter#011(coro-test-2)
>>> Sep  6 09:03:42 coro-test-2 pengine[1544]:   notice: Calculated
>>> Transition 1: /var/lib/pacemaker/pengine/pe-input-108.bz2
>>> Sep  6 09:03:42 coro-test-2 kernel: [59556.693578] dlm: dlm user daemon
>>> left 1 lockspaces
>>> Sep  6 09:03:43 coro-test-2 crmd[1545]:   notice: Operation
>>> scsi-shooter_start_0: ok (node=coro-test-2, call=20, rc=0, cib-update=56,
>>> confirmed=true)
>>> Sep  6 09:03:43 coro-test-2 crmd[1545]:   notice: Transition 1
>>> (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0,
>>> Source=/var/lib/pacemaker/pengine/pe-input-108.bz2): Complete
>>> Sep  6 09:03:43 coro-test-2 crmd[1545]:   notice: State transition
>>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>>> origin=notify_crmd ]
>>> Sep  6 09:04:05 coro-test-2 controld(dlm)[28324]: ERROR: Uncontrolled
>>> lockspace exists, system must reboot. Executing suicide fencing
>>> Sep  6 09:04:07 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:07 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:08 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28344]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:08 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28344]
>>> stderr: [  ]
>>> Sep  6 09:04:08 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28344]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:08 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28344]
>>> stderr: [  ]
>>> Sep  6 09:04:08 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28344]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:08 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28344]
>>> stderr: [  ]
>>> Sep  6 09:04:08 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28344]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:08 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28344]
>>> stderr: [  ]
>>> Sep  6 09:04:08 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28344]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:08 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28344]
>>> stderr: [  ]
>>> Sep  6 09:04:09 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:09 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28357]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28357]
>>> stderr: [  ]
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28357]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28357]
>>> stderr: [  ]
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28357]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28357]
>>> stderr: [  ]
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28357]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28357]
>>> stderr: [  ]
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28357]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28357]
>>> stderr: [  ]
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:    error: Operation
>>> 'reboot' [28357] (call 2 from stonith_admin.controld.28343) for host
>>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker
>>> error)
>>> Sep  6 09:04:09 coro-test-2 stonith-ng[1541]:    error: Operation reboot
>>> of coro-test-2 by coro-test-2 for
>>> stonith_admin.controld.28343 at coro-test-2.fd726e0b: Generic Pacemaker
>>> error
>>> Sep  6 09:04:09 coro-test-2 crmd[1545]:   notice:
>>> coro-test-2-dlm_monitor_30000:17 [ Command failed: Generic Pacemaker
>>> error\n ]
>>> Sep  6 09:04:09 coro-test-2 crmd[1545]:   notice: State transition S_IDLE
>>> -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
>>> origin=abort_transition_graph ]
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 2: /var/lib/pacemaker/pengine/pe-warn-68.bz2
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:04:09 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 3: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:04:13 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:13 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:13 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28370]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:13 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28370]
>>> stderr: [  ]
>>> Sep  6 09:04:13 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28370]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:13 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28370]
>>> stderr: [  ]
>>> Sep  6 09:04:13 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28370]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:13 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28370]
>>> stderr: [  ]
>>> Sep  6 09:04:13 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28370]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:13 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28370]
>>> stderr: [  ]
>>> Sep  6 09:04:13 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28370]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:13 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28370]
>>> stderr: [  ]
>>> Sep  6 09:04:14 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:14 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28383]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28383]
>>> stderr: [  ]
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28383]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28383]
>>> stderr: [  ]
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28383]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28383]
>>> stderr: [  ]
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28383]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28383]
>>> stderr: [  ]
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28383]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28383]
>>> stderr: [  ]
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28383] (call 3 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:04:14 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.171aafb6: Generic
>>> Pacemaker error
>>> Sep  6 09:04:14 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:04:14 coro-test-2 crmd[1545]:   notice: Transition 3
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:04:14 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 4: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:04:19 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:19 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:19 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28432]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:19 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28432]
>>> stderr: [  ]
>>> Sep  6 09:04:19 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28432]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:19 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28432]
>>> stderr: [  ]
>>> Sep  6 09:04:19 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28432]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:19 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28432]
>>> stderr: [  ]
>>> Sep  6 09:04:19 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28432]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:19 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28432]
>>> stderr: [  ]
>>> Sep  6 09:04:19 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28432]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:19 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28432]
>>> stderr: [  ]
>>> Sep  6 09:04:20 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:20 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28445]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28445]
>>> stderr: [  ]
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28445]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28445]
>>> stderr: [  ]
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28445]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28445]
>>> stderr: [  ]
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28445]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28445]
>>> stderr: [  ]
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28445]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28445]
>>> stderr: [  ]
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28445] (call 4 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:04:20 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.0b907f76: Generic
>>> Pacemaker error
>>> Sep  6 09:04:20 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:04:20 coro-test-2 crmd[1545]:   notice: Transition 4
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:04:20 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 5: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:04:24 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:24 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:24 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28477]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:24 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28477]
>>> stderr: [  ]
>>> Sep  6 09:04:24 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28477]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:24 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28477]
>>> stderr: [  ]
>>> Sep  6 09:04:24 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28477]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:24 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28477]
>>> stderr: [  ]
>>> Sep  6 09:04:24 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28477]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:24 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28477]
>>> stderr: [  ]
>>> Sep  6 09:04:24 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28477]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:24 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28477]
>>> stderr: [  ]
>>> Sep  6 09:04:25 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:25 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28490]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28490]
>>> stderr: [  ]
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28490]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28490]
>>> stderr: [  ]
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28490]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28490]
>>> stderr: [  ]
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28490]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28490]
>>> stderr: [  ]
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28490]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28490]
>>> stderr: [  ]
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28490] (call 5 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:04:25 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.559102cb: Generic
>>> Pacemaker error
>>> Sep  6 09:04:25 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:04:25 coro-test-2 crmd[1545]:   notice: Transition 5
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:04:25 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 6: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:04:39 coro-test-2 controld(dlm)[28520]: ERROR: Uncontrolled
>>> lockspace exists, system must reboot. Executing suicide fencing
>>> Sep  6 09:04:39 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:39 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:39 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28540]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:39 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28540]
>>> stderr: [  ]
>>> Sep  6 09:04:39 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28540]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:39 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28540]
>>> stderr: [  ]
>>> Sep  6 09:04:39 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28540]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:39 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28540]
>>> stderr: [  ]
>>> Sep  6 09:04:39 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28540]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:39 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28540]
>>> stderr: [  ]
>>> Sep  6 09:04:39 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28540]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:39 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28540]
>>> stderr: [  ]
>>> Sep  6 09:04:40 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:40 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28553]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28553]
>>> stderr: [  ]
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28553]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28553]
>>> stderr: [  ]
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28553]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28553]
>>> stderr: [  ]
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28553]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28553]
>>> stderr: [  ]
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28553]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28553]
>>> stderr: [  ]
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28553] (call 6 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:04:40 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.58604e64: Generic
>>> Pacemaker error
>>> Sep  6 09:04:40 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:04:40 coro-test-2 crmd[1545]:   notice: Transition 6
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:04:40 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 7: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:04:49 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:49 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:49 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28568]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:49 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28568]
>>> stderr: [  ]
>>> Sep  6 09:04:49 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28568]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:49 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28568]
>>> stderr: [  ]
>>> Sep  6 09:04:49 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28568]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:49 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28568]
>>> stderr: [  ]
>>> Sep  6 09:04:49 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28568]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:49 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28568]
>>> stderr: [  ]
>>> Sep  6 09:04:49 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28568]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:49 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28568]
>>> stderr: [  ]
>>> Sep  6 09:04:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:50 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28581]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28581]
>>> stderr: [  ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28581]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28581]
>>> stderr: [  ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28581]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28581]
>>> stderr: [  ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28581]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28581]
>>> stderr: [  ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28581]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28581]
>>> stderr: [  ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:    error: Operation
>>> 'reboot' [28581] (call 2 from stonith_admin.controld.28539) for host
>>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker
>>> error)
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:    error: Operation reboot
>>> of coro-test-2 by coro-test-2 for
>>> stonith_admin.controld.28539 at coro-test-2.29f17c2f: Generic Pacemaker
>>> error
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning:
>>> new_event_notification (1541-28539-12): Broken pipe (32)
>>> Sep  6 09:04:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:51 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28594]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28594]
>>> stderr: [  ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28594]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28594]
>>> stderr: [  ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28594]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28594]
>>> stderr: [  ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28594]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28594]
>>> stderr: [  ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28594]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28594]
>>> stderr: [  ]
>>> Sep  6 09:04:52 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:04:52 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28607]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28607]
>>> stderr: [  ]
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28607]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28607]
>>> stderr: [  ]
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28607]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28607]
>>> stderr: [  ]
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28607]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28607]
>>> stderr: [  ]
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28607]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28607]
>>> stderr: [  ]
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28607] (call 7 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:04:52 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.1c099fde: Generic
>>> Pacemaker error
>>> Sep  6 09:04:52 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:04:52 coro-test-2 crmd[1545]:   notice: Transition 7
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:04:52 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 8: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:05:01 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:01 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:01 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28643]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:01 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28643]
>>> stderr: [  ]
>>> Sep  6 09:05:01 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28643]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:01 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28643]
>>> stderr: [  ]
>>> Sep  6 09:05:01 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28643]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:01 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28643]
>>> stderr: [  ]
>>> Sep  6 09:05:01 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28643]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:01 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28643]
>>> stderr: [  ]
>>> Sep  6 09:05:01 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28643]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:01 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28643]
>>> stderr: [  ]
>>> Sep  6 09:05:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:02 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28656]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28656]
>>> stderr: [  ]
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28656]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28656]
>>> stderr: [  ]
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28656]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28656]
>>> stderr: [  ]
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28656]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28656]
>>> stderr: [  ]
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28656]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28656]
>>> stderr: [  ]
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28656] (call 8 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:05:02 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.41c7a339: Generic
>>> Pacemaker error
>>> Sep  6 09:05:02 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:05:02 coro-test-2 crmd[1545]:   notice: Transition 8
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:05:02 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 9: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:05:16 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:16 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:16 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28669]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:16 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28669]
>>> stderr: [  ]
>>> Sep  6 09:05:16 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28669]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:16 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28669]
>>> stderr: [  ]
>>> Sep  6 09:05:16 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28669]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:16 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28669]
>>> stderr: [  ]
>>> Sep  6 09:05:16 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28669]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:16 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28669]
>>> stderr: [  ]
>>> Sep  6 09:05:16 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28669]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:16 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28669]
>>> stderr: [  ]
>>> Sep  6 09:05:17 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:17 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28682]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28682]
>>> stderr: [  ]
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28682]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28682]
>>> stderr: [  ]
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28682]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28682]
>>> stderr: [  ]
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28682]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28682]
>>> stderr: [  ]
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28682]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28682]
>>> stderr: [  ]
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28682] (call 9 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:05:18 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.16fd9d3b: Generic
>>> Pacemaker error
>>> Sep  6 09:05:18 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:05:18 coro-test-2 crmd[1545]:   notice: Transition 9
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:05:18 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 10: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:05:21 coro-test-2 controld(dlm)[28695]: ERROR: Uncontrolled
>>> lockspace exists, system must reboot. Executing suicide fencing
>>> Sep  6 09:05:29 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:29 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:29 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28751]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:29 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28751]
>>> stderr: [  ]
>>> Sep  6 09:05:29 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28751]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:29 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28751]
>>> stderr: [  ]
>>> Sep  6 09:05:29 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28751]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:29 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28751]
>>> stderr: [  ]
>>> Sep  6 09:05:29 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28751]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:29 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28751]
>>> stderr: [  ]
>>> Sep  6 09:05:29 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28751]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:29 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28751]
>>> stderr: [  ]
>>> Sep  6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28764]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28764]
>>> stderr: [  ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28764]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28764]
>>> stderr: [  ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28764]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28764]
>>> stderr: [  ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28764]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28764]
>>> stderr: [  ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28764]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28764]
>>> stderr: [  ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28764] (call 10 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.c6433fd3: Generic
>>> Pacemaker error
>>> Sep  6 09:05:30 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:05:30 coro-test-2 crmd[1545]:   notice: Transition 10
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:05:30 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 11: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28777]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28777]
>>> stderr: [  ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28777]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28777]
>>> stderr: [  ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28777]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28777]
>>> stderr: [  ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28777]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28777]
>>> stderr: [  ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28777]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:30 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28777]
>>> stderr: [  ]
>>> Sep  6 09:05:31 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:31 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28790]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28790]
>>> stderr: [  ]
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28790]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28790]
>>> stderr: [  ]
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28790]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28790]
>>> stderr: [  ]
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28790]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28790]
>>> stderr: [  ]
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28790]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28790]
>>> stderr: [  ]
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:    error: Operation
>>> 'reboot' [28790] (call 2 from stonith_admin.controld.28714) for host
>>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker
>>> error)
>>> Sep  6 09:05:31 coro-test-2 stonith-ng[1541]:    error: Operation reboot
>>> of coro-test-2 by coro-test-2 for
>>> stonith_admin.controld.28714 at coro-test-2.ffe64359: Generic Pacemaker
>>> error
>>> Sep  6 09:05:36 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:36 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:36 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28803]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:36 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28803]
>>> stderr: [  ]
>>> Sep  6 09:05:36 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28803]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:36 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28803]
>>> stderr: [  ]
>>> Sep  6 09:05:36 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28803]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:36 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28803]
>>> stderr: [  ]
>>> Sep  6 09:05:36 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28803]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:36 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28803]
>>> stderr: [  ]
>>> Sep  6 09:05:36 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28803]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:36 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28803]
>>> stderr: [  ]
>>> Sep  6 09:05:37 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:37 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28816]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28816]
>>> stderr: [  ]
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28816]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28816]
>>> stderr: [  ]
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28816]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28816]
>>> stderr: [  ]
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28816]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28816]
>>> stderr: [  ]
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28816]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28816]
>>> stderr: [  ]
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28816] (call 11 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:05:37 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.7dfe97e8: Generic
>>> Pacemaker error
>>> Sep  6 09:05:37 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:05:37 coro-test-2 crmd[1545]:   notice: Transition 11
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:05:37 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 12: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:05:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:50 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:50 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28829]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:50 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28829]
>>> stderr: [  ]
>>> Sep  6 09:05:50 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28829]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:50 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28829]
>>> stderr: [  ]
>>> Sep  6 09:05:50 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28829]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:50 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28829]
>>> stderr: [  ]
>>> Sep  6 09:05:50 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28829]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:50 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28829]
>>> stderr: [  ]
>>> Sep  6 09:05:50 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28829]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:50 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28829]
>>> stderr: [  ]
>>> Sep  6 09:05:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:05:51 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28842]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28842]
>>> stderr: [  ]
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28842]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28842]
>>> stderr: [  ]
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28842]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28842]
>>> stderr: [  ]
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28842]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28842]
>>> stderr: [  ]
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28842]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28842]
>>> stderr: [  ]
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28842] (call 12 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:05:51 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.fb005423: Generic
>>> Pacemaker error
>>> Sep  6 09:05:51 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:05:51 coro-test-2 crmd[1545]:   notice: Transition 12
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:   notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:  warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:  warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:   notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:  warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:   notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:   notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:   notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:   notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep  6 09:05:51 coro-test-2 pengine[1544]:  warning: Calculated
>>> Transition 13: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep  6 09:06:01 coro-test-2 controld(dlm)[28874]: ERROR: Uncontrolled
>>> lockspace exists, system must reboot. Executing suicide fencing
>>> Sep  6 09:06:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:06:02 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:06:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28894]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:06:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28894]
>>> stderr: [  ]
>>> Sep  6 09:06:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28894]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:06:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28894]
>>> stderr: [  ]
>>> Sep  6 09:06:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28894]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:06:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28894]
>>> stderr: [  ]
>>> Sep  6 09:06:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28894]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:06:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28894]
>>> stderr: [  ]
>>> Sep  6 09:06:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28894]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:06:02 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28894]
>>> stderr: [  ]
>>> Sep  6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28907]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28907]
>>> stderr: [  ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28907]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28907]
>>> stderr: [  ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28907]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28907]
>>> stderr: [  ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28907]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28907]
>>> stderr: [  ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28907]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28907]
>>> stderr: [  ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:    error: Operation 'off'
>>> [28907] (call 13 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:    error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.e8cc12be: Generic
>>> Pacemaker error
>>> Sep  6 09:06:03 coro-test-2 crmd[1545]:   notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep  6 09:06:03 coro-test-2 crmd[1545]:   notice: Transition 13
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep  6 09:06:03 coro-test-2 crmd[1545]:   notice: State transition
>>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>>> origin=notify_crmd ]
>>> Sep  6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep  6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28922]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28922]
>>> stderr: [  ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28922]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28922]
>>> stderr: [  ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28922]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28922]
>>> stderr: [  ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28922]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28922]
>>> stderr: [  ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28922]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep  6 09:06:03 coro-test-2 stonith-ng[1541]:  warning: fence_scsi[28922]
>>> stderr: [  ]
>>> Sep  6 09:06:04
>>>
>>
> 
> -- 
> Patrick Whitney
> DevOps Engineer -- Tools






More information about the Users mailing list