[ClusterLabs] Antw: Re: 2 node cluster dlm/clvm trouble
Ulrich Windl
Ulrich.Windl at rz.uni-regensburg.de
Tue Sep 11 14:00:30 UTC 2018
Hi!
I think the simple answer is that DLM wants to be shut down orderly, just as cLVM and OCFS2 do.
Regards,
Ulrich
>>> Patrick Whitney <pwhitney at luminoso.com> schrieb am 11.09.2018 um 15:02 in
Nachricht
<CAE0zLk9OnX99vE0o9Xj+XtKLnP+uMUQFU6=2ygdNq4z_uxjaaA at mail.gmail.com>:
> What I'm having trouble understanding is why dlm flattens the remaining
> "running" node when the already fenced node is shutdown... I'm having
> trouble understanding how power fencing would cause dlm to behave any
> differently than just shutting down the fenced node.
>
> -Pat
>
>
> On Thu, Sep 6, 2018 at 2:35 PM Vladislav Bogdanov <bubble at hoster-ok.com>
> wrote:
>
>> On September 6, 2018 2:36:03 PM UTC, Patrick Whitney <
>> pwhitney at luminoso.com> wrote:
>>>
>>> Good Morning Everyone,
>>>
>>> I'm hoping someone with more experience with corosync and pacemaker can
>>> see what I am doing wrong.
>>>
>>> I've got a test setup of 2 nodes, with dlm and clvm setup as clones, and
>>> using fence_scsi as my fencing agent.
>>>
>>> I've got it to the point where the cluster is up, and reports it is
>>> happy. I then began testing fencing. When issuing 'pcs stonith fence' it
>>> appears to work; that is, the scsi reservation is pulled and the output of
>>> 'pcs status' looks sane, and I'm able to access resources on the un-fenced
>>> node.
>>>
>>> Things go awry when I shutdown (init 0) the fenced node... my unfenced
>>> node decides to fence itself (which looks like it was initiated by dlm due
>>> to an abandoned lockspace).
>>>
>>> I suspect this is due to misconfiguration, since I'm new to the toolset,
>>> but I'm not quite sure what I need to change.
>>>
>>> Any and all input is appreciated!
>>>
>>> Below is a chronology of events; my corosync config and cib.xml; command
>>> output; and annotated logs.
>>>
>>> Again, any hints, suggestions, wild guesses, or premonitions are welcomed
>>> -- I'm stuck! Please let me know if there is additional information which
>>> would be helpful.
>>>
>>> Many thanks,
>>> -Patrick W.
>>>
>>> Sep 6 08:54:14 -- Cluster is up and running; UI reports everything
>>> healthy.
>>>
>>> Sep 6 08:55:44 -- 'pcs stonith fence' called against node 1
>>> (coro-test-1);
>>> UI reports everything as expected -- that
>>> is, resources show only running on unfenced node and they're available.
>>> Oddly, although the UI says dlm is stopped
>>> on fenced node, the dlm_controld is still running.
>>>
>>> Sep 6 09:03:38 -- node 1 is shutdown, and node 2 falls to pieces.
>>> - First, corosync sees lost member --
>>> seems like this is appropriate, to me.
>>> - Next, dlm_controld calls to fence
>>> everything
>>> - stonith-ng tries to fence node 1 (but
>>> its already fenced!)
>>> - dlm closes connection to "node 2" (does
>>> dlm "nodes" map to cluster nodes? I'm not sure they do)
>>> - clvmd dlm lockspace is now abandoned;
>>> cluster attempts to fence the remaining node
>>> (But can't because scsi_fence doesn't
>>> work like that).
>>>
>>> ***
>>> ****** -- Configuration --
>>> ***
>>> root at coro-test-2:~# pcs --version
>>> 0.9.149
>>> root at coro-test-2:~# pacemakerd --version
>>> Pacemaker 1.1.14
>>> Written by Andrew Beekhof
>>> root at coro-test-2:~# corosync -v
>>> Corosync Cluster Engine, version '2.3.5'
>>> Copyright (c) 2006-2009 Red Hat, Inc.
>>> root at coro-test-2:~#
>>>
>>> #
>>> ## COROSYNC
>>> #
>>> root at coro-test-1:~# more /etc/corosync/corosync.conf
>>> totem {
>>> version: 2
>>> secauth: off
>>> cluster_name: my_first_cluster
>>> transport: udpu
>>> }
>>>
>>> nodelist {
>>> node {
>>> ring0_addr: coro-test-1
>>> nodeid: 1
>>> }
>>>
>>> node {
>>> ring0_addr: coro-test-2
>>> nodeid: 2
>>> }
>>> }
>>>
>>> quorum {
>>> provider: corosync_votequorum
>>> two_node: 1
>>> }
>>>
>>> logging {
>>> to_logfile: yes
>>> logfile: /var/log/corosync/corosync.log
>>> to_syslog: yes
>>> }
>>>
>>> ##
>>> ### PACEMAKER CIB.XML
>>> ##
>>> root at coro-test-1:/var/lib/pacemaker/cib# cat cib.xml
>>> <cib crm_feature_set="3.0.10" validate-with="pacemaker-2.4" epoch="30"
>>> num_updates="0" admin_epoch="0" cib-last-written="Thu Sep 6 10:12:46 2018"
>>> update-origin="coro-test-2" update-client="cibadmin" update-user="root"
>>> have-quorum="1" dc-uuid="1">
>>> <configuration>
>>> <crm_config>
>>> <cluster_property_set id="cib-bootstrap-options">
>>> <nvpair id="cib-bootstrap-options-have-watchdog"
>>> name="have-watchdog" value="false"/>
>>> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
>>> value="1.1.14-70404b0"/>
>>> <nvpair id="cib-bootstrap-options-cluster-infrastructure"
>>> name="cluster-infrastructure" value="corosync"/>
>>> <nvpair id="cib-bootstrap-options-cluster-name"
>>> name="cluster-name" value="my_first_cluster"/>
>>> <nvpair id="cib-bootstrap-options-no-quorum-policy"
>>> name="no-quorum-policy" value="ignore"/>
>>> <nvpair id="cib-bootstrap-options-stonith-enabled"
>>> name="stonith-enabled" value="true"/>
>>> <nvpair id="cib-bootstrap-options-stonith-action"
>>> name="stonith-action" value="off"/>
>>> </cluster_property_set>
>>> </crm_config>
>>> <nodes>
>>> <node id="1" uname="coro-test-1"/>
>>> <node id="2" uname="coro-test-2"/>
>>> </nodes>
>>> <resources>
>>> <clone id="dlm-clone">
>>> <primitive class="ocf" id="dlm" provider="pacemaker"
>>> type="controld">
>>> <instance_attributes id="dlm-instance_attributes"/>
>>> <operations>
>>> <op id="dlm-start-interval-0s" interval="0s" name="start"
>>> timeout="90"/>
>>> <op id="dlm-stop-interval-0s" interval="0s" name="stop"
>>> timeout="100"/>
>>> <op id="dlm-monitor-interval-30s" interval="30s"
>>> name="monitor" on-fail="fence"/>
>>> </operations>
>>> </primitive>
>>> <meta_attributes id="dlm-clone-meta_attributes">
>>> <nvpair id="dlm-clone-meta_attributes-interleave"
>>> name="interleave" value="true"/>
>>> <nvpair id="dlm-clone-meta_attributes-ordered" name="ordered"
>>> value="true"/>
>>> </meta_attributes>
>>> </clone>
>>> <clone id="clvmd-clone">
>>> <primitive class="ocf" id="clvmd" provider="heartbeat"
>>> type="clvm">
>>> <instance_attributes id="clvmd-instance_attributes"/>
>>> <operations>
>>> <op id="clvmd-start-interval-0s" interval="0s" name="start"
>>> timeout="90"/>
>>> <op id="clvmd-stop-interval-0s" interval="0s" name="stop"
>>> timeout="90"/>
>>> <op id="clvmd-monitor-interval-30s" interval="30s"
>>> name="monitor" on-fail="fence"/>
>>> </operations>
>>> </primitive>
>>> <meta_attributes id="clvmd-clone-meta_attributes">
>>> <nvpair id="clvmd-clone-meta_attributes-interleave"
>>> name="interleave" value="true"/>
>>> <nvpair id="clvmd-clone-meta_attributes-ordered" name="ordered"
>>> value="true"/>
>>> </meta_attributes>
>>> </clone>
>>> <primitive class="stonith" id="scsi-shooter" type="fence_scsi">
>>> <instance_attributes id="scsi-shooter-instance_attributes">
>>> <nvpair id="scsi-shooter-instance_attributes-pcmk_host_list"
>>> name="pcmk_host_list" value="coro-test-1 coro-test-2"/>
>>> <nvpair
>>> id="scsi-shooter-instance_attributes-pcmk_monitor_action"
>>> name="pcmk_monitor_action" value="metadata"/>
>>> <nvpair
>>> id="scsi-shooter-instance_attributes-pcmk_reboot_action"
>>> name="pcmk_reboot_action" value="off"/>
>>> <nvpair id="scsi-shooter-instance_attributes-pcmk_delay_max"
>>> name="pcmk_delay_max" value="15"/>
>>> <nvpair id="scsi-shooter-instance_attributes-devices"
>>> name="devices" value="/dev/sdb1"/>
>>> </instance_attributes>
>>> <meta_attributes id="scsi-shooter-meta_attributes">
>>> <nvpair id="scsi-shooter-meta_attributes-provides"
>>> name="provides" value="unfencing"/>
>>> </meta_attributes>
>>> <operations>
>>> <op id="scsi-shooter-monitor-interval-60s" interval="60s"
>>> name="monitor"/>
>>> </operations>
>>> </primitive>
>>> </resources>
>>> <constraints>
>>> <rsc_order first="dlm-clone" first-action="start"
>>> id="order-dlm-clone-clvmd-clone-mandatory" then="clvmd-clone"
>>> then-action="start"/>
>>> <rsc_colocation id="colocation-clvmd-clone-dlm-clone-INFINITY"
>>> rsc="clvmd-clone" score="INFINITY" with-rsc="dlm-clone"/>
>>> </constraints>
>>> </configuration>
>>> </cib>
>>>
>>> ***
>>> ****** -- Console Output -- Comments begin with "#"
>>> ***
>>>
>>> #
>>> ##
>>> ### -- Moments before testing starts (Node 2)--
>>> ##
>>> #
>>> root at coro-test-2:~# date; pcs status
>>> Thu Sep 6 08:54:14 EDT 2018
>>> Cluster name: my_first_cluster
>>> Last updated: Thu Sep 6 08:54:14 2018 Last change: Wed Sep 5 16:28:53
>>> 2018 by root via cibadmin on coro-test-2
>>> Stack: corosync
>>> Current DC: coro-test-1 (version 1.1.14-70404b0) - partition with quorum
>>> 2 nodes and 5 resources configured
>>>
>>> Online: [ coro-test-1 coro-test-2 ]
>>>
>>> Full list of resources:
>>>
>>> Clone Set: dlm-clone [dlm]
>>> Started: [ coro-test-1 coro-test-2 ]
>>> Clone Set: clvmd-clone [clvmd]
>>> Started: [ coro-test-1 coro-test-2 ]
>>> scsi-shooter (stonith:fence_scsi): Started coro-test-1
>>>
>>> PCSD Status:
>>> coro-test-1: Online
>>> coro-test-2: Online
>>>
>>> Daemon Status:
>>> corosync: active/enabled
>>> pacemaker: active/enabled
>>> pcsd: active/enabled
>>> root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
>>> PR generation=0x41, 2 registered reservation keys follow:
>>> 0x3ff20000
>>> 0x3ff20001
>>> root at coro-test-2:~# dlm_tool status
>>> cluster nodeid 2 quorate 1 ring seq 496 496
>>> daemon now 58996 fence_pid 0
>>> node 1 M add 92 rem 0 fail 0 fence 0 at 0 0
>>> node 2 M add 92 rem 0 fail 0 fence 0 at 0 0
>>> root at coro-test-2:~# dlm_tool ls
>>> dlm lockspaces
>>> name clvmd
>>> id 0x4104eefa
>>> flags 0x00000000
>>> change member 2 joined 1 remove 0 failed 0 seq 1,1
>>> members 1 2
>>>
>>>
>>> #
>>> ##
>>> ### -- Node 1 is fenced --
>>> ##
>>> #
>>> root at coro-test-2:~# date; pcs stonith fence coro-test-1
>>> Thu Sep 6 08:55:44 EDT 2018
>>> Node: coro-test-1 fenced
>>> root at coro-test-2:~# pcs status
>>> Cluster name: my_first_cluster
>>> Last updated: Thu Sep 6 08:56:40 2018 Last change: Wed Sep 5 16:28:53
>>> 2018 by root via cibadmin on coro-test-2
>>> Stack: corosync
>>> Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum
>>> 2 nodes and 5 resources configured
>>>
>>> Node coro-test-1: pending
>>> Online: [ coro-test-2 ]
>>>
>>> Full list of resources:
>>>
>>> Clone Set: dlm-clone [dlm]
>>> Started: [ coro-test-2 ]
>>> Stopped: [ coro-test-1 ]
>>> Clone Set: clvmd-clone [clvmd]
>>> Started: [ coro-test-2 ]
>>> Stopped: [ coro-test-1 ]
>>> scsi-shooter (stonith:fence_scsi): Stopped
>>>
>>> PCSD Status:
>>> coro-test-1: Online
>>> coro-test-2: Online
>>>
>>> Daemon Status:
>>> corosync: active/enabled
>>> pacemaker: active/enabled
>>> pcsd: active/enabled
>>> root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
>>> PR generation=0x42, 1 registered reservation key follows:
>>> 0x3ff20001
>>> root at coro-test-2:~# dlm_tool ls
>>> dlm lockspaces
>>> name clvmd
>>> id 0x4104eefa
>>> flags 0x00000000
>>> change member 2 joined 1 remove 0 failed 0 seq 1,1
>>> members 1 2
>>>
>>> root at coro-test-2:~# dlm_tool status
>>> cluster nodeid 2 quorate 1 ring seq 496 496
>>> daemon now 59166 fence_pid 0
>>> node 1 M add 92 rem 0 fail 0 fence 0 at 0 0
>>> node 2 M add 92 rem 0 fail 0 fence 0 at 0 0
>>> root at coro-test-2:~#
>>>
>>> #
>>> ## NOTE: DLM still running on other node, coro-test-1, at this point.
>>> #
>>> root at coro-test-1:~# ps -aef | grep dlm
>>> root 1633 1 0 Sep05 ? 00:00:00 dlm_controld -s 0
>>> root 1752 2 0 Sep05 ? 00:00:00 [dlm_scand]
>>> root 1754 2 0 Sep05 ? 00:00:00 [dlm_recv]
>>> root 1755 2 0 Sep05 ? 00:00:00 [dlm_send]
>>> root 1756 2 0 Sep05 ? 00:00:00 [dlm_recoverd]
>>> root 29673 1850 0 08:58 pts/0 00:00:00 grep --color=auto dlm
>>> root at coro-test-1:~#
>>>
>>>
>>> #
>>> ##
>>> ### -- node 2 output after node 1 is shutdown
>>> ##
>>> #
>>> #### NODE 1 SHUTDOWN ####
>>> root at coro-test-1:~# pcs status; date; init 0
>>> Error: cluster is not currently running on this node
>>> Thu Sep 6 09:03:38 EDT 2018
>>>
>>> #### NODE 2 UI OUTPUT ####
>>> root at coro-test-2:~# pcs status
>>> Cluster name: my_first_cluster
>>> Last updated: Thu Sep 6 09:04:16 2018 Last change: Wed Sep 5 16:28:53
>>> 2018 by root via cibadmin on coro-test-2
>>> Stack: corosync
>>> Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum
>>> 2 nodes and 5 resources configured
>>>
>>> Node coro-test-2: UNCLEAN (online)
>>> OFFLINE: [ coro-test-1 ]
>>>
>>> Full list of resources:
>>>
>>> Clone Set: dlm-clone [dlm]
>>> dlm (ocf::pacemaker:controld): FAILED coro-test-2
>>> Stopped: [ coro-test-1 ]
>>> Clone Set: clvmd-clone [clvmd]
>>> Started: [ coro-test-2 ]
>>> Stopped: [ coro-test-1 ]
>>> scsi-shooter (stonith:fence_scsi): Started coro-test-2
>>>
>>> Failed Actions:
>>> * dlm_monitor_30000 on coro-test-2 'unknown error' (1): call=17,
>>> status=complete, exitreason='none',
>>> last-rc-change='Thu Sep 6 09:04:09 2018', queued=0ms, exec=0ms
>>>
>>>
>>> PCSD Status:
>>> coro-test-1: Offline
>>> coro-test-2: Online
>>>
>>> Daemon Status:
>>> corosync: active/enabled
>>> pacemaker: active/enabled
>>> pcsd: active/enabled
>>> root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
>>> PR generation=0x42, 1 registered reservation key follows:
>>> 0x3ff20001
>>> root at coro-test-2:~# dlm_tool status
>>> cannot connect to dlm_controld
>>> root at coro-test-2:~# dlm_tool ls
>>> root at coro-test-2:~#
>>>
>>> ***
>>> ****** -- Syslog Output -- Comments begin with "#"
>>> ***
>>> #
>>> ###
>>> ##### NODE 2 FENCES NODE 1 (pcs stonith fence coro-test-1)
>>> ###
>>> #
>>>
>>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: fence_scsi[27468]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-1' ]
>>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: fence_scsi[27468]
>>> stderr: [ ]
>>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation
>>> 'reboot' [27468] (call 2 from stonith_admin.27448) for host 'coro-test-1'
>>> with device 'scsi-shooter' returned: 0 (OK)
>>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation reboot
>>> of coro-test-1 by coro-test-2 for stonith_admin.27448 at coro-test-2.5886f6b5:
>>> OK
>>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning:
>>> new_event_notification (1541-27448-12): Broken pipe (32)
>>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition
>>> S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK
>>> origin=peer_update_callback ]
>>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition
>>> S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED
>>> origin=election_timeout_popped ]
>>> Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: crm_update_peer_proc:
>>> Node coro-test-1[1] - state is now lost (was member)
>>> Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Removing coro-test-1/1
>>> from the membership list
>>> Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Purged 1 peers with
>>> id=1 and/or uname=coro-test-1 from the membership cache
>>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice:
>>> crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was member)
>>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Removing
>>> coro-test-1/1 from the membership list
>>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Purged 1 peers
>>> with id=1 and/or uname=coro-test-1 from the membership cache
>>> Sep 6 08:55:55 coro-test-2 cib[1540]: notice: crm_update_peer_proc:
>>> Node coro-test-1[1] - state is now lost (was member)
>>> Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Removing coro-test-1/1
>>> from the membership list
>>> Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Purged 1 peers with id=1
>>> and/or uname=coro-test-1 from the membership cache
>>> Sep 6 08:55:55 coro-test-2 crmd[1545]: warning: FSA: Input
>>> I_ELECTION_DC from do_election_check() received in state S_INTEGRATION
>>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action
>>> dlm:0_monitor_0 on coro-test-1 is unrunnable (pending)
>>> Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action
>>> clvmd:0_monitor_0 on coro-test-1 is unrunnable (pending)
>>> Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action
>>> scsi-shooter_monitor_0 on coro-test-1 is unrunnable (pending)
>>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Unfencing
>>> coro-test-1: node discovery
>>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Start
>>> scsi-shooter#011(coro-test-2 - blocked)
>>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: error: Operation on of
>>> coro-test-1 by <no-one> for crmd.1545 at coro-test-2.b06d9d39: No such
>>> device
>>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition 0
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=2,
>>> Source=/var/lib/pacemaker/pengine/pe-input-107.bz2): Complete
>>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition
>>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>>> origin=notify_crmd ]
>>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Calculated
>>> Transition 0: /var/lib/pacemaker/pengine/pe-input-107.bz2
>>> Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPREQUEST of 10.0.2.5 on
>>> enp0s3 to 10.0.2.3 port 67 (xid=0x1c247c88)
>>> Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPACK of 10.0.2.5 from
>>> 10.0.2.3
>>> Sep 6 08:58:22 coro-test-2 dhclient[1070]: bound to 10.0.2.5 -- renewal
>>> in 466 seconds.
>>> Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPREQUEST of 192.168.57.4 on
>>> enp0s8 to 192.168.57.2 port 67 (xid=0x5dfb57d7)
>>> Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPACK of 192.168.57.4 from
>>> 192.168.57.2
>>> Sep 6 08:59:20 coro-test-2 dhclient[860]: bound to 192.168.57.4 --
>>> renewal in 599 seconds.
>>>
>>> #
>>> ###
>>> ##### NODE 1 IS SHUTDOWN
>>> ###
>>> #
>>>
>>> ##
>>> ### Cluster appears to see the loss of node 1
>>> ##
>>> Sep 6 09:03:38 coro-test-2 corosync[1505]: notice [TOTEM ] A new
>>> membership (10.0.2.5:500) was formed. Members left: 1
>>> Sep 6 09:03:38 coro-test-2 corosync[1505]: [TOTEM ] A new membership (
>>> 10.0.2.5:500) was formed. Members left: 1
>>> Sep 6 09:03:38 coro-test-2 pacemakerd[1537]: notice:
>>> crm_reap_unseen_nodes: Node coro-test-1[1] - state is now lost (was member)
>>> Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 fence request 1 pid
>>> 28247 procdown time 1536239018 fence_all dlm_stonith
>>> Sep 6 09:03:38 coro-test-2 crmd[1545]: notice: crm_reap_unseen_nodes:
>>> Node coro-test-1[1] - state is now lost (was member)
>>> Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 tell corosync to
>>> remove nodeid 1 from cluster
>>>
>>> ##
>>> ### DLM appears to close off connection to NODE 1
>>> ##
>>> Sep 6 09:03:38 coro-test-2 kernel: [59552.844406] dlm: closing
>>> connection to node 1
>>> Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Found 2
>>> entries for 1/(null): 0 in progress, 1 completed
>>> Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Node 1/(null)
>>> last kicked at: 1536238555
>>>
>>> ##
>>> ### Stonith attempts to fence node 1 (again; it was already fenced)
>>> ##
>>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28248]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-1' ]
>>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28248]
>>> stderr: [ ]
>>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation
>>> 'reboot' [28248] (call 2 from stonith-api.28247) for host 'coro-test-1'
>>> with device 'scsi-shooter' returned: 0 (OK)
>>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation reboot
>>> of coro-test-1 by coro-test-2 for stonith-api.28247 at coro-test-2.2365d094:
>>> OK
>>> Sep 6 09:03:42 coro-test-2 stonith-api[28247]: stonith_api_kick: Node
>>> 1/(null) kicked: reboot
>>> Sep 6 09:03:42 coro-test-2 crmd[1545]: notice: State transition S_IDLE
>>> -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
>>> origin=abort_transition_graph ]
>>> Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>>
>>> ##
>>> ### I begin to get confused here: Why is dlm closing the connection
>>> to the node 2 (itself??)
>>> ##
>>> Sep 6 09:03:42 coro-test-2 kernel: [59556.684699] dlm: closing
>>> connection to node 2
>>>
>>> ##
>>> ### LOCKSPACE GOES ABANDONED!! why? WHY????
>>> ##
>>> Sep 6 09:03:42 coro-test-2 dlm_controld[1657]: 59554 abandoned lockspace
>>> clvmd
>>>
>>> ##
>>> ### Alas, my cluster begins to topple itself
>>> ##
>>> Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Start
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Calculated
>>> Transition 1: /var/lib/pacemaker/pengine/pe-input-108.bz2
>>> Sep 6 09:03:42 coro-test-2 kernel: [59556.693578] dlm: dlm user daemon
>>> left 1 lockspaces
>>> Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Operation
>>> scsi-shooter_start_0: ok (node=coro-test-2, call=20, rc=0, cib-update=56,
>>> confirmed=true)
>>> Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Transition 1
>>> (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0,
>>> Source=/var/lib/pacemaker/pengine/pe-input-108.bz2): Complete
>>> Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: State transition
>>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>>> origin=notify_crmd ]
>>> Sep 6 09:04:05 coro-test-2 controld(dlm)[28324]: ERROR: Uncontrolled
>>> lockspace exists, system must reboot. Executing suicide fencing
>>> Sep 6 09:04:07 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:07 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>>> stderr: [ ]
>>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>>> stderr: [ ]
>>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>>> stderr: [ ]
>>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>>> stderr: [ ]
>>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>>> stderr: [ ]
>>> Sep 6 09:04:09 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:09 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>>> stderr: [ ]
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>>> stderr: [ ]
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>>> stderr: [ ]
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>>> stderr: [ ]
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>>> stderr: [ ]
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation
>>> 'reboot' [28357] (call 2 from stonith_admin.controld.28343) for host
>>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker
>>> error)
>>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation reboot
>>> of coro-test-2 by coro-test-2 for
>>> stonith_admin.controld.28343 at coro-test-2.fd726e0b: Generic Pacemaker
>>> error
>>> Sep 6 09:04:09 coro-test-2 crmd[1545]: notice:
>>> coro-test-2-dlm_monitor_30000:17 [ Command failed: Generic Pacemaker
>>> error\n ]
>>> Sep 6 09:04:09 coro-test-2 crmd[1545]: notice: State transition S_IDLE
>>> -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
>>> origin=abort_transition_graph ]
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 2: /var/lib/pacemaker/pengine/pe-warn-68.bz2
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 3: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:04:13 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:13 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>>> stderr: [ ]
>>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>>> stderr: [ ]
>>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>>> stderr: [ ]
>>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>>> stderr: [ ]
>>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>>> stderr: [ ]
>>> Sep 6 09:04:14 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:14 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>>> stderr: [ ]
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>>> stderr: [ ]
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>>> stderr: [ ]
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>>> stderr: [ ]
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>>> stderr: [ ]
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28383] (call 3 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.171aafb6: Generic
>>> Pacemaker error
>>> Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition 3
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 4: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:04:19 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:19 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>>> stderr: [ ]
>>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>>> stderr: [ ]
>>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>>> stderr: [ ]
>>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>>> stderr: [ ]
>>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>>> stderr: [ ]
>>> Sep 6 09:04:20 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:20 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>>> stderr: [ ]
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>>> stderr: [ ]
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>>> stderr: [ ]
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>>> stderr: [ ]
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>>> stderr: [ ]
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28445] (call 4 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.0b907f76: Generic
>>> Pacemaker error
>>> Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition 4
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 5: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:04:24 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:24 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>>> stderr: [ ]
>>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>>> stderr: [ ]
>>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>>> stderr: [ ]
>>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>>> stderr: [ ]
>>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>>> stderr: [ ]
>>> Sep 6 09:04:25 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:25 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>>> stderr: [ ]
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>>> stderr: [ ]
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>>> stderr: [ ]
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>>> stderr: [ ]
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>>> stderr: [ ]
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28490] (call 5 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.559102cb: Generic
>>> Pacemaker error
>>> Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition 5
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 6: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:04:39 coro-test-2 controld(dlm)[28520]: ERROR: Uncontrolled
>>> lockspace exists, system must reboot. Executing suicide fencing
>>> Sep 6 09:04:39 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:39 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>>> stderr: [ ]
>>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>>> stderr: [ ]
>>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>>> stderr: [ ]
>>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>>> stderr: [ ]
>>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>>> stderr: [ ]
>>> Sep 6 09:04:40 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:40 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>>> stderr: [ ]
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>>> stderr: [ ]
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>>> stderr: [ ]
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>>> stderr: [ ]
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>>> stderr: [ ]
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28553] (call 6 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.58604e64: Generic
>>> Pacemaker error
>>> Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition 6
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 7: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:04:49 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:49 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>>> stderr: [ ]
>>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>>> stderr: [ ]
>>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>>> stderr: [ ]
>>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>>> stderr: [ ]
>>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>>> stderr: [ ]
>>> Sep 6 09:04:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:50 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>>> stderr: [ ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>>> stderr: [ ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>>> stderr: [ ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>>> stderr: [ ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>>> stderr: [ ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation
>>> 'reboot' [28581] (call 2 from stonith_admin.controld.28539) for host
>>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker
>>> error)
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation reboot
>>> of coro-test-2 by coro-test-2 for
>>> stonith_admin.controld.28539 at coro-test-2.29f17c2f: Generic Pacemaker
>>> error
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>>> new_event_notification (1541-28539-12): Broken pipe (32)
>>> Sep 6 09:04:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:51 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>>> stderr: [ ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>>> stderr: [ ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>>> stderr: [ ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>>> stderr: [ ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>>> stderr: [ ]
>>> Sep 6 09:04:52 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:04:52 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>>> stderr: [ ]
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>>> stderr: [ ]
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>>> stderr: [ ]
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>>> stderr: [ ]
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>>> stderr: [ ]
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28607] (call 7 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.1c099fde: Generic
>>> Pacemaker error
>>> Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition 7
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 8: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:05:01 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:01 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>>> stderr: [ ]
>>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>>> stderr: [ ]
>>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>>> stderr: [ ]
>>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>>> stderr: [ ]
>>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>>> stderr: [ ]
>>> Sep 6 09:05:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:02 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>>> stderr: [ ]
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>>> stderr: [ ]
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>>> stderr: [ ]
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>>> stderr: [ ]
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>>> stderr: [ ]
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28656] (call 8 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.41c7a339: Generic
>>> Pacemaker error
>>> Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition 8
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 9: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:05:16 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:16 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>>> stderr: [ ]
>>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>>> stderr: [ ]
>>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>>> stderr: [ ]
>>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>>> stderr: [ ]
>>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>>> stderr: [ ]
>>> Sep 6 09:05:17 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:17 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>>> stderr: [ ]
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>>> stderr: [ ]
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>>> stderr: [ ]
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>>> stderr: [ ]
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>>> stderr: [ ]
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28682] (call 9 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.16fd9d3b: Generic
>>> Pacemaker error
>>> Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition 9
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 10: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:05:21 coro-test-2 controld(dlm)[28695]: ERROR: Uncontrolled
>>> lockspace exists, system must reboot. Executing suicide fencing
>>> Sep 6 09:05:29 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:29 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>>> stderr: [ ]
>>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>>> stderr: [ ]
>>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>>> stderr: [ ]
>>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>>> stderr: [ ]
>>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>>> stderr: [ ]
>>> Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>>> stderr: [ ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>>> stderr: [ ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>>> stderr: [ ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>>> stderr: [ ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>>> stderr: [ ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28764] (call 10 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.c6433fd3: Generic
>>> Pacemaker error
>>> Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition 10
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 11: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>>> stderr: [ ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>>> stderr: [ ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>>> stderr: [ ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>>> stderr: [ ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>>> stderr: [ ]
>>> Sep 6 09:05:31 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:31 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>>> stderr: [ ]
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>>> stderr: [ ]
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>>> stderr: [ ]
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>>> stderr: [ ]
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>>> stderr: [ ]
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation
>>> 'reboot' [28790] (call 2 from stonith_admin.controld.28714) for host
>>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker
>>> error)
>>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation reboot
>>> of coro-test-2 by coro-test-2 for
>>> stonith_admin.controld.28714 at coro-test-2.ffe64359: Generic Pacemaker
>>> error
>>> Sep 6 09:05:36 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:36 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>>> stderr: [ ]
>>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>>> stderr: [ ]
>>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>>> stderr: [ ]
>>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>>> stderr: [ ]
>>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>>> stderr: [ ]
>>> Sep 6 09:05:37 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:37 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>>> stderr: [ ]
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>>> stderr: [ ]
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>>> stderr: [ ]
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>>> stderr: [ ]
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>>> stderr: [ ]
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28816] (call 11 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.7dfe97e8: Generic
>>> Pacemaker error
>>> Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition 11
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 12: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:05:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:50 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>>> stderr: [ ]
>>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>>> stderr: [ ]
>>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>>> stderr: [ ]
>>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>>> stderr: [ ]
>>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>>> stderr: [ ]
>>> Sep 6 09:05:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:05:51 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>>> stderr: [ ]
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>>> stderr: [ ]
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>>> stderr: [ ]
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>>> stderr: [ ]
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>>> stderr: [ ]
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28842] (call 12 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.fb005423: Generic
>>> Pacemaker error
>>> Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition 12
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: On loss of CCM
>>> Quorum: Ignore
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Processing failed op
>>> monitor for dlm:0 on coro-test-2: unknown error (1)
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Node coro-test-2
>>> will be fenced because of resource failure(s)
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>>> with instance of dlm-clone
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>>> with instance of dlm-clone
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Scheduling Node
>>> coro-test-2 for STONITH
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop of failed
>>> resource dlm:0 is implicit after coro-test-2 is fenced
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop
>>> dlm:0#011(coro-test-2)
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop
>>> clvmd:0#011(coro-test-2)
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop
>>> scsi-shooter#011(coro-test-2)
>>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Calculated
>>> Transition 13: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>>> Sep 6 09:06:01 coro-test-2 controld(dlm)[28874]: ERROR: Uncontrolled
>>> lockspace exists, system must reboot. Executing suicide fencing
>>> Sep 6 09:06:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:06:02 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>>> stderr: [ ]
>>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>>> stderr: [ ]
>>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>>> stderr: [ ]
>>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>>> stderr: [ ]
>>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>>> stderr: [ ]
>>> Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>>> stderr: [ ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>>> stderr: [ ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>>> stderr: [ ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>>> stderr: [ ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>>> stderr: [ ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>>> [28907] (call 13 from crmd.1545) for host 'coro-test-2' with device
>>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation off of
>>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.e8cc12be: Generic
>>> Pacemaker error
>>> Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition aborted:
>>> Stonith failed (source=tengine_stonith_callback:748, 0)
>>> Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition 13
>>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>>> Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: State transition
>>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>>> origin=notify_crmd ]
>>> Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>>> can not fence yourself.
>>> Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>>> 'port=coro-test-2' ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>>> stderr: [ ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>>> yourself. ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>>> stderr: [ ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>>> stderr: [ ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>>> stderr: [ ERROR:root:Please use '-h' for usage ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>>> stderr: [ ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>>> stderr: [ Please use '-h' for usage ]
>>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>>> stderr: [ ]
>>> Sep 6 09:06:04
>>>
>>
>
> --
> Patrick Whitney
> DevOps Engineer -- Tools
More information about the Users
mailing list