[ClusterLabs] 2 node cluster dlm/clvm trouble
Patrick Whitney
pwhitney at luminoso.com
Tue Sep 11 09:02:06 EDT 2018
What I'm having trouble understanding is why dlm flattens the remaining
"running" node when the already fenced node is shutdown... I'm having
trouble understanding how power fencing would cause dlm to behave any
differently than just shutting down the fenced node.
-Pat
On Thu, Sep 6, 2018 at 2:35 PM Vladislav Bogdanov <bubble at hoster-ok.com>
wrote:
> On September 6, 2018 2:36:03 PM UTC, Patrick Whitney <
> pwhitney at luminoso.com> wrote:
>>
>> Good Morning Everyone,
>>
>> I'm hoping someone with more experience with corosync and pacemaker can
>> see what I am doing wrong.
>>
>> I've got a test setup of 2 nodes, with dlm and clvm setup as clones, and
>> using fence_scsi as my fencing agent.
>>
>> I've got it to the point where the cluster is up, and reports it is
>> happy. I then began testing fencing. When issuing 'pcs stonith fence' it
>> appears to work; that is, the scsi reservation is pulled and the output of
>> 'pcs status' looks sane, and I'm able to access resources on the un-fenced
>> node.
>>
>> Things go awry when I shutdown (init 0) the fenced node... my unfenced
>> node decides to fence itself (which looks like it was initiated by dlm due
>> to an abandoned lockspace).
>>
>> I suspect this is due to misconfiguration, since I'm new to the toolset,
>> but I'm not quite sure what I need to change.
>>
>> Any and all input is appreciated!
>>
>> Below is a chronology of events; my corosync config and cib.xml; command
>> output; and annotated logs.
>>
>> Again, any hints, suggestions, wild guesses, or premonitions are welcomed
>> -- I'm stuck! Please let me know if there is additional information which
>> would be helpful.
>>
>> Many thanks,
>> -Patrick W.
>>
>> Sep 6 08:54:14 -- Cluster is up and running; UI reports everything
>> healthy.
>>
>> Sep 6 08:55:44 -- 'pcs stonith fence' called against node 1
>> (coro-test-1);
>> UI reports everything as expected -- that
>> is, resources show only running on unfenced node and they're available.
>> Oddly, although the UI says dlm is stopped
>> on fenced node, the dlm_controld is still running.
>>
>> Sep 6 09:03:38 -- node 1 is shutdown, and node 2 falls to pieces.
>> - First, corosync sees lost member --
>> seems like this is appropriate, to me.
>> - Next, dlm_controld calls to fence
>> everything
>> - stonith-ng tries to fence node 1 (but
>> its already fenced!)
>> - dlm closes connection to "node 2" (does
>> dlm "nodes" map to cluster nodes? I'm not sure they do)
>> - clvmd dlm lockspace is now abandoned;
>> cluster attempts to fence the remaining node
>> (But can't because scsi_fence doesn't
>> work like that).
>>
>> ***
>> ****** -- Configuration --
>> ***
>> root at coro-test-2:~# pcs --version
>> 0.9.149
>> root at coro-test-2:~# pacemakerd --version
>> Pacemaker 1.1.14
>> Written by Andrew Beekhof
>> root at coro-test-2:~# corosync -v
>> Corosync Cluster Engine, version '2.3.5'
>> Copyright (c) 2006-2009 Red Hat, Inc.
>> root at coro-test-2:~#
>>
>> #
>> ## COROSYNC
>> #
>> root at coro-test-1:~# more /etc/corosync/corosync.conf
>> totem {
>> version: 2
>> secauth: off
>> cluster_name: my_first_cluster
>> transport: udpu
>> }
>>
>> nodelist {
>> node {
>> ring0_addr: coro-test-1
>> nodeid: 1
>> }
>>
>> node {
>> ring0_addr: coro-test-2
>> nodeid: 2
>> }
>> }
>>
>> quorum {
>> provider: corosync_votequorum
>> two_node: 1
>> }
>>
>> logging {
>> to_logfile: yes
>> logfile: /var/log/corosync/corosync.log
>> to_syslog: yes
>> }
>>
>> ##
>> ### PACEMAKER CIB.XML
>> ##
>> root at coro-test-1:/var/lib/pacemaker/cib# cat cib.xml
>> <cib crm_feature_set="3.0.10" validate-with="pacemaker-2.4" epoch="30"
>> num_updates="0" admin_epoch="0" cib-last-written="Thu Sep 6 10:12:46 2018"
>> update-origin="coro-test-2" update-client="cibadmin" update-user="root"
>> have-quorum="1" dc-uuid="1">
>> <configuration>
>> <crm_config>
>> <cluster_property_set id="cib-bootstrap-options">
>> <nvpair id="cib-bootstrap-options-have-watchdog"
>> name="have-watchdog" value="false"/>
>> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
>> value="1.1.14-70404b0"/>
>> <nvpair id="cib-bootstrap-options-cluster-infrastructure"
>> name="cluster-infrastructure" value="corosync"/>
>> <nvpair id="cib-bootstrap-options-cluster-name"
>> name="cluster-name" value="my_first_cluster"/>
>> <nvpair id="cib-bootstrap-options-no-quorum-policy"
>> name="no-quorum-policy" value="ignore"/>
>> <nvpair id="cib-bootstrap-options-stonith-enabled"
>> name="stonith-enabled" value="true"/>
>> <nvpair id="cib-bootstrap-options-stonith-action"
>> name="stonith-action" value="off"/>
>> </cluster_property_set>
>> </crm_config>
>> <nodes>
>> <node id="1" uname="coro-test-1"/>
>> <node id="2" uname="coro-test-2"/>
>> </nodes>
>> <resources>
>> <clone id="dlm-clone">
>> <primitive class="ocf" id="dlm" provider="pacemaker"
>> type="controld">
>> <instance_attributes id="dlm-instance_attributes"/>
>> <operations>
>> <op id="dlm-start-interval-0s" interval="0s" name="start"
>> timeout="90"/>
>> <op id="dlm-stop-interval-0s" interval="0s" name="stop"
>> timeout="100"/>
>> <op id="dlm-monitor-interval-30s" interval="30s"
>> name="monitor" on-fail="fence"/>
>> </operations>
>> </primitive>
>> <meta_attributes id="dlm-clone-meta_attributes">
>> <nvpair id="dlm-clone-meta_attributes-interleave"
>> name="interleave" value="true"/>
>> <nvpair id="dlm-clone-meta_attributes-ordered" name="ordered"
>> value="true"/>
>> </meta_attributes>
>> </clone>
>> <clone id="clvmd-clone">
>> <primitive class="ocf" id="clvmd" provider="heartbeat"
>> type="clvm">
>> <instance_attributes id="clvmd-instance_attributes"/>
>> <operations>
>> <op id="clvmd-start-interval-0s" interval="0s" name="start"
>> timeout="90"/>
>> <op id="clvmd-stop-interval-0s" interval="0s" name="stop"
>> timeout="90"/>
>> <op id="clvmd-monitor-interval-30s" interval="30s"
>> name="monitor" on-fail="fence"/>
>> </operations>
>> </primitive>
>> <meta_attributes id="clvmd-clone-meta_attributes">
>> <nvpair id="clvmd-clone-meta_attributes-interleave"
>> name="interleave" value="true"/>
>> <nvpair id="clvmd-clone-meta_attributes-ordered" name="ordered"
>> value="true"/>
>> </meta_attributes>
>> </clone>
>> <primitive class="stonith" id="scsi-shooter" type="fence_scsi">
>> <instance_attributes id="scsi-shooter-instance_attributes">
>> <nvpair id="scsi-shooter-instance_attributes-pcmk_host_list"
>> name="pcmk_host_list" value="coro-test-1 coro-test-2"/>
>> <nvpair
>> id="scsi-shooter-instance_attributes-pcmk_monitor_action"
>> name="pcmk_monitor_action" value="metadata"/>
>> <nvpair
>> id="scsi-shooter-instance_attributes-pcmk_reboot_action"
>> name="pcmk_reboot_action" value="off"/>
>> <nvpair id="scsi-shooter-instance_attributes-pcmk_delay_max"
>> name="pcmk_delay_max" value="15"/>
>> <nvpair id="scsi-shooter-instance_attributes-devices"
>> name="devices" value="/dev/sdb1"/>
>> </instance_attributes>
>> <meta_attributes id="scsi-shooter-meta_attributes">
>> <nvpair id="scsi-shooter-meta_attributes-provides"
>> name="provides" value="unfencing"/>
>> </meta_attributes>
>> <operations>
>> <op id="scsi-shooter-monitor-interval-60s" interval="60s"
>> name="monitor"/>
>> </operations>
>> </primitive>
>> </resources>
>> <constraints>
>> <rsc_order first="dlm-clone" first-action="start"
>> id="order-dlm-clone-clvmd-clone-mandatory" then="clvmd-clone"
>> then-action="start"/>
>> <rsc_colocation id="colocation-clvmd-clone-dlm-clone-INFINITY"
>> rsc="clvmd-clone" score="INFINITY" with-rsc="dlm-clone"/>
>> </constraints>
>> </configuration>
>> </cib>
>>
>> ***
>> ****** -- Console Output -- Comments begin with "#"
>> ***
>>
>> #
>> ##
>> ### -- Moments before testing starts (Node 2)--
>> ##
>> #
>> root at coro-test-2:~# date; pcs status
>> Thu Sep 6 08:54:14 EDT 2018
>> Cluster name: my_first_cluster
>> Last updated: Thu Sep 6 08:54:14 2018 Last change: Wed Sep 5 16:28:53
>> 2018 by root via cibadmin on coro-test-2
>> Stack: corosync
>> Current DC: coro-test-1 (version 1.1.14-70404b0) - partition with quorum
>> 2 nodes and 5 resources configured
>>
>> Online: [ coro-test-1 coro-test-2 ]
>>
>> Full list of resources:
>>
>> Clone Set: dlm-clone [dlm]
>> Started: [ coro-test-1 coro-test-2 ]
>> Clone Set: clvmd-clone [clvmd]
>> Started: [ coro-test-1 coro-test-2 ]
>> scsi-shooter (stonith:fence_scsi): Started coro-test-1
>>
>> PCSD Status:
>> coro-test-1: Online
>> coro-test-2: Online
>>
>> Daemon Status:
>> corosync: active/enabled
>> pacemaker: active/enabled
>> pcsd: active/enabled
>> root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
>> PR generation=0x41, 2 registered reservation keys follow:
>> 0x3ff20000
>> 0x3ff20001
>> root at coro-test-2:~# dlm_tool status
>> cluster nodeid 2 quorate 1 ring seq 496 496
>> daemon now 58996 fence_pid 0
>> node 1 M add 92 rem 0 fail 0 fence 0 at 0 0
>> node 2 M add 92 rem 0 fail 0 fence 0 at 0 0
>> root at coro-test-2:~# dlm_tool ls
>> dlm lockspaces
>> name clvmd
>> id 0x4104eefa
>> flags 0x00000000
>> change member 2 joined 1 remove 0 failed 0 seq 1,1
>> members 1 2
>>
>>
>> #
>> ##
>> ### -- Node 1 is fenced --
>> ##
>> #
>> root at coro-test-2:~# date; pcs stonith fence coro-test-1
>> Thu Sep 6 08:55:44 EDT 2018
>> Node: coro-test-1 fenced
>> root at coro-test-2:~# pcs status
>> Cluster name: my_first_cluster
>> Last updated: Thu Sep 6 08:56:40 2018 Last change: Wed Sep 5 16:28:53
>> 2018 by root via cibadmin on coro-test-2
>> Stack: corosync
>> Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum
>> 2 nodes and 5 resources configured
>>
>> Node coro-test-1: pending
>> Online: [ coro-test-2 ]
>>
>> Full list of resources:
>>
>> Clone Set: dlm-clone [dlm]
>> Started: [ coro-test-2 ]
>> Stopped: [ coro-test-1 ]
>> Clone Set: clvmd-clone [clvmd]
>> Started: [ coro-test-2 ]
>> Stopped: [ coro-test-1 ]
>> scsi-shooter (stonith:fence_scsi): Stopped
>>
>> PCSD Status:
>> coro-test-1: Online
>> coro-test-2: Online
>>
>> Daemon Status:
>> corosync: active/enabled
>> pacemaker: active/enabled
>> pcsd: active/enabled
>> root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
>> PR generation=0x42, 1 registered reservation key follows:
>> 0x3ff20001
>> root at coro-test-2:~# dlm_tool ls
>> dlm lockspaces
>> name clvmd
>> id 0x4104eefa
>> flags 0x00000000
>> change member 2 joined 1 remove 0 failed 0 seq 1,1
>> members 1 2
>>
>> root at coro-test-2:~# dlm_tool status
>> cluster nodeid 2 quorate 1 ring seq 496 496
>> daemon now 59166 fence_pid 0
>> node 1 M add 92 rem 0 fail 0 fence 0 at 0 0
>> node 2 M add 92 rem 0 fail 0 fence 0 at 0 0
>> root at coro-test-2:~#
>>
>> #
>> ## NOTE: DLM still running on other node, coro-test-1, at this point.
>> #
>> root at coro-test-1:~# ps -aef | grep dlm
>> root 1633 1 0 Sep05 ? 00:00:00 dlm_controld -s 0
>> root 1752 2 0 Sep05 ? 00:00:00 [dlm_scand]
>> root 1754 2 0 Sep05 ? 00:00:00 [dlm_recv]
>> root 1755 2 0 Sep05 ? 00:00:00 [dlm_send]
>> root 1756 2 0 Sep05 ? 00:00:00 [dlm_recoverd]
>> root 29673 1850 0 08:58 pts/0 00:00:00 grep --color=auto dlm
>> root at coro-test-1:~#
>>
>>
>> #
>> ##
>> ### -- node 2 output after node 1 is shutdown
>> ##
>> #
>> #### NODE 1 SHUTDOWN ####
>> root at coro-test-1:~# pcs status; date; init 0
>> Error: cluster is not currently running on this node
>> Thu Sep 6 09:03:38 EDT 2018
>>
>> #### NODE 2 UI OUTPUT ####
>> root at coro-test-2:~# pcs status
>> Cluster name: my_first_cluster
>> Last updated: Thu Sep 6 09:04:16 2018 Last change: Wed Sep 5 16:28:53
>> 2018 by root via cibadmin on coro-test-2
>> Stack: corosync
>> Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum
>> 2 nodes and 5 resources configured
>>
>> Node coro-test-2: UNCLEAN (online)
>> OFFLINE: [ coro-test-1 ]
>>
>> Full list of resources:
>>
>> Clone Set: dlm-clone [dlm]
>> dlm (ocf::pacemaker:controld): FAILED coro-test-2
>> Stopped: [ coro-test-1 ]
>> Clone Set: clvmd-clone [clvmd]
>> Started: [ coro-test-2 ]
>> Stopped: [ coro-test-1 ]
>> scsi-shooter (stonith:fence_scsi): Started coro-test-2
>>
>> Failed Actions:
>> * dlm_monitor_30000 on coro-test-2 'unknown error' (1): call=17,
>> status=complete, exitreason='none',
>> last-rc-change='Thu Sep 6 09:04:09 2018', queued=0ms, exec=0ms
>>
>>
>> PCSD Status:
>> coro-test-1: Offline
>> coro-test-2: Online
>>
>> Daemon Status:
>> corosync: active/enabled
>> pacemaker: active/enabled
>> pcsd: active/enabled
>> root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
>> PR generation=0x42, 1 registered reservation key follows:
>> 0x3ff20001
>> root at coro-test-2:~# dlm_tool status
>> cannot connect to dlm_controld
>> root at coro-test-2:~# dlm_tool ls
>> root at coro-test-2:~#
>>
>> ***
>> ****** -- Syslog Output -- Comments begin with "#"
>> ***
>> #
>> ###
>> ##### NODE 2 FENCES NODE 1 (pcs stonith fence coro-test-1)
>> ###
>> #
>>
>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: fence_scsi[27468]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-1' ]
>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: fence_scsi[27468]
>> stderr: [ ]
>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation
>> 'reboot' [27468] (call 2 from stonith_admin.27448) for host 'coro-test-1'
>> with device 'scsi-shooter' returned: 0 (OK)
>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation reboot
>> of coro-test-1 by coro-test-2 for stonith_admin.27448 at coro-test-2.5886f6b5:
>> OK
>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning:
>> new_event_notification (1541-27448-12): Broken pipe (32)
>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition
>> S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK
>> origin=peer_update_callback ]
>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition
>> S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED
>> origin=election_timeout_popped ]
>> Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: crm_update_peer_proc:
>> Node coro-test-1[1] - state is now lost (was member)
>> Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Removing coro-test-1/1
>> from the membership list
>> Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Purged 1 peers with
>> id=1 and/or uname=coro-test-1 from the membership cache
>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice:
>> crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was member)
>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Removing
>> coro-test-1/1 from the membership list
>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Purged 1 peers
>> with id=1 and/or uname=coro-test-1 from the membership cache
>> Sep 6 08:55:55 coro-test-2 cib[1540]: notice: crm_update_peer_proc:
>> Node coro-test-1[1] - state is now lost (was member)
>> Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Removing coro-test-1/1
>> from the membership list
>> Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Purged 1 peers with id=1
>> and/or uname=coro-test-1 from the membership cache
>> Sep 6 08:55:55 coro-test-2 crmd[1545]: warning: FSA: Input
>> I_ELECTION_DC from do_election_check() received in state S_INTEGRATION
>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action
>> dlm:0_monitor_0 on coro-test-1 is unrunnable (pending)
>> Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action
>> clvmd:0_monitor_0 on coro-test-1 is unrunnable (pending)
>> Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action
>> scsi-shooter_monitor_0 on coro-test-1 is unrunnable (pending)
>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Unfencing
>> coro-test-1: node discovery
>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Start
>> scsi-shooter#011(coro-test-2 - blocked)
>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: error: Operation on of
>> coro-test-1 by <no-one> for crmd.1545 at coro-test-2.b06d9d39: No such
>> device
>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition 0
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=2,
>> Source=/var/lib/pacemaker/pengine/pe-input-107.bz2): Complete
>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition
>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>> origin=notify_crmd ]
>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Calculated
>> Transition 0: /var/lib/pacemaker/pengine/pe-input-107.bz2
>> Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPREQUEST of 10.0.2.5 on
>> enp0s3 to 10.0.2.3 port 67 (xid=0x1c247c88)
>> Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPACK of 10.0.2.5 from
>> 10.0.2.3
>> Sep 6 08:58:22 coro-test-2 dhclient[1070]: bound to 10.0.2.5 -- renewal
>> in 466 seconds.
>> Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPREQUEST of 192.168.57.4 on
>> enp0s8 to 192.168.57.2 port 67 (xid=0x5dfb57d7)
>> Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPACK of 192.168.57.4 from
>> 192.168.57.2
>> Sep 6 08:59:20 coro-test-2 dhclient[860]: bound to 192.168.57.4 --
>> renewal in 599 seconds.
>>
>> #
>> ###
>> ##### NODE 1 IS SHUTDOWN
>> ###
>> #
>>
>> ##
>> ### Cluster appears to see the loss of node 1
>> ##
>> Sep 6 09:03:38 coro-test-2 corosync[1505]: notice [TOTEM ] A new
>> membership (10.0.2.5:500) was formed. Members left: 1
>> Sep 6 09:03:38 coro-test-2 corosync[1505]: [TOTEM ] A new membership (
>> 10.0.2.5:500) was formed. Members left: 1
>> Sep 6 09:03:38 coro-test-2 pacemakerd[1537]: notice:
>> crm_reap_unseen_nodes: Node coro-test-1[1] - state is now lost (was member)
>> Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 fence request 1 pid
>> 28247 procdown time 1536239018 fence_all dlm_stonith
>> Sep 6 09:03:38 coro-test-2 crmd[1545]: notice: crm_reap_unseen_nodes:
>> Node coro-test-1[1] - state is now lost (was member)
>> Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 tell corosync to
>> remove nodeid 1 from cluster
>>
>> ##
>> ### DLM appears to close off connection to NODE 1
>> ##
>> Sep 6 09:03:38 coro-test-2 kernel: [59552.844406] dlm: closing
>> connection to node 1
>> Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Found 2
>> entries for 1/(null): 0 in progress, 1 completed
>> Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Node 1/(null)
>> last kicked at: 1536238555
>>
>> ##
>> ### Stonith attempts to fence node 1 (again; it was already fenced)
>> ##
>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28248]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-1' ]
>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28248]
>> stderr: [ ]
>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation
>> 'reboot' [28248] (call 2 from stonith-api.28247) for host 'coro-test-1'
>> with device 'scsi-shooter' returned: 0 (OK)
>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation reboot
>> of coro-test-1 by coro-test-2 for stonith-api.28247 at coro-test-2.2365d094:
>> OK
>> Sep 6 09:03:42 coro-test-2 stonith-api[28247]: stonith_api_kick: Node
>> 1/(null) kicked: reboot
>> Sep 6 09:03:42 coro-test-2 crmd[1545]: notice: State transition S_IDLE
>> -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
>> origin=abort_transition_graph ]
>> Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>>
>> ##
>> ### I begin to get confused here: Why is dlm closing the connection
>> to the node 2 (itself??)
>> ##
>> Sep 6 09:03:42 coro-test-2 kernel: [59556.684699] dlm: closing
>> connection to node 2
>>
>> ##
>> ### LOCKSPACE GOES ABANDONED!! why? WHY????
>> ##
>> Sep 6 09:03:42 coro-test-2 dlm_controld[1657]: 59554 abandoned lockspace
>> clvmd
>>
>> ##
>> ### Alas, my cluster begins to topple itself
>> ##
>> Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Start
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Calculated
>> Transition 1: /var/lib/pacemaker/pengine/pe-input-108.bz2
>> Sep 6 09:03:42 coro-test-2 kernel: [59556.693578] dlm: dlm user daemon
>> left 1 lockspaces
>> Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Operation
>> scsi-shooter_start_0: ok (node=coro-test-2, call=20, rc=0, cib-update=56,
>> confirmed=true)
>> Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Transition 1
>> (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0,
>> Source=/var/lib/pacemaker/pengine/pe-input-108.bz2): Complete
>> Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: State transition
>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>> origin=notify_crmd ]
>> Sep 6 09:04:05 coro-test-2 controld(dlm)[28324]: ERROR: Uncontrolled
>> lockspace exists, system must reboot. Executing suicide fencing
>> Sep 6 09:04:07 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:07 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>> stderr: [ ]
>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>> stderr: [ ]
>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>> stderr: [ ]
>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>> stderr: [ ]
>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344]
>> stderr: [ ]
>> Sep 6 09:04:09 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:09 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>> stderr: [ ]
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>> stderr: [ ]
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>> stderr: [ ]
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>> stderr: [ ]
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357]
>> stderr: [ ]
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation
>> 'reboot' [28357] (call 2 from stonith_admin.controld.28343) for host
>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker
>> error)
>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation reboot
>> of coro-test-2 by coro-test-2 for
>> stonith_admin.controld.28343 at coro-test-2.fd726e0b: Generic Pacemaker
>> error
>> Sep 6 09:04:09 coro-test-2 crmd[1545]: notice:
>> coro-test-2-dlm_monitor_30000:17 [ Command failed: Generic Pacemaker
>> error\n ]
>> Sep 6 09:04:09 coro-test-2 crmd[1545]: notice: State transition S_IDLE
>> -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
>> origin=abort_transition_graph ]
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 2: /var/lib/pacemaker/pengine/pe-warn-68.bz2
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 3: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:04:13 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:13 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>> stderr: [ ]
>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>> stderr: [ ]
>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>> stderr: [ ]
>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>> stderr: [ ]
>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370]
>> stderr: [ ]
>> Sep 6 09:04:14 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:14 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>> stderr: [ ]
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>> stderr: [ ]
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>> stderr: [ ]
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>> stderr: [ ]
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383]
>> stderr: [ ]
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28383] (call 3 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.171aafb6: Generic
>> Pacemaker error
>> Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition 3
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 4: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:04:19 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:19 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>> stderr: [ ]
>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>> stderr: [ ]
>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>> stderr: [ ]
>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>> stderr: [ ]
>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432]
>> stderr: [ ]
>> Sep 6 09:04:20 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:20 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>> stderr: [ ]
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>> stderr: [ ]
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>> stderr: [ ]
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>> stderr: [ ]
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445]
>> stderr: [ ]
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28445] (call 4 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.0b907f76: Generic
>> Pacemaker error
>> Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition 4
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 5: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:04:24 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:24 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>> stderr: [ ]
>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>> stderr: [ ]
>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>> stderr: [ ]
>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>> stderr: [ ]
>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477]
>> stderr: [ ]
>> Sep 6 09:04:25 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:25 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>> stderr: [ ]
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>> stderr: [ ]
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>> stderr: [ ]
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>> stderr: [ ]
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490]
>> stderr: [ ]
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28490] (call 5 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.559102cb: Generic
>> Pacemaker error
>> Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition 5
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 6: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:04:39 coro-test-2 controld(dlm)[28520]: ERROR: Uncontrolled
>> lockspace exists, system must reboot. Executing suicide fencing
>> Sep 6 09:04:39 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:39 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>> stderr: [ ]
>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>> stderr: [ ]
>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>> stderr: [ ]
>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>> stderr: [ ]
>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540]
>> stderr: [ ]
>> Sep 6 09:04:40 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:40 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>> stderr: [ ]
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>> stderr: [ ]
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>> stderr: [ ]
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>> stderr: [ ]
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553]
>> stderr: [ ]
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28553] (call 6 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.58604e64: Generic
>> Pacemaker error
>> Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition 6
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 7: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:04:49 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:49 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>> stderr: [ ]
>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>> stderr: [ ]
>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>> stderr: [ ]
>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>> stderr: [ ]
>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568]
>> stderr: [ ]
>> Sep 6 09:04:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:50 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>> stderr: [ ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>> stderr: [ ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>> stderr: [ ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>> stderr: [ ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581]
>> stderr: [ ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation
>> 'reboot' [28581] (call 2 from stonith_admin.controld.28539) for host
>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker
>> error)
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation reboot
>> of coro-test-2 by coro-test-2 for
>> stonith_admin.controld.28539 at coro-test-2.29f17c2f: Generic Pacemaker
>> error
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>> new_event_notification (1541-28539-12): Broken pipe (32)
>> Sep 6 09:04:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:51 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>> stderr: [ ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>> stderr: [ ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>> stderr: [ ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>> stderr: [ ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594]
>> stderr: [ ]
>> Sep 6 09:04:52 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:04:52 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>> stderr: [ ]
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>> stderr: [ ]
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>> stderr: [ ]
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>> stderr: [ ]
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607]
>> stderr: [ ]
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28607] (call 7 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.1c099fde: Generic
>> Pacemaker error
>> Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition 7
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 8: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:05:01 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:01 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>> stderr: [ ]
>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>> stderr: [ ]
>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>> stderr: [ ]
>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>> stderr: [ ]
>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643]
>> stderr: [ ]
>> Sep 6 09:05:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:02 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>> stderr: [ ]
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>> stderr: [ ]
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>> stderr: [ ]
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>> stderr: [ ]
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656]
>> stderr: [ ]
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28656] (call 8 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.41c7a339: Generic
>> Pacemaker error
>> Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition 8
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 9: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:05:16 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:16 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>> stderr: [ ]
>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>> stderr: [ ]
>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>> stderr: [ ]
>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>> stderr: [ ]
>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669]
>> stderr: [ ]
>> Sep 6 09:05:17 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:17 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>> stderr: [ ]
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>> stderr: [ ]
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>> stderr: [ ]
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>> stderr: [ ]
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682]
>> stderr: [ ]
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28682] (call 9 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.16fd9d3b: Generic
>> Pacemaker error
>> Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition 9
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 10: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:05:21 coro-test-2 controld(dlm)[28695]: ERROR: Uncontrolled
>> lockspace exists, system must reboot. Executing suicide fencing
>> Sep 6 09:05:29 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:29 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>> stderr: [ ]
>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>> stderr: [ ]
>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>> stderr: [ ]
>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>> stderr: [ ]
>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751]
>> stderr: [ ]
>> Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>> stderr: [ ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>> stderr: [ ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>> stderr: [ ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>> stderr: [ ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764]
>> stderr: [ ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28764] (call 10 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.c6433fd3: Generic
>> Pacemaker error
>> Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition 10
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 11: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>> stderr: [ ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>> stderr: [ ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>> stderr: [ ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>> stderr: [ ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777]
>> stderr: [ ]
>> Sep 6 09:05:31 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:31 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>> stderr: [ ]
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>> stderr: [ ]
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>> stderr: [ ]
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>> stderr: [ ]
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790]
>> stderr: [ ]
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation
>> 'reboot' [28790] (call 2 from stonith_admin.controld.28714) for host
>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker
>> error)
>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation reboot
>> of coro-test-2 by coro-test-2 for
>> stonith_admin.controld.28714 at coro-test-2.ffe64359: Generic Pacemaker
>> error
>> Sep 6 09:05:36 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:36 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>> stderr: [ ]
>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>> stderr: [ ]
>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>> stderr: [ ]
>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>> stderr: [ ]
>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803]
>> stderr: [ ]
>> Sep 6 09:05:37 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:37 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>> stderr: [ ]
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>> stderr: [ ]
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>> stderr: [ ]
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>> stderr: [ ]
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816]
>> stderr: [ ]
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28816] (call 11 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.7dfe97e8: Generic
>> Pacemaker error
>> Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition 11
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 12: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:05:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:50 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>> stderr: [ ]
>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>> stderr: [ ]
>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>> stderr: [ ]
>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>> stderr: [ ]
>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829]
>> stderr: [ ]
>> Sep 6 09:05:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:05:51 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>> stderr: [ ]
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>> stderr: [ ]
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>> stderr: [ ]
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>> stderr: [ ]
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842]
>> stderr: [ ]
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28842] (call 12 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.fb005423: Generic
>> Pacemaker error
>> Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition 12
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: On loss of CCM
>> Quorum: Ignore
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Processing failed op
>> monitor for dlm:0 on coro-test-2: unknown error (1)
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Node coro-test-2
>> will be fenced because of resource failure(s)
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0
>> with instance of dlm-clone
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1
>> with instance of dlm-clone
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Scheduling Node
>> coro-test-2 for STONITH
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop of failed
>> resource dlm:0 is implicit after coro-test-2 is fenced
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop
>> dlm:0#011(coro-test-2)
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop
>> clvmd:0#011(coro-test-2)
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop
>> scsi-shooter#011(coro-test-2)
>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Calculated
>> Transition 13: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>> Sep 6 09:06:01 coro-test-2 controld(dlm)[28874]: ERROR: Uncontrolled
>> lockspace exists, system must reboot. Executing suicide fencing
>> Sep 6 09:06:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:06:02 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>> stderr: [ ]
>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>> stderr: [ ]
>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>> stderr: [ ]
>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>> stderr: [ ]
>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894]
>> stderr: [ ]
>> Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>> stderr: [ ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>> stderr: [ ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>> stderr: [ ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>> stderr: [ ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907]
>> stderr: [ ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>> [28907] (call 13 from crmd.1545) for host 'coro-test-2' with device
>> 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation off of
>> coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.e8cc12be: Generic
>> Pacemaker error
>> Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition aborted:
>> Stonith failed (source=tengine_stonith_callback:748, 0)
>> Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition 13
>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>> Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: State transition
>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>> origin=notify_crmd ]
>> Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You
>> can not fence yourself.
>> Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>> stderr: [ WARNING:root:Parse error: Ignoring unknown option
>> 'port=coro-test-2' ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>> stderr: [ ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>> yourself. ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>> stderr: [ ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>> stderr: [ ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>> stderr: [ ERROR:root:Please use '-h' for usage ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>> stderr: [ ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>> stderr: [ Please use '-h' for usage ]
>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922]
>> stderr: [ ]
>> Sep 6 09:06:04
>>
>
--
Patrick Whitney
DevOps Engineer -- Tools
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20180911/f6d7fd9e/attachment-0002.html>
More information about the Users
mailing list