[ClusterLabs] 2 node cluster dlm/clvm trouble
Vladislav Bogdanov
bubble at hoster-ok.com
Thu Sep 6 14:34:23 EDT 2018
On September 6, 2018 2:36:03 PM UTC, Patrick Whitney <pwhitney at luminoso.com> wrote:
>Good Morning Everyone,
>
>I'm hoping someone with more experience with corosync and pacemaker can
>see
>what I am doing wrong.
>
>I've got a test setup of 2 nodes, with dlm and clvm setup as clones,
>and
>using fence_scsi as my fencing agent.
>
>I've got it to the point where the cluster is up, and reports it is
>happy.
>I then began testing fencing. When issuing 'pcs stonith fence' it
>appears
>to work; that is, the scsi reservation is pulled and the output of 'pcs
>status' looks sane, and I'm able to access resources on the un-fenced
>node.
>
>Things go awry when I shutdown (init 0) the fenced node... my unfenced
>node
>decides to fence itself (which looks like it was initiated by dlm due
>to an
>abandoned lockspace).
>
>I suspect this is due to misconfiguration, since I'm new to the
>toolset,
>but I'm not quite sure what I need to change.
>
>Any and all input is appreciated!
>
>Below is a chronology of events; my corosync config and cib.xml;
>command
>output; and annotated logs.
>
>Again, any hints, suggestions, wild guesses, or premonitions are
>welcomed
>-- I'm stuck! Please let me know if there is additional information
>which
>would be helpful.
>
>Many thanks,
>-Patrick W.
>
>Sep 6 08:54:14 -- Cluster is up and running; UI reports everything
> healthy.
>
>Sep 6 08:55:44 -- 'pcs stonith fence' called against node 1
>(coro-test-1);
> UI reports everything as expected -- that
>is, resources show only running on unfenced node and they're available.
> Oddly, although the UI says dlm is stopped
>on fenced node, the dlm_controld is still running.
>
>Sep 6 09:03:38 -- node 1 is shutdown, and node 2 falls to pieces.
> - First, corosync sees lost member -- seems
>like this is appropriate, to me.
> - Next, dlm_controld calls to fence
>everything
> - stonith-ng tries to fence node 1 (but its
>already fenced!)
> - dlm closes connection to "node 2" (does
>dlm "nodes" map to cluster nodes? I'm not sure they do)
> - clvmd dlm lockspace is now abandoned;
>cluster attempts to fence the remaining node
> (But can't because scsi_fence doesn't work
>like that).
>
>***
>****** -- Configuration --
>***
>root at coro-test-2:~# pcs --version
>0.9.149
>root at coro-test-2:~# pacemakerd --version
>Pacemaker 1.1.14
>Written by Andrew Beekhof
>root at coro-test-2:~# corosync -v
>Corosync Cluster Engine, version '2.3.5'
>Copyright (c) 2006-2009 Red Hat, Inc.
>root at coro-test-2:~#
>
>#
>## COROSYNC
>#
>root at coro-test-1:~# more /etc/corosync/corosync.conf
>totem {
> version: 2
> secauth: off
> cluster_name: my_first_cluster
> transport: udpu
>}
>
>nodelist {
> node {
> ring0_addr: coro-test-1
> nodeid: 1
> }
>
> node {
> ring0_addr: coro-test-2
> nodeid: 2
> }
>}
>
>quorum {
> provider: corosync_votequorum
> two_node: 1
>}
>
>logging {
> to_logfile: yes
> logfile: /var/log/corosync/corosync.log
> to_syslog: yes
>}
>
>##
>### PACEMAKER CIB.XML
>##
>root at coro-test-1:/var/lib/pacemaker/cib# cat cib.xml
><cib crm_feature_set="3.0.10" validate-with="pacemaker-2.4" epoch="30"
>num_updates="0" admin_epoch="0" cib-last-written="Thu Sep 6 10:12:46
>2018"
>update-origin="coro-test-2" update-client="cibadmin" update-user="root"
>have-quorum="1" dc-uuid="1">
> <configuration>
> <crm_config>
> <cluster_property_set id="cib-bootstrap-options">
> <nvpair id="cib-bootstrap-options-have-watchdog"
>name="have-watchdog" value="false"/>
> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
>value="1.1.14-70404b0"/>
> <nvpair id="cib-bootstrap-options-cluster-infrastructure"
>name="cluster-infrastructure" value="corosync"/>
> <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name"
>value="my_first_cluster"/>
> <nvpair id="cib-bootstrap-options-no-quorum-policy"
>name="no-quorum-policy" value="ignore"/>
> <nvpair id="cib-bootstrap-options-stonith-enabled"
>name="stonith-enabled" value="true"/>
> <nvpair id="cib-bootstrap-options-stonith-action"
>name="stonith-action" value="off"/>
> </cluster_property_set>
> </crm_config>
> <nodes>
> <node id="1" uname="coro-test-1"/>
> <node id="2" uname="coro-test-2"/>
> </nodes>
> <resources>
> <clone id="dlm-clone">
> <primitive class="ocf" id="dlm" provider="pacemaker"
>type="controld">
> <instance_attributes id="dlm-instance_attributes"/>
> <operations>
> <op id="dlm-start-interval-0s" interval="0s" name="start"
>timeout="90"/>
> <op id="dlm-stop-interval-0s" interval="0s" name="stop"
>timeout="100"/>
> <op id="dlm-monitor-interval-30s" interval="30s" name="monitor"
>on-fail="fence"/>
> </operations>
> </primitive>
> <meta_attributes id="dlm-clone-meta_attributes">
> <nvpair id="dlm-clone-meta_attributes-interleave"
>name="interleave" value="true"/>
> <nvpair id="dlm-clone-meta_attributes-ordered" name="ordered"
>value="true"/>
> </meta_attributes>
> </clone>
> <clone id="clvmd-clone">
> <primitive class="ocf" id="clvmd" provider="heartbeat" type="clvm">
> <instance_attributes id="clvmd-instance_attributes"/>
> <operations>
> <op id="clvmd-start-interval-0s" interval="0s" name="start"
>timeout="90"/>
> <op id="clvmd-stop-interval-0s" interval="0s" name="stop"
>timeout="90"/>
> <op id="clvmd-monitor-interval-30s" interval="30s"
>name="monitor" on-fail="fence"/>
> </operations>
> </primitive>
> <meta_attributes id="clvmd-clone-meta_attributes">
> <nvpair id="clvmd-clone-meta_attributes-interleave"
>name="interleave" value="true"/>
> <nvpair id="clvmd-clone-meta_attributes-ordered" name="ordered"
>value="true"/>
> </meta_attributes>
> </clone>
> <primitive class="stonith" id="scsi-shooter" type="fence_scsi">
> <instance_attributes id="scsi-shooter-instance_attributes">
> <nvpair id="scsi-shooter-instance_attributes-pcmk_host_list"
>name="pcmk_host_list" value="coro-test-1 coro-test-2"/>
> <nvpair id="scsi-shooter-instance_attributes-pcmk_monitor_action"
>name="pcmk_monitor_action" value="metadata"/>
> <nvpair id="scsi-shooter-instance_attributes-pcmk_reboot_action"
>name="pcmk_reboot_action" value="off"/>
> <nvpair id="scsi-shooter-instance_attributes-pcmk_delay_max"
>name="pcmk_delay_max" value="15"/>
> <nvpair id="scsi-shooter-instance_attributes-devices"
>name="devices" value="/dev/sdb1"/>
> </instance_attributes>
> <meta_attributes id="scsi-shooter-meta_attributes">
> <nvpair id="scsi-shooter-meta_attributes-provides"
>name="provides" value="unfencing"/>
> </meta_attributes>
> <operations>
> <op id="scsi-shooter-monitor-interval-60s" interval="60s"
>name="monitor"/>
> </operations>
> </primitive>
> </resources>
> <constraints>
> <rsc_order first="dlm-clone" first-action="start"
>id="order-dlm-clone-clvmd-clone-mandatory" then="clvmd-clone"
>then-action="start"/>
> <rsc_colocation id="colocation-clvmd-clone-dlm-clone-INFINITY"
>rsc="clvmd-clone" score="INFINITY" with-rsc="dlm-clone"/>
> </constraints>
> </configuration>
></cib>
>
>***
>****** -- Console Output -- Comments begin with "#"
>***
>
>#
>##
>### -- Moments before testing starts (Node 2)--
>##
>#
>root at coro-test-2:~# date; pcs status
>Thu Sep 6 08:54:14 EDT 2018
>Cluster name: my_first_cluster
>Last updated: Thu Sep 6 08:54:14 2018 Last change: Wed Sep 5 16:28:53
>2018 by root via cibadmin on coro-test-2
>Stack: corosync
>Current DC: coro-test-1 (version 1.1.14-70404b0) - partition with
>quorum
>2 nodes and 5 resources configured
>
>Online: [ coro-test-1 coro-test-2 ]
>
>Full list of resources:
>
> Clone Set: dlm-clone [dlm]
> Started: [ coro-test-1 coro-test-2 ]
> Clone Set: clvmd-clone [clvmd]
> Started: [ coro-test-1 coro-test-2 ]
> scsi-shooter (stonith:fence_scsi): Started coro-test-1
>
>PCSD Status:
> coro-test-1: Online
> coro-test-2: Online
>
>Daemon Status:
> corosync: active/enabled
> pacemaker: active/enabled
> pcsd: active/enabled
>root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
> PR generation=0x41, 2 registered reservation keys follow:
> 0x3ff20000
> 0x3ff20001
>root at coro-test-2:~# dlm_tool status
>cluster nodeid 2 quorate 1 ring seq 496 496
>daemon now 58996 fence_pid 0
>node 1 M add 92 rem 0 fail 0 fence 0 at 0 0
>node 2 M add 92 rem 0 fail 0 fence 0 at 0 0
>root at coro-test-2:~# dlm_tool ls
>dlm lockspaces
>name clvmd
>id 0x4104eefa
>flags 0x00000000
>change member 2 joined 1 remove 0 failed 0 seq 1,1
>members 1 2
>
>
>#
>##
>### -- Node 1 is fenced --
>##
>#
>root at coro-test-2:~# date; pcs stonith fence coro-test-1
>Thu Sep 6 08:55:44 EDT 2018
>Node: coro-test-1 fenced
>root at coro-test-2:~# pcs status
>Cluster name: my_first_cluster
>Last updated: Thu Sep 6 08:56:40 2018 Last change: Wed Sep 5 16:28:53
>2018 by root via cibadmin on coro-test-2
>Stack: corosync
>Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with
>quorum
>2 nodes and 5 resources configured
>
>Node coro-test-1: pending
>Online: [ coro-test-2 ]
>
>Full list of resources:
>
> Clone Set: dlm-clone [dlm]
> Started: [ coro-test-2 ]
> Stopped: [ coro-test-1 ]
> Clone Set: clvmd-clone [clvmd]
> Started: [ coro-test-2 ]
> Stopped: [ coro-test-1 ]
> scsi-shooter (stonith:fence_scsi): Stopped
>
>PCSD Status:
> coro-test-1: Online
> coro-test-2: Online
>
>Daemon Status:
> corosync: active/enabled
> pacemaker: active/enabled
> pcsd: active/enabled
>root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
> PR generation=0x42, 1 registered reservation key follows:
> 0x3ff20001
>root at coro-test-2:~# dlm_tool ls
>dlm lockspaces
>name clvmd
>id 0x4104eefa
>flags 0x00000000
>change member 2 joined 1 remove 0 failed 0 seq 1,1
>members 1 2
>
>root at coro-test-2:~# dlm_tool status
>cluster nodeid 2 quorate 1 ring seq 496 496
>daemon now 59166 fence_pid 0
>node 1 M add 92 rem 0 fail 0 fence 0 at 0 0
>node 2 M add 92 rem 0 fail 0 fence 0 at 0 0
>root at coro-test-2:~#
>
>#
>## NOTE: DLM still running on other node, coro-test-1, at this point.
>#
>root at coro-test-1:~# ps -aef | grep dlm
>root 1633 1 0 Sep05 ? 00:00:00 dlm_controld -s 0
>root 1752 2 0 Sep05 ? 00:00:00 [dlm_scand]
>root 1754 2 0 Sep05 ? 00:00:00 [dlm_recv]
>root 1755 2 0 Sep05 ? 00:00:00 [dlm_send]
>root 1756 2 0 Sep05 ? 00:00:00 [dlm_recoverd]
>root 29673 1850 0 08:58 pts/0 00:00:00 grep --color=auto dlm
>root at coro-test-1:~#
>
>
>#
>##
>### -- node 2 output after node 1 is shutdown
>##
>#
>#### NODE 1 SHUTDOWN ####
>root at coro-test-1:~# pcs status; date; init 0
>Error: cluster is not currently running on this node
>Thu Sep 6 09:03:38 EDT 2018
>
>#### NODE 2 UI OUTPUT ####
>root at coro-test-2:~# pcs status
>Cluster name: my_first_cluster
>Last updated: Thu Sep 6 09:04:16 2018 Last change: Wed Sep 5 16:28:53
>2018 by root via cibadmin on coro-test-2
>Stack: corosync
>Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with
>quorum
>2 nodes and 5 resources configured
>
>Node coro-test-2: UNCLEAN (online)
>OFFLINE: [ coro-test-1 ]
>
>Full list of resources:
>
> Clone Set: dlm-clone [dlm]
> dlm (ocf::pacemaker:controld): FAILED coro-test-2
> Stopped: [ coro-test-1 ]
> Clone Set: clvmd-clone [clvmd]
> Started: [ coro-test-2 ]
> Stopped: [ coro-test-1 ]
> scsi-shooter (stonith:fence_scsi): Started coro-test-2
>
>Failed Actions:
>* dlm_monitor_30000 on coro-test-2 'unknown error' (1): call=17,
>status=complete, exitreason='none',
> last-rc-change='Thu Sep 6 09:04:09 2018', queued=0ms, exec=0ms
>
>
>PCSD Status:
> coro-test-1: Offline
> coro-test-2: Online
>
>Daemon Status:
> corosync: active/enabled
> pacemaker: active/enabled
> pcsd: active/enabled
>root at coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1
> PR generation=0x42, 1 registered reservation key follows:
> 0x3ff20001
>root at coro-test-2:~# dlm_tool status
>cannot connect to dlm_controld
>root at coro-test-2:~# dlm_tool ls
>root at coro-test-2:~#
>
>***
>****** -- Syslog Output -- Comments begin with "#"
>***
>#
>###
>##### NODE 2 FENCES NODE 1 (pcs stonith fence coro-test-1)
>###
>#
>
>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[27468]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-1' ]
>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[27468]
>stderr: [ ]
>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation
>'reboot'
>[27468] (call 2 from stonith_admin.27448) for host 'coro-test-1' with
>device 'scsi-shooter' returned: 0 (OK)
>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation
>reboot of
>coro-test-1 by coro-test-2 for
>stonith_admin.27448 at coro-test-2.5886f6b5: OK
>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning:
>new_event_notification (1541-27448-12): Broken pipe (32)
>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition
>S_NOT_DC
>-> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK
>origin=peer_update_callback ]
>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition
>S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED
>origin=election_timeout_popped ]
>Sep 6 08:55:55 coro-test-2 attrd[1543]: notice:
>crm_update_peer_proc:
>Node coro-test-1[1] - state is now lost (was member)
>Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Removing
>coro-test-1/1
>from the membership list
>Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Purged 1 peers with
>id=1
>and/or uname=coro-test-1 from the membership cache
>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice:
>crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was
>member)
>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Removing
>coro-test-1/1 from the membership list
>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Purged 1 peers
>with
>id=1 and/or uname=coro-test-1 from the membership cache
>Sep 6 08:55:55 coro-test-2 cib[1540]: notice: crm_update_peer_proc:
>Node
>coro-test-1[1] - state is now lost (was member)
>Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Removing coro-test-1/1
>from the membership list
>Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Purged 1 peers with
>id=1
>and/or uname=coro-test-1 from the membership cache
>Sep 6 08:55:55 coro-test-2 crmd[1545]: warning: FSA: Input
>I_ELECTION_DC
>from do_election_check() received in state S_INTEGRATION
>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action
>dlm:0_monitor_0
>on coro-test-1 is unrunnable (pending)
>Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action
>clvmd:0_monitor_0 on coro-test-1 is unrunnable (pending)
>Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action
>scsi-shooter_monitor_0 on coro-test-1 is unrunnable (pending)
>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Unfencing
>coro-test-1:
>node discovery
>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Start
> scsi-shooter#011(coro-test-2 - blocked)
>Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: error: Operation on of
>coro-test-1 by <no-one> for crmd.1545 at coro-test-2.b06d9d39: No such
>device
>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition 0
>(Complete=1,
>Pending=0, Fired=0, Skipped=0, Incomplete=2,
>Source=/var/lib/pacemaker/pengine/pe-input-107.bz2): Complete
>Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition
>S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>origin=notify_crmd ]
>Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Calculated
>Transition
>0: /var/lib/pacemaker/pengine/pe-input-107.bz2
>Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPREQUEST of 10.0.2.5 on
>enp0s3 to 10.0.2.3 port 67 (xid=0x1c247c88)
>Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPACK of 10.0.2.5 from
>10.0.2.3
>Sep 6 08:58:22 coro-test-2 dhclient[1070]: bound to 10.0.2.5 --
>renewal in
>466 seconds.
>Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPREQUEST of 192.168.57.4
>on
>enp0s8 to 192.168.57.2 port 67 (xid=0x5dfb57d7)
>Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPACK of 192.168.57.4 from
>192.168.57.2
>Sep 6 08:59:20 coro-test-2 dhclient[860]: bound to 192.168.57.4 --
>renewal
>in 599 seconds.
>
>#
>###
>##### NODE 1 IS SHUTDOWN
>###
>#
>
>##
>### Cluster appears to see the loss of node 1
>##
>Sep 6 09:03:38 coro-test-2 corosync[1505]: notice [TOTEM ] A new
>membership (10.0.2.5:500) was formed. Members left: 1
>Sep 6 09:03:38 coro-test-2 corosync[1505]: [TOTEM ] A new membership
>(
>10.0.2.5:500) was formed. Members left: 1
>Sep 6 09:03:38 coro-test-2 pacemakerd[1537]: notice:
>crm_reap_unseen_nodes: Node coro-test-1[1] - state is now lost (was
>member)
>Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 fence request 1
>pid
>28247 procdown time 1536239018 fence_all dlm_stonith
>Sep 6 09:03:38 coro-test-2 crmd[1545]: notice:
>crm_reap_unseen_nodes:
>Node coro-test-1[1] - state is now lost (was member)
>Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 tell corosync to
>remove nodeid 1 from cluster
>
>##
>### DLM appears to close off connection to NODE 1
>##
>Sep 6 09:03:38 coro-test-2 kernel: [59552.844406] dlm: closing
>connection
>to node 1
>Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Found 2
>entries
>for 1/(null): 0 in progress, 1 completed
>Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Node
>1/(null)
>last kicked at: 1536238555
>
>##
>### Stonith attempts to fence node 1 (again; it was already fenced)
>##
>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28248]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-1' ]
>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28248]
>stderr: [ ]
>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation
>'reboot'
>[28248] (call 2 from stonith-api.28247) for host 'coro-test-1' with
>device
>'scsi-shooter' returned: 0 (OK)
>Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation
>reboot of
>coro-test-1 by coro-test-2 for stonith-api.28247 at coro-test-2.2365d094:
>OK
>Sep 6 09:03:42 coro-test-2 stonith-api[28247]: stonith_api_kick: Node
>1/(null) kicked: reboot
>Sep 6 09:03:42 coro-test-2 crmd[1545]: notice: State transition
>S_IDLE
>-> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
>origin=abort_transition_graph ]
>Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>
>##
>### I begin to get confused here: Why is dlm closing the
>connection to
>the node 2 (itself??)
>##
>Sep 6 09:03:42 coro-test-2 kernel: [59556.684699] dlm: closing
>connection
>to node 2
>
>##
>### LOCKSPACE GOES ABANDONED!! why? WHY????
>##
>Sep 6 09:03:42 coro-test-2 dlm_controld[1657]: 59554 abandoned
>lockspace
>clvmd
>
>##
>### Alas, my cluster begins to topple itself
>##
>Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Start
> scsi-shooter#011(coro-test-2)
>Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Calculated
>Transition
>1: /var/lib/pacemaker/pengine/pe-input-108.bz2
>Sep 6 09:03:42 coro-test-2 kernel: [59556.693578] dlm: dlm user daemon
>left 1 lockspaces
>Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Operation
>scsi-shooter_start_0: ok (node=coro-test-2, call=20, rc=0,
>cib-update=56,
>confirmed=true)
>Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Transition 1
>(Complete=2,
>Pending=0, Fired=0, Skipped=0, Incomplete=0,
>Source=/var/lib/pacemaker/pengine/pe-input-108.bz2): Complete
>Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: State transition
>S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>origin=notify_crmd ]
>Sep 6 09:04:05 coro-test-2 controld(dlm)[28324]: ERROR: Uncontrolled
>lockspace exists, system must reboot. Executing suicide fencing
>Sep 6 09:04:07 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:07 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28344]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28344]
>stderr: [ ]
>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28344]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28344]
>stderr: [ ]
>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28344]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28344]
>stderr: [ ]
>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28344]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28344]
>stderr: [ ]
>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28344]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28344]
>stderr: [ ]
>Sep 6 09:04:09 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:09 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28357]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28357]
>stderr: [ ]
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28357]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28357]
>stderr: [ ]
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28357]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28357]
>stderr: [ ]
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28357]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28357]
>stderr: [ ]
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28357]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28357]
>stderr: [ ]
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation
>'reboot'
>[28357] (call 2 from stonith_admin.controld.28343) for host
>'coro-test-2'
>with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation
>reboot of
>coro-test-2 by coro-test-2 for
>stonith_admin.controld.28343 at coro-test-2.fd726e0b: Generic Pacemaker
>error
>Sep 6 09:04:09 coro-test-2 crmd[1545]: notice:
>coro-test-2-dlm_monitor_30000:17 [ Command failed: Generic Pacemaker
>error\n ]
>Sep 6 09:04:09 coro-test-2 crmd[1545]: notice: State transition
>S_IDLE
>-> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
>origin=abort_transition_graph ]
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>2: /var/lib/pacemaker/pengine/pe-warn-68.bz2
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>3: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:04:13 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:13 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28370]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28370]
>stderr: [ ]
>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28370]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28370]
>stderr: [ ]
>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28370]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28370]
>stderr: [ ]
>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28370]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28370]
>stderr: [ ]
>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28370]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28370]
>stderr: [ ]
>Sep 6 09:04:14 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:14 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28383]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28383]
>stderr: [ ]
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28383]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28383]
>stderr: [ ]
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28383]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28383]
>stderr: [ ]
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28383]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28383]
>stderr: [ ]
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28383]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28383]
>stderr: [ ]
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28383] (call 3 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.171aafb6: Generic
>Pacemaker error
>Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition 3
>(Complete=1,
>Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>4: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:04:19 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:19 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28432]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28432]
>stderr: [ ]
>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28432]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28432]
>stderr: [ ]
>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28432]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28432]
>stderr: [ ]
>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28432]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28432]
>stderr: [ ]
>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28432]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28432]
>stderr: [ ]
>Sep 6 09:04:20 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:20 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28445]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28445]
>stderr: [ ]
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28445]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28445]
>stderr: [ ]
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28445]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28445]
>stderr: [ ]
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28445]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28445]
>stderr: [ ]
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28445]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28445]
>stderr: [ ]
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28445] (call 4 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.0b907f76: Generic
>Pacemaker error
>Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition 4
>(Complete=1,
>Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>5: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:04:24 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:24 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28477]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28477]
>stderr: [ ]
>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28477]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28477]
>stderr: [ ]
>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28477]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28477]
>stderr: [ ]
>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28477]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28477]
>stderr: [ ]
>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28477]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28477]
>stderr: [ ]
>Sep 6 09:04:25 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:25 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28490]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28490]
>stderr: [ ]
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28490]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28490]
>stderr: [ ]
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28490]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28490]
>stderr: [ ]
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28490]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28490]
>stderr: [ ]
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28490]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28490]
>stderr: [ ]
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28490] (call 5 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.559102cb: Generic
>Pacemaker error
>Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition 5
>(Complete=1,
>Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>6: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:04:39 coro-test-2 controld(dlm)[28520]: ERROR: Uncontrolled
>lockspace exists, system must reboot. Executing suicide fencing
>Sep 6 09:04:39 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:39 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28540]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28540]
>stderr: [ ]
>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28540]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28540]
>stderr: [ ]
>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28540]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28540]
>stderr: [ ]
>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28540]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28540]
>stderr: [ ]
>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28540]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28540]
>stderr: [ ]
>Sep 6 09:04:40 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:40 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28553]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28553]
>stderr: [ ]
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28553]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28553]
>stderr: [ ]
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28553]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28553]
>stderr: [ ]
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28553]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28553]
>stderr: [ ]
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28553]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28553]
>stderr: [ ]
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28553] (call 6 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.58604e64: Generic
>Pacemaker error
>Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition 6
>(Complete=1,
>Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>7: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:04:49 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:49 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28568]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28568]
>stderr: [ ]
>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28568]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28568]
>stderr: [ ]
>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28568]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28568]
>stderr: [ ]
>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28568]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28568]
>stderr: [ ]
>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28568]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28568]
>stderr: [ ]
>Sep 6 09:04:50 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:50 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28581]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28581]
>stderr: [ ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28581]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28581]
>stderr: [ ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28581]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28581]
>stderr: [ ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28581]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28581]
>stderr: [ ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28581]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28581]
>stderr: [ ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation
>'reboot'
>[28581] (call 2 from stonith_admin.controld.28539) for host
>'coro-test-2'
>with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation
>reboot of
>coro-test-2 by coro-test-2 for
>stonith_admin.controld.28539 at coro-test-2.29f17c2f: Generic Pacemaker
>error
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>new_event_notification (1541-28539-12): Broken pipe (32)
>Sep 6 09:04:51 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:51 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28594]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28594]
>stderr: [ ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28594]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28594]
>stderr: [ ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28594]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28594]
>stderr: [ ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28594]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28594]
>stderr: [ ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28594]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28594]
>stderr: [ ]
>Sep 6 09:04:52 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:04:52 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28607]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28607]
>stderr: [ ]
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28607]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28607]
>stderr: [ ]
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28607]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28607]
>stderr: [ ]
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28607]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28607]
>stderr: [ ]
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28607]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28607]
>stderr: [ ]
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28607] (call 7 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.1c099fde: Generic
>Pacemaker error
>Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition 7
>(Complete=1,
>Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>8: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:05:01 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:01 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28643]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28643]
>stderr: [ ]
>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28643]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28643]
>stderr: [ ]
>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28643]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28643]
>stderr: [ ]
>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28643]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28643]
>stderr: [ ]
>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28643]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28643]
>stderr: [ ]
>Sep 6 09:05:02 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:02 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28656]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28656]
>stderr: [ ]
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28656]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28656]
>stderr: [ ]
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28656]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28656]
>stderr: [ ]
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28656]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28656]
>stderr: [ ]
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28656]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28656]
>stderr: [ ]
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28656] (call 8 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.41c7a339: Generic
>Pacemaker error
>Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition 8
>(Complete=1,
>Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>9: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:05:16 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:16 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28669]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28669]
>stderr: [ ]
>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28669]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28669]
>stderr: [ ]
>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28669]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28669]
>stderr: [ ]
>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28669]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28669]
>stderr: [ ]
>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28669]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28669]
>stderr: [ ]
>Sep 6 09:05:17 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:17 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28682]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28682]
>stderr: [ ]
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28682]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28682]
>stderr: [ ]
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28682]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28682]
>stderr: [ ]
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28682]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28682]
>stderr: [ ]
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28682]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28682]
>stderr: [ ]
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28682] (call 9 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.16fd9d3b: Generic
>Pacemaker error
>Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition 9
>(Complete=1,
>Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>10: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:05:21 coro-test-2 controld(dlm)[28695]: ERROR: Uncontrolled
>lockspace exists, system must reboot. Executing suicide fencing
>Sep 6 09:05:29 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:29 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28751]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28751]
>stderr: [ ]
>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28751]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28751]
>stderr: [ ]
>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28751]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28751]
>stderr: [ ]
>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28751]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28751]
>stderr: [ ]
>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28751]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28751]
>stderr: [ ]
>Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28764]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28764]
>stderr: [ ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28764]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28764]
>stderr: [ ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28764]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28764]
>stderr: [ ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28764]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28764]
>stderr: [ ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28764]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28764]
>stderr: [ ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28764] (call 10 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.c6433fd3: Generic
>Pacemaker error
>Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition 10
>(Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>11: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28777]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28777]
>stderr: [ ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28777]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28777]
>stderr: [ ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28777]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28777]
>stderr: [ ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28777]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28777]
>stderr: [ ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28777]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28777]
>stderr: [ ]
>Sep 6 09:05:31 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:31 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28790]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28790]
>stderr: [ ]
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28790]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28790]
>stderr: [ ]
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28790]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28790]
>stderr: [ ]
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28790]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28790]
>stderr: [ ]
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28790]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28790]
>stderr: [ ]
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation
>'reboot'
>[28790] (call 2 from stonith_admin.controld.28714) for host
>'coro-test-2'
>with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation
>reboot of
>coro-test-2 by coro-test-2 for
>stonith_admin.controld.28714 at coro-test-2.ffe64359: Generic Pacemaker
>error
>Sep 6 09:05:36 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:36 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28803]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28803]
>stderr: [ ]
>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28803]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28803]
>stderr: [ ]
>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28803]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28803]
>stderr: [ ]
>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28803]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28803]
>stderr: [ ]
>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28803]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28803]
>stderr: [ ]
>Sep 6 09:05:37 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:37 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28816]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28816]
>stderr: [ ]
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28816]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28816]
>stderr: [ ]
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28816]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28816]
>stderr: [ ]
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28816]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28816]
>stderr: [ ]
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28816]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28816]
>stderr: [ ]
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28816] (call 11 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.7dfe97e8: Generic
>Pacemaker error
>Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition 11
>(Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>12: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:05:50 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:50 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28829]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28829]
>stderr: [ ]
>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28829]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28829]
>stderr: [ ]
>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28829]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28829]
>stderr: [ ]
>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28829]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28829]
>stderr: [ ]
>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28829]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28829]
>stderr: [ ]
>Sep 6 09:05:51 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:05:51 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28842]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28842]
>stderr: [ ]
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28842]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28842]
>stderr: [ ]
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28842]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28842]
>stderr: [ ]
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28842]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28842]
>stderr: [ ]
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28842]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28842]
>stderr: [ ]
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28842] (call 12 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.fb005423: Generic
>Pacemaker error
>Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition 12
>(Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: On loss of CCM
>Quorum:
>Ignore
>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Processing failed
>op
>monitor for dlm:0 on coro-test-2: unknown error (1)
>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Node coro-test-2
>will
>be fenced because of resource failure(s)
>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:0
>with instance of dlm-clone
>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair
>clvmd:1
>with instance of dlm-clone
>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Scheduling Node
>coro-test-2 for STONITH
>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop of failed
>resource dlm:0 is implicit after coro-test-2 is fenced
>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop
>dlm:0#011(coro-test-2)
>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop
>clvmd:0#011(coro-test-2)
>Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop
>scsi-shooter#011(coro-test-2)
>Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Calculated
>Transition
>13: /var/lib/pacemaker/pengine/pe-warn-69.bz2
>Sep 6 09:06:01 coro-test-2 controld(dlm)[28874]: ERROR: Uncontrolled
>lockspace exists, system must reboot. Executing suicide fencing
>Sep 6 09:06:02 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:06:02 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28894]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28894]
>stderr: [ ]
>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28894]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28894]
>stderr: [ ]
>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28894]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28894]
>stderr: [ ]
>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28894]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28894]
>stderr: [ ]
>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28894]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28894]
>stderr: [ ]
>Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28907]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28907]
>stderr: [ ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28907]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28907]
>stderr: [ ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28907]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28907]
>stderr: [ ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28907]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28907]
>stderr: [ ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28907]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28907]
>stderr: [ ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation 'off'
>[28907] (call 13 from crmd.1545) for host 'coro-test-2' with device
>'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation off
>of
>coro-test-2 by coro-test-2 for crmd.1545 at coro-test-2.e8cc12be: Generic
>Pacemaker error
>Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition aborted:
>Stonith failed (source=tengine_stonith_callback:748, 0)
>Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition 13
>(Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9,
>Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete
>Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: State transition
>S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
>origin=notify_crmd ]
>Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28922]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28922]
>stderr: [ ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28922]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28922]
>stderr: [ ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28922]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28922]
>stderr: [ ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28922]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28922]
>stderr: [ ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28922]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28922]
>stderr: [ ]
>Sep 6 09:06:04 coro-test-2 fence_scsi: Failed: keys cannot be same.
>You
>can not fence yourself.
>Sep 6 09:06:04 coro-test-2 fence_scsi: Please use '-h' for usage
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28935]
>stderr: [ WARNING:root:Parse error: Ignoring unknown option
>'port=coro-test-2' ]
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28935]
>stderr: [ ]
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28935]
>stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
>yourself. ]
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28935]
>stderr: [ ]
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28935]
>stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28935]
>stderr: [ ]
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28935]
>stderr: [ ERROR:root:Please use '-h' for usage ]
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28935]
>stderr: [ ]
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28935]
>stderr: [ Please use '-h' for usage ]
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: warning:
>fence_scsi[28935]
>stderr: [ ]
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: error: Operation
>'reboot'
>[28935] (call 2 from stonith_admin.controld.28893) for host
>'coro-test-2'
>with device 'scsi-shooter' returned: -201 (Generic Pacemaker error)
>Sep 6 09:06:04 coro-test-2 stonith-ng[1541]: error: Operation
>reboot of
>coro-test-2 by coro-test-2 for
>stonith_admin.controld.28893 at coro-test-2.185da33b: Generic Pacemaker
>error
>Sep 6 09:06:08 coro-test-2 dhclient[1070]: DHCPREQUEST of 10.0.2.5 on
>enp0s3 to 10.0.2.3 port 67 (xid=0x1c247c88)
>Sep 6 09:06:08 coro-test-2 dhclient[1070]: DHCPACK of 10.0.2.5 from
>10.0.2.3
>Sep 6 09:06:08 coro-test-2 dhclient[1070]: bound to 10.0.2.5 --
>renewal in
>571 seconds.
>
>
>--
>Patrick Whitney
>DevOps Engineer -- Tools
Hi
I believe fabric fencing is not enough for dlm. You definitely need power fencing here so dlm kernel counterpart is reset properly.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20180906/e17e83cd/attachment-0002.html>
More information about the Users
mailing list