[ClusterLabs] Cold star of one node only
Gienek Nowacki
nowackig at gmail.com
Tue Sep 13 16:27:26 EDT 2016
Hi,
I'm still testing (before production running) the solution with
pacemaker+corosync+drbd+dlm+gfs2 on Centos7 with double-primary config.
I have two nodes: wirt1v and wirt2v - each node contains LVM partition
with DRBD (/dev/drbd2) and filesystem mounted as /virtfs2. Filesystems
/virtfs2 contain the images of virtual machines.
My problem is so - I can't start the cluster and the resources on one node
only (cold start) when the second node is completely powered off.
Is it in such configuration at all posssible - is it posible to start one
node only?
Could you help me, please?
The configs and log (during cold start) are attached.
Thanks in advance,
Gienek Nowacki
==============================================================
#---------------------------------
### result: cat /etc/redhat-release ###
CentOS Linux release 7.2.1511 (Core)
#---------------------------------
### result: uname -a ###
Linux wirt1v.example.com 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18
19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
#---------------------------------
### result: cat /etc/hosts ###
127.0.0.1 localhost localhost.localdomain localhost4
localhost4.localdomain4
172.31.0.23 wirt1.example.com wirt1
172.31.0.24 wirt2.example.com wirt2
1.1.1.1 wirt1v.example.com wirt1v
1.1.1.2 wirt2v.example.com wirt2v
#---------------------------------
### result: cat /etc/drbd.conf ###
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
#---------------------------------
### result: cat /etc/drbd.d/global_common.conf ###
common {
protocol C;
syncer {
verify-alg sha1;
}
startup {
become-primary-on both;
wfc-timeout 30;
outdated-wfc-timeout 20;
degr-wfc-timeout 30;
}
disk {
fencing resource-and-stonith;
}
handlers {
fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
split-brain
"/usr/lib/drbd/notify-split-brain.sh linuxadmin at example.com";
pri-lost-after-sb
"/usr/lib/drbd/notify-split-brain.sh linuxadmin at example.com";
out-of-sync
"/usr/lib/drbd/notify-out-of-sync.sh linuxadmin at example.com";
local-io-error
"/usr/lib/drbd/notify-io-error.sh linuxadmin at example.com";
}
net {
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
}
}
#---------------------------------
### result: cat /etc/drbd.d/drbd2.res ###
resource drbd2 {
meta-disk internal;
device /dev/drbd2;
on wirt1v.example.com {
disk /dev/vg02/drbd2;
address 1.1.1.1:7782;
}
on wirt2v.example.com {
disk /dev/vg02/drbd2;
address 1.1.1.2:7782;
}
}
#---------------------------------
### result: cat /etc/corosync/corosync.conf ###
totem {
version: 2
secauth: off
cluster_name: klasterek
transport: udpu
}
nodelist {
node {
ring0_addr: wirt1v
nodeid: 1
}
node {
ring0_addr: wirt2v
nodeid: 2
}
}
quorum {
provider: corosync_votequorum
two_node: 1
}
logging {
to_logfile: yes
logfile: /var/log/cluster/corosync.log
to_syslog: yes
}
#---------------------------------
### result: mount | grep virtfs2 ###
/dev/drbd2 on /virtfs2 type gfs2 (rw,relatime,seclabel)
#---------------------------------
### result: pcs status ###
Cluster name: klasterek
Last updated: Tue Sep 13 20:01:40 2016 Last change: Tue Sep 13
18:31:33 2016 by root via crm_resource on wirt1v
Stack: corosync
Current DC: wirt1v (version 1.1.13-10.el7_2.4-44eb2dd) - partition with
quorum
2 nodes and 8 resources configured
Online: [ wirt1v wirt2v ]
Full list of resources:
Master/Slave Set: Drbd2-clone [Drbd2]
Masters: [ wirt1v wirt2v ]
Clone Set: Virtfs2-clone [Virtfs2]
Started: [ wirt1v wirt2v ]
Clone Set: dlm-clone [dlm]
Started: [ wirt1v wirt2v ]
fencing-idrac1 (stonith:fence_idrac): Started wirt1v
fencing-idrac2 (stonith:fence_idrac): Started wirt2v
PCSD Status:
wirt1v: Online
wirt2v: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
#---------------------------------
### result: pcs property ###
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: klasterek
dc-version: 1.1.13-10.el7_2.4-44eb2dd
have-watchdog: false
no-quorum-policy: ignore
stonith-enabled: true
symmetric-cluster: true
#---------------------------------
### result: pcs cluster cib ###
<cib crm_feature_set="3.0.10" validate-with="pacemaker-2.3" epoch="69"
num_updates="38" admin_epoch="0" cib-last-written="Tue Sep 13 18:31:33
2016" update-origin="wirt1v" update-client="crm_resource"
update-user="root" have-quorum="1" dc-uuid="1">
<configuration>
<crm_config>
<cluster_property_set id="cib-bootstrap-options">
<nvpair id="cib-bootstrap-options-have-watchdog"
name="have-watchdog" value="false"/>
<nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
value="1.1.13-10.el7_2.4-44eb2dd"/>
<nvpair id="cib-bootstrap-options-cluster-infrastructure"
name="cluster-infrastructure" value="corosync"/>
<nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name"
value="klasterek"/>
<nvpair id="cib-bootstrap-options-no-quorum-policy"
name="no-quorum-policy" value="ignore"/>
<nvpair id="cib-bootstrap-options-symmetric-cluster"
name="symmetric-cluster" value="true"/>
<nvpair id="cib-bootstrap-options-stonith-enabled"
name="stonith-enabled" value="true"/>
</cluster_property_set>
</crm_config>
<nodes>
<node id="1" uname="wirt1v"/>
<node id="2" uname="wirt2v"/>
</nodes>
<resources>
<master id="Drbd2-clone">
<primitive class="ocf" id="Drbd2" provider="linbit" type="drbd">
<instance_attributes id="Drbd2-instance_attributes">
<nvpair id="Drbd2-instance_attributes-drbd_resource"
name="drbd_resource" value="drbd2"/>
</instance_attributes>
<operations>
<op id="Drbd2-start-interval-0s" interval="0s" name="start"
timeout="240"/>
<op id="Drbd2-promote-interval-0s" interval="0s" name="promote"
timeout="90"/>
<op id="Drbd2-demote-interval-0s" interval="0s" name="demote"
timeout="90"/>
<op id="Drbd2-stop-interval-0s" interval="0s" name="stop"
timeout="100"/>
<op id="Drbd2-monitor-interval-60s" interval="60s"
name="monitor"/>
</operations>
</primitive>
<meta_attributes id="Drbd2-clone-meta_attributes">
<nvpair id="Drbd2-clone-meta_attributes-master-max"
name="master-max" value="2"/>
<nvpair id="Drbd2-clone-meta_attributes-master-node-max"
name="master-node-max" value="1"/>
<nvpair id="Drbd2-clone-meta_attributes-clone-max"
name="clone-max" value="2"/>
<nvpair id="Drbd2-clone-meta_attributes-clone-node-max"
name="clone-node-max" value="1"/>
<nvpair id="Drbd2-clone-meta_attributes-notify" name="notify"
value="true"/>
<nvpair id="Drbd2-clone-meta_attributes-globally-unique"
name="globally-unique" value="false"/>
<nvpair id="Drbd2-clone-meta_attributes-interleave"
name="interleave" value="true"/>
<nvpair id="Drbd2-clone-meta_attributes-ordered" name="ordered"
value="true"/>
</meta_attributes>
</master>
<clone id="Virtfs2-clone">
<primitive class="ocf" id="Virtfs2" provider="heartbeat"
type="Filesystem">
<instance_attributes id="Virtfs2-instance_attributes">
<nvpair id="Virtfs2-instance_attributes-device" name="device"
value="/dev/drbd2"/>
<nvpair id="Virtfs2-instance_attributes-directory"
name="directory" value="/virtfs2"/>
<nvpair id="Virtfs2-instance_attributes-fstype" name="fstype"
value="gfs2"/>
</instance_attributes>
<operations>
<op id="Virtfs2-start-interval-0s" interval="0s" name="start"
timeout="60"/>
<op id="Virtfs2-stop-interval-0s" interval="0s" name="stop"
timeout="60"/>
<op id="Virtfs2-monitor-interval-20" interval="20"
name="monitor" timeout="40"/>
</operations>
</primitive>
<meta_attributes id="Virtfs2-clone-meta_attributes">
<nvpair id="Virtfs2-interleave" name="interleave" value="true"/>
</meta_attributes>
</clone>
<clone id="dlm-clone">
<primitive class="ocf" id="dlm" provider="pacemaker"
type="controld">
<instance_attributes id="dlm-instance_attributes"/>
<operations>
<op id="dlm-start-interval-0s" interval="0s" name="start"
timeout="90"/>
<op id="dlm-stop-interval-0s" interval="0s" name="stop"
timeout="100"/>
<op id="dlm-monitor-interval-60s" interval="60s"
name="monitor"/>
</operations>
</primitive>
<meta_attributes id="dlm-clone-meta_attributes">
<nvpair id="dlm-clone-max" name="clone-max" value="2"/>
<nvpair id="dlm-clone-node-max" name="clone-node-max" value="1"/>
<nvpair id="dlm-interleave" name="interleave" value="true"/>
<nvpair id="dlm-ordered" name="ordered" value="true"/>
</meta_attributes>
</clone>
<primitive class="stonith" id="fencing-idrac1" type="fence_idrac">
<instance_attributes id="fencing-idrac1-instance_attributes">
<nvpair id="fencing-idrac1-instance_attributes-pcmk_host_list"
name="pcmk_host_list" value="wirt1v"/>
<nvpair id="fencing-idrac1-instance_attributes-ipaddr"
name="ipaddr" value="172.31.0.223"/>
<nvpair id="fencing-idrac1-instance_attributes-lanplus"
name="lanplus" value="on"/>
<nvpair id="fencing-idrac1-instance_attributes-login"
name="login" value="root"/>
<nvpair id="fencing-idrac1-instance_attributes-passwd"
name="passwd" value="my1secret2password3"/>
<nvpair id="fencing-idrac1-instance_attributes-action"
name="action" value="reboot"/>
</instance_attributes>
<operations>
<op id="fencing-idrac1-monitor-interval-60" interval="60"
name="monitor"/>
</operations>
</primitive>
<primitive class="stonith" id="fencing-idrac2" type="fence_idrac">
<instance_attributes id="fencing-idrac2-instance_attributes">
<nvpair id="fencing-idrac2-instance_attributes-pcmk_host_list"
name="pcmk_host_list" value="wirt2v"/>
<nvpair id="fencing-idrac2-instance_attributes-ipaddr"
name="ipaddr" value="172.31.0.224"/>
<nvpair id="fencing-idrac2-instance_attributes-lanplus"
name="lanplus" value="on"/>
<nvpair id="fencing-idrac2-instance_attributes-login"
name="login" value="root"/>
<nvpair id="fencing-idrac2-instance_attributes-passwd"
name="passwd" value="my1secret2password3"/>
<nvpair id="fencing-idrac2-instance_attributes-action"
name="action" value="reboot"/>
</instance_attributes>
<operations>
<op id="fencing-idrac2-monitor-interval-60" interval="60"
name="monitor"/>
</operations>
</primitive>
</resources>
<constraints>
<rsc_colocation id="colocation-Virtfs2-clone-Drbd2-clone-INFINITY"
rsc="Virtfs2-clone" score="INFINITY" with-rsc="Drbd2-clone"
with-rsc-role="Master"/>
<rsc_order first="Drbd2-clone" first-action="promote"
id="order-Drbd2-clone-Virtfs2-clone-mandatory" then="Virtfs2-clone"
then-action="start"/>
<rsc_order first="dlm-clone" first-action="start"
id="order-dlm-clone-Virtfs2-clone-mandatory" then="Virtfs2-clone"
then-action="start"/>
<rsc_colocation id="colocation-Virtfs2-clone-dlm-clone-INFINITY"
rsc="Virtfs2-clone" score="INFINITY" with-rsc="dlm-clone"/>
</constraints>
<rsc_defaults>
<meta_attributes id="rsc_defaults-options">
<nvpair id="rsc_defaults-options-resource-stickiness"
name="resource-stickiness" value="100"/>
</meta_attributes>
</rsc_defaults>
</configuration>
<status>
<node_state id="1" uname="wirt1v" in_ccm="true" crmd="online"
crm-debug-origin="do_update_resource" join="member" expected="member">
<lrm id="1">
<lrm_resources>
<lrm_resource id="fencing-idrac1" type="fence_idrac"
class="stonith">
<lrm_rsc_op id="fencing-idrac1_last_0"
operation_key="fencing-idrac1_start_0" operation="start"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="55:0:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;55:0:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt1v" call-id="27" rc-code="0" op-status="0" interval="0"
last-run="1473786030" last-rc-change="1473786030" exec-time="54"
queue-time="0" op-digest="c5f495355c70285327a4ecd128166155"
op-secure-params=" passwd "
op-secure-digest="58f15e2aeb9ef41c7d7016ac60c95b3d"/>
<lrm_rsc_op id="fencing-idrac1_monitor_60000"
operation_key="fencing-idrac1_monitor_60000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="51:1:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;51:1:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt1v" call-id="29" rc-code="0" op-status="0" interval="60000"
last-rc-change="1473786031" exec-time="54" queue-time="0"
op-digest="2c3a04590a892a02a6109a0e8bd4b89a" op-secure-params=" passwd "
op-secure-digest="58f15e2aeb9ef41c7d7016ac60c95b3d"/>
</lrm_resource>
<lrm_resource id="fencing-idrac2" type="fence_idrac"
class="stonith">
<lrm_rsc_op id="fencing-idrac2_last_0"
operation_key="fencing-idrac2_monitor_0" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="8:0:7:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:7;8:0:7:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt1v" call-id="24" rc-code="7" op-status="0" interval="0"
last-run="1473786029" last-rc-change="1473786029" exec-time="0"
queue-time="0" op-digest="62957a33f7a67eda09c15e3f933f2d0b"
op-secure-params=" passwd "
op-secure-digest="65925748cee98be7e9d827ae5f2eb74f"/>
</lrm_resource>
<lrm_resource id="Drbd2" type="drbd" class="ocf"
provider="linbit">
<lrm_rsc_op id="Drbd2_last_0" operation_key="Drbd2_promote_0"
operation="promote" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="10:2:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;10:2:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt1v" call-id="33" rc-code="0" op-status="0" interval="0"
last-run="1473786032" last-rc-change="1473786032" exec-time="64"
queue-time="1" op-digest="d0c8a735862843030d8426a5218ceb92"/>
</lrm_resource>
<lrm_resource id="Virtfs2" type="Filesystem" class="ocf"
provider="heartbeat">
<lrm_rsc_op id="Virtfs2_last_0" operation_key="Virtfs2_start_0"
operation="start" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="41:3:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;41:3:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt1v" call-id="35" rc-code="0" op-status="0" interval="0"
last-run="1473786032" last-rc-change="1473786032" exec-time="1372"
queue-time="0" op-digest="8dbd904c2115508ebcf3dffe8e7c6d82"/>
<lrm_rsc_op id="Virtfs2_monitor_20000"
operation_key="Virtfs2_monitor_20000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="42:3:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;42:3:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt1v" call-id="36" rc-code="0" op-status="0" interval="20000"
last-rc-change="1473786034" exec-time="64" queue-time="0"
op-digest="051271837d1a8eccc0af38fbd8c406e4"/>
</lrm_resource>
<lrm_resource id="dlm" type="controld" class="ocf"
provider="pacemaker">
<lrm_rsc_op id="dlm_last_0" operation_key="dlm_start_0"
operation="start" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="47:0:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;47:0:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt1v" call-id="26" rc-code="0" op-status="0" interval="0"
last-run="1473786030" last-rc-change="1473786030" exec-time="1098"
queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
<lrm_rsc_op id="dlm_monitor_60000"
operation_key="dlm_monitor_60000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="42:1:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;42:1:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt1v" call-id="28" rc-code="0" op-status="0" interval="60000"
last-rc-change="1473786031" exec-time="34" queue-time="0"
op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
</lrm_resource>
</lrm_resources>
</lrm>
<transient_attributes id="1">
<instance_attributes id="status-1">
<nvpair id="status-1-shutdown" name="shutdown" value="0"/>
<nvpair id="status-1-probe_complete" name="probe_complete"
value="true"/>
<nvpair id="status-1-master-Drbd2" name="master-Drbd2"
value="10000"/>
</instance_attributes>
</transient_attributes>
</node_state>
<node_state id="2" uname="wirt2v" in_ccm="true" crmd="online"
crm-debug-origin="do_update_resource" join="member" expected="member">
<lrm id="2">
<lrm_resources>
<lrm_resource id="fencing-idrac1" type="fence_idrac"
class="stonith">
<lrm_rsc_op id="fencing-idrac1_last_0"
operation_key="fencing-idrac1_monitor_0" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="13:0:7:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:7;13:0:7:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt2v" call-id="20" rc-code="7" op-status="0" interval="0"
last-run="1473786029" last-rc-change="1473786029" exec-time="3"
queue-time="0" op-digest="c5f495355c70285327a4ecd128166155"
op-secure-params=" passwd "
op-secure-digest="58f15e2aeb9ef41c7d7016ac60c95b3d"/>
</lrm_resource>
<lrm_resource id="fencing-idrac2" type="fence_idrac"
class="stonith">
<lrm_rsc_op id="fencing-idrac2_last_0"
operation_key="fencing-idrac2_start_0" operation="start"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="57:0:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;57:0:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt2v" call-id="25" rc-code="0" op-status="0" interval="0"
last-run="1473786030" last-rc-change="1473786030" exec-time="62"
queue-time="0" op-digest="62957a33f7a67eda09c15e3f933f2d0b"
op-secure-params=" passwd "
op-secure-digest="65925748cee98be7e9d827ae5f2eb74f"/>
<lrm_rsc_op id="fencing-idrac2_monitor_60000"
operation_key="fencing-idrac2_monitor_60000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="54:1:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;54:1:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt2v" call-id="26" rc-code="0" op-status="0" interval="60000"
last-rc-change="1473786031" exec-time="74" queue-time="0"
op-digest="02c5ce42002631d918b41adc571d64b8" op-secure-params=" passwd "
op-secure-digest="65925748cee98be7e9d827ae5f2eb74f"/>
</lrm_resource>
<lrm_resource id="dlm" type="controld" class="ocf"
provider="pacemaker">
<lrm_rsc_op id="dlm_last_0" operation_key="dlm_start_0"
operation="start" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="43:1:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;43:1:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt2v" call-id="27" rc-code="0" op-status="0" interval="0"
last-run="1473786031" last-rc-change="1473786031" exec-time="1102"
queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
<lrm_rsc_op id="dlm_monitor_60000"
operation_key="dlm_monitor_60000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="50:2:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;50:2:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt2v" call-id="30" rc-code="0" op-status="0" interval="60000"
last-rc-change="1473786032" exec-time="32" queue-time="0"
op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
</lrm_resource>
<lrm_resource id="Drbd2" type="drbd" class="ocf"
provider="linbit">
<lrm_rsc_op id="Drbd2_last_0" operation_key="Drbd2_promote_0"
operation="promote" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="13:2:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;13:2:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt2v" call-id="32" rc-code="0" op-status="0" interval="0"
last-run="1473786032" last-rc-change="1473786032" exec-time="55"
queue-time="0" op-digest="d0c8a735862843030d8426a5218ceb92"/>
</lrm_resource>
<lrm_resource id="Virtfs2" type="Filesystem" class="ocf"
provider="heartbeat">
<lrm_rsc_op id="Virtfs2_last_0" operation_key="Virtfs2_start_0"
operation="start" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="43:3:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;43:3:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt2v" call-id="34" rc-code="0" op-status="0" interval="0"
last-run="1473786032" last-rc-change="1473786032" exec-time="939"
queue-time="0" op-digest="8dbd904c2115508ebcf3dffe8e7c6d82"/>
<lrm_rsc_op id="Virtfs2_monitor_20000"
operation_key="Virtfs2_monitor_20000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="44:3:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
transition-magic="0:0;44:3:0:5f2f0724-e33d-4494-90b2-9e06a0e2b0df"
on_node="wirt2v" call-id="35" rc-code="0" op-status="0" interval="20000"
last-rc-change="1473786033" exec-time="39" queue-time="0"
op-digest="051271837d1a8eccc0af38fbd8c406e4"/>
</lrm_resource>
</lrm_resources>
</lrm>
<transient_attributes id="2">
<instance_attributes id="status-2">
<nvpair id="status-2-shutdown" name="shutdown" value="0"/>
<nvpair id="status-2-probe_complete" name="probe_complete"
value="true"/>
<nvpair id="status-2-master-Drbd2" name="master-Drbd2"
value="10000"/>
</instance_attributes>
</transient_attributes>
</node_state>
</status>
</cib>
#-------- The End --------------------
### result: pcs config ###
Cluster Name: klasterek
Corosync Nodes:
wirt1v wirt2v
Pacemaker Nodes:
wirt1v wirt2v
Resources:
Master: Drbd2-clone
Meta Attrs: master-max=2 master-node-max=1 clone-max=2 clone-node-max=1
notify=true globally-unique=false interleave=true ordered=true
Resource: Drbd2 (class=ocf provider=linbit type=drbd)
Attributes: drbd_resource=drbd2
Operations: start interval=0s timeout=240 (Drbd2-start-interval-0s)
promote interval=0s timeout=90 (Drbd2-promote-interval-0s)
demote interval=0s timeout=90 (Drbd2-demote-interval-0s)
stop interval=0s timeout=100 (Drbd2-stop-interval-0s)
monitor interval=60s (Drbd2-monitor-interval-60s)
Clone: Virtfs2-clone
Meta Attrs: interleave=true
Resource: Virtfs2 (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/drbd2 directory=/virtfs2 fstype=gfs2
Operations: start interval=0s timeout=60 (Virtfs2-start-interval-0s)
stop interval=0s timeout=60 (Virtfs2-stop-interval-0s)
monitor interval=20 timeout=40 (Virtfs2-monitor-interval-20)
Clone: dlm-clone
Meta Attrs: clone-max=2 clone-node-max=1 interleave=true ordered=true
Resource: dlm (class=ocf provider=pacemaker type=controld)
Operations: start interval=0s timeout=90 (dlm-start-interval-0s)
stop interval=0s timeout=100 (dlm-stop-interval-0s)
monitor interval=60s (dlm-monitor-interval-60s)
Stonith Devices:
Resource: fencing-idrac1 (class=stonith type=fence_idrac)
Attributes: pcmk_host_list=wirt1v ipaddr=172.31.0.223 lanplus=on
login=root passwd=my1secret2password3 action=reboot
Operations: monitor interval=60 (fencing-idrac1-monitor-interval-60)
Resource: fencing-idrac2 (class=stonith type=fence_idrac)
Attributes: pcmk_host_list=wirt2v ipaddr=172.31.0.224 lanplus=on
login=root passwd=my1secret2password3 action=reboot
Operations: monitor interval=60 (fencing-idrac2-monitor-interval-60)
Fencing Levels:
Location Constraints:
Ordering Constraints:
promote Drbd2-clone then start Virtfs2-clone (kind:Mandatory)
(id:order-Drbd2-clone-Virtfs2-clone-mandatory)
start dlm-clone then start Virtfs2-clone (kind:Mandatory)
(id:order-dlm-clone-Virtfs2-clone-mandatory)
Colocation Constraints:
Virtfs2-clone with Drbd2-clone (score:INFINITY) (with-rsc-role:Master)
(id:colocation-Virtfs2-clone-Drbd2-clone-INFINITY)
Virtfs2-clone with dlm-clone (score:INFINITY)
(id:colocation-Virtfs2-clone-dlm-clone-INFINITY)
Resources Defaults:
resource-stickiness: 100
Operations Defaults:
No defaults set
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: klasterek
dc-version: 1.1.13-10.el7_2.4-44eb2dd
have-watchdog: false
no-quorum-policy: ignore
stonith-enabled: true
symmetric-cluster: true
#---------------------------------
# /var/log/messages
Sep 13 22:00:19 wirt1v systemd: Starting Corosync Cluster Engine...
Sep 13 22:00:19 wirt1v corosync[5720]: [MAIN ] Corosync Cluster Engine
('2.3.4'): started and ready to provide service.
Sep 13 22:00:19 wirt1v corosync[5720]: [MAIN ] Corosync built-in features:
dbus systemd xmlconf snmp pie relro bindnow
Sep 13 22:00:19 wirt1v corosync[5721]: [TOTEM ] Initializing transport
(UDP/IP Unicast).
Sep 13 22:00:19 wirt1v corosync[5721]: [TOTEM ] Initializing
transmit/receive security (NSS) crypto: none hash: none
Sep 13 22:00:19 wirt1v corosync[5721]: [TOTEM ] The network interface
[1.1.1.1] is now up.
Sep 13 22:00:19 wirt1v corosync[5721]: [SERV ] Service engine loaded:
corosync configuration map access [0]
Sep 13 22:00:19 wirt1v corosync[5721]: [QB ] server name: cmap
Sep 13 22:00:19 wirt1v corosync[5721]: [SERV ] Service engine loaded:
corosync configuration service [1]
Sep 13 22:00:19 wirt1v corosync[5721]: [QB ] server name: cfg
Sep 13 22:00:19 wirt1v corosync[5721]: [SERV ] Service engine loaded:
corosync cluster closed process group service v1.01 [2]
Sep 13 22:00:19 wirt1v corosync[5721]: [QB ] server name: cpg
Sep 13 22:00:19 wirt1v corosync[5721]: [SERV ] Service engine loaded:
corosync profile loading service [4]
Sep 13 22:00:19 wirt1v corosync[5721]: [QUORUM] Using quorum provider
corosync_votequorum
Sep 13 22:00:19 wirt1v corosync[5721]: [VOTEQ ] Waiting for all cluster
members. Current votes: 1 expected_votes: 2
Sep 13 22:00:19 wirt1v corosync[5721]: [SERV ] Service engine loaded:
corosync vote quorum service v1.0 [5]
Sep 13 22:00:19 wirt1v corosync[5721]: [QB ] server name: votequorum
Sep 13 22:00:19 wirt1v corosync[5721]: [SERV ] Service engine loaded:
corosync cluster quorum service v0.1 [3]
Sep 13 22:00:19 wirt1v corosync[5721]: [QB ] server name: quorum
Sep 13 22:00:19 wirt1v corosync[5721]: [TOTEM ] adding new UDPU member
{1.1.1.1}
Sep 13 22:00:19 wirt1v corosync[5721]: [TOTEM ] adding new UDPU member
{1.1.1.2}
Sep 13 22:00:19 wirt1v corosync[5721]: [TOTEM ] A new membership (
1.1.1.1:708) was formed. Members joined: 1
Sep 13 22:00:19 wirt1v corosync[5721]: [VOTEQ ] Waiting for all cluster
members. Current votes: 1 expected_votes: 2
Sep 13 22:00:19 wirt1v corosync[5721]: [VOTEQ ] Waiting for all cluster
members. Current votes: 1 expected_votes: 2
Sep 13 22:00:19 wirt1v corosync[5721]: [VOTEQ ] Waiting for all cluster
members. Current votes: 1 expected_votes: 2
Sep 13 22:00:19 wirt1v corosync[5721]: [QUORUM] Members[1]: 1
Sep 13 22:00:19 wirt1v corosync[5721]: [MAIN ] Completed service
synchronization, ready to provide service.
Sep 13 22:00:20 wirt1v corosync: Starting Corosync Cluster Engine
(corosync): [ OK ]
Sep 13 22:00:20 wirt1v systemd: Started Corosync Cluster Engine.
Sep 13 22:00:20 wirt1v systemd: Started Pacemaker High Availability Cluster
Manager.
Sep 13 22:00:20 wirt1v systemd: Starting Pacemaker High Availability
Cluster Manager...
Sep 13 22:00:20 wirt1v pacemakerd[5740]: notice: Additional logging
available in /var/log/pacemaker.log
Sep 13 22:00:20 wirt1v pacemakerd[5740]: notice: Switching to
/var/log/cluster/corosync.log
Sep 13 22:00:20 wirt1v pacemakerd[5740]: notice: Additional logging
available in /var/log/cluster/corosync.log
Sep 13 22:00:20 wirt1v pacemakerd[5740]: notice: Configured corosync to
accept connections from group 189: OK (1)
Sep 13 22:00:20 wirt1v pacemakerd[5740]: notice: Starting Pacemaker
1.1.13-10.el7_2.4 (Build: 44eb2dd): generated-manpages agent-manpages
ncurses libqb-logging libqb-ipc upstart systemd nagios corosync-native
atomic-attrd acls
Sep 13 22:00:20 wirt1v pacemakerd[5740]: notice: Tracking existing lrmd
process (pid=3413)
Sep 13 22:00:20 wirt1v pacemakerd[5740]: notice: Tracking existing pengine
process (pid=3415)
Sep 13 22:00:20 wirt1v pacemakerd[5740]: notice: Quorum lost
Sep 13 22:00:20 wirt1v pacemakerd[5740]: notice: pcmk_quorum_notification:
Node wirt1v[1] - state is now member (was (null))
Sep 13 22:00:20 wirt1v stonith-ng[5742]: notice: Additional logging
available in /var/log/cluster/corosync.log
Sep 13 22:00:20 wirt1v cib[5741]: notice: Additional logging available in
/var/log/cluster/corosync.log
Sep 13 22:00:20 wirt1v stonith-ng[5742]: notice: Connecting to cluster
infrastructure: corosync
Sep 13 22:00:20 wirt1v attrd[5743]: notice: Additional logging available
in /var/log/cluster/corosync.log
Sep 13 22:00:20 wirt1v attrd[5743]: notice: Connecting to cluster
infrastructure: corosync
Sep 13 22:00:20 wirt1v crmd[5744]: notice: Additional logging available in
/var/log/cluster/corosync.log
Sep 13 22:00:20 wirt1v crmd[5744]: notice: CRM Git Version:
1.1.13-10.el7_2.4 (44eb2dd)
Sep 13 22:00:20 wirt1v cib[5741]: notice: Connecting to cluster
infrastructure: corosync
Sep 13 22:00:20 wirt1v attrd[5743]: notice: crm_update_peer_proc: Node
wirt1v[1] - state is now member (was (null))
Sep 13 22:00:20 wirt1v stonith-ng[5742]: notice: crm_update_peer_proc:
Node wirt1v[1] - state is now member (was (null))
Sep 13 22:00:20 wirt1v cib[5741]: notice: crm_update_peer_proc: Node
wirt1v[1] - state is now member (was (null))
Sep 13 22:00:21 wirt1v crmd[5744]: notice: Connecting to cluster
infrastructure: corosync
Sep 13 22:00:21 wirt1v crmd[5744]: notice: Quorum lost
Sep 13 22:00:21 wirt1v stonith-ng[5742]: notice: Watching for stonith
topology changes
Sep 13 22:00:21 wirt1v stonith-ng[5742]: notice: On loss of CCM Quorum:
Ignore
Sep 13 22:00:21 wirt1v crmd[5744]: notice: pcmk_quorum_notification: Node
wirt1v[1] - state is now member (was (null))
Sep 13 22:00:21 wirt1v crmd[5744]: notice: Notifications disabled
Sep 13 22:00:21 wirt1v crmd[5744]: notice: The local CRM is operational
Sep 13 22:00:21 wirt1v crmd[5744]: notice: State transition S_STARTING ->
S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Sep 13 22:00:22 wirt1v stonith-ng[5742]: notice: Added 'fencing-idrac1' to
the device list (1 active devices)
Sep 13 22:00:22 wirt1v stonith-ng[5742]: notice: Added 'fencing-idrac2' to
the device list (2 active devices)
Sep 13 22:00:42 wirt1v crmd[5744]: warning: FSA: Input I_DC_TIMEOUT from
crm_timer_popped() received in state S_PENDING
Sep 13 22:00:42 wirt1v crmd[5744]: notice: State transition S_ELECTION ->
S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED
origin=election_timeout_popped ]
Sep 13 22:00:42 wirt1v crmd[5744]: warning: FSA: Input I_ELECTION_DC from
do_election_check() received in state S_INTEGRATION
Sep 13 22:00:42 wirt1v crmd[5744]: notice: Notifications disabled
Sep 13 22:00:42 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:42 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:42 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:42 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:42 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:42 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:42 wirt1v pengine[3415]: warning: Calculated Transition 84:
/var/lib/pacemaker/pengine/pe-warn-294.bz2
Sep 13 22:00:42 wirt1v crmd[5744]: notice: Initiating action 4: monitor
Drbd2:0_monitor_0 on wirt1v (local)
Sep 13 22:00:42 wirt1v crmd[5744]: notice: Initiating action 5: monitor
Virtfs2:0_monitor_0 on wirt1v (local)
Sep 13 22:00:42 wirt1v crmd[5744]: notice: Initiating action 6: monitor
dlm:0_monitor_0 on wirt1v (local)
Sep 13 22:00:42 wirt1v crmd[5744]: notice: Initiating action 7: monitor
fencing-idrac1_monitor_0 on wirt1v (local)
Sep 13 22:00:42 wirt1v crmd[5744]: notice: Initiating action 8: monitor
fencing-idrac2_monitor_0 on wirt1v (local)
Sep 13 22:00:42 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (50) on wirt2v (timeout=60000)
Sep 13 22:00:42 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:42 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: e87b942f-997d-42ad-91ad-dfa501f4ede0 (0)
Sep 13 22:00:42 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:42 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:42 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:42 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:42 wirt1v Filesystem(Virtfs2)[5753]: WARNING: Couldn't find
device [/dev/drbd2]. Expected /dev/??? to exist
Sep 13 22:00:42 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Operation
fencing-idrac1_monitor_0: not running (node=wirt1v, call=33, rc=7,
cib-update=31, confirmed=true)
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Operation
fencing-idrac2_monitor_0: not running (node=wirt1v, call=35, rc=7,
cib-update=32, confirmed=true)
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Operation dlm_monitor_0: not
running (node=wirt1v, call=31, rc=7, cib-update=33, confirmed=true)
Sep 13 22:00:43 wirt1v crmd[5744]: error: pcmkRegisterNode: Triggered
assert at xml.c:594 : node->type == XML_ELEMENT_NODE
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Operation Drbd2_monitor_0: not
running (node=wirt1v, call=27, rc=7, cib-update=34, confirmed=true)
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Operation Virtfs2_monitor_0:
not running (node=wirt1v, call=29, rc=7, cib-update=35, confirmed=true)
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Initiating action 3:
probe_complete probe_complete-wirt1v on wirt1v (local) - no waiting
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Transition aborted by
status-1-probe_complete, probe_complete=true: Transient attribute change
(create cib=0.69.11, source=abort_unless_down:319,
path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'],
0)
Sep 13 22:00:43 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:43 wirt1v stonith-ng[5742]: error: Operation 'reboot' [5849]
(call 2 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:43 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5849 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:43 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5849 [ ]
Sep 13 22:00:43 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5849 [ ]
Sep 13 22:00:43 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:43 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.e87b942f: No route to host
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Stonith operation
2/50:84:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Stonith operation 2 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=e87b942f-997d-42ad-91ad-dfa501f4ede0) by client crmd.5744
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Transition 84 (Complete=12,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-294.bz2): Complete
Sep 13 22:00:43 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:43 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:43 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:43 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:43 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:43 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:43 wirt1v pengine[3415]: warning: Calculated Transition 85:
/var/lib/pacemaker/pengine/pe-warn-295.bz2
Sep 13 22:00:43 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (45) on wirt2v (timeout=60000)
Sep 13 22:00:43 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:43 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: 880b2614-09d2-47df-b740-e1d24732e6c5 (0)
Sep 13 22:00:43 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:43 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:43 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:43 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:43 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:44 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:44 wirt1v stonith-ng[5742]: error: Operation 'reboot' [5879]
(call 3 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:44 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5879 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:44 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5879 [ ]
Sep 13 22:00:44 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5879 [ ]
Sep 13 22:00:44 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:44 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.880b2614: No route to host
Sep 13 22:00:44 wirt1v crmd[5744]: notice: Stonith operation
3/45:85:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:44 wirt1v crmd[5744]: notice: Stonith operation 3 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:44 wirt1v crmd[5744]: notice: Transition aborted: Stonith
failed (source=tengine_stonith_callback:733, 0)
Sep 13 22:00:44 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=880b2614-09d2-47df-b740-e1d24732e6c5) by client crmd.5744
Sep 13 22:00:44 wirt1v crmd[5744]: notice: Transition 85 (Complete=5,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-295.bz2): Complete
Sep 13 22:00:44 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:44 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:44 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:44 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:44 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:44 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:44 wirt1v pengine[3415]: warning: Calculated Transition 86:
/var/lib/pacemaker/pengine/pe-warn-295.bz2
Sep 13 22:00:44 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (45) on wirt2v (timeout=60000)
Sep 13 22:00:44 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:44 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: 4c7af8ee-ffa6-4381-8d98-073d5abba631 (0)
Sep 13 22:00:44 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:44 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:44 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:44 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:44 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:45 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:45 wirt1v stonith-ng[5742]: error: Operation 'reboot' [5893]
(call 4 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:45 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5893 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:45 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5893 [ ]
Sep 13 22:00:45 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5893 [ ]
Sep 13 22:00:45 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:45 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.4c7af8ee: No route to host
Sep 13 22:00:45 wirt1v crmd[5744]: notice: Stonith operation
4/45:86:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:45 wirt1v crmd[5744]: notice: Stonith operation 4 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:45 wirt1v crmd[5744]: notice: Transition aborted: Stonith
failed (source=tengine_stonith_callback:733, 0)
Sep 13 22:00:45 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=4c7af8ee-ffa6-4381-8d98-073d5abba631) by client crmd.5744
Sep 13 22:00:45 wirt1v crmd[5744]: notice: Transition 86 (Complete=5,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-295.bz2): Complete
Sep 13 22:00:45 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:45 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:45 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:45 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:45 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:45 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:45 wirt1v pengine[3415]: warning: Calculated Transition 87:
/var/lib/pacemaker/pengine/pe-warn-295.bz2
Sep 13 22:00:45 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (45) on wirt2v (timeout=60000)
Sep 13 22:00:45 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:45 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: 268e4c7b-0340-4cf5-9c88-4f3c203f1499 (0)
Sep 13 22:00:45 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:45 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:45 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:45 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:46 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:47 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:47 wirt1v stonith-ng[5742]: error: Operation 'reboot' [5907]
(call 5 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:47 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5907 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:47 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5907 [ ]
Sep 13 22:00:47 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5907 [ ]
Sep 13 22:00:47 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:47 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.268e4c7b: No route to host
Sep 13 22:00:47 wirt1v crmd[5744]: notice: Stonith operation
5/45:87:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:47 wirt1v crmd[5744]: notice: Stonith operation 5 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:47 wirt1v crmd[5744]: notice: Transition aborted: Stonith
failed (source=tengine_stonith_callback:733, 0)
Sep 13 22:00:47 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=268e4c7b-0340-4cf5-9c88-4f3c203f1499) by client crmd.5744
Sep 13 22:00:47 wirt1v crmd[5744]: notice: Transition 87 (Complete=5,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-295.bz2): Complete
Sep 13 22:00:47 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:47 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:47 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:47 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:47 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:47 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:47 wirt1v pengine[3415]: warning: Calculated Transition 88:
/var/lib/pacemaker/pengine/pe-warn-295.bz2
Sep 13 22:00:47 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (45) on wirt2v (timeout=60000)
Sep 13 22:00:47 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:47 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: 8c5bf217-030f-400a-b1f8-7aa19918954f (0)
Sep 13 22:00:47 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:47 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:47 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:47 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:47 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:48 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:48 wirt1v stonith-ng[5742]: error: Operation 'reboot' [5921]
(call 6 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:48 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5921 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:48 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5921 [ ]
Sep 13 22:00:48 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5921 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:48 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5921 [ ]
Sep 13 22:00:48 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5921 [ ]
Sep 13 22:00:48 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:48 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.8c5bf217: No route to host
Sep 13 22:00:48 wirt1v crmd[5744]: notice: Stonith operation
6/45:88:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:48 wirt1v crmd[5744]: notice: Stonith operation 6 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:48 wirt1v crmd[5744]: notice: Transition aborted: Stonith
failed (source=tengine_stonith_callback:733, 0)
Sep 13 22:00:48 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=8c5bf217-030f-400a-b1f8-7aa19918954f) by client crmd.5744
Sep 13 22:00:48 wirt1v crmd[5744]: notice: Transition 88 (Complete=5,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-295.bz2): Complete
Sep 13 22:00:48 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:48 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:48 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:48 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:48 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:48 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:48 wirt1v pengine[3415]: warning: Calculated Transition 89:
/var/lib/pacemaker/pengine/pe-warn-295.bz2
Sep 13 22:00:48 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (45) on wirt2v (timeout=60000)
Sep 13 22:00:48 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:48 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: 25e51799-e072-4622-bbb3-1430bdb20536 (0)
Sep 13 22:00:48 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:48 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:48 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:48 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:48 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:49 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:49 wirt1v stonith-ng[5742]: error: Operation 'reboot' [5935]
(call 7 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:49 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5935 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:49 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5935 [ ]
Sep 13 22:00:49 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5935 [ ]
Sep 13 22:00:49 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:49 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.25e51799: No route to host
Sep 13 22:00:49 wirt1v crmd[5744]: notice: Stonith operation
7/45:89:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:49 wirt1v crmd[5744]: notice: Stonith operation 7 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:49 wirt1v crmd[5744]: notice: Transition aborted: Stonith
failed (source=tengine_stonith_callback:733, 0)
Sep 13 22:00:49 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=25e51799-e072-4622-bbb3-1430bdb20536) by client crmd.5744
Sep 13 22:00:49 wirt1v crmd[5744]: notice: Transition 89 (Complete=5,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-295.bz2): Complete
Sep 13 22:00:49 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:49 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:49 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:49 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:49 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:49 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:49 wirt1v pengine[3415]: warning: Calculated Transition 90:
/var/lib/pacemaker/pengine/pe-warn-295.bz2
Sep 13 22:00:49 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (45) on wirt2v (timeout=60000)
Sep 13 22:00:49 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:49 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: 7f520e61-b613-49e4-9213-1958d8a68c6a (0)
Sep 13 22:00:49 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:49 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:49 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:49 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:49 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:50 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:50 wirt1v stonith-ng[5742]: error: Operation 'reboot' [5949]
(call 8 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:50 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5949 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:50 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5949 [ ]
Sep 13 22:00:50 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5949 [ ]
Sep 13 22:00:50 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:50 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.7f520e61: No route to host
Sep 13 22:00:50 wirt1v crmd[5744]: notice: Stonith operation
8/45:90:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:50 wirt1v crmd[5744]: notice: Stonith operation 8 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:50 wirt1v crmd[5744]: notice: Transition aborted: Stonith
failed (source=tengine_stonith_callback:733, 0)
Sep 13 22:00:50 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=7f520e61-b613-49e4-9213-1958d8a68c6a) by client crmd.5744
Sep 13 22:00:50 wirt1v crmd[5744]: notice: Transition 90 (Complete=5,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-295.bz2): Complete
Sep 13 22:00:50 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:50 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:50 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:50 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:50 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:50 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:50 wirt1v pengine[3415]: warning: Calculated Transition 91:
/var/lib/pacemaker/pengine/pe-warn-295.bz2
Sep 13 22:00:50 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (45) on wirt2v (timeout=60000)
Sep 13 22:00:50 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:50 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: 25b67d0b-5b8f-4cd8-82c2-4421474c111c (0)
Sep 13 22:00:50 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:50 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:50 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:50 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:50 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:51 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:51 wirt1v stonith-ng[5742]: error: Operation 'reboot' [5963]
(call 9 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:51 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5963 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:51 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5963 [ ]
Sep 13 22:00:51 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5963 [ ]
Sep 13 22:00:51 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:51 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.25b67d0b: No route to host
Sep 13 22:00:51 wirt1v crmd[5744]: notice: Stonith operation
9/45:91:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:51 wirt1v crmd[5744]: notice: Stonith operation 9 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:51 wirt1v crmd[5744]: notice: Transition aborted: Stonith
failed (source=tengine_stonith_callback:733, 0)
Sep 13 22:00:51 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=25b67d0b-5b8f-4cd8-82c2-4421474c111c) by client crmd.5744
Sep 13 22:00:51 wirt1v crmd[5744]: notice: Transition 91 (Complete=5,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-295.bz2): Complete
Sep 13 22:00:51 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:51 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:51 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:51 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:51 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:51 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:51 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:51 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:51 wirt1v pengine[3415]: warning: Calculated Transition 92:
/var/lib/pacemaker/pengine/pe-warn-295.bz2
Sep 13 22:00:51 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (45) on wirt2v (timeout=60000)
Sep 13 22:00:51 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:51 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: 292a57e9-fd1b-4630-8c10-0d48a268fd68 (0)
Sep 13 22:00:51 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:51 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:51 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:51 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:51 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:52 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:52 wirt1v stonith-ng[5742]: error: Operation 'reboot' [5977]
(call 10 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:52 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5977 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:52 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5977 [ ]
Sep 13 22:00:52 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5977 [ ]
Sep 13 22:00:52 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:52 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.292a57e9: No route to host
Sep 13 22:00:52 wirt1v crmd[5744]: notice: Stonith operation
10/45:92:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:52 wirt1v crmd[5744]: notice: Stonith operation 10 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:52 wirt1v crmd[5744]: notice: Transition aborted: Stonith
failed (source=tengine_stonith_callback:733, 0)
Sep 13 22:00:52 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=292a57e9-fd1b-4630-8c10-0d48a268fd68) by client crmd.5744
Sep 13 22:00:52 wirt1v crmd[5744]: notice: Transition 92 (Complete=5,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-295.bz2): Complete
Sep 13 22:00:52 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:52 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:52 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:52 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:52 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:52 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:52 wirt1v pengine[3415]: warning: Calculated Transition 93:
/var/lib/pacemaker/pengine/pe-warn-295.bz2
Sep 13 22:00:52 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (45) on wirt2v (timeout=60000)
Sep 13 22:00:52 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:52 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: f324baad-ef9b-44e6-9e09-02176fa447ef (0)
Sep 13 22:00:52 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:52 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:52 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:52 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:53 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:54 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:54 wirt1v stonith-ng[5742]: error: Operation 'reboot' [5991]
(call 11 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:54 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5991 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:54 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5991 [ ]
Sep 13 22:00:54 wirt1v stonith-ng[5742]: warning: fencing-idrac2:5991 [ ]
Sep 13 22:00:54 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:54 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.f324baad: No route to host
Sep 13 22:00:54 wirt1v crmd[5744]: notice: Stonith operation
11/45:93:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:54 wirt1v crmd[5744]: notice: Stonith operation 11 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:54 wirt1v crmd[5744]: notice: Transition aborted: Stonith
failed (source=tengine_stonith_callback:733, 0)
Sep 13 22:00:54 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=f324baad-ef9b-44e6-9e09-02176fa447ef) by client crmd.5744
Sep 13 22:00:54 wirt1v crmd[5744]: notice: Transition 93 (Complete=5,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-295.bz2): Complete
Sep 13 22:00:54 wirt1v pengine[3415]: notice: On loss of CCM Quorum: Ignore
Sep 13 22:00:54 wirt1v pengine[3415]: warning: Scheduling Node wirt2v for
STONITH
Sep 13 22:00:54 wirt1v pengine[3415]: notice: Start Drbd2:0#011(wirt1v)
Sep 13 22:00:54 wirt1v pengine[3415]: notice: Start dlm:0#011(wirt1v)
Sep 13 22:00:54 wirt1v pengine[3415]: notice: Start
fencing-idrac1#011(wirt1v)
Sep 13 22:00:54 wirt1v pengine[3415]: notice: Start
fencing-idrac2#011(wirt1v)
Sep 13 22:00:54 wirt1v pengine[3415]: warning: Calculated Transition 94:
/var/lib/pacemaker/pengine/pe-warn-295.bz2
Sep 13 22:00:54 wirt1v crmd[5744]: notice: Executing reboot fencing
operation (45) on wirt2v (timeout=60000)
Sep 13 22:00:54 wirt1v stonith-ng[5742]: notice: Client crmd.5744.8928b80c
wants to fence (reboot) 'wirt2v' with device '(any)'
Sep 13 22:00:54 wirt1v stonith-ng[5742]: notice: Initiating remote
operation reboot for wirt2v: 61af386a-ce3f-438f-b83b-90dee4bdb1c6 (0)
Sep 13 22:00:54 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:54 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:54 wirt1v stonith-ng[5742]: notice: fencing-idrac1 can not
fence (reboot) wirt2v: static-list
Sep 13 22:00:54 wirt1v stonith-ng[5742]: notice: fencing-idrac2 can fence
(reboot) wirt2v: static-list
Sep 13 22:00:54 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:55 wirt1v fence_idrac: Failed: Unable to obtain correct plug
status or plug is not available
Sep 13 22:00:55 wirt1v stonith-ng[5742]: error: Operation 'reboot' [6005]
(call 12 from crmd.5744) for host 'wirt2v' with device 'fencing-idrac2'
returned: -201 (Generic Pacemaker error)
Sep 13 22:00:55 wirt1v stonith-ng[5742]: warning: fencing-idrac2:6005 [
Failed: Unable to obtain correct plug status or plug is not available ]
Sep 13 22:00:55 wirt1v stonith-ng[5742]: warning: fencing-idrac2:6005 [ ]
Sep 13 22:00:55 wirt1v stonith-ng[5742]: warning: fencing-idrac2:6005 [ ]
Sep 13 22:00:55 wirt1v stonith-ng[5742]: notice: Couldn't find anyone to
fence (reboot) wirt2v with any device
Sep 13 22:00:55 wirt1v stonith-ng[5742]: error: Operation reboot of
wirt2v by <no-one> for crmd.5744 at wirt1v.61af386a: No route to host
Sep 13 22:00:55 wirt1v crmd[5744]: notice: Stonith operation
12/45:94:0:dd848cfe-edbc-41f4-bd55-f0cad5f7204f: No route to host (-113)
Sep 13 22:00:55 wirt1v crmd[5744]: notice: Stonith operation 12 for wirt2v
failed (No route to host): aborting transition.
Sep 13 22:00:55 wirt1v crmd[5744]: notice: Transition aborted: Stonith
failed (source=tengine_stonith_callback:733, 0)
Sep 13 22:00:55 wirt1v crmd[5744]: notice: Peer wirt2v was not terminated
(reboot) by <anyone> for wirt1v: No route to host
(ref=61af386a-ce3f-438f-b83b-90dee4bdb1c6) by client crmd.5744
Sep 13 22:00:55 wirt1v crmd[5744]: notice: Transition 94 (Complete=5,
Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pacemaker/pengine/pe-warn-295.bz2): Complete
Sep 13 22:00:55 wirt1v crmd[5744]: notice: Too many failures to fence
wirt2v (11), giving up
Sep 13 22:00:55 wirt1v crmd[5744]: notice: State transition
S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
origin=notify_crmd ]
# -------------------- end of /var/log/messages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20160913/5618336e/attachment-0002.html>
More information about the Users
mailing list