[ClusterLabs] Three node cluster becomes completely fenced if one node leaves
Seth Reid
sreid at vendini.com
Fri Mar 24 17:13:37 EDT 2017
> On 24/03/17 04:44 PM, Seth Reid wrote:
> > I have a three node Pacemaker/GFS2 cluster on Ubuntu 16.04. Its not in
> > production yet because I'm having a problem during fencing. When I
> > disable the network interface of any one machine, the disabled machines
> > is properly fenced leaving me, briefly, with a two node cluster. A
> > second node is then fenced off immediately, and the remaining node
> > appears to try to fence itself off. This leave two nodes with
> > corosync/pacemaker stopped, and the remaining machine still in the
> > cluster but showing an offline node and an UNCLEAN node. What can be
> > causing this behavior?
>
> It looks like the fence attempt failed, leaving the cluster hung. When
> you say all nodes were fenced, did all nodes actually reboot? Or did the
> two surviving nodes just lock up? If the later, then that is the proper
> response to a failed fence (DLM stays blocked).
>
The action if "off", so we aren't rebooting. The logs do still say reboot
though. In terms of actual fencing, only node 2 gets fenced, in that its
keys get removed from the shared volume. Node 1's keys don't get removed so
that is the failed fence. Node2 fence succeeds.
Of the remaining nodes, node 1 is offline in that corosync and pacemaker
are no longer running, so it can't access cluster resources. Node 3 shows
node 1 as Online but in a clean state. Neither node 1 or node 3 can write
to the cluster, but node 3 still has corosync and pacemaker running.
Here are the commands I used to build the cluster. I meant to put these in
the original post.
(single machine)$> pcs property set no-quorum-policy=freeze
(single machine)$> pcs property set stonith-enabled=true
(single machine)$> pcs property set symmetric-cluster=true
(single machine)$> pcs cluster enable --all
(single machine)$> pcs stonith create fence_wh fence_scsi
debug="/var/log/cluster/fence-debug.log" vgs_path="/sbin/vgs"
sg_persist_path="/usr/bin/sg_persist" sg_turs_path="/usr/bin/sg_turs"
pcmk_reboot_action="off" pcmk_host_list="b013-cl b014-cl b015-cl"
pcmk_monitor_action="metadata" meta provides="unfencing" --force
(single machine)$> pcs resource create dlm ocf:pacemaker:controld op
monitor interval=30s on-fail=fence clone interleave=true ordered=true
(single machine)$> pcs resource create clvmd ocf:heartbeat:clvm op monitor
interval=30s on-fail=fence clone interleave=true ordered=true
(single machine)$> pcs constraint order start dlm-clone then clvmd-clone
(single machine)$> pcs constraint colocation add clvmd-clone with dlm-clone
(single machine)$> mkfs.gfs2 -p lock_dlm -t webhosts:share_data -j 3
/dev/mapper/share-data
(single machine)$> pcs resource create gfs2share Filesystem
device="/dev/mapper/share-data" directory="/share" fstype="gfs2"
options="noatime,nodiratime" op monitor interval=10s on-fail=fence clone
interleave=true
(single machine)$> pcs constraint order start clvmd-clone then
gfs2share-clone
(single machine)$> pcs constraint colocation add gfs2share-clone with
clvmd-clone
>
> > Each machine has a dedicated network interface for the cluster, and
> > there is a vlan on the switch devoted to just these interfaces.
> > In the following, I disabled the interface on node id 2 (b014). Node 1
> > (b013) is fenced as well. Node 2 (b015) is still up.
> >
> > Logs from b013:
> > Mar 24 16:35:01 b013 CRON[19133]: (root) CMD (command -v debian-sa1 >
> > /dev/null && debian-sa1 1 1)
> > Mar 24 16:35:13 b013 corosync[2134]: notice [TOTEM ] A processor
> > failed, forming new configuration.
> > Mar 24 16:35:13 b013 corosync[2134]: [TOTEM ] A processor failed,
> > forming new configuration.
> > Mar 24 16:35:17 b013 corosync[2134]: notice [TOTEM ] A new membership
> > (192.168.100.13:576 <http://192.168.100.13:576>) was formed. Members
> left: 2
> > Mar 24 16:35:17 b013 corosync[2134]: notice [TOTEM ] Failed to receive
> > the leave message. failed: 2
> > Mar 24 16:35:17 b013 corosync[2134]: [TOTEM ] A new membership
> > (192.168.100.13:576 <http://192.168.100.13:576>) was formed. Members
> left: 2
> > Mar 24 16:35:17 b013 corosync[2134]: [TOTEM ] Failed to receive the
> > leave message. failed: 2
> > Mar 24 16:35:17 b013 attrd[2223]: notice: crm_update_peer_proc: Node
> > b014-cl[2] - state is now lost (was member)
> > Mar 24 16:35:17 b013 cib[2220]: notice: crm_update_peer_proc: Node
> > b014-cl[2] - state is now lost (was member)
> > Mar 24 16:35:17 b013 cib[2220]: notice: Removing b014-cl/2 from the
> > membership list
> > Mar 24 16:35:17 b013 cib[2220]: notice: Purged 1 peers with id=2
> > and/or uname=b014-cl from the membership cache
> > Mar 24 16:35:17 b013 pacemakerd[2187]: notice: crm_reap_unseen_nodes:
> > Node b014-cl[2] - state is now lost (was member)
> > Mar 24 16:35:17 b013 attrd[2223]: notice: Removing b014-cl/2 from the
> > membership list
> > Mar 24 16:35:17 b013 attrd[2223]: notice: Purged 1 peers with id=2
> > and/or uname=b014-cl from the membership cache
> > Mar 24 16:35:17 b013 stonith-ng[2221]: notice: crm_update_peer_proc:
> > Node b014-cl[2] - state is now lost (was member)
> > Mar 24 16:35:17 b013 stonith-ng[2221]: notice: Removing b014-cl/2 from
> > the membership list
> > Mar 24 16:35:17 b013 stonith-ng[2221]: notice: Purged 1 peers with
> > id=2 and/or uname=b014-cl from the membership cache
> > Mar 24 16:35:17 b013 dlm_controld[2727]: 3091 fence request 2 pid 19223
> > nodedown time 1490387717 fence_all dlm_stonith
> > Mar 24 16:35:17 b013 kernel: [ 3091.800118] dlm: closing connection to
> > node 2
> > Mar 24 16:35:17 b013 crmd[2227]: notice: crm_reap_unseen_nodes: Node
> > b014-cl[2] - state is now lost (was member)
> > Mar 24 16:35:17 b013 dlm_stonith: stonith_api_time: Found 0 entries for
> > 2/(null): 0 in progress, 0 completed
> > Mar 24 16:35:18 b013 stonith-ng[2221]: notice: Operation reboot of
> > b014-cl by b015-cl for stonith-api.19223 at b013-cl.7aeb2ffb: OK
> > Mar 24 16:35:18 b013 stonith-api[19223]: stonith_api_kick: Node 2/(null)
> > kicked: reboot
> > Mar 24 16:35:18 b013 kernel: [ 3092.421495] dlm: closing connection to
> > node 3
> > Mar 24 16:35:18 b013 kernel: [ 3092.422246] dlm: closing connection to
> > node 1
> > Mar 24 16:35:18 b013 dlm_controld[2727]: 3092 abandoned lockspace
> share_data
> > Mar 24 16:35:18 b013 dlm_controld[2727]: 3092 abandoned lockspace clvmd
> > Mar 24 16:35:18 b013 kernel: [ 3092.426545] dlm: dlm user daemon left 2
> > lockspaces
> > Mar 24 16:35:18 b013 systemd[1]: corosync.service: Main process exited,
> > code=exited, status=255/n/a
> > Mar 24 16:35:18 b013 cib[2220]: error: Connection to the CPG API
> > failed: Library error (2)
> > Mar 24 16:35:18 b013 systemd[1]: corosync.service: Unit entered failed
> > state.
> > Mar 24 16:35:18 b013 attrd[2223]: error: Connection to cib_rw failed
> > Mar 24 16:35:18 b013 systemd[1]: corosync.service: Failed with result
> > 'exit-code'.
> > Mar 24 16:35:18 b013 attrd[2223]: error: Connection to
> > cib_rw[0x560754147990] closed (I/O condition=17)
> > Mar 24 16:35:18 b013 systemd[1]: pacemaker.service: Main process exited,
> > code=exited, status=107/n/a
> > Mar 24 16:35:18 b013 pacemakerd[2187]: error: Connection to the CPG
> > API failed: Library error (2)
> > Mar 24 16:35:18 b013 systemd[1]: pacemaker.service: Unit entered failed
> > state.
> > Mar 24 16:35:18 b013 attrd[2223]: notice: Disconnecting client
> > 0x560754149000, pid=2227...
> > Mar 24 16:35:18 b013 systemd[1]: pacemaker.service: Failed with result
> > 'exit-code'.
> > Mar 24 16:35:18 b013 lrmd[2222]: warning: new_event_notification
> > (2222-2227-8): Bad file descriptor (9)
> > Mar 24 16:35:18 b013 stonith-ng[2221]: error: Connection to cib_rw
> failed
> > Mar 24 16:35:18 b013 stonith-ng[2221]: error: Connection to
> > cib_rw[0x5579c03ecdd0] closed (I/O condition=17)
> > Mar 24 16:35:18 b013 lrmd[2222]: error: Connection to stonith-ng
> failed
> > Mar 24 16:35:18 b013 lrmd[2222]: error: Connection to
> > stonith-ng[0x55888c8ef820] closed (I/O condition=17)
> > Mar 24 16:37:02 b013 kernel: [ 3196.469475] dlm: node 0: socket error
> > sending to node 2, port 21064, sk_err=113/113
> > Mar 24 16:37:02 b013 kernel: [ 3196.470675] dlm: node 0: socket error
> > sending to node 2, port 21064, sk_err=113/113
> > Mar 24 16:37:46 b013 kernel: [ 3240.833544] INFO: task gfs2_quotad:3054
> > blocked for more than 120 seconds.
> > Mar 24 16:37:46 b013 kernel: [ 3240.834565] Not tainted
> > 4.4.0-66-generic #87-Ubuntu
> > Mar 24 16:37:46 b013 kernel: [ 3240.835413] "echo 0 >
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > Mar 24 16:37:46 b013 kernel: [ 3240.836656] gfs2_quotad D
> > ffff880fd747fa38 0 3054 2 0x00000000
> > Mar 24 16:37:46 b013 kernel: [ 3240.836663] ffff880fd747fa38
> > 00000001d8144018 ffff880fd975f2c0 ffff880fd7a972c0
> > Mar 24 16:37:46 b013 kernel: [ 3240.836666] ffff880fd7480000
> > ffff887fd81447b8 ffff887fd81447d0 ffff881fd7af00b0
> > Mar 24 16:37:46 b013 kernel: [ 3240.836669] 0000000000000004
> > ffff880fd747fa50 ffffffff818384d5 ffff880fd7a972c0
> > Mar 24 16:37:46 b013 kernel: [ 3240.836672] Call Trace:
> > Mar 24 16:37:46 b013 kernel: [ 3240.836688] [<ffffffff818384d5>]
> > schedule+0x35/0x80
> > Mar 24 16:37:46 b013 kernel: [ 3240.836695] [<ffffffff8183b380>]
> > rwsem_down_read_failed+0xe0/0x140
> > Mar 24 16:37:46 b013 kernel: [ 3240.836701] [<ffffffff81406574>]
> > call_rwsem_down_read_failed+0x14/0x30
> > Mar 24 16:37:46 b013 kernel: [ 3240.836704] [<ffffffff8183a920>] ?
> > down_read+0x20/0x30
> > Mar 24 16:37:46 b013 kernel: [ 3240.836726] [<ffffffffc0583324>]
> > dlm_lock+0x84/0x1f0 [dlm]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836731] [<ffffffff810b57e3>] ?
> > check_preempt_wakeup+0x193/0x220
> > Mar 24 16:37:46 b013 kernel: [ 3240.836755] [<ffffffffc06a5da0>] ?
> > gdlm_recovery_result+0x130/0x130 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836764] [<ffffffffc06a5050>] ?
> > gdlm_cancel+0x30/0x30 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836769] [<ffffffff810ab579>] ?
> > ttwu_do_wakeup+0x19/0xe0
> > Mar 24 16:37:46 b013 kernel: [ 3240.836779] [<ffffffffc06a5499>]
> > gdlm_lock+0x1d9/0x300 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836788] [<ffffffffc06a5050>] ?
> > gdlm_cancel+0x30/0x30 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836798] [<ffffffffc06a5da0>] ?
> > gdlm_recovery_result+0x130/0x130 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836807] [<ffffffffc0686e5f>]
> > do_xmote+0x16f/0x290 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836816] [<ffffffffc068705c>]
> > run_queue+0xdc/0x2d0 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836824] [<ffffffffc06875ef>]
> > gfs2_glock_nq+0x20f/0x410 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836834] [<ffffffffc06a2006>]
> > gfs2_statfs_sync+0x76/0x1c0 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836841] [<ffffffff810ed018>] ?
> > del_timer_sync+0x48/0x50
> > Mar 24 16:37:46 b013 kernel: [ 3240.836851] [<ffffffffc06a1ffc>] ?
> > gfs2_statfs_sync+0x6c/0x1c0 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836861] [<ffffffffc0697fe3>]
> > quotad_check_timeo.part.18+0x23/0x80 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836871] [<ffffffffc069ad01>]
> > gfs2_quotad+0x241/0x2d0 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836876] [<ffffffff810c41e0>] ?
> > wake_atomic_t_function+0x60/0x60
> > Mar 24 16:37:46 b013 kernel: [ 3240.836886] [<ffffffffc069aac0>] ?
> > gfs2_wake_up_statfs+0x40/0x40 [gfs2]
> > Mar 24 16:37:46 b013 kernel: [ 3240.836890] [<ffffffff810a0ba8>]
> > kthread+0xd8/0xf0
> > Mar 24 16:37:46 b013 kernel: [ 3240.836893] [<ffffffff810a0ad0>] ?
> > kthread_create_on_node+0x1e0/0x1e0
> > Mar 24 16:37:46 b013 kernel: [ 3240.836897] [<ffffffff8183c98f>]
> > ret_from_fork+0x3f/0x70
> > Mar 24 16:37:46 b013 kernel: [ 3240.836900] [<ffffffff810a0ad0>] ?
> > kthread_create_on_node+0x1e0/0x1e0
> >
> > Logs from b015:
> > Mar 24 16:35:01 b015 CRON[19781]: (root) CMD (command -v debian-sa1 >
> > /dev/null && debian-sa1 1 1)
> > Mar 24 16:35:13 b015 corosync[2105]: notice [TOTEM ] A processor
> > failed, forming new configuration.
> > Mar 24 16:35:13 b015 corosync[2105]: [TOTEM ] A processor failed,
> > forming new configuration.
> > Mar 24 16:35:17 b015 corosync[2105]: notice [TOTEM ] A new membership
> > (192.168.100.13:576 <http://192.168.100.13:576>) was formed. Members
> left: 2
> > Mar 24 16:35:17 b015 corosync[2105]: notice [TOTEM ] Failed to receive
> > the leave message. failed: 2
> > Mar 24 16:35:17 b015 corosync[2105]: [TOTEM ] A new membership
> > (192.168.100.13:576 <http://192.168.100.13:576>) was formed. Members
> left: 2
> > Mar 24 16:35:17 b015 corosync[2105]: [TOTEM ] Failed to receive the
> > leave message. failed: 2
> > Mar 24 16:35:17 b015 attrd[2253]: notice: crm_update_peer_proc: Node
> > b014-cl[2] - state is now lost (was member)
> > Mar 24 16:35:17 b015 attrd[2253]: notice: Removing b014-cl/2 from the
> > membership list
> > Mar 24 16:35:17 b015 attrd[2253]: notice: Purged 1 peers with id=2
> > and/or uname=b014-cl from the membership cache
> > Mar 24 16:35:17 b015 stonith-ng[2251]: notice: crm_update_peer_proc:
> > Node b014-cl[2] - state is now lost (was member)
> > Mar 24 16:35:17 b015 stonith-ng[2251]: notice: Removing b014-cl/2 from
> > the membership list
> > Mar 24 16:35:17 b015 cib[2249]: notice: crm_update_peer_proc: Node
> > b014-cl[2] - state is now lost (was member)
> > Mar 24 16:35:17 b015 crmd[2255]: notice: State transition S_IDLE ->
> > S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> > origin=abort_transition_graph ]
> > Mar 24 16:35:17 b015 kernel: [ 3478.622093] dlm: closing connection to
> > node 2
> > Mar 24 16:35:17 b015 stonith-ng[2251]: notice: Purged 1 peers with
> > id=2 and/or uname=b014-cl from the membership cache
> > Mar 24 16:35:17 b015 cib[2249]: notice: Removing b014-cl/2 from the
> > membership list
> > Mar 24 16:35:17 b015 cib[2249]: notice: Purged 1 peers with id=2
> > and/or uname=b014-cl from the membership cache
> > Mar 24 16:35:17 b015 crmd[2255]: notice: crm_reap_unseen_nodes: Node
> > b014-cl[2] - state is now lost (was member)
> > Mar 24 16:35:17 b015 pacemakerd[2159]: notice: crm_reap_unseen_nodes:
> > Node b014-cl[2] - state is now lost (was member)
> > Mar 24 16:35:18 b015 systemd[1]:
> > dev-disk-by\x2did-scsi\x2d36782bcb0007085a70000081958aee1ff.device: Dev
> > dev-disk-by\x2did-scsi\x2d36782bcb0007085a70000081958aee1ff.device
> > appeared twice with different sysfs paths
> > /sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0/host7/
> port-7:0/end_device-7:0/target7:0:0/7:0:0:0/block/sdc
> > and /sys/devices/virtual/block/dm-0
> > Mar 24 16:35:18 b015 systemd[1]:
> > dev-disk-by\x2did-wwn\x2d0x6782bcb0007085a70000081958aee1ff.device: Dev
> > dev-disk-by\x2did-wwn\x2d0x6782bcb0007085a70000081958aee1ff.device
> > appeared twice with different sysfs paths
> > /sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0/host7/
> port-7:0/end_device-7:0/target7:0:0/7:0:0:0/block/sdc
> > and /sys/devices/virtual/block/dm-0
> > Mar 24 16:35:18 b015 systemd[1]:
> > dev-disk-by\x2did-scsi\x2d36782bcb0007085a70000081958aee1ff.device: Dev
> > dev-disk-by\x2did-scsi\x2d36782bcb0007085a70000081958aee1ff.device
> > appeared twice with different sysfs paths
> > /sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0/host7/
> port-7:0/end_device-7:0/target7:0:0/7:0:0:0/block/sdc
> > and /sys/devices/virtual/block/dm-0
> > Mar 24 16:35:18 b015 systemd[1]:
> > dev-disk-by\x2did-wwn\x2d0x6782bcb0007085a70000081958aee1ff.device: Dev
> > dev-disk-by\x2did-wwn\x2d0x6782bcb0007085a70000081958aee1ff.device
> > appeared twice with different sysfs paths
> > /sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0/host7/
> port-7:0/end_device-7:0/target7:0:0/7:0:0:0/block/sdc
> > and /sys/devices/virtual/block/dm-0
> > Mar 24 16:35:18 b015 stonith-ng[2251]: warning: fence_scsi[19818]
> > stderr: [ WARNING:root:Parse error: Ignoring unknown option
> 'port=b014-cl' ]
> > Mar 24 16:35:18 b015 stonith-ng[2251]: warning: fence_scsi[19818]
> > stderr: [ ]
> > Mar 24 16:35:18 b015 stonith-ng[2251]: notice: Operation 'reboot'
> > [19818] (call 2 from stonith-api.19223) for host 'b014-cl' with device
> > 'fence_wh' returned: 0 (OK)
> > Mar 24 16:35:18 b015 stonith-ng[2251]: notice: Operation reboot of
> > b014-cl by b015-cl for stonith-api.19223 at b013-cl.7aeb2ffb: OK
> > Mar 24 16:35:18 b015 dlm_controld[2656]: 3479 fence request 2 pid 19880
> > nodedown time 1490387717 fence_all dlm_stonith
> > Mar 24 16:35:18 b015 dlm_controld[2656]: 3479 tell corosync to remove
> > nodeid 1 from cluster
> > Mar 24 16:35:18 b015 systemd[1]:
> > dev-disk-by\x2did-scsi\x2d36782bcb0007085a70000081958aee1ff.device: Dev
> > dev-disk-by\x2did-scsi\x2d36782bcb0007085a70000081958aee1ff.device
> > appeared twice with different sysfs paths
> > /sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0/host7/
> port-7:0/end_device-7:0/target7:0:0/7:0:0:0/block/sdc
> > and /sys/devices/virtual/block/dm-0
> > Mar 24 16:35:18 b015 systemd[1]:
> > dev-disk-by\x2did-wwn\x2d0x6782bcb0007085a70000081958aee1ff.device: Dev
> > dev-disk-by\x2did-wwn\x2d0x6782bcb0007085a70000081958aee1ff.device
> > appeared twice with different sysfs paths
> > /sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0/host7/
> port-7:0/end_device-7:0/target7:0:0/7:0:0:0/block/sdc
> > and /sys/devices/virtual/block/dm-0
> > Mar 24 16:35:18 b015 dlm_controld[2656]: 3479 tell corosync to remove
> > nodeid 1 from cluster
> > Mar 24 16:35:18 b015 dlm_stonith: stonith_api_time: Found 2 entries for
> > 2/(null): 0 in progress, 2 completed
> > Mar 24 16:35:18 b015 dlm_stonith: stonith_api_time: Node 2/(null) last
> > kicked at: 1490387718
> > Mar 24 16:35:18 b015 kernel: [ 3479.266118] dlm: closing connection to
> > node 1
> > Mar 24 16:35:18 b015 kernel: [ 3479.266270] dlm: closing connection to
> > node 3
> > Mar 24 16:35:18 b015 dlm_controld[2656]: 3479 abandoned lockspace
> share_data
> > Mar 24 16:35:18 b015 dlm_controld[2656]: 3479 abandoned lockspace clvmd
> > Mar 24 16:35:18 b015 kernel: [ 3479.268325] dlm: dlm user daemon left 2
> > lockspaces
> > Mar 24 16:35:21 b015 corosync[2105]: notice [TOTEM ] A processor
> > failed, forming new configuration.
> > Mar 24 16:35:21 b015 corosync[2105]: [TOTEM ] A processor failed,
> > forming new configuration.
> > Mar 24 16:35:26 b015 corosync[2105]: notice [TOTEM ] A new membership
> > (192.168.100.15:580 <http://192.168.100.15:580>) was formed. Members
> left: 1
> > Mar 24 16:35:26 b015 corosync[2105]: notice [TOTEM ] Failed to receive
> > the leave message. failed: 1
> > Mar 24 16:35:26 b015 corosync[2105]: [TOTEM ] A new membership
> > (192.168.100.15:580 <http://192.168.100.15:580>) was formed. Members
> left: 1
> > Mar 24 16:35:26 b015 corosync[2105]: [TOTEM ] Failed to receive the
> > leave message. failed: 1
> > Mar 24 16:35:26 b015 attrd[2253]: notice: crm_update_peer_proc: Node
> > b013-cl[1] - state is now lost (was member)
> > Mar 24 16:35:26 b015 attrd[2253]: notice: Removing b013-cl/1 from the
> > membership list
> > Mar 24 16:35:26 b015 stonith-ng[2251]: notice: crm_update_peer_proc:
> > Node b013-cl[1] - state is now lost (was member)
> > Mar 24 16:35:26 b015 attrd[2253]: notice: Purged 1 peers with id=1
> > and/or uname=b013-cl from the membership cache
> > Mar 24 16:35:26 b015 stonith-ng[2251]: notice: Removing b013-cl/1 from
> > the membership list
> > Mar 24 16:35:26 b015 pacemakerd[2159]: notice: Membership 580: quorum
> > lost (1)
> > Mar 24 16:35:26 b015 cib[2249]: notice: crm_update_peer_proc: Node
> > b013-cl[1] - state is now lost (was member)
> > Mar 24 16:35:26 b015 stonith-ng[2251]: notice: Purged 1 peers with
> > id=1 and/or uname=b013-cl from the membership cache
> > Mar 24 16:35:26 b015 pacemakerd[2159]: notice: crm_reap_unseen_nodes:
> > Node b013-cl[1] - state is now lost (was member)
> > Mar 24 16:35:26 b015 cib[2249]: notice: Removing b013-cl/1 from the
> > membership list
> > Mar 24 16:35:26 b015 cib[2249]: notice: Purged 1 peers with id=1
> > and/or uname=b013-cl from the membership cache
> > Mar 24 16:35:26 b015 crmd[2255]: notice: Membership 580: quorum lost
> (1)
> > Mar 24 16:35:26 b015 crmd[2255]: notice: crm_reap_unseen_nodes: Node
> > b013-cl[1] - state is now lost (was member)
> > Mar 24 16:35:26 b015 pengine[2254]: notice: We do not have quorum -
> > fencing and resource management disabled
> > Mar 24 16:35:26 b015 pengine[2254]: warning: Node b013-cl is unclean
> > because the node is no longer part of the cluster
> > Mar 24 16:35:26 b015 pengine[2254]: warning: Node b013-cl is unclean
> > Mar 24 16:35:26 b015 pengine[2254]: warning: Action dlm:1_stop_0 on
> > b013-cl is unrunnable (offline)
> > Mar 24 16:35:26 b015 pengine[2254]: warning: Action dlm:1_stop_0 on
> > b013-cl is unrunnable (offline)
> > Mar 24 16:35:26 b015 pengine[2254]: warning: Action clvmd:1_stop_0 on
> > b013-cl is unrunnable (offline)
> > Mar 24 16:35:26 b015 pengine[2254]: warning: Action clvmd:1_stop_0 on
> > b013-cl is unrunnable (offline)
> > Mar 24 16:35:26 b015 pengine[2254]: warning: Action gfs2share:1_stop_0
> > on b013-cl is unrunnable (offline)
> > Mar 24 16:35:26 b015 pengine[2254]: warning: Action gfs2share:1_stop_0
> > on b013-cl is unrunnable (offline)
> > Mar 24 16:35:26 b015 pengine[2254]: warning: Node b013-cl is unclean!
> > Mar 24 16:35:26 b015 pengine[2254]: notice: Cannot fence unclean nodes
> > until quorum is attained (or no-quorum-policy is set to ignore)
> > Mar 24 16:35:26 b015 pengine[2254]: notice: Start
> > fence_wh#011(b015-cl - blocked)
> > Mar 24 16:35:26 b015 pengine[2254]: notice: Stop dlm:1#011(b013-cl
> > - blocked)
> > Mar 24 16:35:26 b015 pengine[2254]: notice: Stop
> > clvmd:1#011(b013-cl - blocked)
> > Mar 24 16:35:26 b015 pengine[2254]: notice: Stop
> > gfs2share:1#011(b013-cl - blocked)
> > Mar 24 16:35:26 b015 pengine[2254]: warning: Calculated Transition 9:
> > /var/lib/pacemaker/pengine/pe-warn-2669.bz2
> > Mar 24 16:35:26 b015 crmd[2255]: notice: Transition 9 (Complete=6,
> > Pending=0, Fired=0, Skipped=0, Incomplete=0,
> > Source=/var/lib/pacemaker/pengine/pe-warn-2669.bz2): Complete
> > Mar 24 16:35:26 b015 crmd[2255]: notice: State transition
> > S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
> > origin=notify_crmd ]
> > Mar 24 16:35:31 b015 controld(dlm)[20000]: ERROR: Uncontrolled lockspace
> > exists, system must reboot. Executing suicide fencing
> > Mar 24 16:35:31 b015 fence_scsi: Failed: keys cannot be same. You can
> > not fence yourself.
> > Mar 24 16:35:31 b015 fence_scsi: Please use '-h' for usage
> > Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]
> > stderr: [ WARNING:root:Parse error: Ignoring unknown option
> 'port=b015-cl' ]
> > Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]
> > stderr: [ ]
> > Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]
> > stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence
> > yourself. ]
> > Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]
> > stderr: [ ]
> > Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]
> > stderr: [ Failed: keys cannot be same. You can not fence yourself. ]
> > Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]
> > stderr: [ ]
> > Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]
> > stderr: [ ERROR:root:Please use '-h' for usage ]
> > Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]
> > stderr: [ ]
> > Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]
> > stderr: [ Please use '-h' for usage ]
> > Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]
> > stderr: [ ]
> >
> >
> >
> > Software versions:
> > corosync 2.3.5-3ubuntu1
> > pacemaker-common 1.1.14-2ubuntu1.1
> > pcs 0.9.149-1ubuntu1
> > libqb0:amd64 1.0-1ubuntu1
> > gfs2-utils 3.1.6-0ubuntu3
>
> --
> Digimer
> Papers and Projects: https://alteeve.com/w/
> "I am, somehow, less interested in the weight and convolutions of
> Einstein’s brain than in the near certainty that people of equal talent
> have lived and died in cotton fields and sweatshops." - Stephen Jay Gould
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20170324/8b2b1f30/attachment-0003.html>
More information about the Users
mailing list