<div dir="ltr">I will try to install updated packages from ubuntu 16.10 or newer. It can't get worse than not working.<div><br></div><div>Can you think of any logs that might help? I've enabled debug on corosync log, but it really doesn't show anything else other than corosync exiting. Any diagnostic tools you can recommend?</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-------</div>Seth Reid<div>System Operations Engineer</div><div>Vendini, Inc.<br></div><div>415.349.7736</div><div><a href="mailto:sreid@vendini.com" target="_blank">sreid@vendini.com</a></div><div><a href="http://www.vendini.com" target="_blank">www.vendini.com</a></div><div><br></div></div></div></div>
<br><div class="gmail_quote">On Mon, Mar 27, 2017 at 3:10 PM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 03/27/2017 03:54 PM, Seth Reid wrote:<br>
><br>
><br>
><br>
> On Fri, Mar 24, 2017 at 2:10 PM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a><br>
</span><div><div class="h5">> <mailto:<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>>> wrote:<br>
><br>
> On 03/24/2017 03:52 PM, Digimer wrote:<br>
> > On 24/03/17 04:44 PM, Seth Reid wrote:<br>
> >> I have a three node Pacemaker/GFS2 cluster on Ubuntu 16.04. Its not in<br>
> >> production yet because I'm having a problem during fencing. When I<br>
> >> disable the network interface of any one machine, the disabled machines<br>
> >> is properly fenced leaving me, briefly, with a two node cluster. A<br>
> >> second node is then fenced off immediately, and the remaining node<br>
> >> appears to try to fence itself off. This leave two nodes with<br>
> >> corosync/pacemaker stopped, and the remaining machine still in the<br>
> >> cluster but showing an offline node and an UNCLEAN node. What can be<br>
> >> causing this behavior?<br>
> ><br>
> > It looks like the fence attempt failed, leaving the cluster hung. When<br>
> > you say all nodes were fenced, did all nodes actually reboot? Or did the<br>
> > two surviving nodes just lock up? If the later, then that is the proper<br>
> > response to a failed fence (DLM stays blocked).<br>
><br>
> See comments inline ...<br>
><br>
> ><br>
> >> Each machine has a dedicated network interface for the cluster, and<br>
> >> there is a vlan on the switch devoted to just these interfaces.<br>
> >> In the following, I disabled the interface on node id 2 (b014).<br>
> Node 1<br>
> >> (b013) is fenced as well. Node 2 (b015) is still up.<br>
> >><br>
> >> Logs from b013:<br>
> >> Mar 24 16:35:01 b013 CRON[19133]: (root) CMD (command -v debian-sa1 ><br>
> >> /dev/null && debian-sa1 1 1)<br>
> >> Mar 24 16:35:13 b013 corosync[2134]: notice [TOTEM ] A processor<br>
> >> failed, forming new configuration.<br>
> >> Mar 24 16:35:13 b013 corosync[2134]: [TOTEM ] A processor failed,<br>
> >> forming new configuration.<br>
> >> Mar 24 16:35:17 b013 corosync[2134]: notice [TOTEM ] A new<br>
> membership<br>
> >> (<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">192.168.100.13:576</a> <<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">http://192.168.100.13:576</a>><br>
> <<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">http://192.168.100.13:576</a>>) was formed. Members left: 2<br>
> >> Mar 24 16:35:17 b013 corosync[2134]: notice [TOTEM ] Failed to<br>
> receive<br>
> >> the leave message. failed: 2<br>
> >> Mar 24 16:35:17 b013 corosync[2134]: [TOTEM ] A new membership<br>
> >> (<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">192.168.100.13:576</a> <<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">http://192.168.100.13:576</a>><br>
> <<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">http://192.168.100.13:576</a>>) was formed. Members left: 2<br>
> >> Mar 24 16:35:17 b013 corosync[2134]: [TOTEM ] Failed to receive the<br>
> >> leave message. failed: 2<br>
> >> Mar 24 16:35:17 b013 attrd[2223]: notice: crm_update_peer_proc:<br>
> Node<br>
> >> b014-cl[2] - state is now lost (was member)<br>
> >> Mar 24 16:35:17 b013 cib[2220]: notice: crm_update_peer_proc: Node<br>
> >> b014-cl[2] - state is now lost (was member)<br>
> >> Mar 24 16:35:17 b013 cib[2220]: notice: Removing b014-cl/2 from the<br>
> >> membership list<br>
> >> Mar 24 16:35:17 b013 cib[2220]: notice: Purged 1 peers with id=2<br>
> >> and/or uname=b014-cl from the membership cache<br>
> >> Mar 24 16:35:17 b013 pacemakerd[2187]: notice:<br>
> crm_reap_unseen_nodes:<br>
> >> Node b014-cl[2] - state is now lost (was member)<br>
> >> Mar 24 16:35:17 b013 attrd[2223]: notice: Removing b014-cl/2<br>
> from the<br>
> >> membership list<br>
> >> Mar 24 16:35:17 b013 attrd[2223]: notice: Purged 1 peers with id=2<br>
> >> and/or uname=b014-cl from the membership cache<br>
> >> Mar 24 16:35:17 b013 stonith-ng[2221]: notice:<br>
> crm_update_peer_proc:<br>
> >> Node b014-cl[2] - state is now lost (was member)<br>
> >> Mar 24 16:35:17 b013 stonith-ng[2221]: notice: Removing<br>
> b014-cl/2 from<br>
> >> the membership list<br>
> >> Mar 24 16:35:17 b013 stonith-ng[2221]: notice: Purged 1 peers with<br>
> >> id=2 and/or uname=b014-cl from the membership cache<br>
> >> Mar 24 16:35:17 b013 dlm_controld[2727]: 3091 fence request 2 pid<br>
> 19223<br>
> >> nodedown time 1490387717 fence_all dlm_stonith<br>
> >> Mar 24 16:35:17 b013 kernel: [ 3091.800118] dlm: closing<br>
> connection to<br>
> >> node 2<br>
> >> Mar 24 16:35:17 b013 crmd[2227]: notice: crm_reap_unseen_nodes:<br>
> Node<br>
> >> b014-cl[2] - state is now lost (was member)<br>
> >> Mar 24 16:35:17 b013 dlm_stonith: stonith_api_time: Found 0<br>
> entries for<br>
> >> 2/(null): 0 in progress, 0 completed<br>
> >> Mar 24 16:35:18 b013 stonith-ng[2221]: notice: Operation reboot of<br>
> >> b014-cl by b015-cl for stonith-api.19223@b013-cl.<wbr>7aeb2ffb: OK<br>
> >> Mar 24 16:35:18 b013 stonith-api[19223]: stonith_api_kick: Node<br>
> 2/(null)<br>
> >> kicked: reboot<br>
><br>
> It looks like the fencing of b014-cl is reported as successful above ...<br>
><br>
> >> Mar 24 16:35:18 b013 kernel: [ 3092.421495] dlm: closing connection to<br>
> >> node 3<br>
> >> Mar 24 16:35:18 b013 kernel: [ 3092.422246] dlm: closing connection to<br>
> >> node 1<br>
> >> Mar 24 16:35:18 b013 dlm_controld[2727]: 3092 abandoned lockspace share_data<br>
> >> Mar 24 16:35:18 b013 dlm_controld[2727]: 3092 abandoned lockspace clvmd<br>
> >> Mar 24 16:35:18 b013 kernel: [ 3092.426545] dlm: dlm user daemon left 2<br>
> >> lockspaces<br>
> >> Mar 24 16:35:18 b013 systemd[1]: corosync.service: Main process exited,<br>
> >> code=exited, status=255/n/a<br>
><br>
> ... but then DLM and corosync exit on this node. Pacemaker can only<br>
> exit, and the node gets fenced.<br>
><br>
> What does your fencing configuration look like?<br>
><br>
><br>
> This is the command I used. b013-cl, for example is a hosts file entry<br>
> so that the cluster only uses the cluster-only network interface.<br>
><br>
> pcs stonith create fence_wh fence_scsi<br>
> debug="/var/log/cluster/fence-<wbr>debug.log" vgs_path="/sbin/vgs"<br>
> sg_persist_path="/usr/bin/sg_<wbr>persist" sg_turs_path="/usr/bin/sg_<wbr>turs"<br>
> pcmk_reboot_action="off" pcmk_host_list="b013-cl b014-cl b015-cl"<br>
> pcmk_monitor_action="metadata" meta provides="unfencing" --force<br>
><br>
> I got the pcmk_monitor_action, pcmk_hosts_list, pcmk_reboot_action, and<br>
> --force from various redhat articles. I've tried getting fencing to<br>
> start without these, and it doesn't work.<br>
<br>
</div></div>It looks good to me. Not sure what's going wrong.<br>
<br>
The big question is why are DLM and corosync exiting after another node<br>
is fenced. Pacemaker is reacting properly once that happens.<br>
<div><div class="h5"><br>
> >> Mar 24 16:35:18 b013 cib[2220]: error: Connection to the CPG API<br>
> >> failed: Library error (2)<br>
> >> Mar 24 16:35:18 b013 systemd[1]: corosync.service: Unit entered<br>
> failed<br>
> >> state.<br>
> >> Mar 24 16:35:18 b013 attrd[2223]: error: Connection to cib_rw<br>
> failed<br>
> >> Mar 24 16:35:18 b013 systemd[1]: corosync.service: Failed with result<br>
> >> 'exit-code'.<br>
> >> Mar 24 16:35:18 b013 attrd[2223]: error: Connection to<br>
> >> cib_rw[0x560754147990] closed (I/O condition=17)<br>
> >> Mar 24 16:35:18 b013 systemd[1]: pacemaker.service: Main process<br>
> exited,<br>
> >> code=exited, status=107/n/a<br>
> >> Mar 24 16:35:18 b013 pacemakerd[2187]: error: Connection to<br>
> the CPG<br>
> >> API failed: Library error (2)<br>
> >> Mar 24 16:35:18 b013 systemd[1]: pacemaker.service: Unit entered<br>
> failed<br>
> >> state.<br>
> >> Mar 24 16:35:18 b013 attrd[2223]: notice: Disconnecting client<br>
> >> 0x560754149000, pid=2227...<br>
> >> Mar 24 16:35:18 b013 systemd[1]: pacemaker.service: Failed with<br>
> result<br>
> >> 'exit-code'.<br>
> >> Mar 24 16:35:18 b013 lrmd[2222]: warning: new_event_notification<br>
> >> (2222-2227-8): Bad file descriptor (9)<br>
> >> Mar 24 16:35:18 b013 stonith-ng[2221]: error: Connection to<br>
> cib_rw failed<br>
> >> Mar 24 16:35:18 b013 stonith-ng[2221]: error: Connection to<br>
> >> cib_rw[0x5579c03ecdd0] closed (I/O condition=17)<br>
> >> Mar 24 16:35:18 b013 lrmd[2222]: error: Connection to<br>
> stonith-ng failed<br>
> >> Mar 24 16:35:18 b013 lrmd[2222]: error: Connection to<br>
> >> stonith-ng[0x55888c8ef820] closed (I/O condition=17)<br>
> >> Mar 24 16:37:02 b013 kernel: [ 3196.469475] dlm: node 0: socket error<br>
> >> sending to node 2, port 21064, sk_err=113/113<br>
> >> Mar 24 16:37:02 b013 kernel: [ 3196.470675] dlm: node 0: socket error<br>
> >> sending to node 2, port 21064, sk_err=113/113<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.833544] INFO: task<br>
> gfs2_quotad:3054<br>
> >> blocked for more than 120 seconds.<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.834565] Not tainted<br>
> >> 4.4.0-66-generic #87-Ubuntu<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.835413] "echo 0 ><br>
> >> /proc/sys/kernel/hung_task_<wbr>timeout_secs" disables this message.<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836656] gfs2_quotad D<br>
> >> ffff880fd747fa38 0 3054 2 0x00000000<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836663] ffff880fd747fa38<br>
> >> 00000001d8144018 ffff880fd975f2c0 ffff880fd7a972c0<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836666] ffff880fd7480000<br>
> >> ffff887fd81447b8 ffff887fd81447d0 ffff881fd7af00b0<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836669] 0000000000000004<br>
> >> ffff880fd747fa50 ffffffff818384d5 ffff880fd7a972c0<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836672] Call Trace:<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836688] [<ffffffff818384d5>]<br>
> >> schedule+0x35/0x80<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836695] [<ffffffff8183b380>]<br>
> >> rwsem_down_read_failed+0xe0/<wbr>0x140<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836701] [<ffffffff81406574>]<br>
> >> call_rwsem_down_read_failed+<wbr>0x14/0x30<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836704] [<ffffffff8183a920>] ?<br>
> >> down_read+0x20/0x30<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836726] [<ffffffffc0583324>]<br>
> >> dlm_lock+0x84/0x1f0 [dlm]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836731] [<ffffffff810b57e3>] ?<br>
> >> check_preempt_wakeup+0x193/<wbr>0x220<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836755] [<ffffffffc06a5da0>] ?<br>
> >> gdlm_recovery_result+0x130/<wbr>0x130 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836764] [<ffffffffc06a5050>] ?<br>
> >> gdlm_cancel+0x30/0x30 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836769] [<ffffffff810ab579>] ?<br>
> >> ttwu_do_wakeup+0x19/0xe0<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836779] [<ffffffffc06a5499>]<br>
> >> gdlm_lock+0x1d9/0x300 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836788] [<ffffffffc06a5050>] ?<br>
> >> gdlm_cancel+0x30/0x30 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836798] [<ffffffffc06a5da0>] ?<br>
> >> gdlm_recovery_result+0x130/<wbr>0x130 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836807] [<ffffffffc0686e5f>]<br>
> >> do_xmote+0x16f/0x290 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836816] [<ffffffffc068705c>]<br>
> >> run_queue+0xdc/0x2d0 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836824] [<ffffffffc06875ef>]<br>
> >> gfs2_glock_nq+0x20f/0x410 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836834] [<ffffffffc06a2006>]<br>
> >> gfs2_statfs_sync+0x76/0x1c0 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836841] [<ffffffff810ed018>] ?<br>
> >> del_timer_sync+0x48/0x50<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836851] [<ffffffffc06a1ffc>] ?<br>
> >> gfs2_statfs_sync+0x6c/0x1c0 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836861] [<ffffffffc0697fe3>]<br>
> >> quotad_check_timeo.part.18+<wbr>0x23/0x80 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836871] [<ffffffffc069ad01>]<br>
> >> gfs2_quotad+0x241/0x2d0 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836876] [<ffffffff810c41e0>] ?<br>
> >> wake_atomic_t_function+0x60/<wbr>0x60<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836886] [<ffffffffc069aac0>] ?<br>
> >> gfs2_wake_up_statfs+0x40/0x40 [gfs2]<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836890] [<ffffffff810a0ba8>]<br>
> >> kthread+0xd8/0xf0<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836893] [<ffffffff810a0ad0>] ?<br>
> >> kthread_create_on_node+0x1e0/<wbr>0x1e0<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836897] [<ffffffff8183c98f>]<br>
> >> ret_from_fork+0x3f/0x70<br>
> >> Mar 24 16:37:46 b013 kernel: [ 3240.836900] [<ffffffff810a0ad0>] ?<br>
> >> kthread_create_on_node+0x1e0/<wbr>0x1e0<br>
> >><br>
> >> Logs from b015:<br>
> >> Mar 24 16:35:01 b015 CRON[19781]: (root) CMD (command -v debian-sa1 ><br>
> >> /dev/null && debian-sa1 1 1)<br>
> >> Mar 24 16:35:13 b015 corosync[2105]: notice [TOTEM ] A processor<br>
> >> failed, forming new configuration.<br>
> >> Mar 24 16:35:13 b015 corosync[2105]: [TOTEM ] A processor failed,<br>
> >> forming new configuration.<br>
> >> Mar 24 16:35:17 b015 corosync[2105]: notice [TOTEM ] A new<br>
> membership<br>
> >> (<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">192.168.100.13:576</a> <<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">http://192.168.100.13:576</a>><br>
> <<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">http://192.168.100.13:576</a>>) was formed. Members left: 2<br>
> >> Mar 24 16:35:17 b015 corosync[2105]: notice [TOTEM ] Failed to<br>
> receive<br>
> >> the leave message. failed: 2<br>
> >> Mar 24 16:35:17 b015 corosync[2105]: [TOTEM ] A new membership<br>
> >> (<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">192.168.100.13:576</a> <<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">http://192.168.100.13:576</a>><br>
> <<a href="http://192.168.100.13:576" rel="noreferrer" target="_blank">http://192.168.100.13:576</a>>) was formed. Members left: 2<br>
> >> Mar 24 16:35:17 b015 corosync[2105]: [TOTEM ] Failed to receive the<br>
> >> leave message. failed: 2<br>
> >> Mar 24 16:35:17 b015 attrd[2253]: notice: crm_update_peer_proc:<br>
> Node<br>
> >> b014-cl[2] - state is now lost (was member)<br>
> >> Mar 24 16:35:17 b015 attrd[2253]: notice: Removing b014-cl/2<br>
> from the<br>
> >> membership list<br>
> >> Mar 24 16:35:17 b015 attrd[2253]: notice: Purged 1 peers with id=2<br>
> >> and/or uname=b014-cl from the membership cache<br>
> >> Mar 24 16:35:17 b015 stonith-ng[2251]: notice:<br>
> crm_update_peer_proc:<br>
> >> Node b014-cl[2] - state is now lost (was member)<br>
> >> Mar 24 16:35:17 b015 stonith-ng[2251]: notice: Removing<br>
> b014-cl/2 from<br>
> >> the membership list<br>
> >> Mar 24 16:35:17 b015 cib[2249]: notice: crm_update_peer_proc: Node<br>
> >> b014-cl[2] - state is now lost (was member)<br>
> >> Mar 24 16:35:17 b015 crmd[2255]: notice: State transition S_IDLE -><br>
> >> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL<br>
> >> origin=abort_transition_graph ]<br>
> >> Mar 24 16:35:17 b015 kernel: [ 3478.622093] dlm: closing<br>
> connection to<br>
> >> node 2<br>
> >> Mar 24 16:35:17 b015 stonith-ng[2251]: notice: Purged 1 peers with<br>
> >> id=2 and/or uname=b014-cl from the membership cache<br>
> >> Mar 24 16:35:17 b015 cib[2249]: notice: Removing b014-cl/2 from the<br>
> >> membership list<br>
> >> Mar 24 16:35:17 b015 cib[2249]: notice: Purged 1 peers with id=2<br>
> >> and/or uname=b014-cl from the membership cache<br>
> >> Mar 24 16:35:17 b015 crmd[2255]: notice: crm_reap_unseen_nodes:<br>
> Node<br>
> >> b014-cl[2] - state is now lost (was member)<br>
> >> Mar 24 16:35:17 b015 pacemakerd[2159]: notice:<br>
> crm_reap_unseen_nodes:<br>
> >> Node b014-cl[2] - state is now lost (was member)<br>
> >> Mar 24 16:35:18 b015 systemd[1]:<br>
> >><br>
> dev-disk-by\x2did-scsi\<wbr>x2d36782bcb0007085a70000081958<wbr>aee1ff.device: Dev<br>
> >> dev-disk-by\x2did-scsi\<wbr>x2d36782bcb0007085a70000081958<wbr>aee1ff.device<br>
> >> appeared twice with different sysfs paths<br>
> >><br>
> /sys/devices/pci0000:00/0000:<wbr>00:03.0/0000:08:00.0/host7/<wbr>port-7:0/end_device-7:0/<wbr>target7:0:0/7:0:0:0/block/sdc<br>
> >> and /sys/devices/virtual/block/dm-<wbr>0<br>
> >> Mar 24 16:35:18 b015 systemd[1]:<br>
> >><br>
> dev-disk-by\x2did-wwn\<wbr>x2d0x6782bcb0007085a7000008195<wbr>8aee1ff.device: Dev<br>
> >> dev-disk-by\x2did-wwn\<wbr>x2d0x6782bcb0007085a7000008195<wbr>8aee1ff.device<br>
> >> appeared twice with different sysfs paths<br>
> >><br>
> /sys/devices/pci0000:00/0000:<wbr>00:03.0/0000:08:00.0/host7/<wbr>port-7:0/end_device-7:0/<wbr>target7:0:0/7:0:0:0/block/sdc<br>
> >> and /sys/devices/virtual/block/dm-<wbr>0<br>
> >> Mar 24 16:35:18 b015 systemd[1]:<br>
> >><br>
> dev-disk-by\x2did-scsi\<wbr>x2d36782bcb0007085a70000081958<wbr>aee1ff.device: Dev<br>
> >> dev-disk-by\x2did-scsi\<wbr>x2d36782bcb0007085a70000081958<wbr>aee1ff.device<br>
> >> appeared twice with different sysfs paths<br>
> >><br>
> /sys/devices/pci0000:00/0000:<wbr>00:03.0/0000:08:00.0/host7/<wbr>port-7:0/end_device-7:0/<wbr>target7:0:0/7:0:0:0/block/sdc<br>
> >> and /sys/devices/virtual/block/dm-<wbr>0<br>
> >> Mar 24 16:35:18 b015 systemd[1]:<br>
> >><br>
> dev-disk-by\x2did-wwn\<wbr>x2d0x6782bcb0007085a7000008195<wbr>8aee1ff.device: Dev<br>
> >> dev-disk-by\x2did-wwn\<wbr>x2d0x6782bcb0007085a7000008195<wbr>8aee1ff.device<br>
> >> appeared twice with different sysfs paths<br>
> >><br>
> /sys/devices/pci0000:00/0000:<wbr>00:03.0/0000:08:00.0/host7/<wbr>port-7:0/end_device-7:0/<wbr>target7:0:0/7:0:0:0/block/sdc<br>
> >> and /sys/devices/virtual/block/dm-<wbr>0<br>
> >> Mar 24 16:35:18 b015 stonith-ng[2251]: warning: fence_scsi[19818]<br>
> >> stderr: [ WARNING:root:Parse error: Ignoring unknown option<br>
> 'port=b014-cl' ]<br>
> >> Mar 24 16:35:18 b015 stonith-ng[2251]: warning: fence_scsi[19818]<br>
> >> stderr: [ ]<br>
> >> Mar 24 16:35:18 b015 stonith-ng[2251]: notice: Operation 'reboot'<br>
> >> [19818] (call 2 from stonith-api.19223) for host 'b014-cl' with<br>
> device<br>
> >> 'fence_wh' returned: 0 (OK)<br>
> >> Mar 24 16:35:18 b015 stonith-ng[2251]: notice: Operation reboot of<br>
> >> b014-cl by b015-cl for stonith-api.19223@b013-cl.<wbr>7aeb2ffb: OK<br>
> >> Mar 24 16:35:18 b015 dlm_controld[2656]: 3479 fence request 2 pid<br>
> 19880<br>
> >> nodedown time 1490387717 fence_all dlm_stonith<br>
> >> Mar 24 16:35:18 b015 dlm_controld[2656]: 3479 tell corosync to remove<br>
> >> nodeid 1 from cluster<br>
> >> Mar 24 16:35:18 b015 systemd[1]:<br>
> >><br>
> dev-disk-by\x2did-scsi\<wbr>x2d36782bcb0007085a70000081958<wbr>aee1ff.device: Dev<br>
> >> dev-disk-by\x2did-scsi\<wbr>x2d36782bcb0007085a70000081958<wbr>aee1ff.device<br>
> >> appeared twice with different sysfs paths<br>
> >><br>
> /sys/devices/pci0000:00/0000:<wbr>00:03.0/0000:08:00.0/host7/<wbr>port-7:0/end_device-7:0/<wbr>target7:0:0/7:0:0:0/block/sdc<br>
> >> and /sys/devices/virtual/block/dm-<wbr>0<br>
> >> Mar 24 16:35:18 b015 systemd[1]:<br>
> >><br>
> dev-disk-by\x2did-wwn\<wbr>x2d0x6782bcb0007085a7000008195<wbr>8aee1ff.device: Dev<br>
> >> dev-disk-by\x2did-wwn\<wbr>x2d0x6782bcb0007085a7000008195<wbr>8aee1ff.device<br>
> >> appeared twice with different sysfs paths<br>
> >><br>
> /sys/devices/pci0000:00/0000:<wbr>00:03.0/0000:08:00.0/host7/<wbr>port-7:0/end_device-7:0/<wbr>target7:0:0/7:0:0:0/block/sdc<br>
> >> and /sys/devices/virtual/block/dm-<wbr>0<br>
> >> Mar 24 16:35:18 b015 dlm_controld[2656]: 3479 tell corosync to remove<br>
> >> nodeid 1 from cluster<br>
> >> Mar 24 16:35:18 b015 dlm_stonith: stonith_api_time: Found 2<br>
> entries for<br>
> >> 2/(null): 0 in progress, 2 completed<br>
> >> Mar 24 16:35:18 b015 dlm_stonith: stonith_api_time: Node 2/(null)<br>
> last<br>
> >> kicked at: 1490387718<br>
> >> Mar 24 16:35:18 b015 kernel: [ 3479.266118] dlm: closing<br>
> connection to<br>
> >> node 1<br>
> >> Mar 24 16:35:18 b015 kernel: [ 3479.266270] dlm: closing<br>
> connection to<br>
> >> node 3<br>
> >> Mar 24 16:35:18 b015 dlm_controld[2656]: 3479 abandoned lockspace<br>
> share_data<br>
> >> Mar 24 16:35:18 b015 dlm_controld[2656]: 3479 abandoned lockspace<br>
> clvmd<br>
> >> Mar 24 16:35:18 b015 kernel: [ 3479.268325] dlm: dlm user daemon<br>
> left 2<br>
> >> lockspaces<br>
> >> Mar 24 16:35:21 b015 corosync[2105]: notice [TOTEM ] A processor<br>
> >> failed, forming new configuration.<br>
> >> Mar 24 16:35:21 b015 corosync[2105]: [TOTEM ] A processor failed,<br>
> >> forming new configuration.<br>
> >> Mar 24 16:35:26 b015 corosync[2105]: notice [TOTEM ] A new<br>
> membership<br>
> >> (<a href="http://192.168.100.15:580" rel="noreferrer" target="_blank">192.168.100.15:580</a> <<a href="http://192.168.100.15:580" rel="noreferrer" target="_blank">http://192.168.100.15:580</a>><br>
> <<a href="http://192.168.100.15:580" rel="noreferrer" target="_blank">http://192.168.100.15:580</a>>) was formed. Members left: 1<br>
> >> Mar 24 16:35:26 b015 corosync[2105]: notice [TOTEM ] Failed to<br>
> receive<br>
> >> the leave message. failed: 1<br>
> >> Mar 24 16:35:26 b015 corosync[2105]: [TOTEM ] A new membership<br>
> >> (<a href="http://192.168.100.15:580" rel="noreferrer" target="_blank">192.168.100.15:580</a> <<a href="http://192.168.100.15:580" rel="noreferrer" target="_blank">http://192.168.100.15:580</a>><br>
> <<a href="http://192.168.100.15:580" rel="noreferrer" target="_blank">http://192.168.100.15:580</a>>) was formed. Members left: 1<br>
> >> Mar 24 16:35:26 b015 corosync[2105]: [TOTEM ] Failed to receive the<br>
> >> leave message. failed: 1<br>
> >> Mar 24 16:35:26 b015 attrd[2253]: notice: crm_update_peer_proc:<br>
> Node<br>
> >> b013-cl[1] - state is now lost (was member)<br>
> >> Mar 24 16:35:26 b015 attrd[2253]: notice: Removing b013-cl/1<br>
> from the<br>
> >> membership list<br>
> >> Mar 24 16:35:26 b015 stonith-ng[2251]: notice:<br>
> crm_update_peer_proc:<br>
> >> Node b013-cl[1] - state is now lost (was member)<br>
> >> Mar 24 16:35:26 b015 attrd[2253]: notice: Purged 1 peers with id=1<br>
> >> and/or uname=b013-cl from the membership cache<br>
> >> Mar 24 16:35:26 b015 stonith-ng[2251]: notice: Removing<br>
> b013-cl/1 from<br>
> >> the membership list<br>
> >> Mar 24 16:35:26 b015 pacemakerd[2159]: notice: Membership 580:<br>
> quorum<br>
> >> lost (1)<br>
> >> Mar 24 16:35:26 b015 cib[2249]: notice: crm_update_peer_proc: Node<br>
> >> b013-cl[1] - state is now lost (was member)<br>
> >> Mar 24 16:35:26 b015 stonith-ng[2251]: notice: Purged 1 peers with<br>
> >> id=1 and/or uname=b013-cl from the membership cache<br>
> >> Mar 24 16:35:26 b015 pacemakerd[2159]: notice:<br>
> crm_reap_unseen_nodes:<br>
> >> Node b013-cl[1] - state is now lost (was member)<br>
> >> Mar 24 16:35:26 b015 cib[2249]: notice: Removing b013-cl/1 from the<br>
> >> membership list<br>
> >> Mar 24 16:35:26 b015 cib[2249]: notice: Purged 1 peers with id=1<br>
> >> and/or uname=b013-cl from the membership cache<br>
> >> Mar 24 16:35:26 b015 crmd[2255]: notice: Membership 580: quorum<br>
> lost (1)<br>
> >> Mar 24 16:35:26 b015 crmd[2255]: notice: crm_reap_unseen_nodes:<br>
> Node<br>
> >> b013-cl[1] - state is now lost (was member)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: notice: We do not have quorum -<br>
> >> fencing and resource management disabled<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: warning: Node b013-cl is unclean<br>
> >> because the node is no longer part of the cluster<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: warning: Node b013-cl is unclean<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: warning: Action dlm:1_stop_0 on<br>
> >> b013-cl is unrunnable (offline)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: warning: Action dlm:1_stop_0 on<br>
> >> b013-cl is unrunnable (offline)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: warning: Action<br>
> clvmd:1_stop_0 on<br>
> >> b013-cl is unrunnable (offline)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: warning: Action<br>
> clvmd:1_stop_0 on<br>
> >> b013-cl is unrunnable (offline)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: warning: Action<br>
> gfs2share:1_stop_0<br>
> >> on b013-cl is unrunnable (offline)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: warning: Action<br>
> gfs2share:1_stop_0<br>
> >> on b013-cl is unrunnable (offline)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: warning: Node b013-cl is<br>
> unclean!<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: notice: Cannot fence<br>
> unclean nodes<br>
> >> until quorum is attained (or no-quorum-policy is set to ignore)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: notice: Start<br>
> >> fence_wh#011(b015-cl - blocked)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: notice: Stop<br>
> dlm:1#011(b013-cl<br>
> >> - blocked)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: notice: Stop<br>
> >> clvmd:1#011(b013-cl - blocked)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: notice: Stop<br>
> >> gfs2share:1#011(b013-cl - blocked)<br>
> >> Mar 24 16:35:26 b015 pengine[2254]: warning: Calculated<br>
> Transition 9:<br>
> >> /var/lib/pacemaker/pengine/pe-<wbr>warn-2669.bz2<br>
> >> Mar 24 16:35:26 b015 crmd[2255]: notice: Transition 9 (Complete=6,<br>
> >> Pending=0, Fired=0, Skipped=0, Incomplete=0,<br>
> >> Source=/var/lib/pacemaker/<wbr>pengine/pe-warn-2669.bz2): Complete<br>
> >> Mar 24 16:35:26 b015 crmd[2255]: notice: State transition<br>
> >> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS<br>
> cause=C_FSA_INTERNAL<br>
> >> origin=notify_crmd ]<br>
> >> Mar 24 16:35:31 b015 controld(dlm)[20000]: ERROR: Uncontrolled<br>
> lockspace<br>
> >> exists, system must reboot. Executing suicide fencing<br>
> >> Mar 24 16:35:31 b015 fence_scsi: Failed: keys cannot be same. You can<br>
> >> not fence yourself.<br>
> >> Mar 24 16:35:31 b015 fence_scsi: Please use '-h' for usage<br>
> >> Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]<br>
> >> stderr: [ WARNING:root:Parse error: Ignoring unknown option<br>
> 'port=b015-cl' ]<br>
> >> Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]<br>
> >> stderr: [ ]<br>
> >> Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]<br>
> >> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence<br>
> >> yourself. ]<br>
> >> Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]<br>
> >> stderr: [ ]<br>
> >> Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]<br>
> >> stderr: [ Failed: keys cannot be same. You can not fence yourself. ]<br>
> >> Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]<br>
> >> stderr: [ ]<br>
> >> Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]<br>
> >> stderr: [ ERROR:root:Please use '-h' for usage ]<br>
> >> Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]<br>
> >> stderr: [ ]<br>
> >> Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]<br>
> >> stderr: [ Please use '-h' for usage ]<br>
> >> Mar 24 16:35:31 b015 stonith-ng[2251]: warning: fence_scsi[20020]<br>
> >> stderr: [ ]<br>
> >><br>
> >><br>
> >><br>
> >> Software versions:<br>
> >> corosync 2.3.5-3ubuntu1<br>
> >> pacemaker-common 1.1.14-2ubuntu1.1<br>
> >> pcs 0.9.149-1ubuntu1<br>
> >> libqb0:amd64 1.0-1ubuntu1<br>
> >> gfs2-utils 3.1.6-0ubuntu3<br>
> >><br>
> >><br>
> >> -------<br>
> >> Seth Reid<br>
> >> System Operations Engineer<br>
> >> Vendini, Inc.<br>
> >> 415.349.7736<br>
> >> <a href="mailto:sreid@vendini.com">sreid@vendini.com</a> <mailto:<a href="mailto:sreid@vendini.com">sreid@vendini.com</a>><br>
</div></div>> <mailto:<a href="mailto:sreid@vendini.com">sreid@vendini.com</a> <mailto:<a href="mailto:sreid@vendini.com">sreid@vendini.com</a>>><br>
> >> <a href="http://www.vendini.com" rel="noreferrer" target="_blank">www.vendini.com</a> <<a href="http://www.vendini.com" rel="noreferrer" target="_blank">http://www.vendini.com</a>> <<a href="http://www.vendini.com" rel="noreferrer" target="_blank">http://www.vendini.com</a>><br>
</blockquote></div><br></div>