[ClusterLabs] Three VM's in cluster, running on multiple libvirt hosts, stonith not working

Andrew Beekhof andrew at beekhof.net
Tue Aug 4 04:09:30 UTC 2015


> On 4 Jun 2015, at 6:15 am, Steve Dainard <sdainard at spd1.com> wrote:
> 
> Actually, looking at my configs I didn't see where the stonith RA's are actually associated with a specific cluster node:
> 
> Stonith Devices:  
>  Resource: NFS1 (class=stonith type=fence_xvm)
>   Attributes: key_file=/etc/cluster/fence_xvm_ceph1.key multicast_address=225.0.0.12 port=NFS1
>   Operations: monitor interval=20s (NFS1-monitor-interval-20s)
>  Resource: NFS2 (class=stonith type=fence_xvm)
>   Attributes: key_file=/etc/cluster/fence_xvm_ceph2.key multicast_address=225.0.1.12 port=NFS2
>   Operations: monitor interval=20s (NFS2-monitor-interval-20s)
>  Resource: NFS3 (class=stonith type=fence_xvm)
>   Attributes: key_file=/etc/cluster/fence_xvm_ceph3.key multicast_address=225.0.2.12 port=NFS3
>   Operations: monitor interval=20s (NFS3-monitor-interval-20s)
> 
> My cluster contains nodes: node1, node2, node3. But pacemaker wouldn't know that guest NFS1 = node1, etc. Looking through the options for fence_xvm I also didn't see parameters containing cluster node name.
> 
> So I changed the VM names to node[1-3] and its working!

Yep, the other option here is to set pcmk_host_map=“node1:NFS1;node2:NFS2;…"

> Also, before I changed the VM name, I also changed the stonith RA names from NFS[1-3] to node[1-3], doubt that makes any difference as its just a name, but at the least its a logical name now for me.
> 
> Now the next issue..
> 
> After stopping the network service on node1, the other two nodes decided they should stonith node1 as expected.

Ah!!  I debugged one like this last week:

Jun 03 11:32:43 [2091] node1.local    pengine:     info: native_print: 	node1	(stonith:fence_xvm):	Started node1 
Jun 03 11:32:43 [2091] node1.local    pengine:     info: native_print: 	node2	(stonith:fence_xvm):	Stopped 
Jun 03 11:32:43 [2091] node1.local    pengine:     info: native_print: 	node3	(stonith:fence_xvm):	Stopped 

DontDoThat(tm)
We’ve fixed it upstream now, but in general its not a good idea to have resources with the same name as your nodes.
"Just" add a “fence” suffix or prefix to the id of your stonith resources.

This is/was the giveaway:

Jun 03 11:32:43 [2091] node1.local    pengine:   notice: check_rsc_parameters: 	Forcing restart of node1 on node1, type changed: remote -> fence_xvm
Jun 03 11:32:43 [2091] node1.local    pengine:   notice: check_rsc_parameters: 	Forcing restart of node1 on node1, class changed: ocf -> stonith


> Then as I'm watching (watch -n 2 pcs status) on node2 where the services have started up I notice it appears that the services are flapping (log file time 12:09:19 - 12:09:56 matches the video I took here https://youtu.be/mnCJ9FZqGjA)
> 
> Logs:
> https://dl.dropboxusercontent.com/u/21916057/pacemaker-node1.log
> https://dl.dropboxusercontent.com/u/21916057/pacemaker-node2.log
> https://dl.dropboxusercontent.com/u/21916057/pacemaker-node3.log
> 
> 
> Then at 12:12:15 (ish) node3 gets fenced (by itself it appears). I see this on node3's hypervisor:
> Jun  3 12:12:16 ceph3 fence_virtd: Rebooting domain node3
> 
> But afterwards the resources are all still running on node2, which I suppose is the 'safe' bet:
> # pcs status
> Cluster name: nfs
> Last updated: Wed Jun  3 13:10:17 2015
> Last change: Wed Jun  3 11:32:42 2015
> Stack: corosync
> Current DC: node2 (2) - partition WITHOUT quorum
> Version: 1.1.12-a14efad
> 3 Nodes configured
> 9 Resources configured
> 
> 
> Online: [ node2 ]
> OFFLINE: [ node1 node3 ]
> 
> Full list of resources:
> 
>  Resource Group: group_rbd_fs_nfs_vip
>      rbd_nfs-ha (ocf::ceph:rbd.in):     Started node2 
>      rbd_home   (ocf::ceph:rbd.in):     Started node2 
>      fs_nfs-ha  (ocf::heartbeat:Filesystem):    Started node2 
>      FS_home    (ocf::heartbeat:Filesystem):    Started node2 
>      nfsserver  (ocf::heartbeat:nfsserver):     Started node2 
>      vip_nfs_private    (ocf::heartbeat:IPaddr):        Started node2 
>  node1  (stonith:fence_xvm):    Stopped 
>  node2  (stonith:fence_xvm):    Started node2 
>  node3  (stonith:fence_xvm):    Stopped 
> 
> PCSD Status:
>   node1: Online
>   node2: Online
>   node3: Online
> 
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/enabled
> [root at node2 ~]# exportfs
> /mnt/home       10.0.231.0/255.255.255.0
> 
> I realize this is a ton of logs to go through, appreciate it if anyone has the time as I'm not sure how to troubleshoot why node3 was fenced when everything was working properly with 2/3 nodes.
> 
> On Wed, Jun 3, 2015 at 10:38 AM, Ken Gaillot <kgaillot at redhat.com> wrote:
> 
> ----- Original Message -----
> > I've tried configuring without pcmk_host_list as well with the same result.
> 
> What do the logs say now?
> 
> > Stonith Devices:
> >  Resource: NFS1 (class=stonith type=fence_xvm)
> >   Attributes: key_file=/etc/cluster/fence_xvm_ceph1.key
> > multicast_address=225.0.0.12 port=NFS1
> >   Operations: monitor interval=20s (NFS1-monitor-interval-20s)
> >  Resource: NFS2 (class=stonith type=fence_xvm)
> >   Attributes: key_file=/etc/cluster/fence_xvm_ceph2.key
> > multicast_address=225.0.1.12 port=NFS2
> >   Operations: monitor interval=20s (NFS2-monitor-interval-20s)
> >  Resource: NFS3 (class=stonith type=fence_xvm)
> >   Attributes: key_file=/etc/cluster/fence_xvm_ceph3.key
> > multicast_address=225.0.2.12 port=NFS3
> >   Operations: monitor interval=20s (NFS3-monitor-interval-20s)
> >
> > I can get the list of VM's from any of the 3 cluster nodes using the
> > multicast address:
> >
> > # fence_xvm -a 225.0.0.12 -k /etc/cluster/fence_xvm_ceph1.key -o list
> > NFS1                 1814d93d-3e40-797f-a3c6-102aaa6a3d01 on
> >
> > # fence_xvm -a 225.0.1.12 -k /etc/cluster/fence_xvm_ceph2.key -o list
> > NFS2                 75ab85fc-40e9-45ae-8b0a-c346d59b24e8 on
> >
> > # fence_xvm -a 225.0.2.12 -k /etc/cluster/fence_xvm_ceph3.key -o list
> > NFS3                 f23cca5d-d50b-46d2-85dd-d8357337fd22 on
> >
> > On Tue, Jun 2, 2015 at 10:07 AM, Ken Gaillot <kgaillot at redhat.com> wrote:
> >
> > > On 06/02/2015 11:40 AM, Steve Dainard wrote:
> > > > Hello,
> > > >
> > > > I have 3 CentOS7 guests running on 3 CentOS7 hypervisors and I can't get
> > > > stonith operations to work.
> > > >
> > > > Config:
> > > >
> > > > Cluster Name: nfs
> > > > Corosync Nodes:
> > > >  node1 node2 node3
> > > > Pacemaker Nodes:
> > > >  node1 node2 node3
> > > >
> > > > Resources:
> > > >  Group: group_rbd_fs_nfs_vip
> > > >   Resource: rbd_nfs-ha (class=ocf provider=ceph type=rbd.in)
> > > >    Attributes: user=admin pool=rbd name=nfs-ha
> > > cephconf=/etc/ceph/ceph.conf
> > > >    Operations: start interval=0s timeout=20 (rbd_nfs-ha-start-timeout-20)
> > > >                stop interval=0s timeout=20 (rbd_nfs-ha-stop-timeout-20)
> > > >                monitor interval=10s timeout=20s
> > > > (rbd_nfs-ha-monitor-interval-10s)
> > > >   Resource: rbd_home (class=ocf provider=ceph type=rbd.in)
> > > >    Attributes: user=admin pool=rbd name=home cephconf=/etc/ceph/ceph.conf
> > > >    Operations: start interval=0s timeout=20 (rbd_home-start-timeout-20)
> > > >                stop interval=0s timeout=20 (rbd_home-stop-timeout-20)
> > > >                monitor interval=10s timeout=20s
> > > > (rbd_home-monitor-interval-10s)
> > > >   Resource: fs_nfs-ha (class=ocf provider=heartbeat type=Filesystem)
> > > >    Attributes: directory=/mnt/nfs-ha fstype=btrfs
> > > > device=/dev/rbd/rbd/nfs-ha fast_stop=no
> > > >    Operations: monitor interval=20s timeout=40s
> > > > (fs_nfs-ha-monitor-interval-20s)
> > > >                start interval=0 timeout=60s (fs_nfs-ha-start-interval-0)
> > > >                stop interval=0 timeout=60s (fs_nfs-ha-stop-interval-0)
> > > >   Resource: FS_home (class=ocf provider=heartbeat type=Filesystem)
> > > >    Attributes: directory=/mnt/home fstype=btrfs device=/dev/rbd/rbd/home
> > > > options=rw,compress-force=lzo fast_stop=no
> > > >    Operations: monitor interval=20s timeout=40s
> > > > (FS_home-monitor-interval-20s)
> > > >                start interval=0 timeout=60s (FS_home-start-interval-0)
> > > >                stop interval=0 timeout=60s (FS_home-stop-interval-0)
> > > >   Resource: nfsserver (class=ocf provider=heartbeat type=nfsserver)
> > > >    Attributes: nfs_shared_infodir=/mnt/nfs-ha
> > > >    Operations: stop interval=0s timeout=20s (nfsserver-stop-timeout-20s)
> > > >                monitor interval=10s timeout=20s
> > > > (nfsserver-monitor-interval-10s)
> > > >                start interval=0 timeout=40s (nfsserver-start-interval-0)
> > > >   Resource: vip_nfs_private (class=ocf provider=heartbeat type=IPaddr)
> > > >    Attributes: ip=10.0.231.49 cidr_netmask=24
> > > >    Operations: start interval=0s timeout=20s
> > > > (vip_nfs_private-start-timeout-20s)
> > > >                stop interval=0s timeout=20s
> > > > (vip_nfs_private-stop-timeout-20s)
> > > >                monitor interval=5 (vip_nfs_private-monitor-interval-5)
> > > >
> > > > Stonith Devices:
> > > >  Resource: NFS1 (class=stonith type=fence_xvm)
> > > >   Attributes: pcmk_host_list=10.0.231.50
> > > > key_file=/etc/cluster/fence_xvm_ceph1.key multicast_address=225.0.0.12
> > > > port=NFS1
> > > >   Operations: monitor interval=20s (NFS1-monitor-interval-20s)
> > > >  Resource: NFS2 (class=stonith type=fence_xvm)
> > > >   Attributes: pcmk_host_list=10.0.231.51
> > > > key_file=/etc/cluster/fence_xvm_ceph2.key multicast_address=225.0.1.12
> > > > port=NFS2
> > > >   Operations: monitor interval=20s (NFS2-monitor-interval-20s)
> > > >  Resource: NFS3 (class=stonith type=fence_xvm)
> > > >   Attributes: pcmk_host_list=10.0.231.52
> > > > key_file=/etc/cluster/fence_xvm_ceph3.key multicast_address=225.0.2.12
> > > > port=NFS3
> > >
> > > I think pcmk_host_list should have the node name rather than the IP
> > > address. If fence_xvm -o list -a whatever shows the right nodes to
> > > fence, you don't even need to set pcmk_host_list.
> > >
> > > >   Operations: monitor interval=20s (NFS3-monitor-interval-20s)
> > > > Fencing Levels:
> > > >
> > > > Location Constraints:
> > > >   Resource: NFS1
> > > >     Enabled on: node1 (score:1) (id:location-NFS1-node1-1)
> > > >     Enabled on: node2 (score:1000) (id:location-NFS1-node2-1000)
> > > >     Enabled on: node3 (score:500) (id:location-NFS1-node3-500)
> > > >   Resource: NFS2
> > > >     Enabled on: node2 (score:1) (id:location-NFS2-node2-1)
> > > >     Enabled on: node3 (score:1000) (id:location-NFS2-node3-1000)
> > > >     Enabled on: node1 (score:500) (id:location-NFS2-node1-500)
> > > >   Resource: NFS3
> > > >     Enabled on: node3 (score:1) (id:location-NFS3-node3-1)
> > > >     Enabled on: node1 (score:1000) (id:location-NFS3-node1-1000)
> > > >     Enabled on: node2 (score:500) (id:location-NFS3-node2-500)
> > > > Ordering Constraints:
> > > > Colocation Constraints:
> > > >
> > > > Cluster Properties:
> > > >  cluster-infrastructure: corosync
> > > >  cluster-name: nfs
> > > >  dc-version: 1.1.12-a14efad
> > > >  have-watchdog: false
> > > >  stonith-enabled: true
> > > >
> > > > When I stop networking services on node1 (stonith resource NFS1) I see
> > > logs
> > > > on the other two cluster nodes attempting to reboot the vm NFS1 without
> > > > success.
> > > >
> > > > Logs:
> > > >
> > > > Jun 01 15:38:17 [2130] nfs3.pcic.uvic.ca    pengine:   notice:
> > > LogActions:
> > > >      Move    rbd_nfs-ha      (Started node1 -> node2)
> > > > Jun 01 15:38:17 [2130] nfs3.pcic.uvic.ca    pengine:   notice:
> > > LogActions:
> > > >      Move    rbd_home        (Started node1 -> node2)
> > > > Jun 01 15:38:17 [2130] nfs3.pcic.uvic.ca    pengine:   notice:
> > > LogActions:
> > > >      Move    fs_nfs-ha       (Started node1 -> node2)
> > > > Jun 01 15:38:17 [2130] nfs3.pcic.uvic.ca    pengine:   notice:
> > > LogActions:
> > > >      Move    FS_home (Started node1 -> node2)
> > > > Jun 01 15:38:17 [2130] nfs3.pcic.uvic.ca    pengine:   notice:
> > > LogActions:
> > > >      Move    nfsserver       (Started node1 -> node2)
> > > > Jun 01 15:38:17 [2130] nfs3.pcic.uvic.ca    pengine:   notice:
> > > LogActions:
> > > >      Move    vip_nfs_private (Started node1 -> node2)
> > > > Jun 01 15:38:17 [2130] nfs3.pcic.uvic.ca    pengine:     info:
> > > LogActions:
> > > >      Leave   NFS1    (Started node2)
> > > > Jun 01 15:38:17 [2130] nfs3.pcic.uvic.ca    pengine:     info:
> > > LogActions:
> > > >      Leave   NFS2    (Started node3)
> > > > Jun 01 15:38:17 [2130] nfs3.pcic.uvic.ca    pengine:   notice:
> > > LogActions:
> > > >      Move    NFS3    (Started node1 -> node2)
> > > > Jun 01 15:38:17 [2130] nfs3.pcic.uvic.ca    pengine:  warning:
> > > > process_pe_message:      Calculated Transition 8:
> > > > /var/lib/pacemaker/pengine/pe-warn-0.bz2
> > > > Jun 01 15:38:17 [2131] nfs3.pcic.uvic.ca       crmd:     info:
> > > > do_state_transition:     State transition S_POLICY_ENGINE ->
> > > > S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
> > > > origin=handle_response ]
> > > > Jun 01 15:38:17 [2131] nfs3.pcic.uvic.ca       crmd:     info:
> > > > do_te_invoke:    Processing graph 8 (ref=pe_calc-dc-1433198297-78)
> > > derived
> > > > from /var/lib/pacemaker/pengine/pe-warn-0.bz2
> > > > Jun 01 15:38:17 [2131] nfs3.pcic.uvic.ca       crmd:   notice:
> > > > te_fence_node:   Executing reboot fencing operation (37) on node1
> > > > (timeout=60000)
> > > > Jun 01 15:38:17 [2127] nfs3.pcic.uvic.ca stonith-ng:   notice:
> > > > handle_request:  Client crmd.2131.f7e79b61 wants to fence (reboot)
> > > 'node1'
> > > > with device '(any)'
> > > > Jun 01 15:38:17 [2127] nfs3.pcic.uvic.ca stonith-ng:   notice:
> > > > initiate_remote_stonith_op:      Initiating remote operation reboot for
> > > > node1: a22a16f3-b699-453e-a090-43a640dd0e3f (0)
> > > > Jun 01 15:38:17 [2127] nfs3.pcic.uvic.ca stonith-ng:   notice:
> > > > can_fence_host_with_device:      NFS1 can not fence (reboot) node1:
> > > > static-list
> > > > Jun 01 15:38:17 [2127] nfs3.pcic.uvic.ca stonith-ng:   notice:
> > > > can_fence_host_with_device:      NFS2 can not fence (reboot) node1:
> > > > static-list
> > > > Jun 01 15:38:17 [2127] nfs3.pcic.uvic.ca stonith-ng:   notice:
> > > > can_fence_host_with_device:      NFS3 can not fence (reboot) node1:
> > > > static-list
> > > > Jun 01 15:38:17 [2127] nfs3.pcic.uvic.ca stonith-ng:     info:
> > > > process_remote_stonith_query:    All queries have arrived, continuing (2,
> > > > 2, 2, a22a16f3-b699-453e-a090-43a640dd0e3f)
> > > > Jun 01 15:38:17 [2127] nfs3.pcic.uvic.ca stonith-ng:   notice:
> > > > stonith_choose_peer:     Couldn't find anyone to fence node1 with <any>
> > > > Jun 01 15:38:17 [2127] nfs3.pcic.uvic.ca stonith-ng:     info:
> > > > call_remote_stonith:     Total remote op timeout set to 60 for fencing of
> > > > node node1 for crmd.2131.a22a16f3
> > > > Jun 01 15:38:17 [2127] nfs3.pcic.uvic.ca stonith-ng:     info:
> > > > call_remote_stonith:     None of the 2 peers have devices capable of
> > > > terminating node1 for crmd.2131 (0)
> > > > Jun 01 15:38:17 [2127] nfs3.pcic.uvic.ca stonith-ng:    error:
> > > > remote_op_done:  Operation reboot of node1 by <no-one> for
> > > > crmd.2131 at node3.a22a16f3: No such device
> > > > Jun 01 15:38:17 [2131] nfs3.pcic.uvic.ca       crmd:   notice:
> > > > tengine_stonith_callback:        Stonith operation
> > > > 2/37:8:0:241ee032-f3a1-4c2b-8427-63af83b54343: No such device (-19)
> > > > Jun 01 15:38:17 [2131] nfs3.pcic.uvic.ca       crmd:   notice:
> > > > tengine_stonith_callback:        Stonith operation 2 for node1 failed (No
> > > > such device): aborting transition.
> > > > Jun 01 15:38:17 [2131] nfs3.pcic.uvic.ca       crmd:   notice:
> > > > abort_transition_graph:  Transition aborted: Stonith failed
> > > > (source=tengine_stonith_callback:697, 0)
> > > > Jun 01 15:38:17 [2131] nfs3.pcic.uvic.ca       crmd:   notice:
> > > > tengine_stonith_notify:  Peer node1 was not terminated (reboot) by
> > > <anyone>
> > > > for node3: No such device (ref=a22a16f3-b699-453e-a090-43a640dd0e3f) by
> > > > client crmd.2131
> > > > Jun 01 15:38:17 [2131] nfs3.pcic.uvic.ca       crmd:   notice:
> > > run_graph:
> > > >     Transition 8 (Complete=1, Pending=0, Fired=0, Skipped=27,
> > > Incomplete=0,
> > > > Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
> > > > Jun 01 15:38:17 [2131] nfs3.pcic.uvic.ca       crmd:   notice:
> > > > too_many_st_failures:    No devices found in cluster to fence node1,
> > > giving
> > > > up
> > > >
> > > > I can manually fence a guest without any issue:
> > > > # fence_xvm -a 225.0.0.12 -k /etc/cluster/fence_xvm_ceph1.key -o reboot
> > > -H
> > > > NFS1
> > > >
> > > > But the cluster doesn't recover resources to another host:
> > >
> > > The cluster doesn't know that the manual fencing succeeded, so it plays
> > > it safe by not moving resources. If you fix the cluster fencing issue,
> > > I'd expect this to work.
> > >
> > > > # pcs status *<-- after manual fencing*
> > > > Cluster name: nfs
> > > > Last updated: Tue Jun  2 08:34:18 2015
> > > > Last change: Mon Jun  1 16:02:58 2015
> > > > Stack: corosync
> > > > Current DC: node3 (3) - partition with quorum
> > > > Version: 1.1.12-a14efad
> > > > 3 Nodes configured
> > > > 9 Resources configured
> > > >
> > > >
> > > > Node node1 (1): UNCLEAN (offline)
> > > > Online: [ node2 node3 ]
> > > >
> > > > Full list of resources:
> > > >
> > > >  Resource Group: group_rbd_fs_nfs_vip
> > > >      rbd_nfs-ha (ocf::ceph:rbd.in):     Started node1
> > > >      rbd_home   (ocf::ceph:rbd.in):     Started node1
> > > >      fs_nfs-ha  (ocf::heartbeat:Filesystem):    Started node1
> > > >      FS_home    (ocf::heartbeat:Filesystem):    Started node1
> > > >      nfsserver  (ocf::heartbeat:nfsserver):     Started node1
> > > >      vip_nfs_private    (ocf::heartbeat:IPaddr):        Started node1
> > > >  NFS1   (stonith:fence_xvm):    Started node2
> > > >  NFS2   (stonith:fence_xvm):    Started node3
> > > >  NFS3   (stonith:fence_xvm):    Started node1
> > > >
> > > > PCSD Status:
> > > >   node1: Online
> > > >   node2: Online
> > > >   node3: Online
> > > >
> > > > Daemon Status:
> > > >   corosync: active/disabled
> > > >   pacemaker: active/disabled
> > > >   pcsd: active/enabled
> > > >
> > > > Fence_virtd config on one of the hypervisors:
> > > > # cat fence_virt.conf
> > > > backends {
> > > >         libvirt {
> > > >                 uri = "qemu:///system";
> > > >         }
> > > >
> > > > }
> > > >
> > > > listeners {
> > > >         multicast {
> > > >                 port = "1229";
> > > >                 family = "ipv4";
> > > >                 interface = "br1";
> > > >                 address = "225.0.0.12";
> > > >                 key_file = "/etc/cluster/fence_xvm_ceph1.key";
> > > >         }
> > > >
> > > > }
> > > >
> > > > fence_virtd {
> > > >         module_path = "/usr/lib64/fence-virt";
> > > >         backend = "libvirt";
> > > >         listener = "multicast";
> > > > }
> > >
> > >
> > > _______________________________________________
> > > Users mailing list: Users at clusterlabs.org
> > > http://clusterlabs.org/mailman/listinfo/users
> > >
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> > >
> >
> 
> --
> -- Ken Gaillot <kgaillot at redhat.com>
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org





More information about the Users mailing list