[ClusterLabs] [DRBD-user] DRBD fencing issue on failover causes resource failure
Thomas Lamprecht
t.lamprecht at proxmox.com
Wed Mar 16 15:27:58 EDT 2016
On 16.03.2016 18:51, Tim Walberg wrote:
> Is there a way to make this work properly without STONITH? I forgot to mention
> that both nodes are virtual machines (QEMU/KVM), which makes STONITH a minor
> challenge. Also, since these symptoms occur even under "pcs cluster standby",
> where STONITH *shouldn't* be invoked, I'm not sure if that's the entire answer.
>
There exists a lot fence agents which make use of the hypervisor which
hosts the VMs, e.g. fence_pve for Proxmox VE virtual machines, vmware,
virtual box, xen are also implemented, libvirt should be but I don't
know for sure.
See:
https://github.com/ClusterLabs/fence-agents
https://fedorahosted.org/cluster/wiki/fence-agents
This is a fairly easy way to setup fencing for me and I use it quite
often for tests.
I didn't set pacemaker up with such an agent but I see no problem which
could prevent that.
cheers,
Thomas
>
> On 03/16/2016 13:34 -0400, Digimer wrote:
>>> On 16/03/16 01:17 PM, Tim Walberg wrote:
>>> > Having an issue on a newly built CentOS 7.2.1511 NFS cluster with DRBD
>>> > (drbd84-utils-8.9.5-1 with kmod-drbd84-8.4.7-1_1). At this point, the
>>> > resources consist of a cluster address, a DRBD device mirroring between
>>> > the two cluster nodes, the file system, and the nfs-server resource. The
>>> > resources all behave properly until an extended failover or outage.
>>> >
>>> > I have tested failover in several ways ("pcs cluster standby", "pcs
>>> > cluster stop", "init 0", "init 6", "echo b > /proc/sysrq-trigger", etc.)
>>> > and the symptoms are that, until the killed node is brought back into
>>> > the cluster, failover never seems to complete. The DRBD device appears
>>> > on the remaining node to be in a "Secondary/Unknown" state, and the
>>> > resources end up looking like:
>>> >
>>> > # pcs status
>>> > Cluster name: nfscluster
>>> > Last updated: Wed Mar 16 12:05:33 2016 Last change: Wed Mar 16
>>> > 12:04:46 2016 by root via cibadmin on nfsnode01
>>> > Stack: corosync
>>> > Current DC: nfsnode01 (version 1.1.13-10.el7_2.2-44eb2dd) - partition
>>> > with quorum
>>> > 2 nodes and 5 resources configured
>>> >
>>> > Online: [ nfsnode01 ]
>>> > OFFLINE: [ nfsnode02 ]
>>> >
>>> > Full list of resources:
>>> >
>>> > nfsVIP (ocf::heartbeat:IPaddr2): Started nfsnode01
>>> > nfs-server (systemd:nfs-server): Stopped
>>> > Master/Slave Set: drbd_master [drbd_dev]
>>> > Slaves: [ nfsnode01 ]
>>> > Stopped: [ nfsnode02 ]
>>> > drbd_fs (ocf::heartbeat:Filesystem): Stopped
>>> >
>>> > PCSD Status:
>>> > nfsnode01: Online
>>> > nfsnode02: Online
>>> >
>>> > Daemon Status:
>>> > corosync: active/enabled
>>> > pacemaker: active/enabled
>>> > pcsd: active/enabled
>>> >
>>> > As soon as I bring the second node back online, the failover completes.
>>> > But this is obviously not a good state, as an extended outage for any
>>> > reason on one node essentially kills the cluster services. There's
>>> > obviously something I've missed in configuring the resources, but I
>>> > haven't been able to pinpoint it yet.
>>> >
>>> > Perusing the logs, it appears that, upon the initial failure, DRBD does
>>> > in fact promote the drbd_master resource, but immediately after that,
>>> > pengine calls for it to be demoted for reasons I haven't been able to
>>> > determine yet, but seems to be tied to the fencing configuration. I can
>>> > see that the crm-fence-peer.sh script is called, but it almost seems
>>> > like it's fencing the wrong node... Indeed, I do see that it adds a
>>> > -INFINITY location constraint for the surviving node, which would
>>> > explain the decision to demote the DRBD master.
>>> >
>>> > My DRBD resource looks like this:
>>> >
>>> > # cat /etc/drbd.d/drbd0.res
>>> > resource drbd0 {
>>> >
>>> > protocol C;
>>> > startup { wfc-timeout 0; degr-wfc-timeout 120; }
>>> >
>>> > disk {
>>> > on-io-error detach;
>>> > fencing resource-only;
>>>
>>> This should be 'resource-and-stonith;', but alone won't do anything
>>> until pacemaker's stonith is working.
>>>
>>> > }
>>> >
>>> > handlers {
>>> > fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
>>> > after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
>>> > }
>>> >
>>> > on nfsnode01 {
>>> > device /dev/drbd0;
>>> > disk /dev/vg_nfs/lv_drbd0;
>>> > meta-disk internal;
>>> > address 10.0.0.2:7788 <http://10.0.0.2:7788>;
>>> > }
>>> >
>>> > on nfsnode02 {
>>> > device /dev/drbd0;
>>> > disk /dev/vg_nfs/lv_drbd0;
>>> > meta-disk internal;
>>> > address 10.0.0.3:7788 <http://10.0.0.3:7788>;
>>> > }
>>> > }
>>> >
>>> > If I comment out the three lines having to do with fencing, the failover
>>> > works properly. But I'd prefer to have the fencing there in the odd
>>> > chance that we end up with a split brain instead of just a node outage...
>>> >
>>> > And, here's "pcs config --full":
>>> >
>>> > # pcs config --full
>>> > Cluster Name: nfscluster
>>> > Corosync Nodes:
>>> > nfsnode01 nfsnode02
>>> > Pacemaker Nodes:
>>> > nfsnode01 nfsnode02
>>> >
>>> > Resources:
>>> > Resource: nfsVIP (class=ocf provider=heartbeat type=IPaddr2)
>>> > Attributes: ip=10.0.0.1 cidr_netmask=24
>>> > Operations: start interval=0s timeout=20s (nfsVIP-start-interval-0s)
>>> > stop interval=0s timeout=20s (nfsVIP-stop-interval-0s)
>>> > monitor interval=15s (nfsVIP-monitor-interval-15s)
>>> > Resource: nfs-server (class=systemd type=nfs-server)
>>> > Operations: monitor interval=60s (nfs-server-monitor-interval-60s)
>>> > Master: drbd_master
>>> > Meta Attrs: master-max=1 master-node-max=1 clone-max=2
>>> > clone-node-max=1 notify=true
>>> > Resource: drbd_dev (class=ocf provider=linbit type=drbd)
>>> > Attributes: drbd_resource=drbd0
>>> > Operations: start interval=0s timeout=240 (drbd_dev-start-interval-0s)
>>> > promote interval=0s timeout=90 (drbd_dev-promote-interval-0s)
>>> > demote interval=0s timeout=90 (drbd_dev-demote-interval-0s)
>>> > stop interval=0s timeout=100 (drbd_dev-stop-interval-0s)
>>> > monitor interval=29s role=Master
>>> > (drbd_dev-monitor-interval-29s)
>>> > monitor interval=31s role=Slave
>>> > (drbd_dev-monitor-interval-31s)
>>> > Resource: drbd_fs (class=ocf provider=heartbeat type=Filesystem)
>>> > Attributes: device=/dev/drbd0 directory=/exports/drbd0 fstype=xfs
>>> > Operations: start interval=0s timeout=60 (drbd_fs-start-interval-0s)
>>> > stop interval=0s timeout=60 (drbd_fs-stop-interval-0s)
>>> > monitor interval=20 timeout=40 (drbd_fs-monitor-interval-20)
>>> >
>>> > Stonith Devices:
>>> > Fencing Levels:
>>> >
>>> > Location Constraints:
>>> > Ordering Constraints:
>>> > start nfsVIP then start nfs-server (kind:Mandatory)
>>> > (id:order-nfsVIP-nfs-server-mandatory)
>>> > start drbd_fs then start nfs-server (kind:Mandatory)
>>> > (id:order-drbd_fs-nfs-server-mandatory)
>>> > promote drbd_master then start drbd_fs (kind:Mandatory)
>>> > (id:order-drbd_master-drbd_fs-mandatory)
>>> > Colocation Constraints:
>>> > nfs-server with nfsVIP (score:INFINITY)
>>> > (id:colocation-nfs-server-nfsVIP-INFINITY)
>>> > nfs-server with drbd_fs (score:INFINITY)
>>> > (id:colocation-nfs-server-drbd_fs-INFINITY)
>>> > drbd_fs with drbd_master (score:INFINITY) (with-rsc-role:Master)
>>> > (id:colocation-drbd_fs-drbd_master-INFINITY)
>>> >
>>> > Resources Defaults:
>>> > resource-stickiness: 100
>>> > failure-timeout: 60
>>> > Operations Defaults:
>>> > No defaults set
>>> >
>>> > Cluster Properties:
>>> > cluster-infrastructure: corosync
>>> > cluster-name: nfscluster
>>> > dc-version: 1.1.13-10.el7_2.2-44eb2dd
>>> > have-watchdog: false
>>> > maintenance-mode: false
>>> > stonith-enabled: false
>>>
>>> Configure *and test* stonith in pacemaker first, then DRBD will hook
>>> into it and use it properly. DRBD simply asks pacemaker to do the fence,
>>> but you currently don't have it setup.
>>>
>>> --
>>> Digimer
>>> Papers and Projects: https://alteeve.ca/w/
>>> What if the cure for cancer is trapped in the mind of a person without
>>> access to education?
>>> _______________________________________________
>>> drbd-user mailing list
>>> drbd-user at lists.linbit.com
>>> http://lists.linbit.com/mailman/listinfo/drbd-user
> End of included message
>
>
>
More information about the Users
mailing list