[ClusterLabs] Fencing libvirt

Ken Gaillot kgaillot at redhat.com
Tue Jun 19 10:25:00 EDT 2018 

On Mon, 2018-06-18 at 21:01 -0400, Jason Gauthier wrote:
> On Mon, Jun 18, 2018 at 11:12 AM Jason Gauthier <jagauthier at gmail.com
> > wrote:
> > 
> > On Mon, Jun 18, 2018 at 10:58 AM Ken Gaillot <kgaillot at redhat.com>
> > wrote:
> > > 
> > > On Mon, 2018-06-18 at 10:10 -0400, Jason Gauthier wrote:
> > > > On Mon, Jun 18, 2018 at 9:55 AM Ken Gaillot <kgaillot at redhat.co
> > > > m>
> > > > wrote:
> > > > > 
> > > > > On Fri, 2018-06-15 at 21:39 -0400, Jason Gauthier wrote:
> > > > > > Greetings,
> > > > > > 
> > > > > >    Previously, I was using fiber channel with block
> > > > > > devices.  I
> > > > > > used
> > > > > > sbd to fence the disks, by creating a small block device,
> > > > > > and
> > > > > > then
> > > > > > using stonith to fence the physical disk block.
> > > > > > 
> > > > > > However, I had some reliability issues with that (I believe
> > > > > > it
> > > > > > was
> > > > > > the
> > > > > > fibre channel interfacing, not clustering).  So, I've moved
> > > > > > everything
> > > > > > to NFS.
> > > > > 
> > > > > Another possibility would be to use qdevice with a third
> > > > > host, so
> > > > > that
> > > > > the cluster has true quorum, and then you can use sbd with
> > > > > hardware
> > > > > watchdog only (no need for a shared disk).
> > > > 
> > > > This is the first time I've heard of qdevice.  So, that said,
> > > > the man
> > > > page indicates it's used on 2 node clusters, or even numbered?
> > > > 
> > > > "It is recommended for clusters with an even number of nodes
> > > > and
> > > > highly recommended for 2 node clusters."
> > > > 
> > > > But that said, I have two nodes and a shared storage.  I could
> > > > make
> > > > the shared storage device a node, which seems a little
> > > > odd.   But
> > > > that
> > > > would make a 3rd. I do have another system, but the electricity
> > > > costs
> > > > are already high for this project :)
> > > 
> > > A qdevice host isn't a cluster node -- qdevice is a lightweight
> > > process
> > > that simply provides a quorum vote.
> > > 
> > > I don't know what your shared storage device is, but if it's
> > > capable of
> > > running qdevice, I don't know of any problems with that (if you
> > > lose
> > > the storage, you can't do anything anyway, so it wouldn't be a
> > > big deal
> > > to lose quorum). It does sound like the shared storage device is
> > > a
> > > single point of failure.
> > 
> > Yes, it is a single point of failure, for sure. I'm using NFS as
> > the
> > shared storage platform.
> > I'll check out qdevice for all 3.
> > 
> > > > > > My only resources are virtual machines running with
> > > > > > KVM.    So, I
> > > > > > am
> > > > > 
> > > > > The VMs are resources, or nodes? The libvirt fence agents are
> > > > > for
> > > > > fencing VMs used as full cluster nodes.
> > > > > 
> > > > 
> > > > The VMs are resources.   So, yeah, libvirt won't work. That
> > > > explanation helps.
> > > > It seems it would be simplest to protect each VM, so that if
> > > > one node
> > > > went "missing" (but still running), the other node would not
> > > > boot the
> > > > VMs and corrupt all the disks.
> > > > That's what  thought libvirt did.  So, I still have to figure
> > > > out
> > > > some
> > > > form of fencing for this.  (Which you suggested above, but I
> > > > have not
> > > > completely processed yet)
> > > 
> > > Power fencing is almost always best. If that's not possible, you
> > > could
> > > try getting sbd working again. If that's not possible, and your
> > > shared
> > > storage device supports SCSI-3 persistent reservations, you could
> > > look
> > > into fence_scsi, which cuts off a node's access to the shared
> > > disk.
> 
> Thanks for the help on this. I started looking at power fencing and
> then discovered that I can use IPMI.  I've never used this before, so
> I wasn't sure.  But sure enough.   I set up two primitives, pointing
> to each other with a reboot action.

IPMI is great, but not alone if it's on-board. If a host loses power
entirely, its on-board IPMI will not respond, so the fencing will fail
and the cluster cannot recover. It needs a backup method, e.g. an
intelligent power switch or sdb.

> Now, going to look into qnet/qdevice for the additional cluster vote.
> 
> 
> > The toss up was between iscsi and NFS, and I decided NFS was less
> > hassle.
> > If I went iscsi, then I could create block devices and go back to
> > an
> > sbd device as well.
> > 
> > I just had a ton of problem with block devices not showing up at
> > boot
> > when using fibre channel.
> > Perhaps the iscsi layer is completely different and that's a non-
> > issue.
> > 
> > > > > > trying to figure out what I should fence.  I saw stonith
> > > > > > has a
> > > > > > module,
> > > > > > external/libvirt, and that seems like it might work. But I
> > > > > > can't
> > > > > > seem
> > > > > > to figure out how to use it with my crm config.
> > > > > > 
> > > > > > I've attempted this:
> > > > > > primitive st_libvirt stonith:external/libvirt \
> > > > > >         params hypervisor_uri="qemu:///system"
> > > > > > hostlist="alpha
> > > > > > beta"
> > > > > > \
> > > > > >         op monitor interval=2h \
> > > > > >         meta target-role=Stoppedprimitive st_libvirt
> > > > > > stonith:external/libvirt \
> > > > > >         params hypervisor_uri="qemu:///system"
> > > > > > hostlist="alpha
> > > > > > beta"
> > > > > > \
> > > > > >         op monitor interval=2h
> > > > > > 
> > > > > > But, I am not sure this is the correct syntax.  The nodes
> > > > > > are
> > > > > > alpha,
> > > > > > and beta.
> > > > > > 
> > > > > > Any pointers appreciated.
> > > 
> > > --
> > > Ken Gaillot <kgaillot at redhat.com>
> > > _______________________________________________
> > > Users mailing list: Users at clusterlabs.org
> > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > 
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scra
> > > tch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot <kgaillot at redhat.com>

More information about the Users mailing list