[ClusterLabs] dead cluster after centos update

[Ken Gaillot]

On Mon, 2017-10-23 at 15:02 -0500, Dimitri Maziuk wrote:
> I've a 2-node ZFS cluster that was working fine until redhat improved
> my
> user experience. Now, it's dead.
> 
> How do I get it back?
> 
> There was two resources: ZFS and IP. ther was fence_scsi that is now
> logging
> > Oct 23 14:22:45 hereland fence_scsi: Failed: nodename or key is
> > required
> > Oct 23 14:22:45 hereland fence_scsi: Please use '-h' for usage
> > Oct 23 14:22:45 hereland stonith-ng[1753]: warning:
> > fence_scsi[1929] stderr: [ Failed: nodename or key is required ]

I'm not familiar with fence_scsi, but the above looks like the issue.
Does your fence device configuration have either nodename or key?

If your existing configuration was working previously and now isn't,
open a support ticket with Red Hat, as it sounds like a regression.

> > Oct 23 14:22:45 hereland stonith-ng[1753]: warning:
> > fence_scsi[1929] stderr: [  ]
> > Oct 23 14:22:45 hereland stonith-ng[1753]: warning:
> > fence_scsi[1929] stderr: [ Please use '-h' for usage ]
> > Oct 23 14:22:45 hereland stonith-ng[1753]: warning:
> > fence_scsi[1929] stderr: [  ]
> > Oct 23 14:22:45 hereland stonith-ng[1753]:  notice: Disabling port
> > list queries for fence-tank (-201): (null)
> > Oct 23 14:22:45 hereland stonith-ng[1753]:  notice: Operation on of
> > hereland-eth1 by <no-one> for crmd.1784 at flemish-eth1.3e4dd918: No
> > such device
> > Oct 23 14:22:45 hereland crmd[1757]:   error: Unfencing of
> > hereland-eth1 by <anyone> failed: No such device (-19)

Probably due to the configuration issue, unfencing fails. fence_scsi
requires unfencing, which in its case means allowing the node access to
the disk. Since that fails, nothing else can proceed.

> 
> I disabled it for now, and
> 
> pcs resource debug-start resource-zfs --full
> 
> works fine: the pool is imported, filesystems are mounted and
> exported
> -- but the resources remain stopped no matter what.
> 
> I don't see anything useful in the logs. How do I unfsck this mess?
-- 
Ken Gaillot <kgaillot at redhat.com>