[Pacemaker] Asymmetric cluster resource locations

[Andrew Beekhof]

On Tue, Oct 25, 2011 at 3:42 PM, Nick Morrison <nick at nick.on.net> wrote:
> Hello,
>
> I've been browsing around the history of the list, and it looks like this topic has been covered by people with lots more experience in clustering than I.  For that reason, I thought I should ask to be sure, just in case my problem is actually an easy one to solve.
>
> My desired architecture is perhaps a little weird.
>
> I'm trying to find a path-of-least-resistance to configuring our four Dell servers into a reasonably-High-Availability cluster.  I want to avoid using any active/active block devices because I don't want to have to think about split brain recovery.
>
> The architecture.  Call them host-a1 and host-a2, and host-b1 and host-b2.
>
> . Each of these is an primary/secondary DRBD pair (managed by pacemaker)
> . Each of the pairs runs failover NFS (managed by pacemaker)
> . host-a[12] export /data/pool-a
> . host-b[12] export /data/pool-b
> . All four hosts run virtual machines, with virtual disk images stored in /data/pool-a or /data/pool-b
> . The one limitation on virtual machines is that they can't run on their own pool's secondary storage, as it's not accessible
> . It's not accessible because trying to make an NFS server mount its own export was failing horribly
>  . (otherwise I'd mount the same virt-a:/data/pool-a on all four machines and life would be easy)
>
> Crazy?  Maybe.  I haven't seen anyone design an architecture like this.  Perhaps everyone is braver than me about GFS2/OCFS2.  I'm willing to accept a few seconds of IO-Wait while the NFS server switches over.
>
> a) am I missing anything?
>
> b) the reason I wrote to the list:
>
> I'm constraining my resources per server pair with -inf: away from the other nodes and 2: to the nodes I want them on.  Except, sometimes I'll find that my NFS-DRBDFS-POOL-A ocf:heartbeat:Filesystem resource that's supposed to be running on host-a1 (along with the DRBD master for pool-a) is suddenly running on host-b2, and setting it to "unmanaged".
>
> I think that's because pacemaker is running a monitor operation on all of the nodes, finding "a filesystem" mounted in /data/pool-a (which it is - via NFS, from the POOL-A pair) and so assuming that someone's gone and started up the service without telling it.  Hence unmanaged.
>
> My question is : am I right?

Yes.

> and, what's the best way to deal with this scenario?

you'd have to teach the RA how to know the difference between the real
/data/pool-a and the remote one.
perhaps just use a different mount point?

> I thought of running two clusters.  This works ok, EXCEPT, I lose the ability to have my virtual machines managed by pacemaker also, because it would have to know about all four nodes.  Maybe I could run three clusters, node-a1+nodea2 for drbd/nfs, nodeb1+nodeb2 for drbd/nfs, and node.* for virtual machines.
>
> Or maybe I just manage my virtual machines by hand.
>
> Your thoughts are appreciated.  If you've come across any good reading on the topic, I'd love to have the link :-)
>
>
> Cheers and beers,
>
> Nick!
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>