[Pacemaker] Best stonith method to avoid split brain on a drbd cluster

[Devin Reade]

Johannes Freygner <hannes at freygner.at> wrote:

> You mean with corosync will work fine, because I am using heartbeat instead.

I suspect that it's a similar situation with heartbeat.  The problem is
pacemaker losing communication before the node cleanly disconnects.

The behavior I saw on my own clusters is that because the init script
values were bad, the node's network interfaces would be brought down
before the node had cleanly left the cluster.  Since the second node
didn't see a clean disconnect and couldn't contact the first node, it
would stonith the first node sometime after the first node's network
was down but before it was halted (which is pretty rude and can be
hard on filesystem integrity).

> The resource wouldn't be started by the other node, because it can't fence the missing node without power on ILO.

The point that I was trying to make is that your nodes shouldn't be
trying to fence each other unless a node is _unexpectedly_ unreachable.
During maintenance, which you presumably do with a controlled shutdown,
there should be no fencing at all because the node-going-into-maintenance
should first disconnect cleanly.  (Because of the bad sequencing where
corosync/pacemaker was shut down after the networks went down, a clean
disconnect wouldn't happen, and then the node would get fenced.)

For a clean shutdown, the cluster should move all resources _before_
the node disconnects, thus not requiring fencing in order to run them
on the other node.

Fencing should be an action of last choice, not the normal mode of operation.

In the case of a true hardware fault, and assuming that you're using
redundant power supplies fed by independent power sources, you wouldn't
see this behavior either unless you were dealing with multiple failures
(which is problematic in various ways).

So whether you're using heartbeat or corosync, I'd look at your startup
and shutdown sequence and ensure that during controlled operations no
fencing is being triggered.

(You can still test your fencing by pulling your non-ILO network
cables instead of pulling the power cord.)

If you're still concerned about the choice of stonith device and have
only one power supply, you can look at something like an APC switched PDU,
but I suspect that you're further ahead (for all of cost, complexity,
and redundancy) in using dual power supplies and the ILO.

Devin
-- 
If it's sinful, it's more fun.