[ClusterLabs] Fencing with a 3-node (1 for quorum only) cluster
dswartz at druber.com
Thu Aug 4 20:18:56 EDT 2016
On 2016-08-04 19:33, Digimer wrote:
> On 04/08/16 07:21 PM, Dan Swartzendruber wrote:
>> On 2016-08-04 19:03, Digimer wrote:
>>> On 04/08/16 06:56 PM, Dan Swartzendruber wrote:
>>>> I'm setting up an HA NFS server to serve up storage to a couple of
>>>> vsphere hosts. I have a virtual IP, and it depends on a ZFS
>>>> agent which imports or exports a pool. So far, with stonith
>>>> it all works perfectly. I was dubious about a 2-node solution, so I
>>>> created a 3rd node which runs as a virtual machine on one of the
>>>> All it is for is quorum. So, looking at fencing next. The primary
>>>> server is a poweredge R905, which has DRAC for fencing. The backup
>>>> storage node is a Supermicro X9-SCL-F (with IPMI). So I would be
>>>> the DRAC agent for the former and the ipmilan for the latter? I was
>>>> reading about location constraints, where you tell each instance of
>>>> fencing agent not to run on the node that would be getting fenced.
>>>> my first thought was to configure the drac agent and tell it not to
>>>> fence node 1, and configure the ipmilan agent and tell it not to
>>>> node 2. The thing is, there is no agent available for the quorum
>>>> Would it make more sense instead to tell the drac agent to only run
>>>> node 2, and the ipmilan agent to only run on node 1? Thanks!
>>> This is a common mistake.
>>> Fencing and quorum solve different problems and are not
>>> In short;
>>> Fencing is a tool when things go wrong.
>>> Quorum is a tool when things are working.
>>> The only impact that having quorum has with regard to fencing is that
>>> avoids a scenario when both nodes try to fence each other and the
>>> one wins (which is itself OK). Even then, you can add 'delay=15' the
>>> node you want to win and it will win is such a case. In the old days,
>>> would also prevent a fence loop if you started the cluster on boot
>>> comms were down. Now though, you set 'wait_for_all' and you won't get
>>> fence loop, so that solves that.
>>> Said another way; Quorum is optional, fencing is not (people often
>>> that backwards).
>>> As for DRAC vs IPMI, no, they are not two things. In fact, I am
>>> certain that fence_drac is a symlink to fence_ipmilan. All DRAC is
>>> with iRMC, iLO, RSA, etc) is "IPMI + features". Fundamentally, the
>>> action; rebooting the node, works via the basic IPMI standard using
>>> DRAC's BMC.
>>> To do proper redundant fencing, which is a great idea, you want
>>> something like switched PDUs. This is how we do it (with two node
>>> clusters). IPMI first, and if that fails, a pair of PDUs (one for
>>> PSU, each PDU going to independent UPSes) as backup.
>> Thanks for the quick response. I didn't mean to give the impression
>> that I didn't know the different between quorum and fencing. The only
>> reason I (currently) have the quorum node was to prevent a deathmatch
>> (which I had read about elsewhere.) If it is as simple as adding a
>> delay as you describe, I'm inclined to go that route. At least on
>> CentOS7, fence_ipmilan and fence_drac are not the same. e.g. they are
>> both python scripts that are totally different.
> The delay is perfectly fine. We've shipped dozens of two-node systems
> over the last five or so years and all were 2-node and none have had
> trouble. Where node failures have occurred, fencing operated properly
> and services were recovered. So in my opinion, in the interest of
> minimizing complexity, I recommend the two-node approach.
> As for the two agents not being symlinked, OK. It still doesn't change
> the core point through that both fence_ipmilan and fence_drac would be
> acting on the same target.
> Note; If you lose power to the mainboard (which we've seen, failed
> mainboard voltage regulator did this once), you lose the IPMI (DRAC)
> BMC. This scenario will leave your cluster blocked without an external
> secondary fence method, like switched PDUs.
More information about the Users