[ClusterLabs] Question about STONITH for VM HA cluster in shared hosts environment

[Ken Gaillot]

On 06/29/2017 12:08 PM, Digimer wrote:
> On 29/06/17 12:39 PM, Andrés Pozo Muñoz wrote:
>> Hi all,
>>
>> I am a newbie to Pacemaker and I can't find the perfect solution for my
>> problem (probably I'm missing something), maybe someone can give me some
>> hint :)
>>
>> My scenario is the following: I want to make a HA cluster composed of 2
>> virtual machines running on top of SHARED virtualization hosts. That is,
>> I have a bunch of hosts running multiple VMs, and I would like to create
>> an HA cluster with 2 VMs (Ubuntus running some app) running in
>> (different) hosts.
>>
>> About the resources, I have no problem, I'll configure some VIP and some
>> lsb services in a group.
>>
>> My concern is about the STONITH, I can't find the perfect config:
>>     * If I disable STONITH I may have the split brain problem.
>>     * If I enable STONITH with external/libvirt fence, I'll have a
>> single point of failure  if the host with the Active VM dies, right?
>>  (Imagine the host running that active VMs dies, the STONITH operation
>> from the other VM will fail and the switch-over will not happen, right?)
>>     * I can't use a 'hardware' (ILO/DRAC) fence, because the host is
>> running a lot of VMs, not only the ones in HA cluster :( I can't reboot
>> it because of some failure in our HA.
>>
>> Is there an optimal configuration for such scenario?
>>
>> I think I'd rather live with the split brain problem, but I just want to
>> know if I missed any config option.
>>
>> Thanks in advance!
>>
>> Cheers,
>> Andrés
> 
> You've realized why a production cluster on VMs is generally not
> recommended. :)
> 
> If the project is important enough to make HA, then management needs to
> allocate the budget to get the proper hardware for the effort, I would
> argue. If you want to keep the services in VMs, that's fine, get a pair
> of nodes and make them an HA cluster to protect the VMs as the services
> (we do this all the time).
> 
> With that, then you pair IPMI and switched PDUs for complete coverage
> (IPMI alone isn't enough, because if the host is destroyed, it will take
> the IPMI BMC with it).

To elaborate on this approach, the underlying hosts could be the cluster
nodes, and the VMs could be resources. If you make all the VMs into
resources, then you get HA for all of them. You can also run Pacemaker
Remote in any of the VMs if you want to monitor resources running inside
them (or move resources from one VM to another).

Commenting on your original question, I'd point out that if pacemaker
chooses to fence one of the underlying hosts, it's not responding
normally, so any other VMs on it are likely toast anyway. You may
already be familiar, but you can set a fencing topology so that
pacemaker tries libvirt first, then kills the host only if that fails.