[ClusterLabs] Question about STONITH for VM HA cluster in shared hosts environment

[Digimer]

On 29/06/17 12:39 PM, Andrés Pozo Muñoz wrote:
> Hi all,
> 
> I am a newbie to Pacemaker and I can't find the perfect solution for my
> problem (probably I'm missing something), maybe someone can give me some
> hint :)
> 
> My scenario is the following: I want to make a HA cluster composed of 2
> virtual machines running on top of SHARED virtualization hosts. That is,
> I have a bunch of hosts running multiple VMs, and I would like to create
> an HA cluster with 2 VMs (Ubuntus running some app) running in
> (different) hosts.
> 
> About the resources, I have no problem, I'll configure some VIP and some
> lsb services in a group.
> 
> My concern is about the STONITH, I can't find the perfect config:
>     * If I disable STONITH I may have the split brain problem.
>     * If I enable STONITH with external/libvirt fence, I'll have a
> single point of failure  if the host with the Active VM dies, right?
>  (Imagine the host running that active VMs dies, the STONITH operation
> from the other VM will fail and the switch-over will not happen, right?)
>     * I can't use a 'hardware' (ILO/DRAC) fence, because the host is
> running a lot of VMs, not only the ones in HA cluster :( I can't reboot
> it because of some failure in our HA.
> 
> Is there an optimal configuration for such scenario?
> 
> I think I'd rather live with the split brain problem, but I just want to
> know if I missed any config option.
> 
> Thanks in advance!
> 
> Cheers,
> Andrés

You've realized why a production cluster on VMs is generally not
recommended. :)

If the project is important enough to make HA, then management needs to
allocate the budget to get the proper hardware for the effort, I would
argue. If you want to keep the services in VMs, that's fine, get a pair
of nodes and make them an HA cluster to protect the VMs as the services
(we do this all the time).

With that, then you pair IPMI and switched PDUs for complete coverage
(IPMI alone isn't enough, because if the host is destroyed, it will take
the IPMI BMC with it).


-- 
Digimer
Papers and Projects: https://alteeve.com/w/
"I am, somehow, less interested in the weight and convolutions of
Einstein’s brain than in the near certainty that people of equal talent
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould