[ClusterLabs] Fence node when network interface goes down

[Andrei Borzenkov]

On Mon, Nov 15, 2021 at 1:18 PM Klaus Wenninger <kwenning at redhat.com> wrote:
>
>
>
> On Mon, Nov 15, 2021 at 10:37 AM S Rogers <sa.rogers1342 at gmail.com> wrote:
>>
>> I had thought about doing that, but the cluster is then dependent on the
>> external system, and if that external system was to go down or become
>> unreachable for any reason then it would falsely cause the cluster to
>> failover or worse it could even take the cluster down completely, if the
>> external system goes down and both nodes cannot ping it.
>
> You wouldn't necessarily have to ban resources from nodes that can't
> reach the external network. It would be enough to make them prefer
> the location that has connection. So if both lose connection  one side
> would still stay up.
> Not to depend on something really external you might use the
> router to your external network as ping target.
> In case of fencing - triggered by whatever - and a potential fence-race

The problem here is that nothing really triggers fencing. What happens, is

- two postgres lose connection over external network, but cluster
nodes retain connectivity over another network
- postgres RA compares "latest timestamp" when selecting the best node
to fail over to
- primary postgres has better timestamp, so RA simply does not
consider secondary as suitable for (atomatic) failover

The only solution here - as long as fencing node on external
connectivity loss is acceptable - is modifying ethmonitor RA to fail
monitor operation in this case.

> you might use the rather new feature priority-fencing-delay (give the node
> that is running valuable resources a benefit in the race) or go for
> fence_heuristics_ping (pseudo fence-resource that together with a
> fencing-topology prevents the node without access to a certain IP
> from fencing the other node).
> https://clusterlabs.org/pacemaker/doc/deprecated/en-US/Pacemaker/2.0/html/Pacemaker_Explained/s-cluster-options.html
> https://github.com/ClusterLabs/fence-agents/blob/master/agents/heuristics_ping/fence_heuristics_ping.py
>
> Klaus
> _______________________________________________
>>
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
>>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/