[ClusterLabs] fencing configuration

[Klaus Wenninger]

On Tue, Jun 7, 2022 at 10:27 AM Zoran Bošnjak <zoran.bosnjak at via.si> wrote:
>
> Hi, I need some help with correct fencing configuration in 5-node cluster.
>
> The speciffic issue is that there are 3 rooms, where in addition to node failure scenario, each room can fail too (for example in case of room power failure or room network failure).
>
> room0: [ node0 ]
> roomA: [ node1, node2 ]
> roomB: [ node3, node4 ]
>
> - ipmi board is present on each node
> - watchdog timer is available
> - shared storage is not available
>
> Please advice, what would be a proper fencing configuration in this case.
>
> The intention is to configure ipmi fencing (using "fence_idrac" agent) plus watchdog timer as a fallback. In other words, I would like to tell the pacemaker: "If fencing is required, try to fence via ipmi. In case of ipmi fence failure, after some timeout assume watchdog has rebooted the node, so it is safe to proceed, as if the (self)fencing had succeeded)."
>
> From the documentation is not clear to me whether this would be:
> a) multiple fencing where ipmi would be first level and sbd would be a second level fencing (where sbd always succeeds)
> b) or this is considered a single level fencing with a timeout

With b) falling back to watchdog-fencing wouldn't work properly
although I remember
some recent change that might make it fall back without issues.
I would try to go for a) as with a reasonably current
pacemaker-version (iirc 2.1.0 and above)
you should be able to make the watchdog-fencing-device visible as with
other fencing-devices
(just use fence_watchdog as the fence-agent - still implemented inside
pacemaker
fence-watchdog-binary actually just provides the meta-data).
Like this you can limit watchdog-fencing to certain-nodes that do
actually provide a proper
hardware-watchdog and you can add it to a topology.

Depending on your infra-structure an alternative solution to using
watchdog-fencing
for your case (where you can't access ipmis in a room with
power-outage) might be
fabric-fencing.

Klaus
>
> I have tried to followed option b) and create stonith resource for each node and setup the stonith-watchdog-timeout, like this:
>
> ---
> # for each node... [0..4]
> export name=...
> export ip=...
> export password=...
> sudo pcs stonith create "fence_ipmi_$name" fence_idrac \
>     lanplus=1 ip="$ip" \
>     username="admin"  password="$password" \
>     pcmk_host_list="$name" op monitor interval=10m timeout=10s
>
> sudo pcs property set stonith-watchdog-timeout=20
>
> # start dummy resource
> sudo pcs resource create dummy ocf:heartbeat:Dummy op monitor interval=30s
> ---
>
> I am not sure if additional location constraints have to be specified for stonith resources. For example: I have noticed that pacemaker will start a stonith resource on the same node as the fencing target. Is this OK?
>
> Should there be any location constraints regarding fencing and rooms?
>
> 'sbd' is running, properties are as follows:
>
> ---
> $ sudo pcs property show
> Cluster Properties:
>  cluster-infrastructure: corosync
>  cluster-name: debian
>  dc-version: 2.0.3-4b1f869f0f
>  have-watchdog: true
>  last-lrm-refresh: 1654583431
>  stonith-enabled: true
>  stonith-watchdog-timeout: 20
> ---
>
> Ipmi fencing (when the ipmi connection is alive) works correctly for each node. The watchdog timer also seems to be working correctly. The problem is that dummy resource is not restarted as expected.
>
> In the test scenario, the dummy resource is currently running on node1. I have simulated node failure by unplugging the ipmi AND host network interfaces from node1. The result was that node1 gets rebooted (by watchdog), but the rest of the pacemaker cluster was unable to fence node1 (this is expected, since node1's ipmi is not accessible). The problem is that dummy resource remains stopped and node1 unclean. I was expecting that stonith-watchdog-timeout kicks in, so that dummy resource gets restarted on some other node which has quorum.
>
> Obviously there is something wrong with my configuration, since this seems to be a reasonably simple scenario for the pacemaker. Appreciate your help.
>
> regards,
> Zoran
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>