[ClusterLabs] Antw: Re: Antw: Re: Antw: [EXT] Coming in Pacemaker 2.1.2: new fencing configuration options

[Andrei Borzenkov]

On 12.10.2021 09:27, Ulrich Windl wrote:
>>>> Andrei Borzenkov <arvidjaar at gmail.com> schrieb am 11.10.2021 um 11:43 in
> Nachricht
> <CAA91j0Ur9wxxzOpVL7MHmnFMp60EbhxhVDCk=zf2YQGjV-SWtg at mail.gmail.com>:
>> On Mon, Oct 11, 2021 at 9:29 AM Ulrich Windl
>> <Ulrich.Windl at rz.uni‑regensburg.de> wrote:
>> ....
>>>>> Also how long would such a delay be: Long enough until the other node
>>>>> is
>>>>> fenced, or long enough until the other node was fenced, booted
>>>>> (assuming it
>>>>> does) and is running pacemaker?
>>>>
>>>> The delay should be on the less‑preferred node, long enough for that
>>>> node to get fenced. The other node, with no delay, will fence it if it
>>>> can. If the other node is for whatever reason unable to fence, the node
>>>> with the delay will fence it after the delay.
>>>
>>> So the "fence intention" will be lost when the node is being fenced?
>>> Otherwise the surviving node would have to clean up the "fence intention".
>>> Or does it mean the "fence intention" does not make it to the CIB and
> stays
>>> local on the node?
>>>
>>
>> Two nodes cannot communicate with each other so the surviving node is
>> not aware of anything the fenced node did or intended to do. When the
> 
> I thought (local) CIB writes do not need a quorum.
> 
>> fenced node reboots and pacemaker starts it should pull CIB from the
>> surviving node, so whatever intentions the fenced node had before
>> reboot should be lost at this point.
> 
> If the surviving node has a CIB newer (as per modification/configuration
> count) the fenced node that is true, but it the fenced node has a newer CIB,
> the surviving node would pull the "other" CIB, right?

Indeed. I honestly did not expect it.

I am not sure what consequences it has in practice though. It is
certainly one more argument against running without mandatory stonith
because in this case both nodes happily continue and it is unpredictable
which one will win after they rejoin.

Assuming we do run with mandatory stonith then we have relatively small
window before DC is killed (because only DC can update CIB). But I am
not sure whether CIB changes will be committed locally until all nodes
are either confirmed to be offline or acknowledged CIB changes. I guess
only Ken can answer it :)


> I think I had a few cases in the past when the "last dying node" did not have
> the "latest" CIB, causing some "extra noise" when the cluster was formed
> again.

Details of what happened are certainly interesting.

> Probably some period to wait for all nodes to join (and thus sync the CIBs)
> before performing any actions would help there.
>