[ClusterLabs] [EXT] [EXT] Re: Can stalled libqb SHM I/O delay Pacemaker's operation timer?

[Brian Cosgrove]

We do not use hardware-watchdog. We did not have SBD watchdog self-fencing
configured. Fencing used external power control.

That's an excellent guardrail for many use-cases. Thanks for the helpful
guidance.

My question remains, though. Can we not prevent this particular kind of
stall at the source? mlock() would guarantee that the SHM IPC pages remain
resident (the executor request, response, and event rings in particular).
It's a pretty narrow potential fix, and when coupled with the
hardware-watchdog and subdaemon-liveness as a backstop this would tighten
Pacemaker guarantees in a meaningful way, yes?

On Wed, Aug 12, 2026 at 4:39 AM Klaus Wenninger via Users <
users at clusterlabs.org> wrote:

>
>
> On Wed, Aug 12, 2026 at 7:57 AM Windl, Ulrich via Users <
> users at clusterlabs.org> wrote:
>
>> Just wondering: Did you have a working hardware watchdog configured?
>> Wouldn’t it have fenced your node then?
>>
>
> Just a side-note:
> If you're using SBD pacemakerd would be observing the
> pacemaker-subdaemons. And if that gets stuck it would stop sending
> pings to SBD daemon and that would make SBD kick in and trigger a reboot -
> via the hardware-watchdog as a final resort if configured.
> That behavior requires a sufficiently current stack of pacemaker & sbd and
> it has to be configured and built accordingly.
> If you're using watchdog-daemon or alike you have to take care that the
> proper pacemaker-daemons are observed in one way or
> the other for the case not the whole machine is hanging (so that the
> watchdog-daemon process would be hanging anyway).
>
> Regards,
> Klaus
>
>>
>>
>> Kind regards,
>>
>> Ulrich Windl
>>
>>
>>
>> *From:* Users <users-bounces at clusterlabs.org> *On Behalf Of *Brian
>> Cosgrove
>> *Sent:* Tuesday, August 11, 2026 5:28 PM
>> *To:* Cluster Labs - All topics related to open-source clustering
>> welcomed <users at clusterlabs.org>
>> *Subject:* [EXT] [EXT] Re: [ClusterLabs] Can stalled libqb SHM I/O delay
>> Pacemaker's operation timer?
>>
>>
>>
>> Thanks for your reply. Those are great tips, however this was a freak
>> firmware bug.
>>
>>
>>
>> "Degraded RAID array" was the wrong wording since that has a precise
>> meaning for RAID. We were not in an ordinary degraded RAID mode with, say,
>> a disk failure in the array. The array itself (or driver, or controller?)
>> was misbehaving. Reads were as far as I can tell, actually unbounded: never
>> returning and not erroring. I don't expect it to recur after firmware
>> updates.
>>
>>
>>
>> It may, however, have revealed a bug or design flaw in Pacemaker as this
>> is exactly the kind of hardware fault I would expect to trigger fencing and
>> promotion of the hot standby.
>>
>>
>>
>> Only manual intervention by an operator (as the controller was stalled)
>> led to a successful STONITH (we ran the script by hand, a DC election then
>> occurred and Pacemaker correctly moved the primary database resource and
>> the resource agent promoted the hot standby).
>>
>>
>>
>> I have a draft simple reproduction if it helps maintainers triage this or
>> alternatively help me understand that this was expected behavior and why.
>>
>>
>>
>> https://github.com/cosgroveb/pacemaker-shm-timer-repro/blob/main/repro.sh
>>
>>
>>
>> On Tue, Aug 11, 2026 at 8:33 AM Windl, Ulrich via Users <
>> users at clusterlabs.org> wrote:
>>
>> Hi!
>>
>>
>>
>> A non-pacemaker answer: I can imagine two solutions for your problem:
>>
>>    1. Limit the reconstruction rate of the RAID
>>    2. Limit the amount of (dirty) filesystem cache (we had read stalls
>>    when someone backed up the database and most of the RAM had been filled
>>    with dirty buffers to write out)
>>
>>
>>
>> Kind regards,
>>
>> Ulrich Windl
>>
>>
>>
>> *From:* Users <users-bounces at clusterlabs.org> *On Behalf Of *Brian
>> Cosgrove
>> *Sent:* Monday, August 10, 2026 10:03 PM
>> *To:* users at clusterlabs.org
>> *Subject:* [EXT] [EXT] [ClusterLabs] Can stalled libqb SHM I/O delay
>> Pacemaker's operation timer?
>>
>>
>>
>> Could Pacemaker’s controller block on a stalled block-device read while
>> faulting in a swapped-
>> out libqb SHM page, before it receives the executor reply and starts the
>> operation timer?
>>
>>
>>
>> Would that behavior be expected?
>>
>>
>>
>> I'm investigating a lockup where the PostgreSQL primary and Pacemaker DC
>> were on the same node when a degraded RAID array caused reads to appear
>> unbounded without returning errors. Pacemaker did not stop the PostgreSQL
>> resource or promote a standby. crm_mon showed the DC as standby (with
>> active resources).
>>
>>
>>
>> Testing that simulates the incident reproduces that production Pacemaker
>> behavior. In that setup a libqb SHM page is swapped out and controller
>> blocks in shmem_fault() before the executor reply and operation timer. The
>> surviving production logs (such as the crm_mon output) match. Corosync
>> membership remained intact in both the incident and the test.
>>
>>
>>
>>
>>
>> --
>>
>> Brian Cosgrove
>>
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
>>
>>
>>
>>
>> --
>>
>> Brian Cosgrove
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
>>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>


-- 
Brian Cosgrove
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20260813/c1edc648/attachment.htm>