[ClusterLabs] [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected
Windl, Ulrich
u.windl at ukr.de
Fri Nov 6 07:26:39 UTC 2026
Usually formation of a new cluster (DC selection) is quite fast and it should be finished before a fenced node is restarted.
Thinks may be different if the node isn’t actually fenced, however.
Why would all nodes loose the network when one node is fenced?
Kind regards,
Ulrich Windl
From: Users <users-bounces at clusterlabs.org> On Behalf Of Michael Riexinger via Users
Sent: Tuesday, November 3, 2026 10:16 AM
To: users at clusterlabs.org
Cc: Michael Riexinger <michael.riexinger at de.clara.net>
Subject: [EXT] [EXT] [ClusterLabs] [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected
Hi,
I'd like to report what looks like a race in the controller's join handling. Bugzilla is
under maintenance right now, so I'm sending it here; I'm happy to file it there once it's
back up.
Summary
-------
The DC finalizes a join by syncing the CIB from a node it has just fenced. Recovery is
delayed by 30 s, and the fenced node's next join is rejected ("failed to sync its CIB"),
so it gets fenced a second time.
Versions
--------
Pacemaker 3.0.0 (Debian 13 package 3.0.0-2), corosync 3.1.9, corosync-qdevice 3.0.3
(net, ffsplit), sbd 1.5.2 (watchdog only). The code involved looks unchanged in 2.1.6,
3.0.1 and current main.
Setup
-----
Two nodes plus a quorum device. Fencing level 1 powers the node off through the cloud API
(pcmk_reboot_action=off); level 2 is watchdog SBD.
What happens
------------
1. All nodes lose the network for about 7 s.
2. Node 1 (DC) keeps quorum through the quorum device and fences node 2.
3. While the fence is in flight, the network returns and node 2 rejoins the corosync
membership.
4. The DC delays the join until the fence completes. It then finalizes the join by syncing
the CIB *from node 2*, which is powered off by then.
5. The sync times out after 30 s. Only then does the DC hold a new election and start the
resources.
6. When node 2 is started again later, the DC rejects its join because the earlier sync
failed. Node 2 leaves and is fenced a second time. The start after that joins normally.
Reproduced in 1 of 3 runs with a 7 s outage of all nodes. It needs the network to return
while the fence is in flight.
DC journal (UTC, node 2 = gw-2)
-------------------------------
15:47:33.525 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2
15:47:34.342 corosync [TOTEM ] A new membership (1.4f) was formed. Members joined: 2
15:47:38.870 pacemaker-controld: warning: Delaying join-2 finalization while transition in progress
15:47:43.204 pacemaker-based: notice: Node gw-2 state is now lost
15:47:46.781 pacemaker-fenced: notice: Operation 'reboot' targeting gw-2 by gw-1 for pacemaker-controld at gw-1: OK (Done)
15:47:46.782 pacemaker-controld: notice: Finalizing join-2 for 1 node (sync'ing CIB from gw-2)
15:48:16.735 pacemaker-controld: error: Could not sync CIB from gw-2 in join-2: Timer expired
15:48:16.735 pacemaker-controld: warning: Input I_ELECTION_DC received in state S_FINALIZE_JOIN from finalize_sync_callback
15:48:16.760 pacemaker-schedulerd: notice: Actions: Start ... (resources start only now)
15:51:44.756 pacemaker-controld: error: Rejecting join-3 request from node gw-2 because we failed to sync its CIB in join-2
15:51:46.800 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2
Code pointers (main at d981f262f81644d9faad4fe6be158401fdb53bf8)
-----------------------------------------------------------------
1. The sync source is chosen at offer time and never revalidated.
do_dc_join_filter_offer() stores the node with the newest CIB in max_generation_from
(daemons/controld/controld_join_dc.c L542, L568). In our case that was gw-2, whose join
offer arrived right after it rejoined.
https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L542
2. Finalization resumes after the fencing transition with the stale source.
do_dc_join_finalize() stalls while the transition (the fence of gw-2) runs (L634-L639).
When it resumes, it only checks whether max_generation_from is the local node
(L628-L632), not whether that node is still an active member, and calls sync_from() on
a node that is powered off (L668-L671).
https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L628-L671
3. The 30 s come from the CIB call timeout.
cib_op_timeout() is max(10 s x (nodes + 1), MIN_CIB_OP_TIMEOUT = 30)
(daemons/controld/controld_cib.c L203, L214-L220), so 30 s for two nodes.
4. The failure is recorded after the node has already left.
finalize_sync_callback() calls record_failed_sync_node() for gw-2 (controld_join_dc.c
L694). The only removals are in peer_update_callback(), when a node changes away from
member (L175) or loses its controller process (L214)
(daemons/controld/controld_callbacks.c). Both had already happened for gw-2 at
15:47:43, before the entry was recorded at 15:48:16.
https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_callbacks.c#L175
5. The stale entry rejects the next join.
do_dc_join_filter_offer() rejects gw-2 on its next join (controld_join_dc.c L486-L490).
The rejection makes gw-2 leave, which removes the entry, so the join after that
succeeds.
Expected
--------
1. A node that is being fenced, or has just been fenced, in the current transition is never
chosen as the CIB sync source.
2. A failed sync from a node that has already left does not cause its next join to be
rejected.
Possible directions
-------------------
- When finalization resumes, check that max_generation_from is still an active member. If
it is not, restart the join or fall back to the local CIB.
- Clear max_generation_from (or restart the join) when that node is lost or fenced during
the join.
- Do not record a failed sync for a node that is no longer active, or clear its entry when
its fencing succeeds.
Impact
------
Resources start 30 s later than necessary. After a restart, the fenced node is fenced once
more, so automated recovery needs two attempts. No data was at risk in our tests.
A note on a fix
---------------
I would have liked to send a patch for this myself. However, the code analysis above was
done with the help of an AI assistant, and under the ClusterLabs AI policy
(https://github.com/ClusterLabs/ai-policies) AI-assisted code is not accepted. So I'm
leaving the fix to you and providing the analysis instead. The test setup and the logs are
ours, and I checked the findings against the source. I can rerun the scenario or collect
more logs (crm_report, full journals) if that helps.
Best regards
Michael Riexinger
Head of Cloud Infrastructure Engineering
—
Claranet GmbH
Hanauer Landstraße 196
60314 Frankfurt am Main
https://www.claranet.com/de<https://www.claranet.com/de/>
Sitz der Gesellschaft: Frankfurt am Main, Deutschland
Geschäftsführer Sebastian Vögel, Olaf Fischer
HRB 50381 AG Frankfurt am Main
Vat-ID DE812918694
Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze.
Näheres entnehmen Sie bitte unserer Datenschutzerklärung: https://www.claranet.com/de/datenschutz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20261106/5b12f276/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5581 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20261106/5b12f276/attachment-0001.p7s>
More information about the Users
mailing list