[ClusterLabs] [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected

Michael Riexinger michael.riexinger at de.clara.net
Tue Nov 3 09:15:30 UTC 2026 

Hi,

I'd like to report what looks like a race in the controller's join handling. Bugzilla is
under maintenance right now, so I'm sending it here; I'm happy to file it there once it's
back up.

Summary
-------
The DC finalizes a join by syncing the CIB from a node it has just fenced. Recovery is
delayed by 30 s, and the fenced node's next join is rejected ("failed to sync its CIB"),
so it gets fenced a second time.

Versions
--------
Pacemaker 3.0.0 (Debian 13 package 3.0.0-2), corosync 3.1.9, corosync-qdevice 3.0.3
(net, ffsplit), sbd 1.5.2 (watchdog only). The code involved looks unchanged in 2.1.6,
3.0.1 and current main.

Setup
-----
Two nodes plus a quorum device. Fencing level 1 powers the node off through the cloud API
(pcmk_reboot_action=off); level 2 is watchdog SBD.

What happens
------------
1. All nodes lose the network for about 7 s.
2. Node 1 (DC) keeps quorum through the quorum device and fences node 2.
3. While the fence is in flight, the network returns and node 2 rejoins the corosync
   membership.
4. The DC delays the join until the fence completes. It then finalizes the join by syncing
   the CIB *from node 2*, which is powered off by then.
5. The sync times out after 30 s. Only then does the DC hold a new election and start the
   resources.
6. When node 2 is started again later, the DC rejects its join because the earlier sync
   failed. Node 2 leaves and is fenced a second time. The start after that joins normally.

Reproduced in 1 of 3 runs with a 7 s outage of all nodes. It needs the network to return
while the fence is in flight.

DC journal (UTC, node 2 = gw-2)
-------------------------------
15:47:33.525 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2
15:47:34.342 corosync [TOTEM ] A new membership (1.4f) was formed. Members joined: 2
15:47:38.870 pacemaker-controld: warning: Delaying join-2 finalization while transition in progress
15:47:43.204 pacemaker-based: notice: Node gw-2 state is now lost
15:47:46.781 pacemaker-fenced: notice: Operation 'reboot' targeting gw-2 by gw-1 for pacemaker-controld at gw-1: OK (Done)
15:47:46.782 pacemaker-controld: notice: Finalizing join-2 for 1 node (sync'ing CIB from gw-2)
15:48:16.735 pacemaker-controld: error: Could not sync CIB from gw-2 in join-2: Timer expired
15:48:16.735 pacemaker-controld: warning: Input I_ELECTION_DC received in state S_FINALIZE_JOIN from finalize_sync_callback
15:48:16.760 pacemaker-schedulerd: notice: Actions: Start ... (resources start only now)
15:51:44.756 pacemaker-controld: error: Rejecting join-3 request from node gw-2 because we failed to sync its CIB in join-2
15:51:46.800 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2

Code pointers (main at d981f262f81644d9faad4fe6be158401fdb53bf8)
-----------------------------------------------------------------
1. The sync source is chosen at offer time and never revalidated.
   do_dc_join_filter_offer() stores the node with the newest CIB in max_generation_from
   (daemons/controld/controld_join_dc.c L542, L568). In our case that was gw-2, whose join
   offer arrived right after it rejoined.
   https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L542

2. Finalization resumes after the fencing transition with the stale source.
   do_dc_join_finalize() stalls while the transition (the fence of gw-2) runs (L634-L639).
   When it resumes, it only checks whether max_generation_from is the local node
   (L628-L632), not whether that node is still an active member, and calls sync_from() on
   a node that is powered off (L668-L671).
   https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L628-L671

3. The 30 s come from the CIB call timeout.
   cib_op_timeout() is max(10 s x (nodes + 1), MIN_CIB_OP_TIMEOUT = 30)
   (daemons/controld/controld_cib.c L203, L214-L220), so 30 s for two nodes.

4. The failure is recorded after the node has already left.
   finalize_sync_callback() calls record_failed_sync_node() for gw-2 (controld_join_dc.c
   L694). The only removals are in peer_update_callback(), when a node changes away from
   member (L175) or loses its controller process (L214)
   (daemons/controld/controld_callbacks.c). Both had already happened for gw-2 at
   15:47:43, before the entry was recorded at 15:48:16.
   https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_callbacks.c#L175

5. The stale entry rejects the next join.
   do_dc_join_filter_offer() rejects gw-2 on its next join (controld_join_dc.c L486-L490).
   The rejection makes gw-2 leave, which removes the entry, so the join after that
   succeeds.

Expected
--------
1. A node that is being fenced, or has just been fenced, in the current transition is never
   chosen as the CIB sync source.
2. A failed sync from a node that has already left does not cause its next join to be
   rejected.

Possible directions
-------------------
- When finalization resumes, check that max_generation_from is still an active member. If
  it is not, restart the join or fall back to the local CIB.
- Clear max_generation_from (or restart the join) when that node is lost or fenced during
  the join.
- Do not record a failed sync for a node that is no longer active, or clear its entry when
  its fencing succeeds.

Impact
------
Resources start 30 s later than necessary. After a restart, the fenced node is fenced once
more, so automated recovery needs two attempts. No data was at risk in our tests.

A note on a fix
---------------
I would have liked to send a patch for this myself. However, the code analysis above was
done with the help of an AI assistant, and under the ClusterLabs AI policy
(https://github.com/ClusterLabs/ai-policies) AI-assisted code is not accepted. So I'm
leaving the fix to you and providing the analysis instead. The test setup and the logs are
ours, and I checked the findings against the source. I can rerun the scenario or collect
more logs (crm_report, full journals) if that helps.

Best regards

Michael Riexinger
Head of Cloud Infrastructure Engineering
—
Claranet GmbH
Hanauer Landstraße 196
60314 Frankfurt am Main

https://www.claranet.com/de<https://www.claranet.com/de/>


Sitz der Gesellschaft: Frankfurt am Main, Deutschland
Geschäftsführer Sebastian Vögel, Olaf Fischer
HRB 50381 AG Frankfurt am Main
Vat-ID DE812918694

Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze.
Näheres entnehmen Sie bitte unserer Datenschutzerklärung: https://www.claranet.com/de/datenschutz

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20261103/b804f802/attachment-0001.htm>

More information about the Users mailing list