[ClusterLabs] [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected

Michael Riexinger michael.riexinger at claranet.com
Mon Nov 9 16:10:18 UTC 2026


Hi Ulrich,

thanks for looking into it. I don't think the timeouts explain it in
this case. The node that came back was the fence target, not the DC,
and it came back because the network recovered, not after a reboot:

- gw-1 was the DC before and after the outage; there was no DC change.
- The fence of gw-2 was requested at 15:47:33.525, and gw-2 was back in
  the corosync membership at 15:47:34.342, 0.8 s later, because the
  network outage ended. No fencing timeout can be shorter than that; a
  power-off through the cloud API alone takes several seconds.
- The fence action is a power-off (pcmk_reboot_action=off), so gw-2
  does not reboot by itself. It stayed off until it was started again
  about four minutes later.
- The fence completed normally after 13 s ("OK (Done)" at
  15:47:46.781), well within its timeout.

The 30 s delay comes after that. At 15:47:43 the cluster already logs
"Node gw-2 state is now lost", but at 15:47:46.782 the DC still
finalizes join-2 by syncing the CIB from gw-2, which is powered off by
then. That sync runs into the CIB operation timeout (30 s minimum), and
only then does the DC hold a new election and start the resources. As
far as I can see, that timeout is not something we configure.

The second effect does not depend on timing at all: when gw-2 is
started again at 15:51:44, its join is rejected with "failed to sync
its CIB in join-2", and it is fenced a second time.

I agree that a longer corosync token timeout would have hidden our 7 s
outage, but a longer outage would take the same path.

To make the question concrete, below is a sketch against main
(0584e6dd98fe). It is only meant to illustrate the idea, not as a
submission: it was written and tested with AI assistance (Claude Opus
5.5, xhigh). It does two things: do_dc_join_finalize() restarts the
join when the node with the newest CIB is no longer active, instead of
asking it for its CIB, and finalize_sync_callback() does not record a
failed sync for a node that has already left.

We tried a backport of it to 3.0.0 (Debian 13) in our lab: two nodes
with a quorum device, a test fence agent that powers the node off 10 s
after the request, and a 7-10 s network outage on all three hosts at
once. Without the patch, 3 of 10 runs hit the sequence above (fence in
flight, the fenced node rejoins while the DC delays the join, and the
DC syncs the CIB from it). All 3 showed the 30 s stall, the rejected
rejoin and the second fence. With the patch, 3 of 15 runs hit the same
sequence; to hit it more often, the later runs started the outage at a
random offset. In all 3, the DC restarted the join as soon as the fence
completed, the resources started within 0.1 s, and the fenced node's
later rejoin was accepted without a second fence. There was exactly one
new election per restart, no loop.

The limits: against main the sketch only compiles and passes make
check; the lab ran the 3.0.0 backport. The finalize_sync_callback()
part was never exercised, because the restart always came first. And
we only saw the problem when the surviving node was the DC: in 7 runs
where the fenced node had been DC before the outage, it won the
election after rejoining and nothing went wrong.

--- a/daemons/controld/controld_join_dc.c
+++ b/daemons/controld/controld_join_dc.c
@@ -639,6 +639,25 @@
         return;
     }

+    if (!pcmk__is_set(controld_globals.fsa_input_register, R_HAVE_CIB)) {
+        const pcmk__node_status_t *source =
+            pcmk__search_node_caches(0, max_generation_from, NULL,
+                                     pcmk__node_search_cluster_member);
+
+        if (!pcmk__cluster_is_node_active(source)) {
+            /* The node with the newest CIB is gone (for example, it was fenced
+             * while this join was delayed), so it can't send us its CIB.
+             * Start over with the nodes that are still active instead of
+             * waiting for the sync to time out.
+             */
+            pcmk__warn("Restarting join-%d because %s, which has the newest "
+                       "CIB, is no longer active",
+                       current_join_id, max_generation_from);
+            register_fsa_error(I_ELECTION_DC, NULL);
+            return;
+        }
+    }
+
     if (pcmk__is_set(controld_globals.fsa_input_register, R_HAVE_CIB)) {
         // Send our CIB out to everyone
         sync_from = pcmk__str_copy(controld_globals.cluster->priv->node_name);
@@ -691,7 +710,17 @@
                    sync_from, current_join_id, pcmk_strerror(rc));

         if (rc != -pcmk_err_old_data) {
-            record_failed_sync_node(sync_from, current_join_id);
+            const pcmk__node_status_t *node =
+                pcmk__search_node_caches(0, sync_from, NULL,
+                                         pcmk__node_search_cluster_member);
+
+            /* Don't remember a failure for a node that has already left: its
+             * entry would only be removed when it leaves again, so its next
+             * join would be rejected.
+             */
+            if (pcmk__cluster_is_node_active(node)) {
+                record_failed_sync_node(sync_from, current_join_id);
+            }
         }

         /* restart the whole join process */


Does that match how you would expect the DC to behave? The lab is still
there, so I'm happy to test a fix or anything else you would like to
see. I’d also be happy to send a PR via GitHub if AI assisted code is ok for you.

Regards,


Michael Riexinger
Head of Cloud Infrastructure Engineering
--
michael.riexinger at claranet.com<mailto:michael.riexinger at claranet.com>
Tel: +49 69 408018-261

Claranet GmbH
Hanauer Landstraße 196
60314 Frankfurt am Main

https://www.claranet.com/de<https://www.claranet.com/de/>




Sitz der Gesellschaft: Frankfurt am Main, Deutschland
Geschäftsführer Sebastian Vögel, Olaf Fischer
HRB 50381 AG Frankfurt am Main
Vat-ID DE812918694

Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze.
Näheres entnehmen Sie bitte unserer Datenschutzerklärung: https://www.claranet.com/de/datenschutz




From: Windl, Ulrich <u.windl at ukr.de>
Date: Monday, 9. November 2026 at 14:26
To: Michael Riexinger <michael.riexinger at claranet.com>; Cluster Labs - All topics related to open-source clustering welcomed <users at clusterlabs.org>
Cc: Michael Riexinger <michael.riexinger at de.clara.net>
Subject: RE: [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected

> The special part is the timing:
> the network came back while the fence was in flight, …

It seems the timeouts are inconsistent: The fencing timeout should be shorter than the time the fenced node takes to reboot (and re-join), so that the new DC is elected before the old one comes back.

Kind regards,
Ulrich Windl

From: Michael Riexinger <michael.riexinger at claranet.com>
Sent: Friday, November 6, 2026 9:57 AM
To: Windl, Ulrich <u.windl at ukr.de>; Cluster Labs - All topics related to open-source clustering welcomed <users at clusterlabs.org>
Cc: Michael Riexinger <michael.riexinger at de.clara.net>
Subject: [EXT] Re: [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected

Hi Ulrich,

thanks for the reply. Let me clarify the order, as it is the other way round:

> Why would all nodes lose the network when one node is fenced?

They don't. The network outage comes first and the fencing is its consequence.
In our test environment the virtual switch was restarted on all hypervisors at once, which
cut every node off for about 7 s. We reproduce that by dropping all traffic on both nodes
and the qdevice host for 7 s. Node 1 keeps quorum through the qdevice and fences node 2.

> Things may be different if the node isn't actually fenced, however.

Node 2 was really fenced: the fence agent powers the VM off and confirms the
power state, and the result was OK at 15:47:46.8. The special part is the timing:
the network came back while the fence was in flight, so node 2 rejoined the
corosync membership at 15:47:34 and stayed a member until the power-off took
effect at 15:47:43.

> Usually formation of a new cluster (DC selection) is quite fast and it should
> be finished before a fenced node is restarted.

The DC election is not the slow part, and the restart is not involved in the
delay. Node 1 stays DC throughout. What takes 30 s is the join finalization that
was pending when node 2 rejoined: the DC waits for the fence to complete and
then syncs the CIB from node 2, which is already powered off:

  15:47:46.782 Finalizing join-2 for 1 node (sync'ing CIB from gw-2)
  15:48:16.735 Could not sync CIB from gw-2 in join-2: Timer expired

Only after that timeout are the resources started, 30 s after the fence
completed and while node 2 is still off. The restart of node 2 only matters for
the second symptom: its next join (15:51:44) is rejected because of the failed
sync recorded in join-2, and it is fenced a second time.

So the question remains whether the DC should check again that the sync source
is still an active member when finalization resumes after fencing.


Best regards

Michael Riexinger
Head of Cloud Infrastructure Engineering
--
michael.riexinger at claranet.com<mailto:michael.riexinger at claranet.com>
Tel: +49 69 408018-261

Claranet GmbH
Hanauer Landstraße 196
60314 Frankfurt am Main

https://www.claranet.com/de<https://www.claranet.com/de/>


[https://claranet.email/signatur/icon/xing.png]<https://www.xing.com/pages/claranetgmbh/>   [https://claranet.email/signatur/icon/linkedin.png] <https://www.linkedin.com/company/claranet-deutschland/>    [https://claranet.email/signatur/icon/youtube.png] <https://www.youtube.com/claranetDE/>

[https://claranet.email/signatur/logo/claranet.png]<https://www.claranet.com/de/>

Sitz der Gesellschaft: Frankfurt am Main, Deutschland
Geschäftsführer Sebastian Vögel, Olaf Fischer
HRB 50381 AG Frankfurt am Main
Vat-ID DE812918694

Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze.
Näheres entnehmen Sie bitte unserer Datenschutzerklärung: https://www.claranet.com/de/datenschutz

[https://claranet.email/signatur/banner/sovereignty.png]<https://www.claranet.com/de/challenges/digitale-souveraenitaet/?utm_source=claranet&utm_medium=signatur&utm_campaign=mc_digsov>


From: Windl, Ulrich <u.windl at ukr.de>
Date: Friday, 6. November 2026 at 08:26
To: Cluster Labs - All topics related to open-source clustering welcomed <users at clusterlabs.org>
Cc: Michael Riexinger <michael.riexinger at de.clara.net>
Subject: RE: [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected
Usually formation of a new cluster (DC selection) is quite fast and it should be finished before a fenced node is restarted.
Thinks may be different if the node isn’t actually fenced, however.

Why would all nodes loose the network when one node is fenced?

Kind regards,
Ulrich Windl

From: Users <users-bounces at clusterlabs.org> On Behalf Of Michael Riexinger via Users
Sent: Tuesday, November 3, 2026 10:16 AM
To: users at clusterlabs.org
Cc: Michael Riexinger <michael.riexinger at de.clara.net>
Subject: [EXT] [EXT] [ClusterLabs] [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected

Hi,

I'd like to report what looks like a race in the controller's join handling. Bugzilla is
under maintenance right now, so I'm sending it here; I'm happy to file it there once it's
back up.

Summary
-------
The DC finalizes a join by syncing the CIB from a node it has just fenced. Recovery is
delayed by 30 s, and the fenced node's next join is rejected ("failed to sync its CIB"),
so it gets fenced a second time.

Versions
--------
Pacemaker 3.0.0 (Debian 13 package 3.0.0-2), corosync 3.1.9, corosync-qdevice 3.0.3
(net, ffsplit), sbd 1.5.2 (watchdog only). The code involved looks unchanged in 2.1.6,
3.0.1 and current main.

Setup
-----
Two nodes plus a quorum device. Fencing level 1 powers the node off through the cloud API
(pcmk_reboot_action=off); level 2 is watchdog SBD.

What happens
------------
1. All nodes lose the network for about 7 s.
2. Node 1 (DC) keeps quorum through the quorum device and fences node 2.
3. While the fence is in flight, the network returns and node 2 rejoins the corosync
   membership.
4. The DC delays the join until the fence completes. It then finalizes the join by syncing
   the CIB *from node 2*, which is powered off by then.
5. The sync times out after 30 s. Only then does the DC hold a new election and start the
   resources.
6. When node 2 is started again later, the DC rejects its join because the earlier sync
   failed. Node 2 leaves and is fenced a second time. The start after that joins normally.

Reproduced in 1 of 3 runs with a 7 s outage of all nodes. It needs the network to return
while the fence is in flight.

DC journal (UTC, node 2 = gw-2)
-------------------------------
15:47:33.525 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2
15:47:34.342 corosync [TOTEM ] A new membership (1.4f) was formed. Members joined: 2
15:47:38.870 pacemaker-controld: warning: Delaying join-2 finalization while transition in progress
15:47:43.204 pacemaker-based: notice: Node gw-2 state is now lost
15:47:46.781 pacemaker-fenced: notice: Operation 'reboot' targeting gw-2 by gw-1 for pacemaker-controld at gw-1: OK (Done)
15:47:46.782 pacemaker-controld: notice: Finalizing join-2 for 1 node (sync'ing CIB from gw-2)
15:48:16.735 pacemaker-controld: error: Could not sync CIB from gw-2 in join-2: Timer expired
15:48:16.735 pacemaker-controld: warning: Input I_ELECTION_DC received in state S_FINALIZE_JOIN from finalize_sync_callback
15:48:16.760 pacemaker-schedulerd: notice: Actions: Start ... (resources start only now)
15:51:44.756 pacemaker-controld: error: Rejecting join-3 request from node gw-2 because we failed to sync its CIB in join-2
15:51:46.800 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2

Code pointers (main at d981f262f81644d9faad4fe6be158401fdb53bf8)
-----------------------------------------------------------------
1. The sync source is chosen at offer time and never revalidated.
   do_dc_join_filter_offer() stores the node with the newest CIB in max_generation_from
   (daemons/controld/controld_join_dc.c L542, L568). In our case that was gw-2, whose join
   offer arrived right after it rejoined.
   https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L542

2. Finalization resumes after the fencing transition with the stale source.
   do_dc_join_finalize() stalls while the transition (the fence of gw-2) runs (L634-L639).
   When it resumes, it only checks whether max_generation_from is the local node
   (L628-L632), not whether that node is still an active member, and calls sync_from() on
   a node that is powered off (L668-L671).
   https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L628-L671

3. The 30 s come from the CIB call timeout.
   cib_op_timeout() is max(10 s x (nodes + 1), MIN_CIB_OP_TIMEOUT = 30)
   (daemons/controld/controld_cib.c L203, L214-L220), so 30 s for two nodes.

4. The failure is recorded after the node has already left.
   finalize_sync_callback() calls record_failed_sync_node() for gw-2 (controld_join_dc.c
   L694). The only removals are in peer_update_callback(), when a node changes away from
   member (L175) or loses its controller process (L214)
   (daemons/controld/controld_callbacks.c). Both had already happened for gw-2 at
   15:47:43, before the entry was recorded at 15:48:16.
   https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_callbacks.c#L175

5. The stale entry rejects the next join.
   do_dc_join_filter_offer() rejects gw-2 on its next join (controld_join_dc.c L486-L490).
   The rejection makes gw-2 leave, which removes the entry, so the join after that
   succeeds.

Expected
--------
1. A node that is being fenced, or has just been fenced, in the current transition is never
   chosen as the CIB sync source.
2. A failed sync from a node that has already left does not cause its next join to be
   rejected.

Possible directions
-------------------
- When finalization resumes, check that max_generation_from is still an active member. If
  it is not, restart the join or fall back to the local CIB.
- Clear max_generation_from (or restart the join) when that node is lost or fenced during
  the join.
- Do not record a failed sync for a node that is no longer active, or clear its entry when
  its fencing succeeds.

Impact
------
Resources start 30 s later than necessary. After a restart, the fenced node is fenced once
more, so automated recovery needs two attempts. No data was at risk in our tests.

A note on a fix
---------------
I would have liked to send a patch for this myself. However, the code analysis above was
done with the help of an AI assistant, and under the ClusterLabs AI policy
(https://github.com/ClusterLabs/ai-policies) AI-assisted code is not accepted. So I'm
leaving the fix to you and providing the analysis instead. The test setup and the logs are
ours, and I checked the findings against the source. I can rerun the scenario or collect
more logs (crm_report, full journals) if that helps.

Best regards

Michael Riexinger
Head of Cloud Infrastructure Engineering
—
Claranet GmbH
Hanauer Landstraße 196
60314 Frankfurt am Main

https://www.claranet.com/de<https://www.claranet.com/de/>


Sitz der Gesellschaft: Frankfurt am Main, Deutschland
Geschäftsführer Sebastian Vögel, Olaf Fischer
HRB 50381 AG Frankfurt am Main
Vat-ID DE812918694

Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze.
Näheres entnehmen Sie bitte unserer Datenschutzerklärung: https://www.claranet.com/de/datenschutz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20261109/84a7a736/attachment-0001.htm>


More information about the Users mailing list