<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Ulrich,</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
thanks for looking into it. I don't think the timeouts explain it in</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
this case. The node that came back was the fence target, not the DC,</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
and it came back because the network recovered, not after a reboot:</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
- gw-1 was the DC before and after the outage; there was no DC change.</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
- The fence of gw-2 was requested at 15:47:33.525, and gw-2 was back in</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
the corosync membership at 15:47:34.342, 0.8 s later, because the</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
network outage ended. No fencing timeout can be shorter than that; a</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
power-off through the cloud API alone takes several seconds.</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
- The fence action is a power-off (pcmk_reboot_action=off), so gw-2</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
does not reboot by itself. It stayed off until it was started again</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
about four minutes later.</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
- The fence completed normally after 13 s ("OK (Done)" at</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
15:47:46.781), well within its timeout.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The 30 s delay comes after that. At 15:47:43 the cluster already logs</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
"Node gw-2 state is now lost", but at 15:47:46.782 the DC still</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
finalizes join-2 by syncing the CIB from gw-2, which is powered off by</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
then. That sync runs into the CIB operation timeout (30 s minimum), and</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
only then does the DC hold a new election and start the resources. As</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
far as I can see, that timeout is not something we configure.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The second effect does not depend on timing at all: when gw-2 is</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
started again at 15:51:44, its join is rejected with "failed to sync</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
its CIB in join-2", and it is fenced a second time.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I agree that a longer corosync token timeout would have hidden our 7 s</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
outage, but a longer outage would take the same path.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
To make the question concrete, below is a sketch against main</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
(0584e6dd98fe). It is only meant to illustrate the idea, not as a</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
submission: it was written and tested with AI assistance (Claude Opus</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
5.5, xhigh). It does two things: do_dc_join_finalize() restarts the</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
join when the node with the newest CIB is no longer active, instead of</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
asking it for its CIB, and finalize_sync_callback() does not record a</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
failed sync for a node that has already left.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We tried a backport of it to 3.0.0 (Debian 13) in our lab: two nodes</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
with a quorum device, a test fence agent that powers the node off 10 s</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
after the request, and a 7-10 s network outage on all three hosts at</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
once. Without the patch, 3 of 10 runs hit the sequence above (fence in</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
flight, the fenced node rejoins while the DC delays the join, and the</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
DC syncs the CIB from it). All 3 showed the 30 s stall, the rejected</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
rejoin and the second fence. With the patch, 3 of 15 runs hit the same</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
sequence; to hit it more often, the later runs started the outage at a</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
random offset. In all 3, the DC restarted the join as soon as the fence</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
completed, the resources started within 0.1 s, and the fenced node's</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
later rejoin was accepted without a second fence. There was exactly one</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
new election per restart, no loop.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The limits: against main the sketch only compiles and passes make</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
check; the lab ran the 3.0.0 backport. The finalize_sync_callback()</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
part was never exercised, because the restart always came first. And</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
we only saw the problem when the surviving node was the DC: in 7 runs</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
where the fenced node had been DC before the outage, it won the</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
election after rejoining and nothing went wrong.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--- a/daemons/controld/controld_join_dc.c</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+++ b/daemons/controld/controld_join_dc.c</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
@@ -639,6 +639,25 @@</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
return;</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
}</div>
<div style="direction: ltr; font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ if (!pcmk__is_set(controld_globals.fsa_input_register, R_HAVE_CIB)) {</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ const pcmk__node_status_t *source =</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ pcmk__search_node_caches(0, max_generation_from, NULL,</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ pcmk__node_search_cluster_member);</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ if (!pcmk__cluster_is_node_active(source)) {</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ /* The node with the newest CIB is gone (for example, it was fenced</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ * while this join was delayed), so it can't send us its CIB.</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ * Start over with the nodes that are still active instead of</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ * waiting for the sync to time out.</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ */</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ pcmk__warn("Restarting join-%d because %s, which has the newest "</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ "CIB, is no longer active",</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ current_join_id, max_generation_from);</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ register_fsa_error(I_ELECTION_DC, NULL);</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ return;</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ }</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ }</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
if (pcmk__is_set(controld_globals.fsa_input_register, R_HAVE_CIB)) {</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
// Send our CIB out to everyone</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
sync_from = pcmk__str_copy(controld_globals.cluster->priv->node_name);</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
@@ -691,7 +710,17 @@</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
sync_from, current_join_id, pcmk_strerror(rc));</div>
<div style="direction: ltr; font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
if (rc != -pcmk_err_old_data) {</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
- record_failed_sync_node(sync_from, current_join_id);</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ const pcmk__node_status_t *node =</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ pcmk__search_node_caches(0, sync_from, NULL,</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ pcmk__node_search_cluster_member);</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ /* Don't remember a failure for a node that has already left: its</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ * entry would only be removed when it leaves again, so its next</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ * join would be rejected.</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ */</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ if (pcmk__cluster_is_node_active(node)) {</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ record_failed_sync_node(sync_from, current_join_id);</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
+ }</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
}</div>
<div style="direction: ltr; font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Courier New, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
/* restart the whole join process */</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Does that match how you would expect the DC to behave? The lab is still</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
there, so I'm happy to test a fix or anything else you would like to</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
see. I’d also be happy to send a PR via GitHub if AI assisted code is ok for you.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Regards,</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
<br>
<span style="font-family: -webkit-standard; text-transform: none;">Michael Riexinger</span><br>
<span style="text-transform: none;"><b>Head of Cloud Infrastructure Engineering</b></span><br>
<span style="font-family: -webkit-standard; text-transform: none;">-- </span><br>
<span style="text-transform: none;"><a href="mailto:michael.riexinger@claranet.com" data-outlook-id="02494fd5-cbcc-4766-8449-66af2049e5c0">michael.riexinger@claranet.com</a><br>
Tel: +49 69 408018-261<br>
<br>
Claranet GmbH<br>
Hanauer Landstraße 196<br>
60314 Frankfurt am Main<br>
<br>
<a href="https://www.claranet.com/de/" data-outlook-id="e7e7f21a-8895-4921-9643-5b5ebb68ca7e">https://www.claranet.com/de</a><br>
<br>
<br>
<br>
<br>
Sitz der Gesellschaft: Frankfurt am Main, Deutschland<br>
Geschäftsführer Sebastian Vögel, Olaf Fischer<br>
HRB 50381 AG Frankfurt am Main<br>
Vat-ID DE812918694<br>
<br>
Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze.<br>
Näheres entnehmen Sie bitte unserer Datenschutzerklärung: <a href="https://www.claranet.com/de/datenschutz" data-outlook-id="51e107c1-8b43-4d6e-8417-c42631fddb17">
https://www.claranet.com/de/datenschutz</a><br>
<br>
</span><br>
</div>
<div id="ms-outlook-mobile-signature" dir="ltr">
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="mail-editor-reference-message-container">
<div style="padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor;">
<div style="text-align: left; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>Windl, Ulrich <u.windl@ukr.de><br>
<b>Date: </b>Monday, 9. November 2026 at 14:26<br>
<b>To: </b>Michael Riexinger <michael.riexinger@claranet.com>; Cluster Labs - All topics related to open-source clustering welcomed <users@clusterlabs.org><br>
<b>Cc: </b>Michael Riexinger <michael.riexinger@de.clara.net><br>
<b>Subject: </b>RE: [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected<br>
<br>
</div>
</div>
<div id="mail-editor-reference-message-body">
<div class="ms-outlook-mobile-reference-message skipProofing">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
</div>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">> The special part is the timing:</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">> the network came back while the fence was in flight, …</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">It seems the timeouts are inconsistent: The fencing timeout should be shorter than the time the fenced node takes to reboot (and re-join), so that the new DC is elected before the old one comes back.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(31, 73, 125);">Kind regards,</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(31, 73, 125);">Ulrich Windl</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
</p>
<div style="padding: 0cm 0cm 0cm 4pt; border-width: medium medium medium 1.5pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor blue;">
<div style="padding: 3pt 0cm 0cm; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(225, 225, 225) currentcolor currentcolor;">
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Calibri, sans-serif; font-size: 11pt;"><b>From:</b> Michael Riexinger <michael.riexinger@claranet.com>
<br>
<b>Sent:</b> Friday, November 6, 2026 9:57 AM<br>
<b>To:</b> Windl, Ulrich <u.windl@ukr.de>; Cluster Labs - All topics related to open-source clustering welcomed <users@clusterlabs.org><br>
<b>Cc:</b> Michael Riexinger <michael.riexinger@de.clara.net><br>
<b>Subject:</b> [EXT] Re: [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected</span></p>
</div>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
</p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Hi Ulrich,</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">thanks for the reply. Let me clarify the order, as it is the other way round:</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">> Why would all nodes lose the network when one node is fenced?</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">They don't. The network outage comes first and the fencing is its consequence.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">In our test environment the virtual switch was restarted on all hypervisors at once, which</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">cut every node off for about 7 s. We reproduce that by dropping all traffic on both nodes </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">and the qdevice host for 7 s. Node 1 keeps quorum through the qdevice and fences node 2.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">> Things may be different if the node isn't actually fenced, however.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Node 2 was really fenced: the fence agent powers the VM off and confirms the</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">power state, and the result was OK at 15:47:46.8. The special part is the timing:</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">the network came back while the fence was in flight, so node 2 rejoined the</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">corosync membership at 15:47:34 and stayed a member until the power-off took</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">effect at 15:47:43.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">> Usually formation of a new cluster (DC selection) is quite fast and it should</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">> be finished before a fenced node is restarted.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">The DC election is not the slow part, and the restart is not involved in the</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">delay. Node 1 stays DC throughout. What takes 30 s is the join finalization that</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">was pending when node 2 rejoined: the DC waits for the fence to complete and</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">then syncs the CIB from node 2, which is already powered off:</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> 15:47:46.782 Finalizing join-2 for 1 node (sync'ing CIB from gw-2)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> 15:48:16.735 Could not sync CIB from gw-2 in join-2: Timer expired</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Only after that timeout are the resources started, 30 s after the fence</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">completed and while node 2 is still off. The restart of node 2 only matters for</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">the second symptom: its next join (15:51:44) is rejected because of the failed</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">sync recorded in join-2, and it is fenced a second time.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">So the question remains whether the DC should check again that the sync source</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">is still an active member when finalization resumes after fencing.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Best regards</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<div id="ms-outlook-mobile-body-separator-line">
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: -webkit-standard, serif; font-size: 12pt; color: black;">Michael Riexinger</span><span style="font-size: 12pt; color: black;"><br>
<b>Head of Cloud Infrastructure Engineering</b><br>
</span><span style="font-family: -webkit-standard, serif; font-size: 12pt; color: black;">-- </span><span style="font-size: 12pt; color: black;"><br>
</span><span style="font-size: 12pt; color: blue;"><a href="mailto:michael.riexinger@claranet.com" data-outlook-id="3a0dbc10-2bd9-41a0-bf03-0d5e0310660c" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u>michael.riexinger@claranet.com</u></a></span><span style="font-size: 12pt; color: black;"><br>
Tel: +49 69 408018-261<br>
<br>
Claranet GmbH<br>
Hanauer Landstraße 196<br>
60314 Frankfurt am Main<br>
<br>
</span><span style="font-size: 12pt; color: blue;"><a href="https://www.claranet.com/de/" data-outlook-id="44ec2453-f927-407f-bf42-87f8ed617ce6" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u>https://www.claranet.com/de</u></a></span><span style="font-size: 12pt; color: black;"><br>
<br>
<br>
</span><span style="font-size: 12pt; color: blue;"><a href="https://www.xing.com/pages/claranetgmbh/" data-outlook-id="e051430f-00ed-4cf9-ab37-6a5aa30fbb25" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u><img src="https://claranet.email/signatur/icon/xing.png" id="_x0000_i1029" width="18" height="19" style="width: 0.1979in; height: 0.2083in; margin-top: 0px; margin-bottom: 0px;"></u></a></span><span style="font-size: 12pt; color: black;">
</span><span style="font-size: 12pt; color: blue;"><a href="https://www.linkedin.com/company/claranet-deutschland/" data-outlook-id="5cc08e7f-0165-4302-bec8-99d0b658a133" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u><img src="https://claranet.email/signatur/icon/linkedin.png" id="_x0000_i1028" width="19" height="19" style="width: 0.2083in; height: 0.2083in; margin-top: 0px; margin-bottom: 0px;"></u></a></span><span style="font-size: 12pt; color: black;">
</span><span style="font-size: 12pt; color: blue;"><a href="https://www.youtube.com/claranetDE/" data-outlook-id="dc68cc85-7c10-4884-820c-52045a7b8476" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u><img src="https://claranet.email/signatur/icon/youtube.png" id="_x0000_i1027" width="28" height="19" style="width: 0.302in; height: 0.2083in; margin-top: 0px; margin-bottom: 0px;"></u></a></span><span style="font-size: 12pt; color: black;"><br>
<br>
</span><span style="font-size: 12pt; color: blue;"><a href="https://www.claranet.com/de/" data-outlook-id="798659a3-5234-4a88-8cc0-7928a4961db7" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u><img src="https://claranet.email/signatur/logo/claranet.png" id="_x0000_i1026" width="285" height="60" style="width: 2.9791in; height: 0.625in; margin-top: 0px; margin-bottom: 0px;"></u></a></span><span style="font-size: 12pt; color: black;"><br>
<br>
Sitz der Gesellschaft: Frankfurt am Main, Deutschland<br>
Geschäftsführer Sebastian Vögel, Olaf Fischer<br>
HRB 50381 AG Frankfurt am Main<br>
Vat-ID DE812918694<br>
<br>
Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze.<br>
Näheres entnehmen Sie bitte unserer Datenschutzerklärung: </span><span style="font-size: 12pt; color: blue;"><a href="https://www.claranet.com/de/datenschutz" data-outlook-id="05608dc4-3788-4ffa-9a9a-122e3687f898" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u>https://www.claranet.com/de/datenschutz</u></a></span><span style="font-size: 12pt; color: black;"><br>
<br>
</span><span style="font-size: 12pt; color: blue;"><a href="https://www.claranet.com/de/challenges/digitale-souveraenitaet/?utm_source=claranet&utm_medium=signatur&utm_campaign=mc_digsov" data-outlook-id="e3b9d68b-4620-4ea9-9c8f-6d3f9d9ff425" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u><img src="https://claranet.email/signatur/banner/sovereignty.png" id="_x0000_i1025" width="450" height="74" style="width: 4.6875in; height: 0.7812in; margin-top: 0px; margin-bottom: 0px;"></u></a></span></p>
</div>
<div id="ms-outlook-mobile-signature" dir="ltr">
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<div id="mail-editor-reference-message-container">
<div style="padding: 3pt 0cm 0cm; border-width: 1pt medium medium; border-style: solid none none; border-color: currentcolor;">
<p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"><b>From: </b>Windl, Ulrich <u.windl@ukr.de><br>
<b>Date: </b>Friday, 6. November 2026 at 08:26<br>
<b>To: </b>Cluster Labs - All topics related to open-source clustering welcomed <users@clusterlabs.org><br>
<b>Cc: </b>Michael Riexinger <michael.riexinger@de.clara.net><br>
<b>Subject: </b>RE: [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected</span></p>
</div>
<div id="mail-editor-reference-message-body">
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);">Usually formation of a new cluster (DC selection) is quite fast and it should be finished before a fenced node is restarted.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);">Thinks may be different if the node isn’t actually fenced, however.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);">Why would all nodes loose the network when one node is fenced?</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(31, 73, 125);">Kind regards,</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(31, 73, 125);">Ulrich Windl</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
</p>
<div style="padding: 0cm 0cm 0cm 4pt; border-width: medium medium medium 1.5pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor blue;">
<div style="padding: 3pt 0cm 0cm; border-width: 1pt medium medium; border-style: solid none none; border-color: currentcolor;">
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Calibri, sans-serif; font-size: 11pt;"><b>From:</b> Users <users-bounces@clusterlabs.org>
<b>On Behalf Of </b>Michael Riexinger via Users<br>
<b>Sent:</b> Tuesday, November 3, 2026 10:16 AM<br>
<b>To:</b> users@clusterlabs.org<br>
<b>Cc:</b> Michael Riexinger <michael.riexinger@de.clara.net><br>
<b>Subject:</b> [EXT] [EXT] [ClusterLabs] [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected</span></p>
</div>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
</p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Hi,</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">I'd like to report what looks like a race in the controller's join handling. Bugzilla is</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">under maintenance right now, so I'm sending it here; I'm happy to file it there once it's</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">back up.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Summary</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">-------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">The DC finalizes a join by syncing the CIB from a node it has just fenced. Recovery is</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">delayed by 30 s, and the fenced node's next join is rejected ("failed to sync its CIB"),</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">so it gets fenced a second time.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Versions</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">--------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Pacemaker 3.0.0 (Debian 13 package 3.0.0-2), corosync 3.1.9, corosync-qdevice 3.0.3</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">(net, ffsplit), sbd 1.5.2 (watchdog only). The code involved looks unchanged in 2.1.6,</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">3.0.1 and current main.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Setup</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">-----</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Two nodes plus a quorum device. Fencing level 1 powers the node off through the cloud API</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">(pcmk_reboot_action=off); level 2 is watchdog SBD.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">What happens</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">------------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">1. All nodes lose the network for about 7 s.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">2. Node 1 (DC) keeps quorum through the quorum device and fences node 2.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">3. While the fence is in flight, the network returns and node 2 rejoins the corosync</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> membership.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">4. The DC delays the join until the fence completes. It then finalizes the join by syncing</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> the CIB *from node 2*, which is powered off by then.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">5. The sync times out after 30 s. Only then does the DC hold a new election and start the</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> resources.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">6. When node 2 is started again later, the DC rejects its join because the earlier sync</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> failed. Node 2 leaves and is fenced a second time. The start after that joins normally.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Reproduced in 1 of 3 runs with a 7 s outage of all nodes. It needs the network to return</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">while the fence is in flight.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">DC journal (UTC, node 2 = gw-2)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">-------------------------------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:33.525 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:34.342 corosync [TOTEM ] A new membership (1.4f) was formed. Members joined: 2</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:38.870 pacemaker-controld: warning: Delaying join-2 finalization while transition in progress</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:43.204 pacemaker-based: notice: Node gw-2 state is now lost</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:46.781 pacemaker-fenced: notice: Operation 'reboot' targeting gw-2 by gw-1 for pacemaker-controld@gw-1: OK (Done)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:46.782 pacemaker-controld: notice: Finalizing join-2 for 1 node (sync'ing CIB from gw-2)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:48:16.735 pacemaker-controld: error: Could not sync CIB from gw-2 in join-2: Timer expired</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:48:16.735 pacemaker-controld: warning: Input I_ELECTION_DC received in state S_FINALIZE_JOIN from finalize_sync_callback</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:48:16.760 pacemaker-schedulerd: notice: Actions: Start ... (resources start only now)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:51:44.756 pacemaker-controld: error: Rejecting join-3 request from node gw-2 because we failed to sync its CIB in join-2</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:51:46.800 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Code pointers (main at d981f262f81644d9faad4fe6be158401fdb53bf8)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">-----------------------------------------------------------------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">1. The sync source is chosen at offer time and never revalidated.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> do_dc_join_filter_offer() stores the node with the newest CIB in max_generation_from</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> (daemons/controld/controld_join_dc.c L542, L568). In our case that was gw-2, whose join</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> offer arrived right after it rejoined.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L542</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">2. Finalization resumes after the fencing transition with the stale source.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> do_dc_join_finalize() stalls while the transition (the fence of gw-2) runs (L634-L639).</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> When it resumes, it only checks whether max_generation_from is the local node</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> (L628-L632), not whether that node is still an active member, and calls sync_from() on</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> a node that is powered off (L668-L671).</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L628-L671</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">3. The 30 s come from the CIB call timeout.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> cib_op_timeout() is max(10 s x (nodes + 1), MIN_CIB_OP_TIMEOUT = 30)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> (daemons/controld/controld_cib.c L203, L214-L220), so 30 s for two nodes.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">4. The failure is recorded after the node has already left.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> finalize_sync_callback() calls record_failed_sync_node() for gw-2 (controld_join_dc.c</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> L694). The only removals are in peer_update_callback(), when a node changes away from</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> member (L175) or loses its controller process (L214)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> (daemons/controld/controld_callbacks.c). Both had already happened for gw-2 at</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> 15:47:43, before the entry was recorded at 15:48:16.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_callbacks.c#L175</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">5. The stale entry rejects the next join.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> do_dc_join_filter_offer() rejects gw-2 on its next join (controld_join_dc.c L486-L490).</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> The rejection makes gw-2 leave, which removes the entry, so the join after that</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> succeeds.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Expected</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">--------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">1. A node that is being fenced, or has just been fenced, in the current transition is never</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> chosen as the CIB sync source.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">2. A failed sync from a node that has already left does not cause its next join to be</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> rejected.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Possible directions</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">-------------------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">- When finalization resumes, check that max_generation_from is still an active member. If</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> it is not, restart the join or fall back to the local CIB.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">- Clear max_generation_from (or restart the join) when that node is lost or fenced during</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> the join.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">- Do not record a failed sync for a node that is no longer active, or clear its entry when</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> its fencing succeeds.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Impact</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Resources start 30 s later than necessary. After a restart, the fenced node is fenced once</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">more, so automated recovery needs two attempts. No data was at risk in our tests.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">A note on a fix</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">---------------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">I would have liked to send a patch for this myself. However, the code analysis above was</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">done with the help of an AI assistant, and under the ClusterLabs AI policy</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">(https://github.com/ClusterLabs/ai-policies) AI-assisted code is not accepted. So I'm</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">leaving the fix to you and providing the analysis instead. The test setup and the logs are</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">ours, and I checked the findings against the source. I can rerun the scenario or collect</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">more logs (crm_report, full journals) if that helps.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<div id="ms-outlook-mobile-signature">
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: -webkit-standard, serif; font-size: 12pt; color: black;">Best regards</span><span style="font-size: 12pt; color: black;"><br>
<br>
</span><span style="font-family: -webkit-standard, serif; font-size: 12pt; color: black;">Michael Riexinger</span><span style="font-size: 12pt; color: black;"><br>
Head of Cloud Infrastructure Engineering<br>
</span><span style="font-family: -webkit-standard, serif; font-size: 12pt;">— </span><span style="font-size: 12pt; color: black;"><br>
Claranet GmbH<br>
Hanauer Landstraße 196<br>
60314 Frankfurt am Main<br>
<br>
</span><span style="font-size: 12pt; color: blue;"><a href="https://www.claranet.com/de/" data-outlook-id="b80eb3de-f7cd-43cf-8d2e-b62c9f348da8" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u>https://www.claranet.com/de</u></a></span><span style="font-size: 12pt; color: black;"><br>
<br>
<br>
Sitz der Gesellschaft: Frankfurt am Main, Deutschland<br>
Geschäftsführer Sebastian Vögel, Olaf Fischer<br>
HRB 50381 AG Frankfurt am Main<br>
Vat-ID DE812918694<br>
<br>
Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze.<br>
Näheres entnehmen Sie bitte unserer Datenschutzerklärung: </span><span style="font-size: 12pt; color: blue;"><a href="https://www.claranet.com/de/datenschutz" data-outlook-id="df770ce2-af5b-4b10-8512-0742f341f058" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u>https://www.claranet.com/de/datenschutz</u></a></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>