<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Ulrich,</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
thanks for the reply. Let me clarify the order, as it is the other way round:</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> Why would all nodes lose the network when one node is fenced?</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
They don't. The network outage comes first and the fencing is its consequence.</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In our test environment the virtual switch was restarted on all hypervisors at once, which</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
cut every node off for about 7 s. We reproduce that by dropping all traffic on both nodes </div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
and the qdevice host for 7 s. Node 1 keeps quorum through the qdevice and fences node 2.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> Things may be different if the node isn't actually fenced, however.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Node 2 was really fenced: the fence agent powers the VM off and confirms the</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
power state, and the result was OK at 15:47:46.8. The special part is the timing:</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
the network came back while the fence was in flight, so node 2 rejoined the</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
corosync membership at 15:47:34 and stayed a member until the power-off took</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
effect at 15:47:43.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> Usually formation of a new cluster (DC selection) is quite fast and it should</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
> be finished before a fenced node is restarted.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The DC election is not the slow part, and the restart is not involved in the</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
delay. Node 1 stays DC throughout. What takes 30 s is the join finalization that</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
was pending when node 2 rejoined: the DC waits for the fence to complete and</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
then syncs the CIB from node 2, which is already powered off:</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
15:47:46.782 Finalizing join-2 for 1 node (sync'ing CIB from gw-2)</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
15:48:16.735 Could not sync CIB from gw-2 in join-2: Timer expired</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Only after that timeout are the resources started, 30 s after the fence</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
completed and while node 2 is still off. The restart of node 2 only matters for</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
the second symptom: its next join (15:51:44) is rejected because of the failed</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
sync recorded in join-2, and it is fenced a second time.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
So the question remains whether the DC should check again that the sync source</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
is still an active member when finalization resumes after fencing.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Best regards</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" dir="ltr">
<br>
</div>
<div id="ms-outlook-mobile-body-separator-line" dir="ltr">
<div style="font-size: 12pt; color: rgb(0, 0, 0);"><span style="font-family: -webkit-standard;"></span><span style="font-family: -webkit-standard; text-transform: none;">Michael Riexinger</span><span style="font-family: Aptos, Arial, Helvetica, sans-serif;"><br>
</span><span style="font-family: Aptos, Arial, Helvetica, sans-serif; text-transform: none;"><b>Head of Cloud Infrastructure Engineering</b></span><span style="font-family: Aptos, Arial, Helvetica, sans-serif;"><br>
</span><span style="font-family: -webkit-standard; text-transform: none;">-- </span><span style="font-family: Aptos, Arial, Helvetica, sans-serif;"><br>
</span><span style="font-family: Aptos, Arial, Helvetica, sans-serif; text-transform: none;"><a href="mailto:michael.riexinger@claranet.com" data-outlook-id="02494fd5-cbcc-4766-8449-66af2049e5c0">michael.riexinger@claranet.com</a><br>
Tel: +49 69 408018-261<br>
<br>
Claranet GmbH<br>
Hanauer Landstraße 196<br>
60314 Frankfurt am Main<br>
<br>
<a href="https://www.claranet.com/de/" data-outlook-id="e7e7f21a-8895-4921-9643-5b5ebb68ca7e">https://www.claranet.com/de</a><br>
<br>
<br>
<a href="https://www.xing.com/pages/claranetgmbh/" data-outlook-id="5dd79b9b-630e-486a-afb2-ecb37ac495fc"><img src="https://claranet.email/signatur/icon/xing.png" id="img-139e5506-a14a-43b5-8625-7674f5bab5fa" height="20" style="height: 20px; max-width: 100%;"></a>
<a href="https://www.linkedin.com/company/claranet-deutschland/" data-outlook-id="518ba5a5-d871-4254-82bf-29731ebf31fa"><img src="https://claranet.email/signatur/icon/linkedin.png" id="img-7bd4fdd5-4d1c-4c6f-a336-dfca1cba4079" height="20" style="height: 20px; max-width: 100%;"></a>
<a href="https://www.youtube.com/claranetDE/" data-outlook-id="f3d3c681-8d76-454d-b4c9-fbd7139990a8"><img src="https://claranet.email/signatur/icon/youtube.png" id="img-383bf1d6-eb70-4383-9c07-a7e723f6e196" height="20" style="height: 20px; max-width: 100%;"></a><br>
<br>
<a href="https://www.claranet.com/de/" data-outlook-id="9852b853-0784-4e10-84bb-cd8f045858d1"><img src="https://claranet.email/signatur/logo/claranet.png" id="img-44597a9e-0ce1-4d64-9bc1-41df20034cdc" height="60" style="height: 60px; max-width: 100%;"></a><br>
<br>
Sitz der Gesellschaft: Frankfurt am Main, Deutschland<br>
Geschäftsführer Sebastian Vögel, Olaf Fischer<br>
HRB 50381 AG Frankfurt am Main<br>
Vat-ID DE812918694<br>
<br>
Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze.<br>
Näheres entnehmen Sie bitte unserer Datenschutzerklärung: <a href="https://www.claranet.com/de/datenschutz" data-outlook-id="51e107c1-8b43-4d6e-8417-c42631fddb17">
https://www.claranet.com/de/datenschutz</a><br>
<br>
<a href="https://www.claranet.com/de/challenges/digitale-souveraenitaet/?utm_source=claranet&utm_medium=signatur&utm_campaign=mc_digsov" data-outlook-id="46dae5f7-fb79-4635-b2d8-54caab7b2ed8"><img src="https://claranet.email/signatur/banner/sovereignty.png" id="img-1f28e08a-e706-47aa-9ca0-060052ea722a" width="450" style="width: 450px; max-width: 100%;"></a></span></div>
</div>
<div id="ms-outlook-mobile-signature" dir="ltr">
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="mail-editor-reference-message-container">
<div style="padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor;">
<div style="text-align: left; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>Windl, Ulrich <u.windl@ukr.de><br>
<b>Date: </b>Friday, 6. November 2026 at 08:26<br>
<b>To: </b>Cluster Labs - All topics related to open-source clustering welcomed <users@clusterlabs.org><br>
<b>Cc: </b>Michael Riexinger <michael.riexinger@de.clara.net><br>
<b>Subject: </b>RE: [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected<br>
<br>
</div>
</div>
<div id="mail-editor-reference-message-body">
<div class="ms-outlook-mobile-reference-message skipProofing">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
</div>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);">Usually formation of a new cluster (DC selection) is quite fast and it should be finished before a fenced node is restarted.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);">Thinks may be different if the node isn’t actually fenced, however.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);">Why would all nodes loose the network when one node is fenced?</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Segoe UI, sans-serif; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(31, 73, 125);">Kind regards,</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(31, 73, 125);">Ulrich Windl</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
</p>
<div style="padding: 0cm 0cm 0cm 4pt; border-width: medium medium medium 1.5pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor blue;">
<div style="padding: 3pt 0cm 0cm; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(225, 225, 225) currentcolor currentcolor;">
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: Calibri, sans-serif; font-size: 11pt;"><b>From:</b> Users <users-bounces@clusterlabs.org>
<b>On Behalf Of </b>Michael Riexinger via Users<br>
<b>Sent:</b> Tuesday, November 3, 2026 10:16 AM<br>
<b>To:</b> users@clusterlabs.org<br>
<b>Cc:</b> Michael Riexinger <michael.riexinger@de.clara.net><br>
<b>Subject:</b> [EXT] [EXT] [ClusterLabs] [Pacemaker] DC syncs CIB from a node it just fenced: 30 s recovery delay, next join rejected</span></p>
</div>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
</p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Hi,</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">I'd like to report what looks like a race in the controller's join handling. Bugzilla is</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">under maintenance right now, so I'm sending it here; I'm happy to file it there once it's</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">back up.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Summary</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">-------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">The DC finalizes a join by syncing the CIB from a node it has just fenced. Recovery is</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">delayed by 30 s, and the fenced node's next join is rejected ("failed to sync its CIB"),</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">so it gets fenced a second time.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Versions</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">--------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Pacemaker 3.0.0 (Debian 13 package 3.0.0-2), corosync 3.1.9, corosync-qdevice 3.0.3</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">(net, ffsplit), sbd 1.5.2 (watchdog only). The code involved looks unchanged in 2.1.6,</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">3.0.1 and current main.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Setup</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">-----</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Two nodes plus a quorum device. Fencing level 1 powers the node off through the cloud API</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">(pcmk_reboot_action=off); level 2 is watchdog SBD.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">What happens</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">------------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">1. All nodes lose the network for about 7 s.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">2. Node 1 (DC) keeps quorum through the quorum device and fences node 2.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">3. While the fence is in flight, the network returns and node 2 rejoins the corosync</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> membership.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">4. The DC delays the join until the fence completes. It then finalizes the join by syncing</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> the CIB *from node 2*, which is powered off by then.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">5. The sync times out after 30 s. Only then does the DC hold a new election and start the</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> resources.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">6. When node 2 is started again later, the DC rejects its join because the earlier sync</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> failed. Node 2 leaves and is fenced a second time. The start after that joins normally.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Reproduced in 1 of 3 runs with a 7 s outage of all nodes. It needs the network to return</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">while the fence is in flight.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">DC journal (UTC, node 2 = gw-2)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">-------------------------------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:33.525 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:34.342 corosync [TOTEM ] A new membership (1.4f) was formed. Members joined: 2</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:38.870 pacemaker-controld: warning: Delaying join-2 finalization while transition in progress</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:43.204 pacemaker-based: notice: Node gw-2 state is now lost</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:46.781 pacemaker-fenced: notice: Operation 'reboot' targeting gw-2 by gw-1 for pacemaker-controld@gw-1: OK (Done)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:47:46.782 pacemaker-controld: notice: Finalizing join-2 for 1 node (sync'ing CIB from gw-2)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:48:16.735 pacemaker-controld: error: Could not sync CIB from gw-2 in join-2: Timer expired</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:48:16.735 pacemaker-controld: warning: Input I_ELECTION_DC received in state S_FINALIZE_JOIN from finalize_sync_callback</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:48:16.760 pacemaker-schedulerd: notice: Actions: Start ... (resources start only now)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:51:44.756 pacemaker-controld: error: Rejecting join-3 request from node gw-2 because we failed to sync its CIB in join-2</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">15:51:46.800 pacemaker-controld: notice: Requesting fencing (reboot) targeting node gw-2</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Code pointers (main at d981f262f81644d9faad4fe6be158401fdb53bf8)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">-----------------------------------------------------------------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">1. The sync source is chosen at offer time and never revalidated.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> do_dc_join_filter_offer() stores the node with the newest CIB in max_generation_from</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> (daemons/controld/controld_join_dc.c L542, L568). In our case that was gw-2, whose join</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> offer arrived right after it rejoined.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L542</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">2. Finalization resumes after the fencing transition with the stale source.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> do_dc_join_finalize() stalls while the transition (the fence of gw-2) runs (L634-L639).</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> When it resumes, it only checks whether max_generation_from is the local node</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> (L628-L632), not whether that node is still an active member, and calls sync_from() on</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> a node that is powered off (L668-L671).</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_join_dc.c#L628-L671</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">3. The 30 s come from the CIB call timeout.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> cib_op_timeout() is max(10 s x (nodes + 1), MIN_CIB_OP_TIMEOUT = 30)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> (daemons/controld/controld_cib.c L203, L214-L220), so 30 s for two nodes.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">4. The failure is recorded after the node has already left.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> finalize_sync_callback() calls record_failed_sync_node() for gw-2 (controld_join_dc.c</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> L694). The only removals are in peer_update_callback(), when a node changes away from</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> member (L175) or loses its controller process (L214)</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> (daemons/controld/controld_callbacks.c). Both had already happened for gw-2 at</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> 15:47:43, before the entry was recorded at 15:48:16.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> https://github.com/ClusterLabs/pacemaker/blob/d981f262f81644d9faad4fe6be158401fdb53bf8/daemons/controld/controld_callbacks.c#L175</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">5. The stale entry rejects the next join.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> do_dc_join_filter_offer() rejects gw-2 on its next join (controld_join_dc.c L486-L490).</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> The rejection makes gw-2 leave, which removes the entry, so the join after that</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> succeeds.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Expected</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">--------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">1. A node that is being fenced, or has just been fenced, in the current transition is never</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> chosen as the CIB sync source.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">2. A failed sync from a node that has already left does not cause its next join to be</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> rejected.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Possible directions</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">-------------------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">- When finalization resumes, check that max_generation_from is still an active member. If</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> it is not, restart the join or fall back to the local CIB.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">- Clear max_generation_from (or restart the join) when that node is lost or fenced during</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> the join.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">- Do not record a failed sync for a node that is no longer active, or clear its entry when</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> its fencing succeeds.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Impact</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">Resources start 30 s later than necessary. After a restart, the fenced node is fenced once</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">more, so automated recovery needs two attempts. No data was at risk in our tests.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">A note on a fix</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">---------------</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">I would have liked to send a patch for this myself. However, the code analysis above was</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">done with the help of an AI assistant, and under the ClusterLabs AI policy</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">(https://github.com/ClusterLabs/ai-policies) AI-assisted code is not accepted. So I'm</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">leaving the fix to you and providing the analysis instead. The test setup and the logs are</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">ours, and I checked the findings against the source. I can rerun the scenario or collect</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;">more logs (crm_report, full journals) if that helps.</span></p>
<p class="MsoNormal" style="margin: 0cm; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-size: 12pt; color: black;"> </span></p>
<div id="ms-outlook-mobile-signature" dir="ltr">
<p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-family: Aptos, serif; font-size: 10pt;">
<span style="font-family: -webkit-standard, serif; font-size: 12pt; color: black;">Best regards</span><span style="font-size: 12pt; color: black;"><br>
<br>
</span><span style="font-family: -webkit-standard, serif; font-size: 12pt; color: black;">Michael Riexinger</span><span style="font-size: 12pt; color: black;"><br>
Head of Cloud Infrastructure Engineering<br>
</span><span style="font-family: -webkit-standard, serif; font-size: 12pt;">— </span><span style="font-size: 12pt; color: black;"><br>
Claranet GmbH<br>
Hanauer Landstraße 196<br>
60314 Frankfurt am Main<br>
<br>
</span><span style="font-size: 12pt; color: blue;"><a href="https://www.claranet.com/de/" data-outlook-id="fef9573d-2261-425a-aee4-da14291ed5f8" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u>https://www.claranet.com/de</u></a></span><span style="font-size: 12pt; color: black;"><br>
<br>
<br>
Sitz der Gesellschaft: Frankfurt am Main, Deutschland<br>
Geschäftsführer Sebastian Vögel, Olaf Fischer<br>
HRB 50381 AG Frankfurt am Main<br>
Vat-ID DE812918694<br>
<br>
Claranet nimmt den Schutz Ihrer personenbezogenen Daten sehr ernst und hält sich strikt an die Regeln der Datenschutzgesetze.<br>
Näheres entnehmen Sie bitte unserer Datenschutzerklärung: </span><span style="font-size: 12pt; color: blue;"><a href="https://www.claranet.com/de/datenschutz" data-outlook-id="3ead0541-f9de-45cf-9569-3abc54beead2" style="color: blue; margin-top: 0px; margin-bottom: 0px;"><u>https://www.claranet.com/de/datenschutz</u></a></span></p>
</div>
</div>
</div>
</div>
</body>
</html>