<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.m3416958625089056106msolistparagraph, li.m3416958625089056106msolistparagraph, div.m3416958625089056106msolistparagraph
{mso-style-name:m_3416958625089056106msolistparagraph;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.m3416958625089056106gmailsignatureprefix
{mso-style-name:m_3416958625089056106gmailsignatureprefix;}
span.gmailsignatureprefix
{mso-style-name:gmail_signature_prefix;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:30813885;
mso-list-template-ids:1715000586;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Just wondering: Did you have a working hardware watchdog configured? Wouldn’t it have fenced your node then?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Kind regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Ulrich Windl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Users <users-bounces@clusterlabs.org>
<b>On Behalf Of </b>Brian Cosgrove<br>
<b>Sent:</b> Tuesday, August 11, 2026 5:28 PM<br>
<b>To:</b> Cluster Labs - All topics related to open-source clustering welcomed <users@clusterlabs.org><br>
<b>Subject:</b> [EXT] [EXT] Re: [ClusterLabs] Can stalled libqb SHM I/O delay Pacemaker's operation timer?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for your reply. Those are great tips, however this was a freak firmware bug. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">"Degraded RAID array" was the wrong wording since that has a precise meaning for RAID. We were not in an ordinary degraded RAID mode with, say, a disk failure in the array. The array itself (or driver, or controller?) was misbehaving. Reads
were as far as I can tell, actually unbounded: never returning and not erroring. I don't expect it to recur after firmware updates.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It may, however, have revealed a bug or design flaw in Pacemaker as this is exactly the kind of hardware fault I would expect to trigger fencing and promotion of the hot standby.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Only manual intervention by an operator (as the controller was stalled) led to a successful STONITH (we ran the script by hand, a DC election then occurred and Pacemaker correctly moved the primary database resource and the resource agent
promoted the hot standby).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have a draft simple reproduction if it helps maintainers triage this or alternatively help me understand that this was expected behavior and why.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://github.com/cosgroveb/pacemaker-shm-timer-repro/blob/main/repro.sh">https://github.com/cosgroveb/pacemaker-shm-timer-repro/blob/main/repro.sh</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">On Tue, Aug 11, 2026 at 8:33 AM Windl, Ulrich via Users <<a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D">Hi!</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D">A non-pacemaker answer: I can imagine two solutions for your problem:</span><o:p></o:p></p>
<ol start="1" type="1">
<li class="m3416958625089056106msolistparagraph" style="color:#1F497D;mso-list:l0 level1 lfo1">
<span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif">Limit the reconstruction rate of the RAID</span><o:p></o:p></li><li class="m3416958625089056106msolistparagraph" style="color:#1F497D;mso-list:l0 level1 lfo1">
<span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif">Limit the amount of (dirty) filesystem cache (we had read stalls when someone backed up the database and most of the RAM had been filled with dirty buffers to write out)</span><o:p></o:p></li></ol>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Kind regards,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Ulrich Windl</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid windowtext 1.5pt;padding:0cm 0cm 0cm 4.0pt;border-color:currentcolor currentcolor currentcolor blue">
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-color:currentcolor currentcolor">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Users <<a href="mailto:users-bounces@clusterlabs.org" target="_blank">users-bounces@clusterlabs.org</a>>
<b>On Behalf Of </b>Brian Cosgrove<br>
<b>Sent:</b> Monday, August 10, 2026 10:03 PM<br>
<b>To:</b> <a href="mailto:users@clusterlabs.org" target="_blank">users@clusterlabs.org</a><br>
<b>Subject:</b> [EXT] [EXT] [ClusterLabs] Can stalled libqb SHM I/O delay Pacemaker's operation timer?</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Could Pacemaker’s controller block on a stalled block-device read while faulting in a swapped-<br>
out libqb SHM page, before it receives the executor reply and starts the operation timer? </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Would that behavior be expected? <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I'm investigating a lockup where the PostgreSQL primary and Pacemaker DC were on the same node when a degraded RAID array caused reads to appear unbounded without returning errors.
Pacemaker did not stop the PostgreSQL resource or promote a standby. crm_mon showed the DC as standby (with active resources).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Testing that simulates the incident reproduces that production Pacemaker behavior. In that setup a libqb SHM page is swapped out and controller blocks in shmem_fault() before the
executor reply and operation timer. The surviving production logs (such as the crm_mon output) match. Corosync membership remained intact in both the incident and the test.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="m3416958625089056106gmailsignatureprefix">--
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Brian Cosgrove<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" target="_blank">https://www.clusterlabs.org/</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span class="gmailsignatureprefix">-- </span><o:p></o:p></p>
<p class="MsoNormal">Brian Cosgrove<o:p></o:p></p>
</div>
</div>
</body>
</html>