<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:245577758;
mso-list-template-ids:305150266;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1
{mso-list-id:1388995666;
mso-list-template-ids:-125377628;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2
{mso-list-id:2010789025;
mso-list-template-ids:-1773217256;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">I think you should provide more information, like SBD configuration, syslog messages, etc.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Usually node fencing via SBD works by writing a message to a shared disk slot.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Once written, SBD/pacemaker expects the node to suicide soon. However multiple timeouts are configurable.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Asking AI, I got this (treat with some care):<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">I'll search for the specific timeout parameters and their interdependencies in Linux Pacemaker clusters with SBD fencing.<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">SBD Timeout Parameters in Linux Pacemaker Clusters<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">In a Pacemaker cluster using SBD (STONITH Block Device) for fencing, there are
<b>four primary configurable timeout parameters</b>, and they have strict interdependencies.
</span><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Understanding these relationships is critical for reliable cluster operation.<o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">
<hr size="1" width="100%" align="center">
</span></div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Timeout Parameters and Their Interdependencies<o:p></o:p></span></b></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="794" style="width:595.5pt;border-collapse:collapse">
<thead>
<tr>
<td valign="top" style="padding:0cm 12.0pt 3.0pt .75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Parameter<o:p></o:p></span></b></p>
</td>
<td valign="top" style="padding:0cm 12.0pt 3.0pt .75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Configuration Location<o:p></o:p></span></b></p>
</td>
<td valign="top" style="padding:0cm 12.0pt 3.0pt .75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Type<o:p></o:p></span></b></p>
</td>
<td valign="top" style="padding:0cm 12.0pt 3.0pt .75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Purpose<o:p></o:p></span></b></p>
</td>
<td valign="top" style="padding:0cm 12.0pt 3.0pt .75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Default<o:p></o:p></span></b></p>
</td>
</tr>
</thead>
<tbody>
<tr>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">SBD_WATCHDOG_TIMEOUT</span></b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">/etc/sysconfig/sbd<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">SBD daemon config<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Hardware watchdog timeout; triggers node self-fence if no kick received<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">5 seconds<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">msgwait</span></b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">SBD device metadata<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">SBD device level<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Time window for message delivery to node slot on SBD device<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Set during device initialization<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">stonith-timeout</span></b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Pacemaker CIB (cluster property)<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Global cluster property<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Maximum time Pacemaker waits for STONITH action (reboot/off) to complete<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">60 seconds<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">stonith-watchdog-timeout</span></b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Pacemaker CIB (cluster property)<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Global cluster property<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Time Pacemaker assumes fencing has completed via watchdog (diskless SBD only)<o:p></o:p></span></p>
</td>
<td valign="top" style="padding:6.0pt 6.0pt 6.0pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">0 (disabled by default)<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">
<hr size="1" width="100%" align="center">
</span></div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Critical Interdependencies<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">The timeout parameters have strict mathematical relationships that
<b>must be maintained</b> for proper cluster behavior:<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">For Disk-Based SBD (with shared storage devices):<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">msgwait >= (watchdog_timeout × 2)stonith-timeout >= msgwait + 20%<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Example:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">
If watchdog timeout is <b>30 seconds</b>:<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoNormal" style="color:#1F497D;mso-list:l2 level1 lfo1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">msgwait must be at least <b>60 seconds</b><o:p></o:p></span></li><li class="MsoNormal" style="color:#1F497D;mso-list:l2 level1 lfo1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">stonith-timeout must be at least <b>72 seconds</b> (60 + 20%)<o:p></o:p></span></li></ul>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">For Diskless SBD (watchdog-only, no shared storage):<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">stonith-watchdog-timeout >= (SBD_WATCHDOG_TIMEOUT × 2)stonith-timeout >= stonith-watchdog-timeout + 20%<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Example:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">
If SBD_WATCHDOG_TIMEOUT is <b>5 seconds</b>:<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoNormal" style="color:#1F497D;mso-list:l0 level1 lfo2"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">stonith-watchdog-timeout must be at least <b>10 seconds</b><o:p></o:p></span></li><li class="MsoNormal" style="color:#1F497D;mso-list:l0 level1 lfo2"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">stonith-timeout must be at least <b>12 seconds</b> (10 + 20%)<o:p></o:p></span></li></ul>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">
<hr size="1" width="100%" align="center">
</span></div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">How These Parameters Interact<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Watchdog Timeout (SBD_WATCHDOG_TIMEOUT)<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">This is the
<b>foundation</b> of the timeout hierarchy. It represents how long the hardware watchdog will wait for a "kick" (heartbeat) from the SBD daemon before forcibly resetting the node. If storage latency or system issues prevent the SBD daemon from operating, the
node self-fences after this timeout expires.<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Message Wait Timeout (msgwait)<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">This is set in the
<b>SBD device metadata</b> during initialization and defines the grace period for a fencing message to be acknowledged as delivered to the target node's slot. It must be
<b>at least twice the watchdog timeout</b> to ensure the node has time to detect the fencing message and self-fence gracefully before the watchdog triggers.<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">STONITH Timeout (stonith-timeout)<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">This is a
<b>Pacemaker cluster property</b> that controls how long the cluster waits for the fencing action to complete. It must exceed msgwait by at least
<b>20%</b> to allow sufficient time for the message to be delivered and processed. If this timeout is too short, the cluster may consider the fencing action failed and retry, causing unnecessary delays.<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">STONITH Watchdog Timeout (stonith-watchdog-timeout)<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">This parameter is
<b>only used for diskless SBD</b> and tells Pacemaker how long to wait before assuming a node has already self-fenced via the watchdog. It must be
<b>at least twice the SBD_WATCHDOG_TIMEOUT</b> to provide a safety margin. Setting this to
<b>0</b> (the default) disables resource recovery and is appropriate only for disk-based SBD configurations.<o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">
<hr size="1" width="100%" align="center">
</span></div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Critical Warnings<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">Alice, pay attention to these constraints:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoNormal" style="color:#1F497D;mso-list:l1 level1 lfo3"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">Do not set stonith-watchdog-timeout until SBD is configured and running on every node</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">, including Pacemaker Remote nodes.<o:p></o:p></span></li><li class="MsoNormal" style="color:#1F497D;mso-list:l1 level1 lfo3"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">If stonith-timeout < stonith-watchdog-timeout in diskless SBD</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">, nodes can become stuck in an UNCLEAN state, blocking failover.<o:p></o:p></span></li><li class="MsoNormal" style="color:#1F497D;mso-list:l1 level1 lfo3"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">For multipath or iSCSI setups</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">, the watchdog timeout should account for path failure detection and failover time. The max_polling_interval in /etc/multipath.conf must be less than the watchdog timeout.<o:p></o:p></span></li><li class="MsoNormal" style="color:#1F497D;mso-list:l1 level1 lfo3"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">Changing watchdog timeout requires coordinating changes across all dependent timeouts</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US"> to maintain the mathematical relationships.<o:p></o:p></span></li><li class="MsoNormal" style="color:#1F497D;mso-list:l1 level1 lfo3"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US">Storage latency is the primary driver</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;mso-fareast-language:EN-US"> of watchdog timeout values; high-latency storage requires longer timeouts, which cascades into longer msgwait and stonith-timeout values.<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US">The interdependency structure ensures that each timeout layer provides sufficient time for the layer below it to complete, preventing
race conditions and cluster deadlock scenarios.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Aptos",serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Kind regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Ulrich Windl<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Users <users-bounces@clusterlabs.org>
<b>On Behalf Of </b>Dmytro Poliarush via Users<br>
<b>Sent:</b> Tuesday, March 17, 2026 12:32 PM<br>
<b>To:</b> users@clusterlabs.org<br>
<b>Cc:</b> Dmytro Poliarush <Dmytro_Poliarush@epam.com><br>
<b>Subject:</b> [EXT] [EXT] [ClusterLabs] pacemaker: 1.1.23 20sec timeout on cluster with disc I/O write delays<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="mso-element:para-border-div;border:solid black 1.0pt;padding:0cm 0cm 0cm 0cm;background:#FFF3CD">
<p style="background:#FFF3CD;border:none;padding:0cm"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">Sicherheits-Hinweis: Diese E-Mail wurde von einer Person außerhalb des UKR gesendet. Seien Sie vorsichtig vor gefälschten Absendern,
wenn Sie auf Links klicken, Anhänge öffnen oder weitere Aktionen ausführen, bevor Sie die Echtheit überprüft haben.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">Hi all,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">Need a small guidance on pacemaker: 1.1.23.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">I'm chasing a stubborn issue in a 2node 2disc SBD cluster.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">When running manual fencing test with `pcs stonith fence` command, I observe an error<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">```<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> Error: unable to fence '<nodehostname>'<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">```<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">Error manifests each time around a `20second` timeout(I assume this is a timeout).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">`time` command is used to track how long execution runs: `time pcs stonith fence`.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">Here is an example:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">```<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> [root@node1 ~]# time pcs stonith fence --debug node2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> Running: /usr/sbin/stonith_admin -B node2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> > Return Value: 194<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> --Debug Output Start--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> --Debug Output End--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> Error: unable to fence 'node2'<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> > real 0m20.791s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> user 0m0.063s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> sys 0m0.033s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> [root@node1 ~]#<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">```<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">For investigation, I've setup a testing cluster with 2 Virtualbox VMs.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">Behaviour was NOT observed on testing cluster until I intentionally added disk write delays with dmsetup tool on one of the nodes.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">Here is an example of setting a 22sec write delay:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">```<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> # Create: read delay = 0 ms, write delay = 22000 ms<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> # Table format: delay <dev> <start> <read_ms> <dev> <start> <write_ms><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> dmsetup --noudevsync create slow-sdc --table "0 ${SIZE} delay /dev/sdc 0 0 /dev/sdc 0 22000"<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> dmsetup mknodes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">```<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">NOTE, that tests with delays upto(including) 19sec pass:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">```<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> [root@node1 ~]# ./suspend-resume-slow-sdc-delay-write.sh 20000<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> [root@node1 ~]# dmsetup table slow-sdc<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">> 0 262144 delay 8:32 0 0 8:32 0 20000<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> [root@node1 ~]# time pcs stonith fence --debug node2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> Running: /usr/sbin/stonith_admin -B node2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> Return Value: 194<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> --Debug Output Start--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> --Debug Output End--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">```<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">> Error: unable to fence 'node2'<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">> real 0m20.588s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> user 0m0.088s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> sys 0m0.021s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">> [root@node1 ~]# ./suspend-resume-slow-sdc-delay-write.sh 19000<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> ++ blockdev --getsize /dev/sdc<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> + SIZE=262144<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> ++ lsblk -dn -o MAJ:MIN /dev/sdc<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> + MAJMIN=' 8:32 '<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> + dmsetup suspend slow-sdc<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> + dmsetup reload slow-sdc --table '0 262144 delay /dev/sdc 0 0 /dev/sdc 0 19000'<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> + dmsetup resume slow-sdc<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> + dmsetup table slow-sdc<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">> 0 262144 delay 8:32 0 0 8:32 0 19000<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> [root@node1 ~]# pcs stonith history cleanup; pcs stonith cleanup # pcs-cleanup-error-cleanup<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> cleaning up fencing-history for node *<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> Cleaned up all resources on all nodes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> [root@node1 ~]#<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> [root@node1 ~]# time pcs stonith fence --debug node2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> Running: /usr/sbin/stonith_admin -B node2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> Return Value: 0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> --Debug Output Start--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> --Debug Output End--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">> Node: node2 fenced<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">> real 0m19.869s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> user 0m0.098s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> sys 0m0.035s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"> [root@node1 ~]#<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">```<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">So here is my question:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">I assume there is a 20sec timeout value hardcoded somewhere in pacemaker 1.1.23 sources.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">This hardcoded value impacts manual fencing in case of disc I/O delays(maybe in some other cases).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">I expect that increasing timeout can mitigate clusters with disc I/O issues similar to ones described above.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">Please note this timeout is NOT: stonith-timeout or stonith-watchdog-timeout.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">Could you please comment if that is a meaningfull assumption and where does the 20sec timeout come from.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",serif;color:black">Regards, Dmytro<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>