<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-IN" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Klaus,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Answer below.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal">how you are tackling the - quite strict - requirement<o:p></o:p></p>
<p class="MsoNormal">of sbd for a watchdog-device (even if it is just softdog - which isn't available for your setup either) to guarantee reliable self-fencing?<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><Deva> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We have modified sbd and using similar pipe based approach communicate with base linix (watchdog_tickle modified) .<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If the sbd daemon has a problem the tickle stops and a daemon in the base linux would identify it and take actions like re-start the docker container.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Deva.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
</p>
<p style="font-family:Calibri;font-size:7pt;color:#737373;margin:5pt;font-style:normal;font-weight:normal;text-decoration:none;" align="Left">
Internal Use - Confidential<br>
</p>
From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Klaus Wenninger <kwenning@redhat.com>
<br>
<b>Sent:</b> 28 March 2025 13:38<br>
<b>To:</b> Cluster Labs - All topics related to open-source clustering welcomed <users@clusterlabs.org><br>
<b>Cc:</b> Narayanan, Devarajan <Devarajan.Narayanan@dell.com><br>
<b>Subject:</b> Re: [ClusterLabs] Queries about a Cluster setup inside a docker<o:p></o:p></span>
<p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p><span style="color:#CE1126">[EXTERNAL EMAIL] <o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Mar 25, 2025 at 3:37<span style="font-family:"Arial",sans-serif"> </span>PM Narayanan, Devarajan via Users <<a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Hi,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">I have a setup where I have multiple docker instances on a base linux (Node1).</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">I have a cluster inside each docker instances which pair with a similar setup on another node (node2) and form 2 node clusters.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">See pic1 below.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">In this setup, the cluster status etc resides in the docker overlay file system I presume.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt"><Query1></span></b><span style="font-size:11.0pt"> Is there a clear list of files which have the cluster status (Basically data of corosync, pacemaker,
sbd, crmsh processes I think)?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt"><Query2></span></b><span style="font-size:11.0pt"> In this setup if I wanted the cluster data to be persistent across “remove and re-run of the
docker instance”, what can I do?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Presuming cluster data will be in /var, /etc, /usr folders, I tried the following solution.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Created volumes for var, etc and usr and then during docker run used the options like “-v var_vol:/var -v etc_vol:/etc -v usr_vol”</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">With this some portion worked and saw some weird behaviour as well.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt"><Query3></span></b><span style="font-size:11.0pt"> Is this a correct way of solving the problem to have a persistent cluster data? Have I missed
mapping any folder?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">FYI, I have given the details about the experiment I tried to verify if the cluster data is consistent below (Experiment).
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Let me know if this makes sense.</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Slightly off topic regarding your questions ...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Just out of curiosity without really having investigated what docker offers to help with I wanted to ask how you are tackling the - quite strict - requirement<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">of sbd for a watchdog-device (even if it is just softdog - which isn't available for your setup either) to guarantee reliable self-fencing?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">sbd in containers would be an interesting field to investigate and thus I'm interested in what is there and what we could do to improve the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">situation. Many years back I for instance implemented a pipe-device offering a similar interface as the kernel-watchdog-drivers provided by<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">each instance of the per container lxc process (iirc the lxc processes then had an integration with watchdog-daemon while an instance of <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">watchdog daemon interacting with the pipe running inside each container - everything way before sbd appeared on the horizon - just as an example).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Klaus<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><u><span style="font-size:11.0pt">Pic1</span></u></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"><img border="0" width="947" height="532" style="width:9.8645in;height:5.5416in" id="m_-1149863384711949803Picture_x0020_3" src="cid:image001.png@01DB9FE8.3160FB30"></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><u><span style="font-size:11.0pt">Experiment</span></u></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Tried the following experiment. Please let me know if this makes sense</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">1) In a proper working cluster, stopped app-1-on-node-2 container on node2 to get the following crm status in app-1-on-node-1</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt">Node List:</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt"> * Online: [ app-1-on-node-1 ]</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt"> * OFFLINE: [ app-1-on-node-2 ]</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">2) Stopped and started the app-1-on-node-1 and checked the crm status. Remained the same as before</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt">Node List:</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt"> * Online: [ app-1-on-node-1 ]</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt"> * OFFLINE: [ app-1-on-node-2 ]</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">3) Remove the container app-1-on-node-1 and run it newly and then checked the crm status.
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> Now the status was changed by not showing the app-1-on-node-2 (Presume the reason for this is old cluster data is not available)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt;color:black;background:yellow">Node List:</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt;color:black;background:yellow"> * Online: [ app-1-on-node-1 ]</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">4) Repeated the step 1 and observed the crm status (This time I used “-v var_vol:/var -v etc_vol:/etc -v usr_vol" during docker run)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt">Node List:</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt"> * Online: [ app-1-on-node-1 ]</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt"> * OFFLINE: [ app-1-on-node-2 ]</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">5) Remove the container app-1-on-node-1 and run it newly (with “-v var_vol:/var -v etc_vol:/etc -v usr_vol" during docker run) and then checked the
crm status. </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">6) Now checked the crm status</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt;color:black;background:lime">Node List:</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt;color:black;background:lime"> * Online: [ app-1-on-node-1 ]</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:11.0pt;color:black;background:lime"> * OFFLINE: [ app-1-on-node-2 ]</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Regards,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Deva.</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:5.0pt"><span style="font-size:7.0pt;font-family:"Calibri",sans-serif;color:#737373">Internal Use - Confidential<o:p></o:p></span></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
Manage your subscription:<br>
<a href="https://urldefense.com/v3/__https:/lists.clusterlabs.org/mailman/listinfo/users__;!!LpKI!mUN_tEyhzd935xq2QwiY_HDqrEQ9CEZjtAzHHca5jkRALA42VbYD_wgSGuBn2uERAcrCHMTtjmz6BdP_k_kZEds$" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users
[lists.clusterlabs.org]</a><br>
<br>
ClusterLabs home: <a href="https://urldefense.com/v3/__https:/www.clusterlabs.org/__;!!LpKI!mUN_tEyhzd935xq2QwiY_HDqrEQ9CEZjtAzHHca5jkRALA42VbYD_wgSGuBn2uERAcrCHMTtjmz6BdP_rt0VjeU$" target="_blank">
https://www.clusterlabs.org/ [clusterlabs.org]</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>