<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.E-MailFormatvorlage18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D;mso-fareast-language:EN-US">If an MD or VG is activated exclusively on one node, it does not need to be shared. If you open if on multiple nodes at the same time (e.g. for any clustered filesystem),
it needs to be shared.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D;mso-fareast-language:EN-US">Kind regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D;mso-fareast-language:EN-US">Ulrich Windl<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Users <users-bounces@clusterlabs.org> <b>On Behalf Of
</b>Artem<br>
<b>Sent:</b> Wednesday, April 16, 2025 6:38 PM<br>
<b>To:</b> Klaus Wenninger <kwenning@redhat.com><br>
<b>Cc:</b> users@clusterlabs.org<br>
<b>Subject:</b> [EXT] Re: [ClusterLabs] crazy setup of fence_sbd, clusterd MD, nvmet<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I’ll reevaluate SBD design.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">But still, are there any thought about clustered MD vs simple MD for main data storage?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">And same question for LVM. Does it improve consistency?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks in advance.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Kindest regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Artem<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">16 апр. 2025 г., в 11:20, Klaus Wenninger <kwenning@redhat.com> написал(а):<o:p></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Apr 15, 2025 at 8:04 PM Andrei Borzenkov <<a href="mailto:arvidjaar@gmail.com">arvidjaar@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">14.04.2025 19:43, Artem wrote:<br>
> Dear gurus, I need your advice.<br>
> <br>
> We want to build a pacemaker cluster with the following resources.<br>
> Could you please evaluate the idea and give feedback?<br>
> <br>
> <br>
> Pairs of nodes with NVMe disks. Disks are shared from one node to<br>
> another via nvmet. Persistent udev names and partition ids.<br>
> MD raid1 is made on top of pairs of disks from different nodes. I<br>
> suspect it must be clustered MD, and it'll require dlm?<br>
> 2 or 4 clustered VLM volume groups are made on top of MD devices.<br>
> Pacemaker location preference rules for half of VGs to one node and<br>
> another half to another node.<br>
> <br>
> Striped LVs on top of VG with FS for Lustre MDT and OST. 2 main nodes<br>
> in Corosync, other OST nodes are configured as remote resources.<br>
> <br>
> OS network is separate from iBMC, and firewall rules deny this<br>
> traffic, so I decided to use SBD for fencing.<br>
> <br>
<br>
SBD requires a shared independent device. Using disks local to each <br>
cluster node for SBD defeats its purpose.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Agreed! Maybe just one more thing to add before it comes up as<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">a possible solution:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">You might think of sharing the disks via some mechanism to the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">respectively other side and use sbd poison-pill with 2 disks.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">This would probably prevent split-brain but imagine the other node<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">has some issue and you want to fence it. You would probably not be<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">able to access the disk shared by it and for successful fencing in<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">a 2 disk scenario you need to be able to write the poison-pill to both.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Such a setup might on the other hand make sense in a 3-node cluster -<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">at least under certain circumstances.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Klaus <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
> <br>
> I only found some pieces of such a stack documented, different OS,<br>
> different years ago. Now I'm trying to make it work together. At the<br>
> moment the clustered MD cannot be created as it fails to create a<br>
> lockspace (due to dlm error?). And dlm-clone doesn't want to start<br>
> either on main nodes or (as it should) on remote nodes. OS = RHEL9.<br>
> <br>
> May be such setup is too complicated? I try to avoid split brain<br>
> situations and uncoordinated writes by 2 mdadm processes on different<br>
> nodes in all failure scenarios.<br>
> I know that a common approach is to use JBODs of SAN arrays. But we<br>
> don't have it for this project.<br>
> <br>
> Thanks in advance.<br>
> Kindest regards,<br>
> Artem<br>
> _______________________________________________<br>
> Manage your subscription:<br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" target="_blank">
https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
> <br>
> ClusterLabs home: <a href="https://www.clusterlabs.org/" target="_blank">https://www.clusterlabs.org/</a><br>
<br>
_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" target="_blank">https://www.clusterlabs.org/</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>