<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-IN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hello All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Split-brain situation occurs due to there is a drop in quorum which leads to Spilt-brain situation and status information is not exchanged
between both two nodes of the cluster. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">This can be avoided if quorum communicates between both the nodes.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I have checked the code. In My opinion these files need to be updated (quorum.py/stonith.py) to avoid the spilt-brain situation to
maintain Active-Passive configuration.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Garima<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Derek Wuelfrath [mailto:dwuelfrath@inverse.ca]
<br>
<b>Sent:</b> 13 November 2017 20:55<br>
<b>To:</b> Cluster Labs - All topics related to open-source clustering welcomed <users@clusterlabs.org><br>
<b>Subject:</b> Re: [ClusterLabs] Pacemaker responsible of DRBD and a systemd resource<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hello Ken !<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Make sure that the systemd service is not enabled. If pacemaker is<br>
managing a service, systemd can't also be trying to start and stop it.<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">It is not. I made sure of this in the first place :)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Beyond that, the question is what log messages are there from around<br>
the time of the issue (on both nodes).<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Well, that’s the thing. There is not much log messages telling what is actually happening. The ’systemd’ resource is not even trying to start (nothing in either log for that resource). Here are the logs from my last attempt:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Scenario:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- Services were running on ‘pancakeFence2’. DRBD was synced and connected<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- I rebooted ‘pancakeFence2’. Services failed to ‘pancakeFence1’<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- After ‘pancakeFence2’ comes back, services are running just fine on ‘pancakeFence1’ but DRBD is in Standalone due to split-brain<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Logs for pancakeFence1: <a href="https://pastebin.com/dVSGPP78">https://pastebin.com/dVSGPP78</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Logs for pancakeFence2: <a href="https://pastebin.com/at8qPkHE">https://pastebin.com/at8qPkHE</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">It really looks like the status checkup mechanism of corosync/pacemaker for a systemd resource force the resource to “start” and therefore, start the ones above that resource in the group (DRBD in instance).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">This does not happen for a regular OCF resource (IPaddr2 per example)<o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><br>
Cheers!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black">-dw<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black">--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black">Derek Wuelfrath<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><a href="mailto:dwuelfrath@inverse.ca">dwuelfrath@inverse.ca</a> :: +1.514.447.4918 (x110) :: +1.866.353.6153 (x110)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black">Inverse inc. :: Leaders behind SOGo (<a href="https://www.sogo.nu/">www.sogo.nu</a>), PacketFence (<a href="https://www.packetfence.org/">www.packetfence.org</a>)
and Fingerbank (<a href="https://www.fingerbank.org">www.fingerbank.org</a>)<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Nov 10, 2017, at 11:39, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">On Thu, 2017-11-09 at 20:27 -0500, Derek Wuelfrath wrote:<br style="font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Hello there,<br>
<br>
First post here but following since a while!<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Welcome!<br>
<br style="font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Here’s my issue,<br>
we are putting in place and running this type of cluster since a<br>
while and never really encountered this kind of problem.<br>
<br>
I recently set up a Corosync / Pacemaker / PCS cluster to manage DRBD<br>
along with different other resources. Part of theses resources are<br>
some systemd resources… this is the part where things are “breaking”.<br>
<br>
Having a two servers cluster running only DRBD or DRBD with an OCF<br>
ipaddr2 resource (Cluser IP in instance) works just fine. I can<br>
easily move from one node to the other without any issue.<br>
As soon as I add a systemd resource to the resource group, things are<br>
breaking. Moving from one node to the other using standby mode works<br>
just fine but as soon as Corosync / Pacemaker restart involves<br>
polling of a systemd resource, it seems like it is trying to start<br>
the whole resource group and therefore, create a split-brain of the<br>
DRBD resource.<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
My first two suggestions would be:<br>
<br>
Make sure that the systemd service is not enabled. If pacemaker is<br>
managing a service, systemd can't also be trying to start and stop it.<br>
<br>
Fencing is the only way pacemaker can resolve split-brains and certain<br>
other situations, so that will help in the recovery.<br>
<br>
Beyond that, the question is what log messages are there from around<br>
the time of the issue (on both nodes).<br>
<br>
<br style="font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
It is the best explanation / description of the situation that I can<br>
give. If it need any clarification, examples, … I am more than open<br>
to share them.<br>
<br>
Any guidance would be appreciated :)<br>
<br>
Here’s the output of a ‘pcs config’<br>
<br>
<a href="https://pastebin.com/1TUvZ4X9">https://pastebin.com/1TUvZ4X9</a><br>
<br>
Cheers!<br>
-dw<br>
<br>
--<br>
Derek Wuelfrath<br>
<a href="mailto:dwuelfrath@inverse.ca">dwuelfrath@inverse.ca</a> :: +1.514.447.4918 (x110) :: +1.866.353.6153<br>
(x110)<br>
Inverse inc. :: Leaders behind SOGo (<a href="http://www.sogo.nu">www.sogo.nu</a>), PacketFence<br>
(<a href="http://www.packetfence.org">www.packetfence.org</a>) and Fingerbank (<a href="http://www.fingerbank.org">www.fingerbank.org</a>)<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">--<span class="apple-converted-space"> </span><br>
Ken Gaillot <</span><a href="mailto:kgaillot@redhat.com"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">kgaillot@redhat.com</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">><br>
<br>
_______________________________________________<br>
Users mailing list:<span class="apple-converted-space"> </span></span><a href="mailto:Users@clusterlabs.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Users@clusterlabs.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
</span><a href="http://lists.clusterlabs.org/mailman/listinfo/users"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://lists.clusterlabs.org/mailman/listinfo/users</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
<br>
Project Home:<span class="apple-converted-space"> </span></span><a href="http://www.clusterlabs.org/"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://www.clusterlabs.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Getting started:<span class="apple-converted-space"> </span></span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
Bugs:<span class="apple-converted-space"> </span></span><a href="http://bugs.clusterlabs.org/"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://bugs.clusterlabs.org</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>