<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:365252544;
mso-list-type:hybrid;
mso-list-template-ids:1939873032 1662048598 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:.75in;
text-indent:-.25in;
font-family:"Segoe UI",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.25in;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.75in;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.25in;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.75in;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.25in;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.75in;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.25in;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.75in;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:1874347099;
mso-list-type:hybrid;
mso-list-template-ids:-413607690 1074939800 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Segoe UI",sans-serif;
mso-fareast-font-family:Calibri;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white">Hi All,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white">We have a scenario on SLES 12 SP3 cluster.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white">The scenario is explained as follows in the order of events:
<o:p></o:p></span></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="color:#242424;margin-left:.25in;mso-list:l0 level1 lfo2">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;background:white">There is a 2-node cluster (FILE-1, FILE-2)<o:p></o:p></span></li><li class="MsoListParagraph" style="color:#242424;margin-left:.25in;mso-list:l0 level1 lfo2">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;background:white">The cluster and the resources were up and running fine initially .
<o:p></o:p></span></li><li class="MsoListParagraph" style="color:#242424;margin-left:.25in;mso-list:l0 level1 lfo2">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;background:white">Then fencing request from pacemaker got issued on both nodes simultaneously
<o:p></o:p></span></li></ul>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:white">Logs from 1<sup>st</sup> node: <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:silver;mso-highlight:silver">2022-02-22T03:26:36.737075+00:00
FILE-1 corosync[12304]: [TOTEM ] Failed to receive the leave message. failed: 2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:silver;mso-highlight:silver">.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:silver;mso-highlight:silver">.</span><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:silver;mso-highlight:silver"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:silver;mso-highlight:silver">2022-02-22T03:26:36.977888+00:00
FILE-1 pacemaker-fenced[12331]: notice: Requesting that FILE-1 perform 'off' action targeting FILE-2</span><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:#FDD472"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:white">Logs from 2nd node: <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:silver;mso-highlight:silver">2022-02-22T03:26:36.738080+00:00
FILE-2 corosync[4989]: [TOTEM ] Failed to receive the leave message. failed: 1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:silver;mso-highlight:silver">.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:silver;mso-highlight:silver">.</span><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:silver;mso-highlight:silver"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:silver;mso-highlight:silver">Feb 22 03:26:38 FILE-2 pacemaker-fenced
[5015] (call_remote_stonith) notice: Requesting that FILE-2 perform 'off' action targeting FILE-1</span><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:#FDD472">
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in"><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:#FDD472"><o:p> </o:p></span></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="color:#4472C4;margin-left:.25in;mso-list:l0 level1 lfo2">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white">When the nodes came up after unfencing, the DC got set after election</span><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:#FDD472"><o:p></o:p></span></li><li class="MsoListParagraph" style="color:#4472C4;margin-left:.25in;mso-list:l0 level1 lfo2">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white">After that the resources which were expected to run on only one node became active on both (all) nodes of the cluster.</span><span style="font-size:10.0pt;font-family:"Courier New";color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:#FDD472"><o:p></o:p></span></li></ul>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#2F5597;mso-style-textfill-fill-color:#2F5597;mso-style-textfill-fill-alpha:100.0%;background:white"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#242424;background:white"><o:p> </o:p></span></p>
<table class="MsoTableGrid" border="1" cellspacing="0" cellpadding="0" style="margin-left:63.25pt;border-collapse:collapse;border:none">
<tbody>
<tr>
<td width="1061" valign="top" style="width:796.0pt;border:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#242424;background:white">27290 2022-02-22T04:16:31.699186+00:00 FILE-2 pacemaker-schedulerd[5018]:
<span style="background:yellow;mso-highlight:yellow">error: Resource stonith-sbd is active on 2 nodes (attempting recovery)</span></span><span style="font-size:10.0pt;font-family:"Courier New";color:#242424"><br>
<span style="background:white">27291 2022-02-22T04:16:31.699397+00:00 FILE-2 pacemaker-schedulerd[5018]: notice: See </span></span><span style="font-size:10.0pt;font-family:"Courier New""><a href="https://wiki.clusterlabs.org/wiki/FAQ#Resource_" target="_blank" title="https://wiki.clusterlabs.org/wiki/faq#resource_"><span style="color:#5B5FC7;background:white;text-decoration:none">https://wiki.clusterlabs.org/wiki/FAQ#Resource_</span></a><span style="color:#242424;background:white"> is_Too_Active
for more information</span><span style="color:#242424"><br>
<span style="background:white">27292 2022-02-22T04:16:31.699590+00:00 FILE-2 pacemaker-schedulerd[5018]:
<span style="background:yellow;mso-highlight:yellow">error: Resource FILE_Filesystem is active on 2 nodes (attem pting recovery)</span></span><br>
<span style="background:white">27293 2022-02-22T04:16:31.699731+00:00 FILE-2 pacemaker-schedulerd[5018]: notice: See </span></span><a href="https://wiki.clusterlabs.org/wiki/FAQ#Resource_" target="_blank" title="https://wiki.clusterlabs.org/wiki/faq#resource_"><span style="color:#5B5FC7;background:white;text-decoration:none">https://wiki.clusterlabs.org/wiki/FAQ#Resource_</span></a><span style="color:#242424;background:white"> is_Too_Active
for more information</span><span style="color:#242424"><br>
<span style="background:white">27294 2022-02-22T04:16:31.699878+00:00 FILE-2 pacemaker-schedulerd[5018]:
<span style="background:yellow;mso-highlight:yellow">error: Resource IP_Floating is active on 2 nodes (attemptin g recovery)</span></span><br>
<span style="background:white">27295 2022-02-22T04:16:31.700027+00:00 FILE-2 pacemaker-schedulerd[5018]: notice: See </span></span><a href="https://wiki.clusterlabs.org/wiki/FAQ#Resource_" target="_blank" title="https://wiki.clusterlabs.org/wiki/faq#resource_"><span style="color:#5B5FC7;background:white;text-decoration:none">https://wiki.clusterlabs.org/wiki/FAQ#Resource_</span></a><span style="color:#242424;background:white"> is_Too_Active
for more information</span><span style="color:#242424"><br>
<span style="background:white">27296 2022-02-22T04:16:31.700203+00:00 FILE-2 pacemaker-schedulerd[5018]:
<span style="background:yellow;mso-highlight:yellow">error: Resource Service_Postgresql is active on 2 nodes (at tempting recovery)</span></span><br>
<span style="background:white">27297 2022-02-22T04:16:31.700354+00:00 FILE-2 pacemaker-schedulerd[5018]: notice: See </span></span><a href="https://wiki.clusterlabs.org/wiki/FAQ#Resource_" target="_blank" title="https://wiki.clusterlabs.org/wiki/faq#resource_"><span style="color:#5B5FC7;background:white;text-decoration:none">https://wiki.clusterlabs.org/wiki/FAQ#Resource_</span></a><span style="color:#242424;background:white"> is_Too_Active
for more information</span><span style="color:#242424"><br>
<span style="background:white">27298 2022-02-22T04:16:31.700501+00:00 FILE-2 pacemaker-schedulerd[5018]: error:
<span style="background:yellow;mso-highlight:yellow">Resource Service_Postgrest is active on 2 nodes (att empting recovery)</span></span><br>
<span style="background:white">27299 2022-02-22T04:16:31.700648+00:00 FILE-2 pacemaker-schedulerd[5018]: notice: See </span></span><a href="https://wiki.clusterlabs.org/wiki/FAQ#Resource_" target="_blank" title="https://wiki.clusterlabs.org/wiki/faq#resource_"><span style="color:#5B5FC7;background:white;text-decoration:none">https://wiki.clusterlabs.org/wiki/FAQ#Resource_</span></a><span style="color:#242424;background:white"> is_Too_Active
for more information</span><span style="color:#242424"><br>
<span style="background:white">27300 2022-02-22T04:16:31.700792+00:00 FILE-2 pacemaker-schedulerd[5018]:
<span style="background:yellow;mso-highlight:yellow">error: Resource Service_esm_primary is active on 2 nodes (a ttempting recovery)</span></span><br>
<span style="background:white">27301 2022-02-22T04:16:31.700939+00:00 FILE-2 pacemaker-schedulerd[5018]: notice: See </span></span><a href="https://wiki.clusterlabs.org/wiki/FAQ#Resource_" target="_blank" title="https://wiki.clusterlabs.org/wiki/faq#resource_"><span style="color:#5B5FC7;background:white;text-decoration:none">https://wiki.clusterlabs.org/wiki/FAQ#Resource_</span></a><span style="color:#242424;background:white"> is_Too_Active
for more information</span><span style="color:#242424"><br>
<span style="background:white">27302 2022-02-22T04:16:31.701086+00:00 FILE-2 pacemaker-schedulerd[5018]:
<span style="background:yellow;mso-highlight:yellow">error: Resource Shared_Cluster_Backup is active on 2 nodes (attempting recovery)</span><o:p></o:p></span></span></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#242424;background:white"><o:p> </o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#242424;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal">Can you guys please help us understand if this is indeed a split-brain scenario ? Under what circumstances can such a scenario be observed?
<o:p></o:p></p>
<p class="MsoNormal">We can have very serious impact if such a case can re-occur inspite of stonith already configured. Hence the ask .<o:p></o:p></p>
<p class="MsoNormal">In case this situation gets reproduced, how can it be handled?
<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">Note: We have stonith configured and it has been working fine so far. In this case also, the initial fencing happened from stonith only.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks in advance!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#242424;background:white"><o:p> </o:p></span></p>
<br>
<p class="msipfooter90245289" align="Left" style="margin:0"><span style="font-size:7.0pt;font-family:Calibri;color:#737373">Internal Use - Confidential</span></p>
</div>
</body>
</html>