<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">All,<o:p></o:p></p>
<p class="MsoNormal">Hello. Hope all is well. I have been researching Oracle Linux 8.2 and ran across a situation that is not well documented. I decided to provide some details to the community in case I am missing something.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Basically, if you increase the totem token above approximately 33000 with the knet transport, then a two node cluster will not properly form. The exact threshold value will slightly fluctuate, depending on hardware type and debugging,
but will consistently fail above 40000.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The failure to form a cluster would occur when running the “pcs cluster start --all" command or if I would start one cluster, let it stabilize, then start the second. When it fails to form a cluster, each side would say they are ONLINE,
but the other side is UNCLEAN(offline) (cluster state: partition WITHOUT quorum). If I define proper stonith resources, then they will not fence since the cluster never makes it to an initial quorum state. So, the cluster will stay in this split state indefinitely.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Changing the transport back to udpu or udp, the higher totem tokens worked as expected.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">From the debug logging, I suspect that the Election Trigger (20 seconds) fires before all nodes are properly identified by the knet transport. I noticed that with a totem token passing 32 seconds, the knet_ping* defaults were pushing up
against that 20 second mark. The output of “corosync-cfgtool -s” will show each node’s link as enabled, but each side will state the other side’s link is not connected. Since each side thinks the other node is not active, they fail to properly send a join
message to the other node during the election. They will essentially form a singleton cluster(??). It is more puzzling when you start one node at a time, waiting for the node to stabilize before starting the other. It is like the first node will never
see the remote knet interfaces become active, regardless of how long you wait.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The solution is to manually set the knet ping_timeout and ping_interval to lower values than the default values derived from the totem token. This seems to allow for the knet transport to determine link status of all nodes before the election
timer pops.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I tested this on both physical hardware and with VMs. Both react similarly.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Bare bones test case to reproduce:<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">yum install pcs pacemaker fence-agents-all<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">firewall-cmd --permanent --add-service=high-availability<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">firewall-cmd --add-service=high-availability<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">systemctl start pcsd.service<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">systemctl enable pcsd.service<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">systemctl disable corosync<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">systemctl disable pacemaker<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">passwd hacluster<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">pcs host auth node1 node2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">pcs cluster setup rhcs_test node1 node2 totem token=41000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">pcs cluster start --all<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Example command to create cluster that will properly form and get quorum:<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">pcs cluster setup rhcs_test node1 node2 totem token=61000 transport knet link ping_interval=1250 ping_timeout=2500<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hope this helps someone in the future.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal">Robert<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:navy">Robert Hayden</span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:navy"> | Lead Technology Architect | Cerner Corporation
</span><o:p></o:p></p>
</div>
<div>
<p style="font-family: arial, sans-serif; font-size: 8pt; color: #6A737B">CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential
and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the
addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.</p>
</div>
</body>
</html>