<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">I’ve been working on porting a product originally based upon Pacemaker 1.0/Heartbeat to Centos7.6, using Pacemaker 2.0 and Corosync. I’ve encountered an issue with “failing over” a master resource from one node to another.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Our cluster consists of two nodes, each of which is a host system running Centos7.6, and supporting a single resource, a proprietary application. That application runs on both nodes concurrently, one as the master, the other as slave.
They communicate state information via an interprocessor link so that if the master instance fails, the slave instance will become master within seconds, and will resume operations from the point of failure of the other application/node. To test this operation,
we do a couple of different tests, one of which is simply to issue a <b><span style="font-family:"Courier New"">kill -9 <pid></span></b> to the master instance of the application. In my testing, I find that rather than app on the other node becoming master,
the app is restarted on the first node, and it becomes master.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here's some information for reference –<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">[root@mgraid-16201289RN00023-0 admin]# crm_mon -1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">Stack: corosync<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">Current DC: mgraid-16201289RN00023-0 (version 1.1.19-8.el7-c3c624ea3d) - partition with quorum<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">Last updated: Mon Aug 12 05:11:56 2019<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">Last change: Fri Aug 9 07:45:22 2019 by root via crm_attribute on mgraid-16201289RN00023-0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">2 nodes configured<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">4 resources configured<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">Online: [ mgraid-16201289RN00023-0 mgraid-16201289RN00023-1 ]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">Active resources:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">Clone Set: mgraid-stonith-clone [mgraid-stonith]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""> Started: [ mgraid-16201289RN00023-0 mgraid-16201289RN00023-1 ]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">Master/Slave Set: ms-SS16201289RN00023 [SS16201289RN00023]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""> Masters: [ mgraid-16201289RN00023-0 ]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""> Slaves: [ mgraid-16201289RN00023-1 ]<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve attached a <b>corosync</b> log file which shows the behavior I described. Our code uses an OCF compliant resource agent named
<b>ss</b>, which invokes a utility named <b><span style="font-family:"Courier New"">ssadm</span></b> to communicate with the application as needed. The
<b>ss</b> implementation executes functions named after the <b><span style="font-family:"Courier New"">$__OCF_ACTION</span></b> variable when it is invoked, e.g.
<b><span style="font-family:"Courier New"">ss_monitor</span></b>, <b><span style="font-family:"Courier New"">ss_demote</span></b>, etc.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">At 07:44:49, the ss agent discovers that the master instance has failed on node
<b><i>mgraid…-0</i></b> as a result of a failed <b><span style="font-family:"Courier New"">ssadm</span></b> request in response to an
<b><span style="font-family:"Courier New"">ss_monitor()</span></b> operation. It issues a
<b><span style="font-family:"Courier New"">crm_master -Q -D</span></b> command with the intent of demoting the master and promoting the slave, on the other node, to master. The
<b><span style="font-family:"Courier New"">ss_demote()</span></b> function finds that the application is no longer running and returns
<b><span style="font-family:"Courier New"">OCF_NOT_RUNNING</span></b> (7). In the older product, this was sufficient to promote the other instance to master, but in the current product, that does not happen. Currently, the failed application is restarted,
as expected, and is promoted to master, but this takes 10’s of seconds.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As far as I understand it, the intent of the <b>crm_master</b> command in our script is to lower the score of the failed resource so that the instance on the other node will become master. I’ve tried modifying our script to use
<b><span style="font-family:"Courier New"">crm_master -Q -v -INFINITY</span></b> instead, to no avail (which is what the attached log shows.)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In summary then, I need to find an answer to the question of how to accomplish the goal of failing over mastership from one node to another as quickly as possible. My reading of the Pacemaker documentation makes no mention of
<b>crm_master</b>, but the Pacemaker 2.0 Configuration Explained document does seem to imply that this technique of lowering the score should work. Can anyone offer any guidance?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal"> Michael Powell<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><img width="161" height="51" style="width:1.677in;height:.5312in" id="Picture_x005f_x0020_1" src="cid:image001.gif@01D54E04.D2AAEAF0" alt="cid:image004.gif@01D15F05.08FA4670"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D"> Michael Powell<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D"> Sr. Staff Engineer<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D"> 15220 NW Greenbrier Pkwy<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D"> Suite 290<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D"> Beaverton, OR 97006<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#1F497D"> T 503-372-7327 M 503-789-3019 H 503-625-5332<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>