<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-AU link=blue vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Thanks for the update Ken!</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='border:none;padding:0cm'><b>From: </b><a href="mailto:kgaillot@redhat.com">Ken Gaillot</a><br><b>Sent: </b>Saturday, 21 October 2017 7:06 AM<br><b>To: </b><a href="mailto:users@clusterlabs.org">Cluster Labs - All topics related to open-source clustering welcomed</a><br><b>Subject: </b>Re: [ClusterLabs] crm_resource --wait</p></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I've narrowed down the cause.</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>When the "standby" transition completes, vm2 has more remaining</p><p class=MsoNormal>utilization capacity than vm1, so the cluster wants to run sv-fencer</p><p class=MsoNormal>there. That should be taken into account in the same transition, but it</p><p class=MsoNormal>isn't, so a second transition is needed to make it happen.</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Still investigating a fix. A workaround is to assign some stickiness or</p><p class=MsoNormal>utilization to sv-fencer.</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>On Wed, 2017-10-11 at 14:01 +1000, Leon Steffens wrote:</p><p class=MsoNormal>> I've attached two files:</p><p class=MsoNormal>> 314 = after standby step</p><p class=MsoNormal>> 315 = after resource update</p><p class=MsoNormal>> </p><p class=MsoNormal>> On Wed, Oct 11, 2017 at 12:22 AM, Ken Gaillot <kgaillot@redhat.com></p><p class=MsoNormal>> wrote:</p><p class=MsoNormal>> > On Tue, 2017-10-10 at 15:19 +1000, Leon Steffens wrote:</p><p class=MsoNormal>> > > Hi Ken,</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > I managed to reproduce this on a simplified version of the</p><p class=MsoNormal>> > cluster,</p><p class=MsoNormal>> > > and on Pacemaker 1.1.15, 1.1.16, as well as 1.1.18-rc1</p><p class=MsoNormal>> > </p><p class=MsoNormal>> > > The steps to create the cluster are:</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > pcs property set stonith-enabled=false</p><p class=MsoNormal>> > > pcs property set placement-strategy=balanced</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > pcs node utilization vm1 cpu=100</p><p class=MsoNormal>> > > pcs node utilization vm2 cpu=100</p><p class=MsoNormal>> > > pcs node utilization vm3 cpu=100</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > pcs property set maintenance-mode=true</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > pcs resource create sv-fencer ocf:pacemaker:Dummy</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > pcs resource create sv ocf:pacemaker:Dummy clone notify=false</p><p class=MsoNormal>> > > pcs resource create std ocf:pacemaker:Dummy meta resource-</p><p class=MsoNormal>> > > stickiness=100</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > pcs resource create partition1 ocf:pacemaker:Dummy meta resource-</p><p class=MsoNormal>> > > stickiness=100</p><p class=MsoNormal>> > > pcs resource create partition2 ocf:pacemaker:Dummy meta resource-</p><p class=MsoNormal>> > > stickiness=100</p><p class=MsoNormal>> > > pcs resource create partition3 ocf:pacemaker:Dummy meta resource-</p><p class=MsoNormal>> > > stickiness=100</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > pcs resource utilization partition1 cpu=5</p><p class=MsoNormal>> > > pcs resource utilization partition2 cpu=5</p><p class=MsoNormal>> > > pcs resource utilization partition3 cpu=5</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > pcs constraint colocation add std with sv-clone INFINITY</p><p class=MsoNormal>> > > pcs constraint colocation add partition1 with sv-clone INFINITY</p><p class=MsoNormal>> > > pcs constraint colocation add partition2 with sv-clone INFINITY</p><p class=MsoNormal>> > > pcs constraint colocation add partition3 with sv-clone INFINITY</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > pcs property set maintenance-mode=false</p><p class=MsoNormal>> > > </p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > I can then reproduce the issues in the following way:</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > $ pcs resource</p><p class=MsoNormal>> > > sv-fencer (ocf::pacemaker:Dummy): Started vm1</p><p class=MsoNormal>> > > Clone Set: sv-clone [sv]</p><p class=MsoNormal>> > > Started: [ vm1 vm2 vm3 ]</p><p class=MsoNormal>> > > std (ocf::pacemaker:Dummy): Started vm2</p><p class=MsoNormal>> > > partition1 (ocf::pacemaker:Dummy): Started vm3</p><p class=MsoNormal>> > > partition2 (ocf::pacemaker:Dummy): Started vm1</p><p class=MsoNormal>> > > partition3 (ocf::pacemaker:Dummy): Started vm2</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > $ pcs cluster standby vm3</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > # Check that all resources have moved off vm3</p><p class=MsoNormal>> > > $ pcs resource</p><p class=MsoNormal>> > > sv-fencer (ocf::pacemaker:Dummy): Started vm1</p><p class=MsoNormal>> > > Clone Set: sv-clone [sv]</p><p class=MsoNormal>> > > Started: [ vm1 vm2 ]</p><p class=MsoNormal>> > > Stopped: [ vm3 ]</p><p class=MsoNormal>> > > std (ocf::pacemaker:Dummy): Started vm2</p><p class=MsoNormal>> > > partition1 (ocf::pacemaker:Dummy): Started vm1</p><p class=MsoNormal>> > > partition2 (ocf::pacemaker:Dummy): Started vm1</p><p class=MsoNormal>> > > partition3 (ocf::pacemaker:Dummy): Started vm2</p><p class=MsoNormal>> > </p><p class=MsoNormal>> > Thanks for the detailed information, this should help me get to the</p><p class=MsoNormal>> > bottom of it. From this description, it sounds like a new</p><p class=MsoNormal>> > transition</p><p class=MsoNormal>> > isn't being triggered when it should.</p><p class=MsoNormal>> > </p><p class=MsoNormal>> > Could you please attach the DC's pe-input file that is listed in</p><p class=MsoNormal>> > the</p><p class=MsoNormal>> > logs after the standby step above? That would simplify analysis.</p><p class=MsoNormal>> > </p><p class=MsoNormal>> > > # Wait for any outstanding actions to complete.</p><p class=MsoNormal>> > > $ crm_resource --wait --timeout 300</p><p class=MsoNormal>> > > Pending actions:</p><p class=MsoNormal>> > > Action 22: sv-fencer_monitor_10000 on vm2</p><p class=MsoNormal>> > > Action 21: sv-fencer_start_0 on vm2</p><p class=MsoNormal>> > > Action 20: sv-fencer_stop_0 on vm1</p><p class=MsoNormal>> > > Error performing operation: Timer expired</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > # Check the resources again - sv-fencer is still on vm1</p><p class=MsoNormal>> > > $ pcs resource</p><p class=MsoNormal>> > > sv-fencer (ocf::pacemaker:Dummy): Started vm1</p><p class=MsoNormal>> > > Clone Set: sv-clone [sv]</p><p class=MsoNormal>> > > Started: [ vm1 vm2 ]</p><p class=MsoNormal>> > > Stopped: [ vm3 ]</p><p class=MsoNormal>> > > std (ocf::pacemaker:Dummy): Started vm2</p><p class=MsoNormal>> > > partition1 (ocf::pacemaker:Dummy): Started vm1</p><p class=MsoNormal>> > > partition2 (ocf::pacemaker:Dummy): Started vm1</p><p class=MsoNormal>> > > partition3 (ocf::pacemaker:Dummy): Started vm2</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > # Perform a random update to the CIB.</p><p class=MsoNormal>> > > $ pcs resource update std op monitor interval=20 timeout=20</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > # Check resource status again - sv_fencer has now moved to vm2</p><p class=MsoNormal>> > (the</p><p class=MsoNormal>> > > action crm_resource was waiting for)</p><p class=MsoNormal>> > > $ pcs resource</p><p class=MsoNormal>> > > sv-fencer (ocf::pacemaker:Dummy): Started vm2 </p><p class=MsoNormal>> > <<<============</p><p class=MsoNormal>> > > Clone Set: sv-clone [sv]</p><p class=MsoNormal>> > > Started: [ vm1 vm2 ]</p><p class=MsoNormal>> > > Stopped: [ vm3 ]</p><p class=MsoNormal>> > > std (ocf::pacemaker:Dummy): Started vm2</p><p class=MsoNormal>> > > partition1 (ocf::pacemaker:Dummy): Started vm1</p><p class=MsoNormal>> > > partition2 (ocf::pacemaker:Dummy): Started vm1</p><p class=MsoNormal>> > > partition3 (ocf::pacemaker:Dummy): Started vm2</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > I do not get the problem if I:</p><p class=MsoNormal>> > > 1) remove the "std" resource; or</p><p class=MsoNormal>> > > 2) remove the co-location constraints; or</p><p class=MsoNormal>> > > 3) remove the utilization attributes for the partition resources.</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > In these cases the sv-fencer resource is happy to stay on vm1,</p><p class=MsoNormal>> > and</p><p class=MsoNormal>> > > crm_resource --wait returns immediately.</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > It looks like the pcs cluster standby call is</p><p class=MsoNormal>> > creating/registering</p><p class=MsoNormal>> > > the actions to move the sv-fencer resource to vm2, but it doesn't</p><p class=MsoNormal>> > > include it in the cluster transition. When the CIB is later</p><p class=MsoNormal>> > updated</p><p class=MsoNormal>> > > by something else, the action is included in that transition.</p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > ></p><p class=MsoNormal>> > > Regards,</p><p class=MsoNormal>> > > Leon</p><p class=MsoNormal>> > </p><p class=MsoNormal>> > _______________________________________________</p><p class=MsoNormal>> > Users mailing list: Users@clusterlabs.org</p><p class=MsoNormal>> > http://lists.clusterlabs.org/mailman/listinfo/users</p><p class=MsoNormal>> > </p><p class=MsoNormal>> > Project Home: http://www.clusterlabs.org</p><p class=MsoNormal>> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc</p><p class=MsoNormal>> > h.pdf</p><p class=MsoNormal>> > Bugs: http://bugs.clusterlabs.org</p><p class=MsoNormal>> > </p><p class=MsoNormal>> </p><p class=MsoNormal>> _______________________________________________</p><p class=MsoNormal>> Users mailing list: Users@clusterlabs.org</p><p class=MsoNormal>> http://lists.clusterlabs.org/mailman/listinfo/users</p><p class=MsoNormal>> </p><p class=MsoNormal>> Project Home: http://www.clusterlabs.org</p><p class=MsoNormal>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.</p><p class=MsoNormal>> pdf</p><p class=MsoNormal>> Bugs: http://bugs.clusterlabs.org</p><p class=MsoNormal>-- </p><p class=MsoNormal>Ken Gaillot <kgaillot@redhat.com></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>_______________________________________________</p><p class=MsoNormal>Users mailing list: Users@clusterlabs.org</p><p class=MsoNormal>http://lists.clusterlabs.org/mailman/listinfo/users</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Project Home: http://www.clusterlabs.org</p><p class=MsoNormal>Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</p><p class=MsoNormal>Bugs: http://bugs.clusterlabs.org</p><p class=MsoNormal><o:p> </o:p></p></div></body></html>