<div dir="ltr">I've attached two files:<div>314 = after standby step</div><div>315 = after resource update</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 11, 2017 at 12:22 AM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Tue, 2017-10-10 at 15:19 +1000, Leon Steffens wrote:<br>
> Hi Ken,<br>
><br>
> I managed to reproduce this on a simplified version of the cluster,<br>
> and on Pacemaker 1.1.15, 1.1.16, as well as 1.1.18-rc1<br>
<br>
> The steps to create the cluster are:<br>
><br>
> pcs property set stonith-enabled=false<br>
> pcs property set placement-strategy=balanced<br>
><br>
> pcs node utilization vm1 cpu=100<br>
> pcs node utilization vm2 cpu=100<br>
> pcs node utilization vm3 cpu=100<br>
><br>
> pcs property set maintenance-mode=true<br>
><br>
> pcs resource create sv-fencer ocf:pacemaker:Dummy<br>
><br>
> pcs resource create sv ocf:pacemaker:Dummy clone notify=false<br>
> pcs resource create std ocf:pacemaker:Dummy meta resource-<br>
> stickiness=100<br>
><br>
> pcs resource create partition1 ocf:pacemaker:Dummy meta resource-<br>
> stickiness=100<br>
> pcs resource create partition2 ocf:pacemaker:Dummy meta resource-<br>
> stickiness=100<br>
> pcs resource create partition3 ocf:pacemaker:Dummy meta resource-<br>
> stickiness=100<br>
><br>
> pcs resource utilization partition1 cpu=5<br>
> pcs resource utilization partition2 cpu=5<br>
> pcs resource utilization partition3 cpu=5<br>
><br>
> pcs constraint colocation add std with sv-clone INFINITY<br>
> pcs constraint colocation add partition1 with sv-clone INFINITY<br>
> pcs constraint colocation add partition2 with sv-clone INFINITY<br>
> pcs constraint colocation add partition3 with sv-clone INFINITY<br>
><br>
> pcs property set maintenance-mode=false<br>
> <br>
><br>
> I can then reproduce the issues in the following way:<br>
><br>
> $ pcs resource<br>
> sv-fencer (ocf::pacemaker:Dummy): Started vm1<br>
> Clone Set: sv-clone [sv]<br>
> Started: [ vm1 vm2 vm3 ]<br>
> std (ocf::pacemaker:Dummy): Started vm2<br>
> partition1 (ocf::pacemaker:Dummy): Started vm3<br>
> partition2 (ocf::pacemaker:Dummy): Started vm1<br>
> partition3 (ocf::pacemaker:Dummy): Started vm2<br>
><br>
> $ pcs cluster standby vm3<br>
><br>
> # Check that all resources have moved off vm3<br>
> $ pcs resource<br>
> sv-fencer (ocf::pacemaker:Dummy): Started vm1<br>
> Clone Set: sv-clone [sv]<br>
> Started: [ vm1 vm2 ]<br>
> Stopped: [ vm3 ]<br>
> std (ocf::pacemaker:Dummy): Started vm2<br>
> partition1 (ocf::pacemaker:Dummy): Started vm1<br>
> partition2 (ocf::pacemaker:Dummy): Started vm1<br>
> partition3 (ocf::pacemaker:Dummy): Started vm2<br>
<br>
</div></div>Thanks for the detailed information, this should help me get to the<br>
bottom of it. From this description, it sounds like a new transition<br>
isn't being triggered when it should.<br>
<br>
Could you please attach the DC's pe-input file that is listed in the<br>
logs after the standby step above? That would simplify analysis.<br>
<div class="HOEnZb"><div class="h5"><br>
> # Wait for any outstanding actions to complete.<br>
> $ crm_resource --wait --timeout 300<br>
> Pending actions:<br>
> Action 22: sv-fencer_monitor_10000 on vm2<br>
> Action 21: sv-fencer_start_0 on vm2<br>
> Action 20: sv-fencer_stop_0 on vm1<br>
> Error performing operation: Timer expired<br>
><br>
> # Check the resources again - sv-fencer is still on vm1<br>
> $ pcs resource<br>
> sv-fencer (ocf::pacemaker:Dummy): Started vm1<br>
> Clone Set: sv-clone [sv]<br>
> Started: [ vm1 vm2 ]<br>
> Stopped: [ vm3 ]<br>
> std (ocf::pacemaker:Dummy): Started vm2<br>
> partition1 (ocf::pacemaker:Dummy): Started vm1<br>
> partition2 (ocf::pacemaker:Dummy): Started vm1<br>
> partition3 (ocf::pacemaker:Dummy): Started vm2<br>
><br>
> # Perform a random update to the CIB.<br>
> $ pcs resource update std op monitor interval=20 timeout=20<br>
><br>
> # Check resource status again - sv_fencer has now moved to vm2 (the<br>
> action crm_resource was waiting for)<br>
> $ pcs resource<br>
> sv-fencer (ocf::pacemaker:Dummy): Started vm2 <<<============<br>
> Clone Set: sv-clone [sv]<br>
> Started: [ vm1 vm2 ]<br>
> Stopped: [ vm3 ]<br>
> std (ocf::pacemaker:Dummy): Started vm2<br>
> partition1 (ocf::pacemaker:Dummy): Started vm1<br>
> partition2 (ocf::pacemaker:Dummy): Started vm1<br>
> partition3 (ocf::pacemaker:Dummy): Started vm2<br>
><br>
> I do not get the problem if I:<br>
> 1) remove the "std" resource; or<br>
> 2) remove the co-location constraints; or<br>
> 3) remove the utilization attributes for the partition resources.<br>
><br>
> In these cases the sv-fencer resource is happy to stay on vm1, and<br>
> crm_resource --wait returns immediately.<br>
><br>
> It looks like the pcs cluster standby call is creating/registering<br>
> the actions to move the sv-fencer resource to vm2, but it doesn't<br>
> include it in the cluster transition. When the CIB is later updated<br>
> by something else, the action is included in that transition.<br>
><br>
><br>
> Regards,<br>
> Leon<br>
<br>
</div></div><div class="HOEnZb"><div class="h5">______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>