[ClusterLabs] Antw: Re: Antw: Unexpected Resource movement after failover
Ken Gaillot
kgaillot at redhat.com
Tue Oct 18 10:25:30 EDT 2016
On 10/17/2016 11:29 PM, Nikhil Utane wrote:
> Thanks Ken.
> I will give it a shot.
>
> http://oss.clusterlabs.org/pipermail/pacemaker/2011-August/011271.html
> On this thread, if I interpret it correctly, his problem was solved when
> he swapped the anti-location constraint
>
> From (mapping to my example)
> cu_2 with cu_4 (score:-INFINITY)
> cu_3 with cu_4 (score:-INFINITY)
> cu_2 with cu_3 (score:-INFINITY)
>
> To
> cu_2 with cu_4 (score:-INFINITY)
> cu_4 with cu_3 (score:-INFINITY)
> cu_3 with cu_2 (score:-INFINITY)
>
> Do you think that would make any difference? The way you explained it,
> sounds to me it might.
It would create a dependency loop:
cu_2 must be placed before cu_3
cu_3 must be placed before cu_4
cu_4 must be placed before cu_2
(loop)
The cluster tries to detect and break such loops, but I wouldn't rely on
that resulting in a particular behavior.
> -Regards
> Nikhil
>
> On Mon, Oct 17, 2016 at 11:36 PM, Ken Gaillot <kgaillot at redhat.com
> <mailto:kgaillot at redhat.com>> wrote:
>
> On 10/17/2016 09:55 AM, Nikhil Utane wrote:
> > I see these prints.
> >
> > pengine: info: rsc_merge_weights:cu_4: Rolling back scores from cu_3
> > pengine: debug: native_assign_node:Assigning Redun_CU4_Wb30 to cu_4
> > pengine: info: rsc_merge_weights:cu_3: Rolling back scores from cu_2
> > pengine: debug: native_assign_node:Assigning Redund_CU5_WB30 to cu_3
> >
> > Looks like rolling back the scores is causing the new decision to
> > relocate the resources.
> > Am I using the scores incorrectly?
>
> No, I think this is expected.
>
> Your anti-colocation constraints place cu_2 and cu_3 relative to cu_4,
> so that means the cluster will place cu_4 first if possible, before
> deciding where the others should go. Similarly, cu_2 has a constraint
> relative to cu_3, so cu_3 gets placed next, and cu_2 is the one left
> out.
>
> The anti-colocation scores of -INFINITY outweigh the stickiness of 100.
> I'm not sure whether setting stickiness to INFINITY would change
> anything; hopefully, it would stop cu_3 from moving, but cu_2 would
> still be stopped.
>
> I don't see a good way around this. The cluster has to place some
> resource first, in order to know not to place some other resource on the
> same node. I don't think there's a way to make them "equal", because
> then none of them could be placed to begin with -- unless you went with
> utilization attributes, as someone else suggested, with
> placement-strategy=balanced:
>
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#idm140521708557280
> <http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#idm140521708557280>
>
> >
> > [root at Redund_CU5_WB30 root]# pcs constraint
> > Location Constraints:
> > Resource: cu_2
> > Enabled on: Redun_CU4_Wb30 (score:0)
> > Enabled on: Redund_CU5_WB30 (score:0)
> > Enabled on: Redund_CU3_WB30 (score:0)
> > Enabled on: Redund_CU1_WB30 (score:0)
> > Resource: cu_3
> > Enabled on: Redun_CU4_Wb30 (score:0)
> > Enabled on: Redund_CU5_WB30 (score:0)
> > Enabled on: Redund_CU3_WB30 (score:0)
> > Enabled on: Redund_CU1_WB30 (score:0)
> > Resource: cu_4
> > Enabled on: Redun_CU4_Wb30 (score:0)
> > Enabled on: Redund_CU5_WB30 (score:0)
> > Enabled on: Redund_CU3_WB30 (score:0)
> > Enabled on: Redund_CU1_WB30 (score:0)
> > Ordering Constraints:
> > Colocation Constraints:
> > cu_2 with cu_4 (score:-INFINITY)
> > cu_3 with cu_4 (score:-INFINITY)
> > cu_2 with cu_3 (score:-INFINITY)
> >
> >
> > On Mon, Oct 17, 2016 at 8:16 PM, Nikhil Utane
> > <nikhil.subscribed at gmail.com <mailto:nikhil.subscribed at gmail.com>
> <mailto:nikhil.subscribed at gmail.com
> <mailto:nikhil.subscribed at gmail.com>>> wrote:
> >
> > This is driving me insane.
> >
> > This is how the resources were started. Redund_CU1_WB30 was the DC
> > which I rebooted.
> > cu_4(ocf::redundancy:RedundancyRA):Started Redund_CU1_WB30
> > cu_2(ocf::redundancy:RedundancyRA):Started Redund_CU5_WB30
> > cu_3(ocf::redundancy:RedundancyRA):Started Redun_CU4_Wb30
> >
> > Since the standby node was not UP. I was expecting resource cu_4 to
> > be waiting to be scheduled.
> > But then it re-arranged everything as below.
> > cu_4(ocf::redundancy:RedundancyRA):Started Redun_CU4_Wb30
> > cu_2(ocf::redundancy:RedundancyRA):Stopped
> > cu_3(ocf::redundancy:RedundancyRA):Started Redund_CU5_WB30
> >
> > There is not much information available in the logs on new DC. It
> > just shows what it has decided to do but nothing to suggest why it
> > did it that way.
> >
> > notice: Start cu_4(Redun_CU4_Wb30)
> > notice: Stop cu_2(Redund_CU5_WB30)
> > notice: Move cu_3(Started Redun_CU4_Wb30 -> Redund_CU5_WB30)
> >
> > I have default stickiness set to 100 which is higher than any score
> > that I have configured.
> > I have migration_threshold set to 1. Should I bump that up instead?
> >
> > -Thanks
> > Nikhil
> >
> > On Sat, Oct 15, 2016 at 12:36 AM, Ken Gaillot <kgaillot at redhat.com <mailto:kgaillot at redhat.com>
> > <mailto:kgaillot at redhat.com <mailto:kgaillot at redhat.com>>> wrote:
> >
> > On 10/14/2016 06:56 AM, Nikhil Utane wrote:
> > > Hi,
> > >
> > > Thank you for the responses so far.
> > > I added reverse colocation as well. However seeing some
> other issue in
> > > resource movement that I am analyzing.
> > >
> > > Thinking further on this, why doesn't "/a not with b"
> does not
> > imply "b
> > > not with a"?/
> > > Coz wouldn't putting "b with a" violate "a not with b"?
> > >
> > > Can someone confirm that colocation is required to be
> configured both ways?
> >
> > The anti-colocation should only be defined one-way. Otherwise,
> > you get a
> > dependency loop (as seen in logs you showed elsewhere).
> >
> > The one-way constraint is enough to keep the resources apart.
> > However,
> > the question is whether the cluster might move resources
> around
> > unnecessarily.
> >
> > For example, "A not with B" means that the cluster will
> place B
> > first,
> > then place A somewhere else. So, if B's node fails, can
> the cluster
> > decide that A's node is now the best place for B, and move
> A to
> > a free
> > node, rather than simply start B on the free node?
> >
> > The cluster does take dependencies into account when placing a
> > resource,
> > so I would hope that wouldn't happen. But I'm not sure.
> Having some
> > stickiness might help, so that A has some preference
> against moving.
> >
> > > -Thanks
> > > Nikhil
> > >
> > > /
> > > /
> > >
> > > On Fri, Oct 14, 2016 at 1:09 PM, Vladislav Bogdanov
> > > <bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>
> <mailto:bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>>
> > <mailto:bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>
> <mailto:bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>>>> wrote:
> > >
> > > On October 14, 2016 10:13:17 AM GMT+03:00, Ulrich Windl
> > > <Ulrich.Windl at rz.uni-regensburg.de
> <mailto:Ulrich.Windl at rz.uni-regensburg.de>
> > <mailto:Ulrich.Windl at rz.uni-regensburg.de
> <mailto:Ulrich.Windl at rz.uni-regensburg.de>>
> > > <mailto:Ulrich.Windl at rz.uni-regensburg.de
> <mailto:Ulrich.Windl at rz.uni-regensburg.de>
> > <mailto:Ulrich.Windl at rz.uni-regensburg.de
> <mailto:Ulrich.Windl at rz.uni-regensburg.de>>>> wrote:
> > > >>>> Nikhil Utane <nikhil.subscribed at gmail.com <mailto:nikhil.subscribed at gmail.com>
> <mailto:nikhil.subscribed at gmail.com
> <mailto:nikhil.subscribed at gmail.com>>
> > > <mailto:nikhil.subscribed at gmail.com
> <mailto:nikhil.subscribed at gmail.com>
> > <mailto:nikhil.subscribed at gmail.com <mailto:nikhil.subscribed at gmail.com>>>>
> schrieb am 13.10.2016 um
> > > >16:43 in
> > > >Nachricht
> > > ><CAGNWmJUbPucnBGXroHkHSbQ0LXovwsLFPkUPg1R8gJqRFqM9Dg at mail.gmail.com
> <mailto:CAGNWmJUbPucnBGXroHkHSbQ0LXovwsLFPkUPg1R8gJqRFqM9Dg at mail.gmail.com>
> > <mailto:CAGNWmJUbPucnBGXroHkHSbQ0LXovwsLFPkUPg1R8gJqRFqM9Dg at mail.gmail.com
> <mailto:CAGNWmJUbPucnBGXroHkHSbQ0LXovwsLFPkUPg1R8gJqRFqM9Dg at mail.gmail.com>>
> > >
> >
> <mailto:CAGNWmJUbPucnBGXroHkHSbQ0LXovwsLFPkUPg1R8gJqRFqM9Dg at mail.gmail.com
> <mailto:CAGNWmJUbPucnBGXroHkHSbQ0LXovwsLFPkUPg1R8gJqRFqM9Dg at mail.gmail.com>
> > <mailto:CAGNWmJUbPucnBGXroHkHSbQ0LXovwsLFPkUPg1R8gJqRFqM9Dg at mail.gmail.com
> <mailto:CAGNWmJUbPucnBGXroHkHSbQ0LXovwsLFPkUPg1R8gJqRFqM9Dg at mail.gmail.com>>>>:
> > > >> Ulrich,
> > > >>
> > > >> I have 4 resources only (not 5, nodes are 5). So then I only need 6
> > > >> constraints, right?
> > > >>
> > > >> [,1] [,2] [,3] [,4] [,5] [,6]
> > > >> [1,] "A" "A" "A" "B" "B" "C"
> > > >> [2,] "B" "C" "D" "C" "D" "D"
> > > >
> > > >Sorry for my confusion. As Andrei Borzenkovsaid in
> > > ><CAA91j0W+epAHFLg9u6VX_X8LgFkf9Rp55g3nocY4oZNA9BbZ+g at mail.gmail.com
> <mailto:CAA91j0W%2BepAHFLg9u6VX_X8LgFkf9Rp55g3nocY4oZNA9BbZ%2Bg at mail.gmail.com>
> > <mailto:CAA91j0W%2BepAHFLg9u6VX_X8LgFkf9Rp55g3nocY4oZNA9BbZ%2Bg at mail.gmail.com
> <mailto:CAA91j0W%252BepAHFLg9u6VX_X8LgFkf9Rp55g3nocY4oZNA9BbZ%252Bg at mail.gmail.com>>
> > >
> >
> <mailto:CAA91j0W%2BepAHFLg9u6VX_X8LgFkf9Rp55g3nocY4oZNA9BbZ%2Bg at mail.gmail.com
> <mailto:CAA91j0W%252BepAHFLg9u6VX_X8LgFkf9Rp55g3nocY4oZNA9BbZ%252Bg at mail.gmail.com>
> >
> <mailto:CAA91j0W%252BepAHFLg9u6VX_X8LgFkf9Rp55g3nocY4oZNA9BbZ%252Bg at mail.gmail.com
> <mailto:CAA91j0W%25252BepAHFLg9u6VX_X8LgFkf9Rp55g3nocY4oZNA9BbZ%25252Bg at mail.gmail.com>>>>
> > > >you probably have to add (A, B) _and_ (B, A)! Thinking about it, I
> > > >wonder whether an easier solution would be using "utilization": If
> > > >every node has one token to give, and every resource needs on token, no
> > > >two resources will run on one node. Sounds like an easier solution to
> > > >me.
> > > >
> > > >Regards,
> > > >Ulrich
> > > >
> > > >
> > > >>
> > > >> I understand that if I configure constraint of R1 with R2 score as
> > > >> -infinity, then the same applies for R2 with R1 score as -infinity
> > > >(don't
> > > >> have to configure it explicitly).
> > > >> I am not having a problem of multiple resources getting schedule on
> > > >the
> > > >> same node. Rather, one working resource is unnecessarily getting
> > > >relocated.
> > > >>
> > > >> -Thanks
> > > >> Nikhil
> > > >>
> > > >>
> > > >> On Thu, Oct 13, 2016 at 7:45 PM, Ulrich Windl <
> > > >> Ulrich.Windl at rz.uni-regensburg.de
> <mailto:Ulrich.Windl at rz.uni-regensburg.de>
> > <mailto:Ulrich.Windl at rz.uni-regensburg.de
> <mailto:Ulrich.Windl at rz.uni-regensburg.de>>
> > > <mailto:Ulrich.Windl at rz.uni-regensburg.de
> <mailto:Ulrich.Windl at rz.uni-regensburg.de>
> > <mailto:Ulrich.Windl at rz.uni-regensburg.de
> <mailto:Ulrich.Windl at rz.uni-regensburg.de>>>> wrote:
> > > >>
> > > >>> Hi!
> > > >>>
> > > >>> Don't you need 10 constraints, excluding every possible pair of your
> > > >5
> > > >>> resources (named A-E here), like in this table (produced with R):
> > > >>>
> > > >>> [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
> > > >>> [1,] "A" "A" "A" "A" "B" "B" "B" "C" "C" "D"
> > > >>> [2,] "B" "C" "D" "E" "C" "D" "E" "D" "E" "E"
> > > >>>
> > > >>> Ulrich
> > > >>>
> > > >>> >>> Nikhil Utane <nikhil.subscribed at gmail.com <mailto:nikhil.subscribed at gmail.com>
> <mailto:nikhil.subscribed at gmail.com
> <mailto:nikhil.subscribed at gmail.com>>
> > > <mailto:nikhil.subscribed at gmail.com
> <mailto:nikhil.subscribed at gmail.com>
> > <mailto:nikhil.subscribed at gmail.com <mailto:nikhil.subscribed at gmail.com>>>>
> schrieb am 13.10.2016
> > > >um
> > > >>> 15:59 in
> > > >>> Nachricht
> > > >>>
> > > ><CAGNWmJW0CWMr3bvR3L9xZCAcJUzyczQbZEzUzpaJxi+Pn7Oj_A at mail.gmail.com
> <mailto:CAGNWmJW0CWMr3bvR3L9xZCAcJUzyczQbZEzUzpaJxi%2BPn7Oj_A at mail.gmail.com>
> > <mailto:CAGNWmJW0CWMr3bvR3L9xZCAcJUzyczQbZEzUzpaJxi%2BPn7Oj_A at mail.gmail.com
> <mailto:CAGNWmJW0CWMr3bvR3L9xZCAcJUzyczQbZEzUzpaJxi%252BPn7Oj_A at mail.gmail.com>>
> > >
> >
> <mailto:CAGNWmJW0CWMr3bvR3L9xZCAcJUzyczQbZEzUzpaJxi%2BPn7Oj_A at mail.gmail.com
> <mailto:CAGNWmJW0CWMr3bvR3L9xZCAcJUzyczQbZEzUzpaJxi%252BPn7Oj_A at mail.gmail.com>
> >
> <mailto:CAGNWmJW0CWMr3bvR3L9xZCAcJUzyczQbZEzUzpaJxi%252BPn7Oj_A at mail.gmail.com
> <mailto:CAGNWmJW0CWMr3bvR3L9xZCAcJUzyczQbZEzUzpaJxi%25252BPn7Oj_A at mail.gmail.com>>>>:
> > > >>> > Hi,
> > > >>> >
> > > >>> > I have 5 nodes and 4 resources configured.
> > > >>> > I have configured constraint such that no two
> > resources can be
> > > >>> co-located.
> > > >>> > I brought down a node (which happened to be DC). I
> > was expecting
> > > >the
> > > >>> > resource on the failed node would be migrated
> to the
> > 5th waiting
> > > >node
> > > >>> (that
> > > >>> > is not running any resource).
> > > >>> > However what happened was the failed node resource
> > was started on
> > > >another
> > > >>> > active node (after stopping it's existing
> resource)
> > and that
> > > >node's
> > > >>> > resource was moved to the waiting node.
> > > >>> >
> > > >>> > What could I be doing wrong?
> > > >>> >
> > > >>> > <nvpair id="cib-bootstrap-options-have-watchdog"
> > value="true"
> > > >>> > name="have-watchdog"/>
> > > >>> > <nvpair id="cib-bootstrap-options-dc-version"
> > > >value="1.1.14-5a6cdd1"
> > > >>> > name="dc-version"/>
> > > >>> > <nvpair
> > id="cib-bootstrap-options-cluster-infrastructure"
> > > >>> value="corosync"
> > > >>> > name="cluster-infrastructure"/>
> > > >>> > <nvpair id="cib-bootstrap-options-stonith-enabled"
> > value="false"
> > > >>> > name="stonith-enabled"/>
> > > >>> > <nvpair
> id="cib-bootstrap-options-no-quorum-policy"
> > value="ignore"
> > > >>> > name="no-quorum-policy"/>
> > > >>> > <nvpair
> > id="cib-bootstrap-options-default-action-timeout"
> > > >value="240"
> > > >>> > name="default-action-timeout"/>
> > > >>> > <nvpair
> id="cib-bootstrap-options-symmetric-cluster"
> > value="false"
> > > >>> > name="symmetric-cluster"/>
> > > >>> >
> > > >>> > # pcs constraint
> > > >>> > Location Constraints:
> > > >>> > Resource: cu_2
> > > >>> > Enabled on: Redun_CU4_Wb30 (score:0)
> > > >>> > Enabled on: Redund_CU2_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU3_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU5_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU1_WB30 (score:0)
> > > >>> > Resource: cu_3
> > > >>> > Enabled on: Redun_CU4_Wb30 (score:0)
> > > >>> > Enabled on: Redund_CU2_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU3_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU5_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU1_WB30 (score:0)
> > > >>> > Resource: cu_4
> > > >>> > Enabled on: Redun_CU4_Wb30 (score:0)
> > > >>> > Enabled on: Redund_CU2_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU3_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU5_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU1_WB30 (score:0)
> > > >>> > Resource: cu_5
> > > >>> > Enabled on: Redun_CU4_Wb30 (score:0)
> > > >>> > Enabled on: Redund_CU2_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU3_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU5_WB30 (score:0)
> > > >>> > Enabled on: Redund_CU1_WB30 (score:0)
> > > >>> > Ordering Constraints:
> > > >>> > Colocation Constraints:
> > > >>> > cu_3 with cu_2 (score:-INFINITY)
> > > >>> > cu_4 with cu_2 (score:-INFINITY)
> > > >>> > cu_4 with cu_3 (score:-INFINITY)
> > > >>> > cu_5 with cu_2 (score:-INFINITY)
> > > >>> > cu_5 with cu_3 (score:-INFINITY)
> > > >>> > cu_5 with cu_4 (score:-INFINITY)
> > > >>> >
> > > >>> > -Thanks
> > > >>> > Nikhil
> > > >>>
> > > >>>
> > > >>>
> > >
> > > Hi,
> > >
> > > use of utilization (balanced strategy) has one caveat:
> > resources are
> > > not moved just because of utilization of one node is
> less,
> > when
> > > nodes have the same allocation score for the resource.
> > > So, after the simultaneus outage of two nodes in a
> 5-node
> > cluster,
> > > it may appear that one node runs two resources and two
> > recovered
> > > nodes run nothing.
> > >
> > > Original 'utilization' strategy only limits resource
> > placement, it
> > > is not considered when choosing a node for a resource.
> > >
> > > Vladislav
>
>
More information about the Users
mailing list