[ClusterLabs] Colocation constraint moving resource

[Andrei Borzenkov]

26.03.2019 17:14, Ken Gaillot пишет:
> On Tue, 2019-03-26 at 14:11 +0100, Thomas Singleton wrote:
>> Dear all
>>
>> I am encountering an issue with colocation constraints. 
>>
>> I have created a 4 nodes cluster (3 "main" and 1 "spare") with 3
>> resources and I wish to have each resource run only on its own node
>> (or
>> on the spare) and resources must never run together on the spare.
>>
>> I understand that this implies a definition of priorities between
>> resources should two nodes fail at the same time. This is the desired
>> behavior. Resource 1 is more important than resource 2 which is more
>> important than resource 3. And thus in the case of multiple nodes
>> failure, the spare node must be running the higher priority resource
>> even if this means the lower priority resources will be stopped.
>>
>> When the resources are created with the adequate priorities but
>> without
>> colocation constraint they are indeed running on each "main" node as
>> expected.
>>
>> Trouble arises when I start adding the colocation constraints.
>>
>> If I add one colocation constraint (resource3 cannot run with
>> resource2), resources remain correctly on their node. 
>>
>> But as soon as I add a second colocation constraint (resource2 cannot
>> run with resource1), resource1 switches to the spare node because the
>> resource1 allocation score on node1 becomes -INFINITY and I cannot
>> understand why ?
>>
>> Setup and commands output below
>>
>> Thank you
>>
>>
>> ****************
>>
>> Resources definition, opt-in cluster
>>
>> # pcs property set symmetric-cluster=false
>> # pcs resource create TestResourceNode1 ocf:pacemaker:Dummy op
>> monitor
>> interval=120s
>> # pcs constraint location TestResourceNode1 prefers node1=100
>> # pcs constraint location TestResourceNode1 prefers nodespare=80
>> # pcs resource create TestResourceNode2 ocf:pacemaker:Dummy op
>> monitor
>> interval=120s
>> # pcs constraint location TestResourceNode2 prefers node2=50
>> # pcs constraint location TestResourceNode2 prefers nodespare=30
>> # pcs resource create TestResourceNode3 ocf:pacemaker:Dummy op
>> monitor
>> interval=120s
>> # pcs constraint location TestResourceNode3 prefers node3=10
>> # pcs constraint location TestResourceNode3 prefers nodespare=3
> 
> Side comment: Using different location constraint scores for each
> resource doesn't establish a priority of resources if they can't all be
> run. For that, there is an actual "priority" meta-attribute for
> resources, so you want to set that for all three.
> 
>> # crm_simulate -sL
>>
>> Current cluster status:
>> Online: [ node1 node2 node3 nodespare ]
>>
>>  TestResourceNode1	(ocf::pacemaker:Dummy):	Started node1
>>  TestResourceNode2	(ocf::pacemaker:Dummy):	Started node2
>>  TestResourceNode3	(ocf::pacemaker:Dummy):	Started node3
>>
>> Allocation scores:
>> native_color: TestResourceNode1 allocation score on node1: 100
>> native_color: TestResourceNode1 allocation score on nodespare: 80
>> native_color: TestResourceNode2 allocation score on node2: 50
>> native_color: TestResourceNode2 allocation score on nodespare: 30
>> native_color: TestResourceNode3 allocation score on node3: 10
>> native_color: TestResourceNode3 allocation score on nodespare: 3
>>
>> # pcs constraint colocation add TestResourceNode3 with
>> TestResourceNode2 score=-INFINITY
>> # crm_simulate -sL
>>
>> Current cluster status:
>> Online: [ node1 node2 node3 nodespare ]
>>
>>  TestResourceNode1	(ocf::pacemaker:Dummy):	Started node1
>>  TestResourceNode2	(ocf::pacemaker:Dummy):	Started node2
>>  TestResourceNode3	(ocf::pacemaker:Dummy):	Started node3
>>
>> Allocation scores:
>> native_color: TestResourceNode1 allocation score on node1: 100
>> native_color: TestResourceNode1 allocation score on nodespare: 80
>> native_color: TestResourceNode2 allocation score on node2: 50
>> native_color: TestResourceNode2 allocation score on nodespare: 27
>> native_color: TestResourceNode3 allocation score on node3: 10
>> native_color: TestResourceNode3 allocation score on nodespare: 3
>>
>> # pcs constraint colocation add TestResourceNode2 with
>> TestResourceNode1 score=-INFINITY
>> # crm_simulate -sL
>>
>> Current cluster status:
>> Online: [ node1 node2 node3 nodespare ]
>>
>>  TestResourceNode1	(ocf::pacemaker:Dummy):	Started
>> nodespare
>>  TestResourceNode2	(ocf::pacemaker:Dummy):	Started node2
>>  TestResourceNode3	(ocf::pacemaker:Dummy):	Started node3
>>
>> Allocation scores:
>> native_color: TestResourceNode1 allocation score on node1: -INFINITY
>> native_color: TestResourceNode1 allocation score on nodespare: 53
>> native_color: TestResourceNode2 allocation score on node2: 50
>> native_color: TestResourceNode2 allocation score on nodespare:
>> -INFINITY
>> native_color: TestResourceNode3 allocation score on node3: 10
>> native_color: TestResourceNode3 allocation score on nodespare: 3
> 
> This seems like a bug to me. Can you attach (or e-mail me privately)
> the pe-input file that led to the above situation?
> 

What apparently happens is the problem with INFINITY math. We have chain

TestResourceNode3 -> TestResourceNode2 -> TestResourceNode1

from colocation constraints

colocate(TestResourceNode3, TestResourceNode2, -INFINITY)
colocate(TestResourceNode2, TestResourceNode1, -INFINITY)

TestResourceNode1 gets score 100 on node1 and then tries to include
score of TestResourceNode2. Factor is -1 (-INFINITY/INFINITY), score on
node is -INFINITY so result is 100 + (-1)*(-INFINITY) == INFINITY. Next
step is to include scores of TestResourceNode3. The problem is,
rsc_merge_wights flips factor:

    if (factor < 0) {
        multiplier = -1;
    }
...
            work = rsc_merge_weights(other, rhs, work,
constraint->node_attribute,
                                     multiplier *
(float)constraint->score / INFINITY, flags|pe_weights_rollback);


so factor becomes (-1)*(-INFINITY/INFINITY) == 1, score of
TestResourceNode3 on node1 is -INFINITY so pacemaker adds
(1)*(-INFINITY) == -INFINITY with final result -INFINITY, blocking
TestResourceNode1 from node1.

Same happens with sparenode. We start with score 80, subtract 30 for
anti-colocation with TestResourceNode2 and then add 3 due to
anti-colocation between TestResourceNode3 and TestResourceNode2.

I am more surprised TestResourceNode2 does not get INFINITY with just
one colocation contstraint, but I freely admit I to not understand the code.