[ClusterLabs] Pacemaker resource is not tried to be recovered after failure on slave node even when failcount is less than migration-threshold

[Pankaj]

Hi,

Could you please help me with below query.

I have a stateful resource, stateful_ms, defined as below. The
migration-threshold is defined as 4 and resource-stickiness as 100.
I have pacemaker cluster of 5 nodes. First time resource(stateful_ms) is up
as MASTER on node-0 and as slave in other nodes.
I made monitor of stateful_ms to fail on node-0. As expected, after
reaching failcount=4 on node-0, resource instance on node-1 is promoted to
be MASTER.
But, when monitor, of stateful_ms in node-1, is made to fail, it was seen
that resource on node-0 is promoted immediately even failcount was 1 on
node-1 against defined migration-threshold=4.

Could you please help me to understand:
1. Why resource instance on node-1 is not promoted again when its failcount
is still less than migration-threshold?
2. How we can make sure than first resource is tried to be recovered on
current node(where it failed), as per migration-threshold, then only any
other node instance is promoted?

Below are the setup details:
resource configure
crm configure primitive stateful_dummy ocf:pacemaker:Stateful op monitor
interval="5" role="Master" timeout="20" op monitor interval="10"
role="Slave" timeout="20" meta resource-stickiness="100"
crm configure ms stateful_ms stateful_dummy meta resource-stickiness="100"
notify="true" master-max="1" interleave="true" migration-threshold=4
failure-timeout=60

# crm status
Stack: corosync
Current DC: NODE-0 (version 1.1.16-94ff4df51a) - partition with quorum
Last updated: Tue Nov 14 12:54:52 2017
Last change: Tue Nov 14 12:54:43 2017 by root via cibadmin on NODE-1

5 nodes configured

 Master/Slave Set: stateful_ms [stateful_dummy]
     Masters: [ NODE-0 ]
     Slaves: [ NODE-1 NODE-2 NODE-3 NODE-4 ]


# crm configure show
node 1: NODE-0
node 2: NODE-1
node 3: NODE-2
node 4: NODE-3
node 5: NODE-4

primitive stateful_dummy ocf:pacemaker:Stateful \
        op monitor interval=5 role=Master timeout=20 \
        op monitor interval=10 role=Slave timeout=20 \
        meta resource-stickiness=100
ms stateful_ms stateful_dummy \
        meta resource-stickiness=100 notify=true master-max=1
interleave=true migration-threshold=4 failure-timeout=60 target-role=Started


#crm resource score
Allocation scores and utilization information:
Original: NODE-0 capacity:
Original: NODE-1 capacity:
Original: NODE-2 capacity:
Original: NODE-3 capacity:
Original: NODE-4 capacity:
clone_color: stateful_ms allocation score on NODE-0: 0
clone_color: stateful_ms allocation score on NODE-1: 0
clone_color: stateful_ms allocation score on NODE-2: 0
clone_color: stateful_ms allocation score on NODE-3: 0
clone_color: stateful_ms allocation score on NODE-4: 0
clone_color: stateful_dummy:0 allocation score on NODE-0: 110
clone_color: stateful_dummy:0 allocation score on NODE-1: 0
clone_color: stateful_dummy:0 allocation score on NODE-2: 0
clone_color: stateful_dummy:0 allocation score on NODE-3: 0
clone_color: stateful_dummy:0 allocation score on NODE-4: 0
clone_color: stateful_dummy:1 allocation score on NODE-0: 0
clone_color: stateful_dummy:1 allocation score on NODE-1: 105
clone_color: stateful_dummy:1 allocation score on NODE-2: 0
clone_color: stateful_dummy:1 allocation score on NODE-3: 0
clone_color: stateful_dummy:1 allocation score on NODE-4: 0
clone_color: stateful_dummy:2 allocation score on NODE-0: 0
clone_color: stateful_dummy:2 allocation score on NODE-1: 0
clone_color: stateful_dummy:2 allocation score on NODE-2: 105
clone_color: stateful_dummy:2 allocation score on NODE-3: 0
clone_color: stateful_dummy:2 allocation score on NODE-4: 0
clone_color: stateful_dummy:3 allocation score on NODE-0: 0
clone_color: stateful_dummy:3 allocation score on NODE-1: 0
clone_color: stateful_dummy:3 allocation score on NODE-2: 0
clone_color: stateful_dummy:3 allocation score on NODE-3: 105
clone_color: stateful_dummy:3 allocation score on NODE-4: 0
clone_color: stateful_dummy:4 allocation score on NODE-0: 0
clone_color: stateful_dummy:4 allocation score on NODE-1: 0
clone_color: stateful_dummy:4 allocation score on NODE-2: 0
clone_color: stateful_dummy:4 allocation score on NODE-3: 0
clone_color: stateful_dummy:4 allocation score on NODE-4: 105
native_color: stateful_dummy:2 allocation score on NODE-0: 0
native_color: stateful_dummy:2 allocation score on NODE-1: 0
native_color: stateful_dummy:2 allocation score on NODE-2: 105
native_color: stateful_dummy:2 allocation score on NODE-3: 0
native_color: stateful_dummy:2 allocation score on NODE-4: 0
native_assign_node: stateful_dummy:2 utilization on NODE-2:
native_color: stateful_dummy:4 allocation score on NODE-0: 0
native_color: stateful_dummy:4 allocation score on NODE-1: 0
native_color: stateful_dummy:4 allocation score on NODE-2: -INFINITY
native_color: stateful_dummy:4 allocation score on NODE-3: 0
native_color: stateful_dummy:4 allocation score on NODE-4: 105
native_assign_node: stateful_dummy:4 utilization on NODE-4:
native_color: stateful_dummy:1 allocation score on NODE-0: 0
native_color: stateful_dummy:1 allocation score on NODE-1: 105
native_color: stateful_dummy:1 allocation score on NODE-2: -INFINITY
native_color: stateful_dummy:1 allocation score on NODE-3: 0
native_color: stateful_dummy:1 allocation score on NODE-4: -INFINITY
native_assign_node: stateful_dummy:1 utilization on NODE-1:
native_color: stateful_dummy:3 allocation score on NODE-0: 0
native_color: stateful_dummy:3 allocation score on NODE-1: -INFINITY
native_color: stateful_dummy:3 allocation score on NODE-2: -INFINITY
native_color: stateful_dummy:3 allocation score on NODE-3: 105
native_color: stateful_dummy:3 allocation score on NODE-4: -INFINITY
native_assign_node: stateful_dummy:3 utilization on NODE-3:
native_color: stateful_dummy:0 allocation score on NODE-0: 110
native_color: stateful_dummy:0 allocation score on NODE-1: -INFINITY
native_color: stateful_dummy:0 allocation score on NODE-2: -INFINITY
native_color: stateful_dummy:0 allocation score on NODE-3: -INFINITY
native_color: stateful_dummy:0 allocation score on NODE-4: -INFINITY
native_assign_node: stateful_dummy:0 utilization on NODE-0:
stateful_dummy:0 promotion score on NODE-0: 10
stateful_dummy:1 promotion score on NODE-1: 5
stateful_dummy:2 promotion score on NODE-2: 5
stateful_dummy:3 promotion score on NODE-3: 5
stateful_dummy:4 promotion score on NODE-4: 5

Regards,
Pankaj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20171127/f58774b1/attachment-0002.html>