[ClusterLabs] ClusterIP won't return to recovered node

[Dan Ragle]

On 6/16/2017 3:08 PM, Ken Gaillot wrote:
> On 06/16/2017 01:18 PM, Dan Ragle wrote:
>>
>>
>> On 6/12/2017 10:30 AM, Ken Gaillot wrote:
>>> On 06/12/2017 09:23 AM, Klaus Wenninger wrote:
>>>> On 06/12/2017 04:02 PM, Ken Gaillot wrote:
>>>>> On 06/10/2017 10:53 AM, Dan Ragle wrote:
>>>>>> So I guess my bottom line question is: How does one tell Pacemaker
>>>>>> that
>>>>>> the individual legs of globally unique clones should *always* be
>>>>>> spread
>>>>>> across the available nodes whenever possible, regardless of the number
>>>>>> of processes on any one of the nodes? For kicks I did try:
>>>>>>
>>>>>> pcs constraint location ClusterIP:0 prefers node1-pcs=INFINITY
>>>>>>
>>>>>> but it responded with an error about an invalid character (:).
>>>>> There isn't a way currently. It will try to do that when initially
>>>>> placing them, but once they've moved together, there's no simple way to
>>>>> tell them to move. I suppose a workaround might be to create a dummy
>>>>> resource that you constrain to that node so it looks like the other
>>>>> node
>>>>> is less busy.
>>>>
>>>> Another ugly dummy resource idea - maybe less fragile -
>>>> and not tried out:
>>>> One could have 2 dummy resources that would rather like
>>>> to live on different nodes - no issue with primitives - and
>>>> do depend collocated on ClusterIP.
>>>> Wouldn't that pull them apart once possible?
>>>
>>> Sounds like a good idea
>>
>> Hmmmm... still no luck with this.
>>
>> Based on your suggestion, I thought this would work (leaving out all the
>> status displays this time):
>>
>> # pcs resource create Test1 systemd:test1
>> # pcs resource create Test2 systemd:test2
>> # pcs constraint location Test1 prefers node1-pcs=INFINITY
>> # pcs constraint location Test2 prefers node1-pcs=INFINITY
>> # pcs resource create Test3 systemd:test3
>> # pcs resource create Test4 systemd:test4
>> # pcs constraint location Test3 prefers node1-pcs=INFINITY
>> # pcs constraint location Test4 prefers node2-pcs=INFINITY
>> # pcs resource create ClusterIP ocf:heartbeat:IPaddr2 ip=162.220.75.138
>> nic=bond0 cidr_netmask=24
>> # pcs resource meta ClusterIP resource-stickiness=0
>> # pcs resource clone ClusterIP clone-max=2 clone-node-max=2
>> globally-unique=true
>> # pcs constraint colocation add ClusterIP-clone with Test3 INFINITY
>> # pcs constraint colocation add ClusterIP-clone with Test4 INFINITY
>>
>> But that simply refuses to run ClusterIP at all ("Resource ClusterIP:0/1
>> cannot run anywhere"). And if I change the last two colocation
>> constraints to a numeric then it runs, but with the same problem I had
>> before (both ClusterIP instances on one node).
>>
>> I also tried it reversing the colocation definition (add Test3 with
>> ClusterIP-clone) and trying differing combinations of scores between the
>> location and colocation constraints, still with no luck.
>>
>> Thanks,
>>
>> Dan
> 
> Ah of course, the colocation with both means they all have to run on the
> same node, which is impossible.
> 
> FYI you can create dummy resources with ocf:pacemaker:Dummy so you don't
> have to write your own agents.

Good to know, thanks.

> 
> OK, this is getting even hackier, but I'm thinking you can use
> utilization for this:
> 
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#idm139683960632560
> 
> * Create two dummy resources, each with a -INFINITY location preference
> for one of the nodes, so each is allowed to run on only one node.
> 
> * Set the priority meta-attribute to a positive number on all your real
> resources, and leave the dummies at 0 (so if the cluster can't run all
> of them, it will stop the dummies first).
> 
> * Set placement-strategy=utilization.
> 
> * Define a utilization attribute, with values for each node and resource
> like this:
> ** Set a utilization of 1 on all resources except the dummies and the
> clone, so that their total utilization is N.
> ** Set a utilization of 100 on the dummies and the clone.
> ** Set a utilization capacity of 200 + N on each node.
> 
> (I'm assuming you never expect to have more than 99 other resources. If
> that's not the case, just raise the 100 usage accordingly.)
> 
> With those values, if only one node is up, that node can host all the
> real resources (including both clone instances), with the dummies
> stopped. If both nodes are up, the only way the cluster can run all
> resources (including the clone instances and dummies) is to spread the
> clone instances out.
> 
> Again, it's hacky, and I haven't tested it, but I think it would work.

Interesting. That does seem to work, at least in my reduction; I've not 
yet tried it in my actual real-world setup yet. A few notes, though:

1. I had to set placement-strategy=balanced. When set to utilization the 
IP clones still would not split following a standby/unstandby of one of 
the nodes.

2. I still had to remember to have resource-stickiness=0 on the 
ClusterIP primitives. Without it, after standby/unstandby the clones 
still both preferred to stay where they were, with one of the dummies 
running on the other node and the second dummy stopped.

3. Rather than set the priority on the "real" resources to 1, I set the 
priority on the dummy resources to -1 (so that, going forward, it's one 
less thing I'd have to remember to do when adding new resources). 
*Seems* to work just as well.

4. If I'm following the logic correctly, I *think* I could set the node 
utilization capacity setting to anything LESS THAN the (total clones (2) 
+ 1 dummy * 100) and greater than or equal to the number of clones * 100 
plus the number of real resources. I.E., for this example anything less 
than 300, but greater than or equal to 204 (since I have four Real 
resources in my final example). Again, this leaves a little room for the 
addition of new resources without having to remember to up the capacity 
for the nodes.

All that said, and for anyone interested, here's the recipe I tried that 
appears to work well. After setting it up this way, I was able to 
standby/unstandby each of the nodes in turn with the clones consistently 
re-splitting after each unstandby (and the RealN resources all remaining 
on node1 if it was available and node otherwise):

# pcs property set placement-strategy=balanced

# pcs node utilization node1-pcs weight=250
# pcs node utilization node2-pcs weight=250

# pcs resource create Real1 ocf:pacemaker:Dummy
# pcs resource create Real2 ocf:pacemaker:Dummy
# pcs resource create Real3 ocf:pacemaker:Dummy
# pcs resource create Real4 ocf:pacemaker:Dummy
# pcs resource utilization Real1 weight=1
# pcs resource utilization Real2 weight=1
# pcs resource utilization Real3 weight=1
# pcs resource utilization Real4 weight=1
# pcs constraint location Real1 prefers node1-pcs=INFINITY
# pcs constraint location Real2 prefers node1-pcs=INFINITY
# pcs constraint location Real3 prefers node1-pcs=INFINITY
# pcs constraint location Real4 prefers node1-pcs=INFINITY

# pcs resource create Dummy1 ocf:pacemaker:Dummy meta priority=-1
# pcs resource create Dummy2 ocf:pacemaker:Dummy meta priority=-1
# pcs constraint location Dummy1 prefers node2-pcs=-INFINITY
# pcs constraint location Dummy2 prefers node1-pcs=-INFINITY
# pcs resource utilization Dummy1 weight=100
# pcs resource utilization Dummy2 weight=100

# pcs resource create ClusterIP ocf:heartbeat:IPaddr2 ip=1.2.3.4 
nic=bond0 cidr_netmask=24 meta resource-stickiness=0
# pcs resource utilization ClusterIP weight=100
# pcs resource clone ClusterIP clone-max=2 clone-node-max=2 
globally-unique=true

Dan

> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>