[ClusterLabs Developers] [deveoplers] maintenance vs is-managed; different levels of the maintenance property

[Yan Gao]

On 11/28/19 1:19 AM, Ken Gaillot wrote:
> On Wed, 2019-11-27 at 12:13 +0000, Yan Gao wrote:
>> Hi all,
>>
>> First thanks for bringing this up, Aleksei.
>>
>> On 11/26/19 3:38 PM, Aleksei Burlakov wrote:
>>> Dear Developers,
>>>
>>> I would like to raise a discussion about an issue/feature about
>>> the
>>> maintenance property applied to different levels of a cluster.
>>>
>>> In order to explain the problem lets consider several examples.
>>>
>>> *Scenario 1*. There is a primitive p1 in a group g1. When applying
>>> the
>>> maintenance property to them, the most specific resource will take
>>> precedence. Namely, if p1 has the maintenance property set, it will
>>> be
>>> used, otherwise, the maintenance property of the g1 is used. It's
>>> ok.
>>>
>>> *Scenario 1*. There is a clone of a group c1, group g1, and a
>>> primitive
>>> p1, that belongs to the group g1. When we apply the maintenance
>>> attribute, the most specific attribute takes precedence, but the
>>> attribute of the clone c1 may have different value than the group
>>> and
>>> the primitive. It's strange, but still ok.
>>>
>>> *Scenario 3*. There is a cluster with one node node1 and one
>>> primitive
>>> p1. This time when setting the maintenance/maintenance-mode
>>> attribute to
>>> the primitive/node1/default-property the p1 will be true when
>>> either one
>>> of them is set to true. It means the most specific attribute don't
>>> precede anymore. One may think it's the feature, but imho its a
>>> *bug*.
> 
> One source of confusion is that maintenance mode can be set cluster-
> wide via the "maintenance-mode" property, for a particular node via the
> "maintenance" node attribute, or for a particular resource via the
> "maintenance" resource meta-attribute.
> 
> Separately, the resource meta-attribute may be set in rsc_defaults, the
> primitive, or any collective resource (clone or group) enclosing it.
> 
> How the different options combine is separate from how resource meta-
> attributes at different levels combine.
> 
> When comparing meta-attributes at different levels, it is true that the
> most specific attribute wins. The primitive wins over the collective
> resource, which wins over the rsc_defaults.
> 
> When comparing the different options, it's a different story. There
> really isn't such a thing as "more specific" -- a node is not more or
> less specific than a resource (consider a clone running on all nodes,
> which is more specific?). So in that case, the resource is put in
> maintenance if either the resource or the node is in maintenance mode.
> 
> The cluster-wide property forces everything into maintenance. Note that
> it's different from setting the resource meta-attribute to true in
> rsc_defaults, which can be overridden by a more specific setting in the
> resource itself.
> 
> I agree it's confusing but I'm not sure a different approach would be
> any more obvious.
> 
>>> *Scenario 4*. There is only a primitive p1. We apply  both: an
>>> is-managed attribute and maintenance attribute This is it will work
>>> as
>>>           is-managed        maintenance        |               p1
>>>                false                      false
>>> |
>>>    unmanaged
>>
>> Since it seems that "maintenance" attribute tends to takes
>> precedence
>> over "is-managed" attribute, this combination probably is the only
>> problematic case. I think it probably needs an "else" in here:
>>
>> https://github.com/ClusterLabs/pacemaker/blob/master/lib/pengine/complex.c#L531
>>
>> :-)
>>
>>>                false                      true
>>>   |
>>>    unmanaged
>>>                true                       false
>>> |
>>>    managed
>>>                true                       true
>>>   |
>>>    unmanaged
>>> that works quite unexpectedly.
>>>
>>> In the more complex scenarios, where the is-managed property is
>>> used
>>> together with the maintenance, the status resources get
>>> unpredictable,
>>> which is definitely *not ok*.
> 
> I think it's inherently confusing that both is-managed and maintenance
> exist. For resources they function identically except monitors will
> continue to run if a resource is unmanaged but not if it's in
> maintenance mode.
> 
> It's especially confusing since is-managed is positive (true is the
> default situation) whereas maintenance is negative (false is the
> default). So their senses are flipped (is-managed=false is comparable
> to maintenance=true).
> 
> It might have been a clearer design if there was one option for
> maintenance mode (no is-managed), and a second option for whether
> monitors continue to run in maintenance mode. Or, resource maintenance
> could take three values (for off, on-with-monitor, and on-without-
> monitor).
> 
> The table of results you got makes sense if you consider that only if
> both options are in their default state (is-managed=true and
> maintenance=false) is the resource in the default (managed) state.
> Flipping either option is an intention to unmanage the resource.
Agreed.

> 
>>> The *counterargument* is that it was always like this and the
>>> customers
>>> are used to this behavior. They may remember that a certain
>>> combination
>>> of attributes leads to a certain status and enforcing the most
>>> specific
>>> rule would lead to the *change of behaviour and backward
>>> incompatibility*.
>>
>> Despite how the conflicting attributes/settings should be handled
>> correctly at resource levels, I think one main question here probably
>> is:
>>
>> Should maintenance mode of node/cluster scope consider any specific
>> conflicting settings of resources? Or should it just put the whole
>> scope
>> (node/cluster) into maintenance mode without exceptions, as how it is
>> now?
> 
> It's a good question. I don't think either answer is obviously
> intuitive, so it boils down to documenting it. Which we should do :)
> 
> There is some room for coming up with better option naming and meaning.
> For example maybe the cluster-wide "maintenance-mode" should be
> something like "force-maintenance" to make clear it takes precedence
> over node and resource maintenance.
Not sure if renaming would introduce even more confusion ... But indeed, 
documentation definitely makes lot of sense.

Based on the whole idea, an inconsistent logic is in here then:

https://github.com/ClusterLabs/pacemaker/commit/9a8cb86573#diff-b4b7b0fdcefcd3eb5087dfbf0d101ec4R471

We should probably remove the "else" there, so that cluster-wide 
maintenance-mode=true ALWAYS takes precedence.

Currently there's a corner case:

* Cluster maintenance-mode=true
* Resource is-managed=true
* Resource maintenance=false

, which makes an exception that the resource will be "managed".

Regards,
   Yan



> 
>>
>> Regards,
>>     Yan
>>
>>>
>>> Please share your opinion about the issue, if we should leave it
>>> working
>>> as is or enforce the most specific rule in though the whole
>>> cluster. And
>>> give a priority to either one of the conflicting attributes (is-
>>> managed
>>> vs maintenance).
>>>
>>> Best regards,
>>> Aleksei Burlakov
>>> SUSE Senior Developer