[ClusterLabs] Antw: Re: Antw: Re: Q: monitor and probe result codes and consequences
kgaillot at redhat.com
Fri May 13 16:25:26 EDT 2016
On 05/13/2016 06:00 AM, Ulrich Windl wrote:
>>>> Dejan Muhamedagic <dejanmm at fastmail.fm> schrieb am 13.05.2016 um 12:16 in
> Nachricht <20160513101626.GA12493 at walrus.homenet>:
>> On Fri, May 13, 2016 at 09:05:54AM +0200, Ulrich Windl wrote:
>>>>>> Ken Gaillot <kgaillot at redhat.com> schrieb am 12.05.2016 um 16:41 in Nachricht
>>> <57349629.40408 at redhat.com>:
>>>> On 05/12/2016 02:56 AM, Ulrich Windl wrote:
>>>>> I have a question regarding an RA written by myself and pacemaker
>>>> 1.1.12-f47ea56 (SLES11 SP4):
>>>>> During "probe" all resources' "monitor" actions are executed (regardless of
>>>> any ordering constraints). Therefore my RA considers a parameter as invalid
>>>> ("file does not exist") (the file will be provided once some supplying
>>>> resource is up) and returns rc=2.
>>>>> OK, this may not be optimal, but pacemaker makes it worse: It does not
>>>> repeat the probe once the resource would start, but keeps the state,
>>>> preventing a resource start:
>>>>> primitive_monitor_0 on h05 'invalid parameter' (2): call=73,
>>>> status=complete, exit-reason='none', last-rc-change='Wed May 11 17:03:39
>>>> queued=0ms, exec=82ms
>>>> Correct, OCF_ERR_CONFIGURED is a "fatal" error:
>>> I think the mistake here is assuming that OCF_ERR_CONFIGURED only depends on
>> the RA parameters, when in fact the validity of RA params may depend on the
>> environment found at the time of checking. And as we all now the environment
>> changes, especially when resources are started and stopped.
It's not a mistake; it's the definition of OCF_ERR_CONFIGURED. The
various documentation could be more clear, but the intent is: the
resource's configuration *in the cluster configuration* is inherently
invalid, and could never work, regardless of what system it was on.
Examples listed include a required parameter not being specified, or a
nonnumeric value supplied for an integer parameter.
OCF_ERR_CONFIGURED is the *only* fatal error (fatal = the operation will
not be retried anywhere).
Of course, as you mentioned later, here you actually have OCF_ERR_ARGS
(2) -- I saw "invalid parameter" and mistakenly thought
>>>>> So you would say that monitor may only return "success" or "not running",
>>>> but I feel the RA should detect the condition that the resource could not
>>>> at all at the present state.
>>>> OCF_ERR_CONFIGURED is meant to indicate that the resource could not
>>>> possibly run *as configured*, regardless of the system's current state.
>>> But how do you handle parameters that describe file or host names (which may
>> exist or not independently of a change in the param's value)?
>> The RA should've exited with OCF_ERR_INSTALLED. That's the code
>> which means that there's something wrong with the environment on
>> this node, but that the resource could be started on another one.
> Really, besides implementation, I don't see why OCF_ERR_INSTALLED is less permanent than OCF_ERR_ARGS.
>>>> So for example, a required parameter is missing or invalid.
>>>> You could possibly use OCF_ERR_ARGS in this case (a "hard" error that
>>>> bans the particular node, and means that the resource's configuration is
>>>> not valid on this particular node).
>>> ("rc=2" _is_ OCF_ERR_ARGS)
OCF_ERR_ARGS and OCF_ERR_INSTALLED are both hard errors (hard = the
operation will not be retried on this node).
Looking at the code, OCF_ERR_INSTALLED is treated differently for probes
(it's not recorded as a failed op), but I think the node would still get
banned from running the resource.
I totally understand your point now: The probe may hit one of these
conditions only because it is called before depended-on resources are
up, but the cluster doesn't really care -- it just wants to know whether
the resource is running. Using ocf_is_probe to mask these errors works,
but could be considered a less-than-ideal workaround.
Now, it rings a bell: Last year, we looked into implementing "ordered
probes", which is essentially what you originally suggested, regarding
re-probing. See order_probes() in pengine/allocate.c; basically, there
are difficult implementation issues that haven't been figured out.
Bottom line, using ocf_is_probe is the best approach currently.
>>>> But, I suspect the right answer here is simply an order constraint
>>>> between the supplying resource and this resource. This resource's start
>>> As I said before the problem is probes: They are all started immediately
>> when a node comes up. And if a probe fails, no start is ever attempted later.
>>> AFAIK probes ignore any colocation or ordering constraints.
>> Try with ocf_is_probe. On probes, if the file's missing, exit
>> with OCF_NOT_RUNNING.
> I did that in the meantime, but I consider this as a work-around. Spontaneously I feel each RA should have a separate "probe" action with the following semantics:
> IF parameters seem valid AND the resource is up RETURN success
> ELSE return not running.
> Specifically this means any invalid parameter will result in "NOT RUNNING".
> I'm not saying having a separate "probe" will make things better; it's just clear that probes may be called from "invalid context" (I name it).
>>>> action, not monitor, should be the one that checks for the existence of
>>>> the supplied file.
>>> So it all depends what an invalid parameter (in the sense of validate-all)
>> actually is. Maybe the documentation should be more clear abou that.
>>> (As a matter of fact, the only RA docs I found yesterday are still hosted at
>> linux-ha (from linbit))
>> The developer's guide does have a note about probes and
>> ocf_is_probe and even some sample code (see the monitor and
>> validate-all actions).
>>>>> Shouldn't pacemaker reprobe resources before it tries to start them?
>>>> Probes are meant to check whether the resource is already active
>>>> anywhere. The decision of whether and where to start the resource takes
>>>> into account the result of the probes, so it doesn't make sense to
>>>> re-probe -- that's what the initial probe was for.
>>> In case a _probe_ returned an error indicating it was not able to decide, it
>> would make sense to reprobe after colocation and ordering constraiints are
>>> One of the first lessions I've learned with HA was that the state of a
>> resource is not Boolean (up or down), but multistate (I wrote on that after I
>> had started to use pacemaker):
>>> A resource can be "down", "starting", "started/up", "stopping",
>> "down/stopped" and "undecided" (the case whare retries can make sense).
>> During states "starting" and "stopping" it does not make sense to enforce a
>> Boolean result like "up" or "down", because the resource may fail to perform
>> the transition, and the resource may hang in the state of "starting" or
>> "stopping" (wich both are definitely not "up").
>>> If you need a specific directory to decide whether a specific resource is up
>> or down, and that directory does not exist, you cannot really make a
>> statement about the resource's state: Most likely it won't be "up", but you
>> cannot say for sure that it is down (e.g. you have a state file on NFS and
>> NFS server is down at the time of checking).
>> The transitioning period (i.e. start, stop, promote, demote) for
>> the resource may take a while, depending on the resource and the
>> actual deployment, which is why setting the timeouts properly is
>> important. Note that probes are meant to run mostly on node
>> startup, that is to establish the initial state.
>> If the RA, on probe, cannot decide the state the resource is in,
>> then it should return OCF_RA_GENERIC and let the resource manager
>> cleanup (by invoking stop). Furthermore, the RA must take into
>> account that some of its dependencies may not be running.
> But if the problem is due to "invalid calling context", then action "stop" will have the same problem (and return an error as well). STONITH deathmatch then?
> Users mailing list: Users at clusterlabs.org
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
More information about the Users