[ClusterLabs] Informing RAs about recovery: failed resource recovery, or any start-stop cycle?

[Ken Gaillot]

On 06/24/2016 05:41 AM, Adam Spiers wrote:
> Andrew Beekhof <abeekhof at redhat.com> wrote:
>> On Fri, Jun 24, 2016 at 1:01 AM, Adam Spiers <aspiers at suse.com> wrote:
>>> Andrew Beekhof <abeekhof at redhat.com> wrote:
>>
>>>>> Well, if you're OK with bending the rules like this then that's good
>>>>> enough for me to say we should at least try it :)
>>>>
>>>> I still say you shouldn't only do it on error.
>>>
>>> When else should it be done?
>>
>> I was thinking whenever a stop() happens.
> 
> OK, seems we are agreed finally :)
> 
>>> IIUC, disabling/enabling the service is independent of the up/down
>>> state which nova tracks automatically, and which based on slightly
>>> more than a skim of the code, is dependent on the state of the RPC
>>> layer.
>>>
>>>>> But how would you avoid repeated consecutive invocations of "nova
>>>>> service-disable" when the monitor action fails, and ditto for "nova
>>>>> service-enable" when it succeeds?
>>>>
>>>> I don't think you can. Not ideal but I'd not have thought a deal breaker.
>>>
>>> Sounds like a massive deal-breaker to me!  With op monitor
>>> interval="10s" and 100 compute nodes, that would mean 10 pointless
>>> calls to nova-api every second.  Am I missing something?
>>
>> I was thinking you would only call it for the "I detected a failure
>> case" and service-enable would still be on start().
>> So the number of pointless calls per second would be capped at one
>> tenth of the number of failed compute nodes.
>>
>> One would hope that all of them weren't dead.
> 
> Oh OK - yeah that wouldn't be nearly as bad.
> 
>>> Also I don't see any benefit to moving the API calls from start/stop
>>> actions to the monitor action.  If there's a failure, Pacemaker will
>>> invoke the stop action, so we can do service-disable there.
>>
>> I agree. Doing it unconditionally at stop() is my preferred option, I
>> was only trying to provide a path that might be close to the behaviour
>> you were looking for.
>>
>>> If the
>>> start action is invoked and we successfully initiate startup of
>>> nova-compute, the RA can undo any service-disable it previously did
>>> (although it should not reverse a service-disable done elsewhere,
>>> e.g. manually by the cloud operator).
>>
>> Agree
> 
> Trying to adjust to this new sensation of agreement ;-)
> 
>>>>> Earlier in this thread I proposed
>>>>> the idea of a tiny temporary file in /run which tracks the last known
>>>>> state and optimizes away the consecutive invocations, but IIRC you
>>>>> were against that.
>>>>
>>>> I'm generally not a fan, but sometimes state files are a necessity.
>>>> Just make sure you think through what a missing file might mean.
>>>
>>> Sure.  A missing file would mean the RA's never called service-disable
>>> before,
>>
>> And that is why I generally don't like state files.
>> The default location for state files doesn't persist across reboots.
>>
>> t1. stop (ie. disable)
>> t2. reboot
>> t3. start with no state file
>> t4. WHY WONT NOVA USE THE NEW COMPUTE NODE STUPID CLUSTERS
> 
> Well then we simply put the state file somewhere which does persist
> across reboots.

There's also the possibility of using a node attribute. If you set a
normal node attribute, it will abort the transition and calculate a new
one, so that's something to take into account. You could set a private
node attribute, which never gets written to the CIB and thus doesn't
abort transitions, but it also does not survive a complete cluster stop.

>>> which means that it shouldn't call service-enable on startup.
>>>
>>>> Unless.... use the state file to store the date at which the last
>>>> start operation occurred?
>>>>
>>>> If we're calling stop() and data - start_date > threshold, then, if
>>>> you must, be optimistic, skip service-disable and assume we'll get
>>>> started again soon.
>>>>
>>>> Otherwise if we're calling stop() and data - start_date <= threshold,
>>>> always call service-disable because we're in a restart loop which is
>>>> not worth optimising for.
>>>>
>>>> ( And always call service-enable at start() )
>>>>
>>>> No Pacemaker feature or Beekhof approval required :-)
>>>
>>> Hmm ...  it's possible I just don't understand this proposal fully,
>>> but it sounds a bit woolly to me, e.g. how would you decide a suitable
>>> threshold?
>>
>> roll a dice?
>>
>>> I think I preferred your other suggestion of just skipping the
>>> optimization, i.e. calling service-disable on the first stop, and
>>> service-enable on (almost) every start.
>>
>> good :)
>>
>> And the use of force-down from your subsequent email sounds excellent
> 
> OK great!  We finally got there :-)  Now I guess I just have to write
> the spec and the actual code ;-)