[ClusterLabs] Continuous master monitor failure of a resource in case some other resource is being promoted

[Andrei Borzenkov]

26.02.2019 18:05, Ken Gaillot пишет:
> On Tue, 2019-02-26 at 06:55 +0300, Andrei Borzenkov wrote:
>> 26.02.2019 1:08, Ken Gaillot пишет:
>>> On Mon, 2019-02-25 at 23:00 +0300, Andrei Borzenkov wrote:
>>>> 25.02.2019 22:36, Andrei Borzenkov пишет:
>>>>>
>>>>>> Could you please help me understand:
>>>>>> 1. Why doesn't pacemaker process the failure of
>>>>>> Stateful_Test_2
>>>>>> resource
>>>>>> immediately after first failure?
>>>>
>>>> I'm still not sure why.
>>>>
>>>>> I vaguely remember something about sequential execution
>>>>> mentioned
>>>>> before
>>>>> but cannot find details.
>>>>>
>>>>>> 2. Why does the monitor failure of Stateful_Test_2 continue
>>>>>> even
>>>>>> after the
>>>>>> promote of Stateful_Test_1 has been completed? Shouldn't it
>>>>>> handle
>>>>>> Stateful_Test_2's failure and take necessary action on it? It
>>>>>> feels as if
>>>>>> that particular failure 'event' has been 'dropped' and
>>>>>> pengine is
>>>>>> not even
>>>>>> aware of the Stateful_Test_2's failure.
>>>>>>
>>>>>
>>>>> Yes. Although crm_mon shows resource as being master on this
>>>>> node,
>>>>> in
>>>>> reality resource is left in failed state forever and monitor
>>>>> result
>>>>> is
>>>>> simply ignored.
>>>>>
>>>>
>>>> Yes, pacemaker reacts only on result change (more precisely, it
>>>> tells
>>>> lrmd to report only the first result and suppress all further
>>>> consecutive duplicates). As the first report gets lost due to low
>>>> failure-timeout, this explains what happens.
>>>
>>> That would explain why the first monitor failure is ignored, but
>>> after
>>> the long-running action completes, a new transition should see the
>>> failure timeout, wipe the resource history, and log a message on
>>> the DC
>>> about "Re-initiated expired calculated failure", at which point the
>>> cluster should schedule recovery.
>>>
>>> Do the logs show such a message?
>>>
>>
>> Yes, this message appears. Not sure how it changes anything because
>> the
>> problem is not that pacemaker does not react to subsequent failures,
>> but
>> that after the first monitor failure lrmd does not report anything to
>> pacemaker at all.
> 
> That's what the "re-initiated" code is meant to work around. It's
> hacky: it changes the restart hash for the operation, making it appear
> to have changed parameter values, which requires a restart (which will
> stop and restart the monitor). At least that's how I think it works by
> looking at the code. If that's not happening, I may be missing
> something.
> 

Well, I do not see in debug logs any output from any place where this
changed hash would be checked.

I opened https://bugs.clusterlabs.org/show_bug.cgi?id=5380