[ClusterLabs] Gracefully stop nodes one by one with disk-less sbd

[Yan Gao]

On 8/12/19 3:24 PM, Klaus Wenninger wrote:
> On 8/12/19 2:30 PM, Yan Gao wrote:
>> Hi Klaus,
>>
>> On 8/12/19 1:39 PM, Klaus Wenninger wrote:
>>> On 8/9/19 9:06 PM, Yan Gao wrote:
>>>> On 8/9/19 6:40 PM, Andrei Borzenkov wrote:
>>>>> 09.08.2019 16:34, Yan Gao пишет:
>>>>>> Hi,
>>>>>>
>>>>>> With disk-less sbd,  it's fine to stop cluster service from the cluster
>>>>>> nodes all at the same time.
>>>>>>
>>>>>> But if to stop the nodes one by one, for example with a 3-node cluster,
>>>>>> after stopping the 2nd node, the only remaining node resets itself with:
>>>>>>
>>>>> That is sort of documented in SBD manual page:
>>>>>
>>>>> --><--
>>>>> However, while the cluster is in such a degraded state, it can
>>>>> neither successfully fence nor be shutdown cleanly (as taking the
>>>>> cluster below the quorum threshold will immediately cause all remaining
>>>>> nodes to self-fence).
>>>>> --><--
>>>>>
>>>>> SBD in shared-nothing mode is basically always in such degraded state
>>>>> and cannot tolerate loss of quorum.
>>>> Well, the context here is it loses quorum *expectedly* since the other
>>>> nodes gracefully shut down.
>>>>
>>>>>> Aug 09 14:30:20 opensuse150-1 sbd[1079]:       pcmk:    debug:
>>>>>> notify_parent: Not notifying parent: state transient (2)
>>>>>> Aug 09 14:30:20 opensuse150-1 sbd[1080]:    cluster:    debug:
>>>>>> notify_parent: Notifying parent: healthy
>>>>>> Aug 09 14:30:20 opensuse150-1 sbd[1078]:  warning: inquisitor_child:
>>>>>> Latency: No liveness for 4 s exceeds threshold of 3 s (healthy servants: 0)
>>>>>>
>>>>>> I can think of the way to manipulate quorum with last_man_standing and
>>>>>> potentially also auto_tie_breaker, not to mention
>>>>>> last_man_standing_window would also be a factor... But is there a better
>>>>>> solution?
>>>>>>
>>>>> Lack of cluster wide shutdown mode was mentioned more than once on this
>>>>> list. I guess the only workaround is to use higher level tools which
>>>>> basically simply try to stop cluster on all nodes at once. It is still
>>>>> susceptible to race condition.
>>>> Gracefully stopping nodes one by one on purpose is still a reasonable
>>>> need though ...
>>> If you do the teardown as e.g. pcs is doing it - first tear down
>>> pacemaker-instances and then corosync/sbd - it is at
>>> least possible to tear down the pacemaker-instances one-by one
>>> without risking a reboot due to quorum-loss.
>>> With kind of current sbd having in
>>> -
>>> https://github.com/ClusterLabs/sbd/commit/824fe834c67fb7bae7feb87607381f9fa8fa2945
>>> -
>>> https://github.com/ClusterLabs/sbd/commit/79b778debfee5b4ab2d099b2bfc7385f45597f70
>>> -
>>> https://github.com/ClusterLabs/sbd/commit/a716a8ddd3df615009bcff3bd96dd9ae64cb5f68
>>> this should be pretty robust although we are still thinking
>>> (probably together with some heartbeat to pacemakerd
>>> that assures pacemakerd is checking liveness of sub-daemons
>>> properly) of having a cleaner way to detect graceful
>>> pacemaker-shutdown.
>> These are all good improvements, thanks!
>>
>> But in this case the remaining node is not shutting down yet, or it's
>> intentionally not being shut down :-) Loss of quorum is as expected, so
>> is following no-quorum-policy, but self-reset is probably too much?
> Hmm ... not sure if I can follow ...
> If you shutdown solely pacemaker one-by-one on all nodes
> and these shutdowns are considered graceful then you are
> not gonna experience any reboots (e.g. 3 node cluster).
> Afterwards you can shutdown corosync one-by-one as well
> without experiencing reboots as without the cib-connection
> sbd isn't gonna check for quorum anymore (all resources
> down so no need to reboot in case of quorum-loss - extra
> care has to be taken care of with unmanaged resources but
> that isn't particular with sbd).
I meant if users would like shut down only 2 out of 3 nodes in the 
cluster and keep the last one online and alive, it's simply not possible 
for now, although the loss of quorum is expected.

Regards,
   Yan