[ClusterLabs] Why Do Nodes Leave the Cluster?

[Strahil Nikolov]

On April 12, 2020 10:36:42 AM GMT+03:00, Eric Robinson <eric.robinson at psmnv.com> wrote:
>> -----Original Message-----
>> From: Strahil Nikolov <hunter86_bg at yahoo.com>
>> Sent: Sunday, April 12, 2020 1:32 AM
>> To: Eric Robinson <eric.robinson at psmnv.com>; Cluster Labs - All
>topics
>> related to open-source clustering welcomed <users at clusterlabs.org>;
>> Andrei Borzenkov <arvidjaar at gmail.com>
>> Subject: RE: [ClusterLabs] Why Do Nodes Leave the Cluster?
>>
>> On April 11, 2020 5:01:37 PM GMT+03:00, Eric Robinson
>> <eric.robinson at psmnv.com> wrote:
>> >
>> >Hi Strahil --
>> >
>> >I hope you won't mind if I revive this old question. In your
>comments
>> >below, you suggested using a 1s  token with a 1.2s consensus. I
>> >currently have 2-node clusters (will soon install a qdevice). I was
>> >reading in the corosync.conf man page where it says...
>> >
>> >"For  two  node  clusters,  a  consensus larger than the join
>timeout
>> >but less than token is safe.  For three node or larger clusters,
>> >consensus should be larger than token."
>> >
>> >Do you still think the consensus should be 1.2 * token in a 2-node
>> >cluster? Why is a smaller consensus considered safe for 2-node
>> >clusters? Should I use a larger consensus anyway?
>> >
>> >--Eric
>> >
>> >
>> >> -----Original Message-----
>> >> From: Strahil Nikolov <hunter86_bg at yahoo.com>
>> >> Sent: Thursday, February 6, 2020 1:07 PM
>> >> To: Eric Robinson <eric.robinson at psmnv.com>; Cluster Labs - All
>> >topics
>> >> related to open-source clustering welcomed
><users at clusterlabs.org>;
>> >> Andrei Borzenkov <arvidjaar at gmail.com>
>> >> Subject: RE: [ClusterLabs] Why Do Nodes Leave the Cluster?
>> >>
>> >> On February 6, 2020 7:35:53 PM GMT+02:00, Eric Robinson
>> >> <eric.robinson at psmnv.com> wrote:
>> >> >Hi Nikolov --
>> >> >
>> >> >> Defaults are 1s  token,  1.2s  consensus which is too small.
>> >> >> In Suse, token is 10s, while consensus  is 1.2 * token -> 12s.
>> >> >> With these settings, cluster  will not react   for 22s.
>> >> >>
>> >> >> I think it's a good start for your cluster .
>> >> >> Don't forget to put  the cluster  in maintenance (pcs property
>set
>> >> >> maintenance-mode=true) before restarting the stack ,  or  even
>> >better
>> >> >- get
>> >> >> some downtime.
>> >> >>
>> >> >> You can use the following article to run a simulation before
>> >removing
>> >> >the
>> >> >> maintenance:
>> >> >> https://www.suse.com/support/kb/doc/?id=7022764
>> >> >>
>> >> >
>> >> >
>> >> >Thanks for the suggestions. Any thoughts on timeouts for DRBD?
>> >> >
>> >> >--Eric
>> >> >
>> >> >Disclaimer : This email and any files transmitted with it are
>> >> >confidential and intended solely for intended recipients. If you
>are
>> >> >not the named addressee you should not disseminate, distribute,
>copy
>> >or
>> >> >alter this email. Any views or opinions presented in this email
>are
>> >> >solely those of the author and might not represent those of
>> >Physician
>> >> >Select Management. Warning: Although Physician Select Management
>> has
>> >> >taken reasonable precautions to ensure no viruses are present in
>> >this
>> >> >email, the company cannot accept responsibility for any loss or
>> >damage
>> >> >arising from the use of this email or attachments.
>> >>
>> >> Hi Eric,
>> >>
>> >> The timeouts can be treated as 'how much time to wait before 
>taking
>> >any
>> >> action'. The workload is not very important (HANA  is something
>> >different).
>> >>
>> >> You can try with 10s (token) , 12s (consensus) and if needed  you
>can
>> >adjust.
>> >>
>> >> Warning: Use a 3 node cluster or at least 2 drbd nodes + qdisk.
>The 2
>> >node
>> >> cluster is vulnerable to split brain, especially when one of the
>> >nodes  is
>> >> syncing  (for example after a patching) and the source is
>> >> fenced/lost/disconnected. It's very hard to extract data from a
>> >semi-synced
>> >> drbd.
>> >>
>> >> Also, if you need guidance for the SELINUX, I can point you to my
>> >guide in the
>> >> centos forum.
>> >>
>> >> Best Regards,
>> >> Strahil Nikolov
>> >Disclaimer : This email and any files transmitted with it are
>> >confidential and intended solely for intended recipients. If you are
>> >not the named addressee you should not disseminate, distribute, copy
>or
>> >alter this email. Any views or opinions presented in this email are
>> >solely those of the author and might not represent those of
>Physician
>> >Select Management. Warning: Although Physician Select Management has
>> >taken reasonable precautions to ensure no viruses are present in
>this
>> >email, the company cannot accept responsibility for any loss or
>damage
>> >arising from the use of this email or attachments.
>>
>> Hey Eric,
>>
>> 1s/1.2s  are the defaults and I guess they are defaults for 2-node
>clusters too.
>>
>> Yet if the man page suggests that - then you should set it up this
>way.
>> On our SLES 2-node  clusters  we  got  token 30s  & consensus 36s
>after
>> several network issues  and they are fine now - just a failiure is
>delayed by a
>> minute.
>>
>> Maybe you can try with consensus 0.8s and default token of 1s.
>> I have noticed that my oVirt LAB RHEL nodes are being fenced with the
>> defaults, so I'm using SUSE's defaults of  10s token, 12s consensus
>even on
>> RHEL 2-node  clusters.
>>
>> Best Regards,
>> Strahil Nikolov
>
>
>I tried token=10000 (10 seconds) and noticed that it took a long time
>for the cluster to react to some things. I finally figured out that
>token_retransmit and token_retransmits_before_loss_const Also come into
>play.
>
>--Eric
>
>
>Disclaimer : This email and any files transmitted with it are
>confidential and intended solely for intended recipients. If you are
>not the named addressee you should not disseminate, distribute, copy or
>alter this email. Any views or opinions presented in this email are
>solely those of the author and might not represent those of Physician
>Select Management. Warning: Although Physician Select Management has
>taken reasonable precautions to ensure no viruses are present in this
>email, the company cannot accept responsibility for any loss or damage
>arising from the use of this email or attachments.

Yes,  it  does  affect failover speeds,  and there  is no "magic bullet" .

Best Regards,
Strahil Nikolov