[ClusterLabs] temporary loss of quorum when member starts to rejoin

[Sherrard Burton]

On 4/6/20 4:10 PM, Andrei Borzenkov wrote:
> 06.04.2020 20:57, Sherrard Burton пишет:
>>
>>
>> On 4/6/20 1:20 PM, Sherrard Burton wrote:
>>>
>>>
>>> On 4/6/20 12:35 PM, Andrei Borzenkov wrote:
>>>> 06.04.2020 17:05, Sherrard Burton пишет:
>>>>>
>>>>> from the quorum node:
>>> ...
>>>>> Apr 05 23:10:17 debug   Client ::ffff:192.168.250.50:54462 (cluster
>>>>> xen-nfs01_xen-nfs02, node_id 1) sent quorum node list.
>>>>> Apr 05 23:10:17 debug     msg seq num = 6
>>>>> Apr 05 23:10:17 debug     quorate = 0
>>>>> Apr 05 23:10:17 debug     node list:
>>>>> Apr 05 23:10:17 debug       node_id = 1, data_center_id = 0, node_state
>>>>> = member
>>>>
>>>> Oops. How comes that node that was rebooted formed cluster all by
>>>> itself, without seeing the second node? Do you have two_nodes and/or
>>>> wait_for_all configured?
>>>>
>>
>> i never thought to check the logs on the rebooted server. hopefully
>> someone can extract some further useful information here:
>>
>>
>> https://pastebin.com/imnYKBMN
>>
> 
> It looks like some timing issue or race condition. After reboot node
> manages to contact qnetd first, before connection to other node is
> established. Qnetd behaves as documented - it sees two equal size
> partitions and favors the partition that includes tie breaker (lowest
> node id). So existing node goes out of quorum. Second later both nodes
> see each other and so quorum is regained.


thank you for taking the time to troll through my debugging output. your 
explanation seems to accurately describe what i am experiencing. of 
course i have no idea how to remedy it. :-)

> 
> I cannot reproduce it, but I also do not use knet. From documentation I
> have impression that knet has artificial delay before it considers links
> operational, so may be that is the reason.

i will do some reading on how knet factors into all of this and respond 
with any questions or discoveries.

> 
>>>
>>> BTW, great eyes. i had not picked up on that little nuance. i had
>>> poured through this particular log a number of times, but it was very
>>> hard for me to discern the starting and stopping points for each
>>> logical group of messages. the indentation made some of it clear. but
>>> when you have a series of lines beginning in the left-most column, it
>>> is not clear whether they belong to the previous group, the next
>>> group, or they are their own group.
>>>
>>> just wanted to note my confusion in case the relevant maintainer
>>> happens across this thread.
>>>
>>> thanks again
>>> _______________________________________________
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
>