[ClusterLabs] large cluster with corosync

Jan Friesse jfriesse at redhat.com
Tue Jan 10 08:06:43 EST 2017 

Arne Jansen napsal(a):
>
>
> On 04.01.2017 13:52, Jan Friesse wrote:
>>
>> Variables you can try tweak.
>> - Definitively start with increase totem.config (default 1000, you can
>> try 10000)
>
> what does that do? Haven't found it in corosync.conf(5)

Sorry, typo. I meant totem.token.

>
>> - If it doesn't help, try increase totem.join (default is 50, 1000 may
>> work) and consider increase totem.send_join (default is 0, 100 may be
>> good idea).
>
> The problem is indeed the flood of join messages overwhelming the
> receive socket. I increased the socket size in source code from 160k

Ok. I'm thinking if it would make sense to make this define user 
configurable or probably even better somehow depending on number of 
configured nodes.

> to 3MB and forming the cluster works fine for 80 nodes. send_join
> looks like a very useful variable to avoid the need for excessively
> large buffers, so I'll test with join=120ms and send_join=60ms.

Sounds good.

Honza

>
> -Arne
>
>> - As a last variable, increase of totem.merge (default is 200, 2000 may
>> do the job).
>>
>> And definitively let us know about results. It's quite hard to test such
>> a big amount of nodes so some of the variable may be sub-optimal. When
>> we know which of variables are victims, we can change their defaults.
>>
>> Regards,
>>   Honza
>>
>>> I want to use corosync + dlm to get a distributed lock manager to have
>>> a uniform locking service throughout the platform as a basis for many
>>> services.
>>>
>>> Thanks,
>>> Arne
>>>
>>>>
>>>> Cheers,
>>>> Kristoffer
>>>>
>>>>>
>>>>> Thanks,
>>>>> Arne
>>>>>
>>>>> Jan 04 10:24:49 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:25:00 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:616) was formed. Members joined: 168101089
>>>>> Jan 04 10:25:00 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:25:11 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:620) was formed. Members joined: 168101090
>>>>> Jan 04 10:25:11 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:25:21 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:624) was formed. Members joined: 168101091
>>>>> Jan 04 10:25:21 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:25:32 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:628) was formed. Members joined: 168101092
>>>>> Jan 04 10:25:32 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:26:09 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:632) was formed. Members joined: 168101141
>>>>> Jan 04 10:26:10 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:26:20 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:636) was formed. Members joined: 168101142
>>>>> Jan 04 10:26:20 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:26:31 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:640) was formed. Members joined: 168101143
>>>>> Jan 04 10:26:31 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:26:42 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:644) was formed. Members joined: 168101144
>>>>> Jan 04 10:26:42 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:26:52 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:648) was formed. Members joined: 168101145
>>>>> Jan 04 10:26:52 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:03 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:652) was formed. Members joined: 168101146
>>>>> Jan 04 10:27:03 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:14 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:656) was formed. Members joined: 168101147
>>>>> Jan 04 10:27:14 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:25 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:660) was formed. Members joined: 168101148
>>>>> Jan 04 10:27:25 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:35 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:664) was formed. Members joined: 168101161
>>>>> Jan 04 10:27:35 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:46 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:668) was formed. Members joined: 168101162
>>>>> Jan 04 10:27:46 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:57 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:672) was formed. Members joined: 168101163
>>>>> Jan 04 10:27:57 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:28:07 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:676) was formed. Members joined: 168101164
>>>>> Jan 04 10:28:07 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:28:18 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:680) was formed. Members joined: 168101165
>>>>> Jan 04 10:28:18 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:28:28 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:684) was formed. Members joined: 168101166
>>>>> Jan 04 10:28:28 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:28:40 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:688) was formed. Members joined: 168101167
>>>>> Jan 04 10:28:40 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:28:50 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:692) was formed. Members joined: 168101168
>>>>> Jan 04 10:28:50 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:29:00 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:696) was formed. Members joined: 168101181
>>>>> Jan 04 10:29:00 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:29:14 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:704) was formed. Members joined: 168100966 168100967
>>>>> 168100968 168100969 168100970 168100971 168100972 168100985 168100986
>>>>> 168100987 168100988 168100989 168100990 168100991 168100992 168101005
>>>>> 168101006 168101007 168101008 168101009 168101010 168101011 168101012
>>>>> 168101025 168101026 168101027 168101028 168101029 168101030 168101031
>>>>> 168101032 168101045 168101046 168101047 168101048 168101049 168101050
>>>>> 168101051 168101052 168
>>>>> Jan 04 10:29:14 [4915] reniar corosync notice  [TOTEM ] Failed to
>>>>> receive the leave message. failed: 168100966 168100967 168100968
>>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>>> 168101052 168101065 168101066
>>>>> Jan 04 10:29:15 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:716) was formed. Members joined: 168100966 168100967
>>>>> 168100968 168100969 168100970 168100971 168100972 168100985 168100986
>>>>> 168100987 168100988 168100989 168100990 168100991 168100992 168101005
>>>>> 168101006 168101007 168101008 168101009 168101010 168101011 168101012
>>>>> 168101025 168101026 168101027 168101028 168101029 168101030 168101031
>>>>> 168101032 168101045 168101046 168101047 168101048 168101049 168101050
>>>>> 168101051 168101052 168
>>>>> Jan 04 10:29:15 [4915] reniar corosync notice  [TOTEM ] Failed to
>>>>> receive the leave message. failed: 168100966 168100967 168100968
>>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>>> 168101052 168101066 168101067
>>>>> Jan 04 10:29:16 [4915] reniar corosync notice  [TOTEM ] Retransmit
>>>>> List:
>>>>> 285
>>>>> Jan 04 10:29:16 [4915] reniar corosync notice  [TOTEM ] Retransmit
>>>>> List:
>>>>> 4db 4dc 4dd 4e1
>>>>> Jan 04 10:29:16 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:732) was formed. Members left: 168100966 168100967
>>>>> 168100968
>>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>>> 168101052 16810
>>>>> Jan 04 10:29:20 [4915] reniar corosync notice  [TOTEM ] Failed to
>>>>> receive the leave message. failed: 168100966 168100967 168100968
>>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>>> 168101052 168101066 168101067
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:736) was formed. Members
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:25 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>>> detected for 503 ms, flushing membership messages.
>>>>> Jan 04 10:29:27 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>>> detected for 1277 ms, flushing membership messages.
>>>>> Jan 04 10:29:27 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>>> process was not scheduled for 1345.9542 ms (threshold is 800.0000 ms).
>>>>> Consider token timeout increase.
>>>>> Jan 04 10:29:29 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>>> detected for 607 ms, flushing membership messages.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>>> process was not scheduled for 12273.4664 ms (threshold is 800.0000
>>>>> ms).
>>>>> Consider token timeout increase.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:916) was formed. Members
>>>>> Jan 04 10:29:42 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>>> detected for 12562 ms, flushing membership messages.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>>> process was not scheduled for 12636.5143 ms (threshold is 800.0000
>>>>> ms).
>>>>> Consider token timeout increase.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:964) was formed. Members
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:59 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>>> process was not scheduled for 1465.7160 ms (threshold is 800.0000 ms).
>>>>> Consider token timeout increase.
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list: Users at clusterlabs.org
>>>>> http://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Users mailing list: Users at clusterlabs.org
>>> http://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://lists.clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

More information about the Users mailing list