[ClusterLabs] large cluster with corosync

Jan Friesse jfriesse at redhat.com
Wed Jan 4 11:14:46 EST 2017 

Arne Jansen napsal(a):
> Hi Honza,
>
> On 04.01.2017 13:52, Jan Friesse wrote:
>
>>>
>>> At least those limits doesn't seem to get enforced, as a 64 node cluster
>>> seems to work, although a bit shaky.
>>
>> No, they are not enforced. 16/32 are official supported number of nodes.
>> Basically, this is number what was tested and known to work reliably.
>> This doesn't mean corosync doesn't work with bigger number of nodes.
>> Eventho I'm quite surprised that 64 nodes really works.
>>
>> Variables you can try tweak.
>> - Definitively start with increase totem.config (default 1000, you can
>> try 10000)
>> - If it doesn't help, try increase totem.join (default is 50, 1000 may
>> work) and consider increase totem.send_join (default is 0, 100 may be
>> good idea).
>> - As a last variable, increase of totem.merge (default is 200, 2000 may
>> do the job).
>>
>> And definitively let us know about results. It's quite hard to test such
>> a big amount of nodes so some of the variable may be sub-optimal. When
>> we know which of variables are victims, we can change their defaults.
>>
>
> Thanks for the tuning hints. One thing I definitely need to do before
> the next test run is to build some kind of rate limiting into the daemon
> to prevent it from bringing our whole network down. I see sendmsg()
> calls in totemudp.c and totemupdu.c. Would it be sufficient to restrict
> those two calls?

I'm not sure why to introduce another rate limiting? I mean, corosync 
itself has rate-limit by:
- allow only node which holds the token to send messages
- allow only limited number of messages to be send when holding token 
(you can tweak them by totem.max_messages and totem.window_size)

Itroduce another rate-limit in totemudp(u).c is probably going to create 
less stable membership and various problems... but who knows, you can 
give it a try.

Honza

>
> Thanks, Arne
>
>> Regards,
>>   Honza
>>
>>> I want to use corosync + dlm to get a distributed lock manager to have
>>> a uniform locking service throughout the platform as a basis for many
>>> services.
>>>
>>> Thanks,
>>> Arne
>>>
>>>>
>>>> Cheers,
>>>> Kristoffer
>>>>
>>>>>
>>>>> Thanks,
>>>>> Arne
>>>>>
>>>>> Jan 04 10:24:49 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:25:00 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:616) was formed. Members joined: 168101089
>>>>> Jan 04 10:25:00 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:25:11 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:620) was formed. Members joined: 168101090
>>>>> Jan 04 10:25:11 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:25:21 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:624) was formed. Members joined: 168101091
>>>>> Jan 04 10:25:21 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:25:32 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:628) was formed. Members joined: 168101092
>>>>> Jan 04 10:25:32 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:26:09 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:632) was formed. Members joined: 168101141
>>>>> Jan 04 10:26:10 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:26:20 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:636) was formed. Members joined: 168101142
>>>>> Jan 04 10:26:20 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:26:31 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:640) was formed. Members joined: 168101143
>>>>> Jan 04 10:26:31 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:26:42 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:644) was formed. Members joined: 168101144
>>>>> Jan 04 10:26:42 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:26:52 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:648) was formed. Members joined: 168101145
>>>>> Jan 04 10:26:52 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:03 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:652) was formed. Members joined: 168101146
>>>>> Jan 04 10:27:03 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:14 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:656) was formed. Members joined: 168101147
>>>>> Jan 04 10:27:14 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:25 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:660) was formed. Members joined: 168101148
>>>>> Jan 04 10:27:25 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:35 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:664) was formed. Members joined: 168101161
>>>>> Jan 04 10:27:35 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:46 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:668) was formed. Members joined: 168101162
>>>>> Jan 04 10:27:46 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:27:57 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:672) was formed. Members joined: 168101163
>>>>> Jan 04 10:27:57 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:28:07 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:676) was formed. Members joined: 168101164
>>>>> Jan 04 10:28:07 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:28:18 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:680) was formed. Members joined: 168101165
>>>>> Jan 04 10:28:18 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:28:28 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:684) was formed. Members joined: 168101166
>>>>> Jan 04 10:28:28 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:28:40 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:688) was formed. Members joined: 168101167
>>>>> Jan 04 10:28:40 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:28:50 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:692) was formed. Members joined: 168101168
>>>>> Jan 04 10:28:50 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:29:00 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:696) was formed. Members joined: 168101181
>>>>> Jan 04 10:29:00 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:29:14 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:704) was formed. Members joined: 168100966 168100967
>>>>> 168100968 168100969 168100970 168100971 168100972 168100985 168100986
>>>>> 168100987 168100988 168100989 168100990 168100991 168100992 168101005
>>>>> 168101006 168101007 168101008 168101009 168101010 168101011 168101012
>>>>> 168101025 168101026 168101027 168101028 168101029 168101030 168101031
>>>>> 168101032 168101045 168101046 168101047 168101048 168101049 168101050
>>>>> 168101051 168101052 168
>>>>> Jan 04 10:29:14 [4915] reniar corosync notice  [TOTEM ] Failed to
>>>>> receive the leave message. failed: 168100966 168100967 168100968
>>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>>> 168101052 168101065 168101066
>>>>> Jan 04 10:29:15 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:716) was formed. Members joined: 168100966 168100967
>>>>> 168100968 168100969 168100970 168100971 168100972 168100985 168100986
>>>>> 168100987 168100988 168100989 168100990 168100991 168100992 168101005
>>>>> 168101006 168101007 168101008 168101009 168101010 168101011 168101012
>>>>> 168101025 168101026 168101027 168101028 168101029 168101030 168101031
>>>>> 168101032 168101045 168101046 168101047 168101048 168101049 168101050
>>>>> 168101051 168101052 168
>>>>> Jan 04 10:29:15 [4915] reniar corosync notice  [TOTEM ] Failed to
>>>>> receive the leave message. failed: 168100966 168100967 168100968
>>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>>> 168101052 168101066 168101067
>>>>> Jan 04 10:29:16 [4915] reniar corosync notice  [TOTEM ] Retransmit
>>>>> List:
>>>>> 285
>>>>> Jan 04 10:29:16 [4915] reniar corosync notice  [TOTEM ] Retransmit
>>>>> List:
>>>>> 4db 4dc 4dd 4e1
>>>>> Jan 04 10:29:16 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:20 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:732) was formed. Members left: 168100966 168100967
>>>>> 168100968
>>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>>> 168101052 16810
>>>>> Jan 04 10:29:20 [4915] reniar corosync notice  [TOTEM ] Failed to
>>>>> receive the leave message. failed: 168100966 168100967 168100968
>>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>>> 168101052 168101066 168101067
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:736) was formed. Members
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:25 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>>> detected for 503 ms, flushing membership messages.
>>>>> Jan 04 10:29:27 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>>> detected for 1277 ms, flushing membership messages.
>>>>> Jan 04 10:29:27 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>>> process was not scheduled for 1345.9542 ms (threshold is 800.0000 ms).
>>>>> Consider token timeout increase.
>>>>> Jan 04 10:29:29 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>>> detected for 607 ms, flushing membership messages.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>>> process was not scheduled for 12273.4664 ms (threshold is 800.0000
>>>>> ms).
>>>>> Consider token timeout increase.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:916) was formed. Members
>>>>> Jan 04 10:29:42 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>>> detected for 12562 ms, flushing membership messages.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>>> process was not scheduled for 12636.5143 ms (threshold is 800.0000
>>>>> ms).
>>>>> Consider token timeout increase.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync notice  [TOTEM ] A new
>>>>> membership
>>>>> (10.5.4.101:964) was formed. Members
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync notice  [MAIN  ] Completed
>>>>> service synchronization, ready to provide service.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>>> message was thrown away during flush operation.
>>>>> Jan 04 10:29:59 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>>> process was not scheduled for 1465.7160 ms (threshold is 800.0000 ms).
>>>>> Consider token timeout increase.
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list: Users at clusterlabs.org
>>>>> http://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Users mailing list: Users at clusterlabs.org
>>> http://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://lists.clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

More information about the Users mailing list