[ClusterLabs] large cluster with corosync

Arne Jansen lists at die-jansens.de
Wed Jan 4 08:14:51 EST 2017 

Hi Honza,

On 04.01.2017 13:52, Jan Friesse wrote:

>>
>> At least those limits doesn't seem to get enforced, as a 64 node cluster
>> seems to work, although a bit shaky.
>
> No, they are not enforced. 16/32 are official supported number of nodes.
> Basically, this is number what was tested and known to work reliably.
> This doesn't mean corosync doesn't work with bigger number of nodes.
> Eventho I'm quite surprised that 64 nodes really works.
>
> Variables you can try tweak.
> - Definitively start with increase totem.config (default 1000, you can
> try 10000)
> - If it doesn't help, try increase totem.join (default is 50, 1000 may
> work) and consider increase totem.send_join (default is 0, 100 may be
> good idea).
> - As a last variable, increase of totem.merge (default is 200, 2000 may
> do the job).
>
> And definitively let us know about results. It's quite hard to test such
> a big amount of nodes so some of the variable may be sub-optimal. When
> we know which of variables are victims, we can change their defaults.
>

Thanks for the tuning hints. One thing I definitely need to do before
the next test run is to build some kind of rate limiting into the daemon
to prevent it from bringing our whole network down. I see sendmsg()
calls in totemudp.c and totemupdu.c. Would it be sufficient to restrict
those two calls?

Thanks, Arne

> Regards,
>   Honza
>
>> I want to use corosync + dlm to get a distributed lock manager to have
>> a uniform locking service throughout the platform as a basis for many
>> services.
>>
>> Thanks,
>> Arne
>>
>>>
>>> Cheers,
>>> Kristoffer
>>>
>>>>
>>>> Thanks,
>>>> Arne
>>>>
>>>> Jan 04 10:24:49 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:25:00 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:616) was formed. Members joined: 168101089
>>>> Jan 04 10:25:00 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:25:11 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:620) was formed. Members joined: 168101090
>>>> Jan 04 10:25:11 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:25:21 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:624) was formed. Members joined: 168101091
>>>> Jan 04 10:25:21 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:25:32 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:628) was formed. Members joined: 168101092
>>>> Jan 04 10:25:32 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:26:09 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:632) was formed. Members joined: 168101141
>>>> Jan 04 10:26:10 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:26:20 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:636) was formed. Members joined: 168101142
>>>> Jan 04 10:26:20 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:26:31 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:640) was formed. Members joined: 168101143
>>>> Jan 04 10:26:31 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:26:42 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:644) was formed. Members joined: 168101144
>>>> Jan 04 10:26:42 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:26:52 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:648) was formed. Members joined: 168101145
>>>> Jan 04 10:26:52 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:27:03 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:652) was formed. Members joined: 168101146
>>>> Jan 04 10:27:03 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:27:14 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:656) was formed. Members joined: 168101147
>>>> Jan 04 10:27:14 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:27:25 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:660) was formed. Members joined: 168101148
>>>> Jan 04 10:27:25 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:27:35 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:664) was formed. Members joined: 168101161
>>>> Jan 04 10:27:35 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:27:46 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:668) was formed. Members joined: 168101162
>>>> Jan 04 10:27:46 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:27:57 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:672) was formed. Members joined: 168101163
>>>> Jan 04 10:27:57 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:28:07 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:676) was formed. Members joined: 168101164
>>>> Jan 04 10:28:07 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:28:18 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:680) was formed. Members joined: 168101165
>>>> Jan 04 10:28:18 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:28:28 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:684) was formed. Members joined: 168101166
>>>> Jan 04 10:28:28 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:28:40 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:688) was formed. Members joined: 168101167
>>>> Jan 04 10:28:40 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:28:50 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:692) was formed. Members joined: 168101168
>>>> Jan 04 10:28:50 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:29:00 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:696) was formed. Members joined: 168101181
>>>> Jan 04 10:29:00 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:29:14 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:704) was formed. Members joined: 168100966 168100967
>>>> 168100968 168100969 168100970 168100971 168100972 168100985 168100986
>>>> 168100987 168100988 168100989 168100990 168100991 168100992 168101005
>>>> 168101006 168101007 168101008 168101009 168101010 168101011 168101012
>>>> 168101025 168101026 168101027 168101028 168101029 168101030 168101031
>>>> 168101032 168101045 168101046 168101047 168101048 168101049 168101050
>>>> 168101051 168101052 168
>>>> Jan 04 10:29:14 [4915] reniar corosync notice  [TOTEM ] Failed to
>>>> receive the leave message. failed: 168100966 168100967 168100968
>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>> 168101052 168101065 168101066
>>>> Jan 04 10:29:15 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:716) was formed. Members joined: 168100966 168100967
>>>> 168100968 168100969 168100970 168100971 168100972 168100985 168100986
>>>> 168100987 168100988 168100989 168100990 168100991 168100992 168101005
>>>> 168101006 168101007 168101008 168101009 168101010 168101011 168101012
>>>> 168101025 168101026 168101027 168101028 168101029 168101030 168101031
>>>> 168101032 168101045 168101046 168101047 168101048 168101049 168101050
>>>> 168101051 168101052 168
>>>> Jan 04 10:29:15 [4915] reniar corosync notice  [TOTEM ] Failed to
>>>> receive the leave message. failed: 168100966 168100967 168100968
>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>> 168101052 168101066 168101067
>>>> Jan 04 10:29:16 [4915] reniar corosync notice  [TOTEM ] Retransmit
>>>> List:
>>>> 285
>>>> Jan 04 10:29:16 [4915] reniar corosync notice  [TOTEM ] Retransmit
>>>> List:
>>>> 4db 4dc 4dd 4e1
>>>> Jan 04 10:29:16 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:20 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:732) was formed. Members left: 168100966 168100967
>>>> 168100968
>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>> 168101052 16810
>>>> Jan 04 10:29:20 [4915] reniar corosync notice  [TOTEM ] Failed to
>>>> receive the leave message. failed: 168100966 168100967 168100968
>>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>>> 168101052 168101066 168101067
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:736) was formed. Members
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:25 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>> detected for 503 ms, flushing membership messages.
>>>> Jan 04 10:29:27 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>> detected for 1277 ms, flushing membership messages.
>>>> Jan 04 10:29:27 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>> process was not scheduled for 1345.9542 ms (threshold is 800.0000 ms).
>>>> Consider token timeout increase.
>>>> Jan 04 10:29:29 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>> detected for 607 ms, flushing membership messages.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>> process was not scheduled for 12273.4664 ms (threshold is 800.0000 ms).
>>>> Consider token timeout increase.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:916) was formed. Members
>>>> Jan 04 10:29:42 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync notice  [TOTEM ] Process pause
>>>> detected for 12562 ms, flushing membership messages.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>> process was not scheduled for 12636.5143 ms (threshold is 800.0000 ms).
>>>> Consider token timeout increase.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync notice  [TOTEM ] A new
>>>> membership
>>>> (10.5.4.101:964) was formed. Members
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync notice  [MAIN  ] Completed
>>>> service synchronization, ready to provide service.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>>> message was thrown away during flush operation.
>>>> Jan 04 10:29:59 [4915] reniar corosync warning [MAIN  ] Corosync main
>>>> process was not scheduled for 1465.7160 ms (threshold is 800.0000 ms).
>>>> Consider token timeout increase.
>>>>
>>>> _______________________________________________
>>>> Users mailing list: Users at clusterlabs.org
>>>> http://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started:
>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>>
>>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://lists.clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

More information about the Users mailing list