[ClusterLabs] large cluster with corosync

Jan Friesse jfriesse at redhat.com
Wed Jan 4 13:52:09 CET 2017 

Arne Jansen napsal(a):
>
>
> On 04.01.2017 11:25, Kristoffer Grönlund wrote:
>> Arne Jansen <lists at die-jansens.de> writes:
>>
>>> Hi,
>>>
>>> I've built corosync for solaris and am trying to build a largish
>>> cluster. I started corosync with default configuration on an
>>> increasing number of nodes, one by one. At around 70 nodes the
>>> cluster breaks down. Below is an excerpt from the logfile on the
>>> first node.
>>> When the cluster breaks down corosync seems to completely flood
>>> the network. It doesn't recover by itself, I have to stop all nodes.
>>> Things get worse if I start corosync on multiple nodes at once.
>>> In this case it already breaks down around 40 nodes.
>>>
>>> Is it supposed to work with such a setup? Does it just need tuning?
>>> My goal is have a cluster of several hundred nodes.
>>
>> Hi,
>>
>> No, corosync has a limit to how many nodes the cluster can
>> contain and still function properly. According to SUSE, the limit is 32
>> nodes, according to Red Hat the limit is 16 (those are the limits for
>> enterprise support as far as I know) - but even having that number
>> of nodes will probably require some tweaking of timeouts in
>> corosync.conf, depending on what kind of network you have.
>>
>> If I recall correctly, there are plans for overcoming this limit in the
>> future but right now that's the situation.
>>
>> There is the pacemaker_remote project which allows for adding
>> additional, non-corosync nodes to a cluster which can run resources but
>> don't participate in quorum and rely on at least some of the core
>> cluster nodes still being available. Using pacemaker_remote could enable
>> a cluster of several hundred nodes.
>>
>> My question would be, why do you need so many nodes in a high
>> availaibility cluster?
>
> At least those limits doesn't seem to get enforced, as a 64 node cluster
> seems to work, although a bit shaky.

No, they are not enforced. 16/32 are official supported number of nodes. 
Basically, this is number what was tested and known to work reliably. 
This doesn't mean corosync doesn't work with bigger number of nodes. 
Eventho I'm quite surprised that 64 nodes really works.

Variables you can try tweak.
- Definitively start with increase totem.config (default 1000, you can 
try 10000)
- If it doesn't help, try increase totem.join (default is 50, 1000 may 
work) and consider increase totem.send_join (default is 0, 100 may be 
good idea).
- As a last variable, increase of totem.merge (default is 200, 2000 may 
do the job).

And definitively let us know about results. It's quite hard to test such 
a big amount of nodes so some of the variable may be sub-optimal. When 
we know which of variables are victims, we can change their defaults.

Regards,
   Honza

> I want to use corosync + dlm to get a distributed lock manager to have
> a uniform locking service throughout the platform as a basis for many
> services.
>
> Thanks,
> Arne
>
>>
>> Cheers,
>> Kristoffer
>>
>>>
>>> Thanks,
>>> Arne
>>>
>>> Jan 04 10:24:49 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:25:00 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:616) was formed. Members joined: 168101089
>>> Jan 04 10:25:00 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:25:11 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:620) was formed. Members joined: 168101090
>>> Jan 04 10:25:11 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:25:21 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:624) was formed. Members joined: 168101091
>>> Jan 04 10:25:21 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:25:32 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:628) was formed. Members joined: 168101092
>>> Jan 04 10:25:32 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:26:09 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:632) was formed. Members joined: 168101141
>>> Jan 04 10:26:10 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:26:20 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:636) was formed. Members joined: 168101142
>>> Jan 04 10:26:20 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:26:31 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:640) was formed. Members joined: 168101143
>>> Jan 04 10:26:31 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:26:42 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:644) was formed. Members joined: 168101144
>>> Jan 04 10:26:42 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:26:52 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:648) was formed. Members joined: 168101145
>>> Jan 04 10:26:52 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:27:03 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:652) was formed. Members joined: 168101146
>>> Jan 04 10:27:03 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:27:14 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:656) was formed. Members joined: 168101147
>>> Jan 04 10:27:14 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:27:25 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:660) was formed. Members joined: 168101148
>>> Jan 04 10:27:25 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:27:35 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:664) was formed. Members joined: 168101161
>>> Jan 04 10:27:35 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:27:46 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:668) was formed. Members joined: 168101162
>>> Jan 04 10:27:46 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:27:57 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:672) was formed. Members joined: 168101163
>>> Jan 04 10:27:57 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:28:07 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:676) was formed. Members joined: 168101164
>>> Jan 04 10:28:07 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:28:18 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:680) was formed. Members joined: 168101165
>>> Jan 04 10:28:18 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:28:28 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:684) was formed. Members joined: 168101166
>>> Jan 04 10:28:28 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:28:40 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:688) was formed. Members joined: 168101167
>>> Jan 04 10:28:40 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:28:50 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:692) was formed. Members joined: 168101168
>>> Jan 04 10:28:50 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:29:00 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:696) was formed. Members joined: 168101181
>>> Jan 04 10:29:00 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:29:14 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:704) was formed. Members joined: 168100966 168100967
>>> 168100968 168100969 168100970 168100971 168100972 168100985 168100986
>>> 168100987 168100988 168100989 168100990 168100991 168100992 168101005
>>> 168101006 168101007 168101008 168101009 168101010 168101011 168101012
>>> 168101025 168101026 168101027 168101028 168101029 168101030 168101031
>>> 168101032 168101045 168101046 168101047 168101048 168101049 168101050
>>> 168101051 168101052 168
>>> Jan 04 10:29:14 [4915] reniar corosync notice  [TOTEM ] Failed to
>>> receive the leave message. failed: 168100966 168100967 168100968
>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>> 168101052 168101065 168101066
>>> Jan 04 10:29:15 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:716) was formed. Members joined: 168100966 168100967
>>> 168100968 168100969 168100970 168100971 168100972 168100985 168100986
>>> 168100987 168100988 168100989 168100990 168100991 168100992 168101005
>>> 168101006 168101007 168101008 168101009 168101010 168101011 168101012
>>> 168101025 168101026 168101027 168101028 168101029 168101030 168101031
>>> 168101032 168101045 168101046 168101047 168101048 168101049 168101050
>>> 168101051 168101052 168
>>> Jan 04 10:29:15 [4915] reniar corosync notice  [TOTEM ] Failed to
>>> receive the leave message. failed: 168100966 168100967 168100968
>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>> 168101052 168101066 168101067
>>> Jan 04 10:29:16 [4915] reniar corosync notice  [TOTEM ] Retransmit List:
>>> 285
>>> Jan 04 10:29:16 [4915] reniar corosync notice  [TOTEM ] Retransmit List:
>>> 4db 4dc 4dd 4e1
>>> Jan 04 10:29:16 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:20 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:732) was formed. Members left: 168100966 168100967 168100968
>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>> 168101052 16810
>>> Jan 04 10:29:20 [4915] reniar corosync notice  [TOTEM ] Failed to
>>> receive the leave message. failed: 168100966 168100967 168100968
>>> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
>>> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
>>> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
>>> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
>>> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
>>> 168101052 168101066 168101067
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:736) was formed. Members
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:25 [4915] reniar corosync notice  [TOTEM ] Process pause
>>> detected for 503 ms, flushing membership messages.
>>> Jan 04 10:29:27 [4915] reniar corosync notice  [TOTEM ] Process pause
>>> detected for 1277 ms, flushing membership messages.
>>> Jan 04 10:29:27 [4915] reniar corosync warning [MAIN  ] Corosync main
>>> process was not scheduled for 1345.9542 ms (threshold is 800.0000 ms).
>>> Consider token timeout increase.
>>> Jan 04 10:29:29 [4915] reniar corosync notice  [TOTEM ] Process pause
>>> detected for 607 ms, flushing membership messages.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [MAIN  ] Corosync main
>>> process was not scheduled for 12273.4664 ms (threshold is 800.0000 ms).
>>> Consider token timeout increase.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:916) was formed. Members
>>> Jan 04 10:29:42 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync notice  [TOTEM ] Process pause
>>> detected for 12562 ms, flushing membership messages.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [MAIN  ] Corosync main
>>> process was not scheduled for 12636.5143 ms (threshold is 800.0000 ms).
>>> Consider token timeout increase.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync notice  [TOTEM ] A new membership
>>> (10.5.4.101:964) was formed. Members
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync notice  [MAIN  ] Completed
>>> service synchronization, ready to provide service.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
>>> message was thrown away during flush operation.
>>> Jan 04 10:29:59 [4915] reniar corosync warning [MAIN  ] Corosync main
>>> process was not scheduled for 1465.7160 ms (threshold is 800.0000 ms).
>>> Consider token timeout increase.
>>>
>>> _______________________________________________
>>> Users mailing list: Users at clusterlabs.org
>>> http://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

More information about the Users mailing list