[ClusterLabs] large cluster with corosync
Kristoffer Grönlund
kgronlund at suse.com
Wed Jan 4 05:25:23 EST 2017
Arne Jansen <lists at die-jansens.de> writes:
> Hi,
>
> I've built corosync for solaris and am trying to build a largish
> cluster. I started corosync with default configuration on an
> increasing number of nodes, one by one. At around 70 nodes the
> cluster breaks down. Below is an excerpt from the logfile on the
> first node.
> When the cluster breaks down corosync seems to completely flood
> the network. It doesn't recover by itself, I have to stop all nodes.
> Things get worse if I start corosync on multiple nodes at once.
> In this case it already breaks down around 40 nodes.
>
> Is it supposed to work with such a setup? Does it just need tuning?
> My goal is have a cluster of several hundred nodes.
Hi,
No, corosync has a limit to how many nodes the cluster can
contain and still function properly. According to SUSE, the limit is 32
nodes, according to Red Hat the limit is 16 (those are the limits for
enterprise support as far as I know) - but even having that number
of nodes will probably require some tweaking of timeouts in
corosync.conf, depending on what kind of network you have.
If I recall correctly, there are plans for overcoming this limit in the
future but right now that's the situation.
There is the pacemaker_remote project which allows for adding
additional, non-corosync nodes to a cluster which can run resources but
don't participate in quorum and rely on at least some of the core
cluster nodes still being available. Using pacemaker_remote could enable
a cluster of several hundred nodes.
My question would be, why do you need so many nodes in a high
availaibility cluster?
Cheers,
Kristoffer
>
> Thanks,
> Arne
>
> Jan 04 10:24:49 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:25:00 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:616) was formed. Members joined: 168101089
> Jan 04 10:25:00 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:25:11 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:620) was formed. Members joined: 168101090
> Jan 04 10:25:11 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:25:21 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:624) was formed. Members joined: 168101091
> Jan 04 10:25:21 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:25:32 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:628) was formed. Members joined: 168101092
> Jan 04 10:25:32 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:26:09 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:632) was formed. Members joined: 168101141
> Jan 04 10:26:10 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:26:20 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:636) was formed. Members joined: 168101142
> Jan 04 10:26:20 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:26:31 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:640) was formed. Members joined: 168101143
> Jan 04 10:26:31 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:26:42 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:644) was formed. Members joined: 168101144
> Jan 04 10:26:42 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:26:52 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:648) was formed. Members joined: 168101145
> Jan 04 10:26:52 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:27:03 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:652) was formed. Members joined: 168101146
> Jan 04 10:27:03 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:27:14 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:656) was formed. Members joined: 168101147
> Jan 04 10:27:14 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:27:25 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:660) was formed. Members joined: 168101148
> Jan 04 10:27:25 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:27:35 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:664) was formed. Members joined: 168101161
> Jan 04 10:27:35 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:27:46 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:668) was formed. Members joined: 168101162
> Jan 04 10:27:46 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:27:57 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:672) was formed. Members joined: 168101163
> Jan 04 10:27:57 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:28:07 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:676) was formed. Members joined: 168101164
> Jan 04 10:28:07 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:28:18 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:680) was formed. Members joined: 168101165
> Jan 04 10:28:18 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:28:28 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:684) was formed. Members joined: 168101166
> Jan 04 10:28:28 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:28:40 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:688) was formed. Members joined: 168101167
> Jan 04 10:28:40 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:28:50 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:692) was formed. Members joined: 168101168
> Jan 04 10:28:50 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:29:00 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:696) was formed. Members joined: 168101181
> Jan 04 10:29:00 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:29:14 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:704) was formed. Members joined: 168100966 168100967
> 168100968 168100969 168100970 168100971 168100972 168100985 168100986
> 168100987 168100988 168100989 168100990 168100991 168100992 168101005
> 168101006 168101007 168101008 168101009 168101010 168101011 168101012
> 168101025 168101026 168101027 168101028 168101029 168101030 168101031
> 168101032 168101045 168101046 168101047 168101048 168101049 168101050
> 168101051 168101052 168
> Jan 04 10:29:14 [4915] reniar corosync notice [TOTEM ] Failed to
> receive the leave message. failed: 168100966 168100967 168100968
> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
> 168101052 168101065 168101066
> Jan 04 10:29:15 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:716) was formed. Members joined: 168100966 168100967
> 168100968 168100969 168100970 168100971 168100972 168100985 168100986
> 168100987 168100988 168100989 168100990 168100991 168100992 168101005
> 168101006 168101007 168101008 168101009 168101010 168101011 168101012
> 168101025 168101026 168101027 168101028 168101029 168101030 168101031
> 168101032 168101045 168101046 168101047 168101048 168101049 168101050
> 168101051 168101052 168
> Jan 04 10:29:15 [4915] reniar corosync notice [TOTEM ] Failed to
> receive the leave message. failed: 168100966 168100967 168100968
> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
> 168101052 168101066 168101067
> Jan 04 10:29:16 [4915] reniar corosync notice [TOTEM ] Retransmit List:
> 285
> Jan 04 10:29:16 [4915] reniar corosync notice [TOTEM ] Retransmit List:
> 4db 4dc 4dd 4e1
> Jan 04 10:29:16 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:20 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:732) was formed. Members left: 168100966 168100967 168100968
> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
> 168101052 16810
> Jan 04 10:29:20 [4915] reniar corosync notice [TOTEM ] Failed to
> receive the leave message. failed: 168100966 168100967 168100968
> 168100969 168100970 168100971 168100972 168100985 168100986 168100987
> 168100988 168100989 168100990 168100991 168100992 168101005 168101006
> 168101007 168101008 168101009 168101010 168101011 168101012 168101025
> 168101026 168101027 168101028 168101029 168101030 168101031 168101032
> 168101045 168101046 168101047 168101048 168101049 168101050 168101051
> 168101052 168101066 168101067
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:736) was formed. Members
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:25 [4915] reniar corosync notice [TOTEM ] Process pause
> detected for 503 ms, flushing membership messages.
> Jan 04 10:29:27 [4915] reniar corosync notice [TOTEM ] Process pause
> detected for 1277 ms, flushing membership messages.
> Jan 04 10:29:27 [4915] reniar corosync warning [MAIN ] Corosync main
> process was not scheduled for 1345.9542 ms (threshold is 800.0000 ms).
> Consider token timeout increase.
> Jan 04 10:29:29 [4915] reniar corosync notice [TOTEM ] Process pause
> detected for 607 ms, flushing membership messages.
> Jan 04 10:29:42 [4915] reniar corosync warning [MAIN ] Corosync main
> process was not scheduled for 12273.4664 ms (threshold is 800.0000 ms).
> Consider token timeout increase.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:916) was formed. Members
> Jan 04 10:29:42 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync notice [TOTEM ] Process pause
> detected for 12562 ms, flushing membership messages.
> Jan 04 10:29:55 [4915] reniar corosync warning [MAIN ] Corosync main
> process was not scheduled for 12636.5143 ms (threshold is 800.0000 ms).
> Consider token timeout increase.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync notice [TOTEM ] A new membership
> (10.5.4.101:964) was formed. Members
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync notice [MAIN ] Completed
> service synchronization, ready to provide service.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE
> message was thrown away during flush operation.
> Jan 04 10:29:59 [4915] reniar corosync warning [MAIN ] Corosync main
> process was not scheduled for 1465.7160 ms (threshold is 800.0000 ms).
> Consider token timeout increase.
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
--
// Kristoffer Grönlund
// kgronlund at suse.com
More information about the Users
mailing list