[ClusterLabs] large cluster with corosync

Arne Jansen lists at die-jansens.de
Wed Jan 4 11:00:08 CET 2017


Hi,

I've built corosync for solaris and am trying to build a largish
cluster. I started corosync with default configuration on an
increasing number of nodes, one by one. At around 70 nodes the
cluster breaks down. Below is an excerpt from the logfile on the
first node.
When the cluster breaks down corosync seems to completely flood
the network. It doesn't recover by itself, I have to stop all nodes.
Things get worse if I start corosync on multiple nodes at once.
In this case it already breaks down around 40 nodes.

Is it supposed to work with such a setup? Does it just need tuning?
My goal is have a cluster of several hundred nodes.

Thanks,
Arne

Jan 04 10:24:49 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:25:00 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:616) was formed. Members joined: 168101089
Jan 04 10:25:00 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:25:11 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:620) was formed. Members joined: 168101090
Jan 04 10:25:11 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:25:21 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:624) was formed. Members joined: 168101091
Jan 04 10:25:21 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:25:32 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:628) was formed. Members joined: 168101092
Jan 04 10:25:32 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:26:09 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:632) was formed. Members joined: 168101141
Jan 04 10:26:10 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:26:20 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:636) was formed. Members joined: 168101142
Jan 04 10:26:20 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:26:31 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:640) was formed. Members joined: 168101143
Jan 04 10:26:31 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:26:42 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:644) was formed. Members joined: 168101144
Jan 04 10:26:42 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:26:52 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:648) was formed. Members joined: 168101145
Jan 04 10:26:52 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:27:03 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:652) was formed. Members joined: 168101146
Jan 04 10:27:03 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:27:14 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:656) was formed. Members joined: 168101147
Jan 04 10:27:14 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:27:25 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:660) was formed. Members joined: 168101148
Jan 04 10:27:25 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:27:35 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:664) was formed. Members joined: 168101161
Jan 04 10:27:35 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:27:46 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:668) was formed. Members joined: 168101162
Jan 04 10:27:46 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:27:57 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:672) was formed. Members joined: 168101163
Jan 04 10:27:57 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:28:07 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:676) was formed. Members joined: 168101164
Jan 04 10:28:07 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:28:18 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:680) was formed. Members joined: 168101165
Jan 04 10:28:18 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:28:28 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:684) was formed. Members joined: 168101166
Jan 04 10:28:28 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:28:40 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:688) was formed. Members joined: 168101167
Jan 04 10:28:40 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:28:50 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:692) was formed. Members joined: 168101168
Jan 04 10:28:50 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:29:00 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:696) was formed. Members joined: 168101181
Jan 04 10:29:00 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:29:14 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:704) was formed. Members joined: 168100966 168100967 
168100968 168100969 168100970 168100971 168100972 168100985 168100986 
168100987 168100988 168100989 168100990 168100991 168100992 168101005 
168101006 168101007 168101008 168101009 168101010 168101011 168101012 
168101025 168101026 168101027 168101028 168101029 168101030 168101031 
168101032 168101045 168101046 168101047 168101048 168101049 168101050 
168101051 168101052 168
Jan 04 10:29:14 [4915] reniar corosync notice  [TOTEM ] Failed to 
receive the leave message. failed: 168100966 168100967 168100968 
168100969 168100970 168100971 168100972 168100985 168100986 168100987 
168100988 168100989 168100990 168100991 168100992 168101005 168101006 
168101007 168101008 168101009 168101010 168101011 168101012 168101025 
168101026 168101027 168101028 168101029 168101030 168101031 168101032 
168101045 168101046 168101047 168101048 168101049 168101050 168101051 
168101052 168101065 168101066
Jan 04 10:29:15 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:716) was formed. Members joined: 168100966 168100967 
168100968 168100969 168100970 168100971 168100972 168100985 168100986 
168100987 168100988 168100989 168100990 168100991 168100992 168101005 
168101006 168101007 168101008 168101009 168101010 168101011 168101012 
168101025 168101026 168101027 168101028 168101029 168101030 168101031 
168101032 168101045 168101046 168101047 168101048 168101049 168101050 
168101051 168101052 168
Jan 04 10:29:15 [4915] reniar corosync notice  [TOTEM ] Failed to 
receive the leave message. failed: 168100966 168100967 168100968 
168100969 168100970 168100971 168100972 168100985 168100986 168100987 
168100988 168100989 168100990 168100991 168100992 168101005 168101006 
168101007 168101008 168101009 168101010 168101011 168101012 168101025 
168101026 168101027 168101028 168101029 168101030 168101031 168101032 
168101045 168101046 168101047 168101048 168101049 168101050 168101051 
168101052 168101066 168101067
Jan 04 10:29:16 [4915] reniar corosync notice  [TOTEM ] Retransmit List: 
285
Jan 04 10:29:16 [4915] reniar corosync notice  [TOTEM ] Retransmit List: 
4db 4dc 4dd 4e1
Jan 04 10:29:16 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:20 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:732) was formed. Members left: 168100966 168100967 168100968 
168100969 168100970 168100971 168100972 168100985 168100986 168100987 
168100988 168100989 168100990 168100991 168100992 168101005 168101006 
168101007 168101008 168101009 168101010 168101011 168101012 168101025 
168101026 168101027 168101028 168101029 168101030 168101031 168101032 
168101045 168101046 168101047 168101048 168101049 168101050 168101051 
168101052 16810
Jan 04 10:29:20 [4915] reniar corosync notice  [TOTEM ] Failed to 
receive the leave message. failed: 168100966 168100967 168100968 
168100969 168100970 168100971 168100972 168100985 168100986 168100987 
168100988 168100989 168100990 168100991 168100992 168101005 168101006 
168101007 168101008 168101009 168101010 168101011 168101012 168101025 
168101026 168101027 168101028 168101029 168101030 168101031 168101032 
168101045 168101046 168101047 168101048 168101049 168101050 168101051 
168101052 168101066 168101067
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:736) was formed. Members
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:21 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:25 [4915] reniar corosync notice  [TOTEM ] Process pause 
detected for 503 ms, flushing membership messages.
Jan 04 10:29:27 [4915] reniar corosync notice  [TOTEM ] Process pause 
detected for 1277 ms, flushing membership messages.
Jan 04 10:29:27 [4915] reniar corosync warning [MAIN  ] Corosync main 
process was not scheduled for 1345.9542 ms (threshold is 800.0000 ms). 
Consider token timeout increase.
Jan 04 10:29:29 [4915] reniar corosync notice  [TOTEM ] Process pause 
detected for 607 ms, flushing membership messages.
Jan 04 10:29:42 [4915] reniar corosync warning [MAIN  ] Corosync main 
process was not scheduled for 12273.4664 ms (threshold is 800.0000 ms). 
Consider token timeout increase.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:916) was formed. Members
Jan 04 10:29:42 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:42 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync notice  [TOTEM ] Process pause 
detected for 12562 ms, flushing membership messages.
Jan 04 10:29:55 [4915] reniar corosync warning [MAIN  ] Corosync main 
process was not scheduled for 12636.5143 ms (threshold is 800.0000 ms). 
Consider token timeout increase.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync notice  [TOTEM ] A new membership 
(10.5.4.101:964) was formed. Members
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync notice  [MAIN  ] Completed 
service synchronization, ready to provide service.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:55 [4915] reniar corosync warning [TOTEM ] JOIN or LEAVE 
message was thrown away during flush operation.
Jan 04 10:29:59 [4915] reniar corosync warning [MAIN  ] Corosync main 
process was not scheduled for 1465.7160 ms (threshold is 800.0000 ms). 
Consider token timeout increase.



More information about the Users mailing list