[ClusterLabs] recommendations for corosync totem timeout for CentOS 7 + VMware?
reichert at numachi.com
Thu Mar 21 12:21:46 EDT 2019
I've followed several tutorials about setting up a simple three-node
cluster, with no resources (yet), under CentOS 7.
I've discovered the cluster won't restart upon rebooting a node.
The other two nodes, however, do claim the cluster is up, as shown
with 'pcs status cluster'.
I tracked down that on the rebooted node, corosync exited with a
'0' status. Nothing outright seems to be what I would call an error
message, but this was recorded:
[MAIN ] Corosync main process was not scheduled for 2145.7053
ms (threshold is 1320.0000 ms). Consider token timeout increase.
This seems related:
High Availability cluster node logs the message "Corosync main
process was not scheduled for X ms (threshold is Y ms). Consider
token timeout increase."
I've confirmed that corosync is running with the maximum realtime
[root at node1 ~]# ps -eo cmd,rtprio | grep -e [c]orosync -e RTPRIO
I am doing my testing in an admittedly underprovisioned VM environment.
I've used this same environment for CentOS 6 / heartbeat-based
solutions, and they were nowhere near as sensitive to these timing
Manually running 'pcs cluster start' does indeed fire everything
up without a hitch, and remains running for days at a crack.
The 'consider token timeout increase' message has me looking at this:
Which makes this assertion:
RHEL 7 or 8
If no token value is specified in the corosync configuration, the
default is 1000 ms, or 1 second for a 2 node cluster, increasing
by 650ms for each additional member.
I have a three-node cluster, and the arithmetic for totem.token
seems to hold:
[root at node3 ~]# corosync-cmapctl | grep totem.token
runtime.config.totem.token (u32) = 1650
runtime.config.totem.token_retransmit (u32) = 392
runtime.config.totem.token_retransmits_before_loss_const (u32) = 4
I'm confused on a number of issues:
- The 'totem.token' value of 1650 doesn't seem to related to the
threshold number in the diagnostic message the corosync service
threshold is 1320.0000 ms
Can someone explain the relationship between these values?
- If I manually set 'totem.token' to a higher value, am I responsible
for tracking the number of nodes in the cluster, to keep in
alignment with what Red Hat's page says?
- Under these conditions, when corosync exits, why does it do so
with a zero status? It seems to me that if it exited at all,
without someone controllably stopping the service, it warrants a
- Is there a recommended way to alter either pacemaker/corosync or
systemd configuration of these services to harden against resource
I don't know if corosync's startup can be deferred until the CPU
load settles, or if the some automatic retry can be set up...
Details of my environment; I'm happy to provide others, if anyone
has any specific questions:
[root at node1 ~]# cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)
[root at node1 ~]# rpm -qa | egrep 'pacemaker|corosync'
Brian Reichert <reichert at numachi.com>
BSD admin/developer at large
More information about the Users