[ClusterLabs] 5 minute repeat warnings

[Ken Gaillot]

On Tue, 2019-11-26 at 08:31 +0000, BASDEN, ALASTAIR G. wrote:
> Hi,
> 
> In our /var/log/messages, we are getting messages repeated every 5 
> minutes:
> Nov 26 08:22:02 c6mds1 crmd[26950]:  notice: State transition S_IDLE
> -> 
> S_POLICY_ENGINE
> Nov 26 08:22:02 c6mds1 pengine[26949]:  notice: On loss of CCM
> Quorum: 
> Ignore
> Nov 26 08:22:02 c6mds1 pengine[26949]:  notice: Calculated
> transition 
> 1397, saving inputs in /var/lib/pacemaker/pengine/pe-input-184.bz2
> Nov 26 08:22:02 c6mds1 crmd[26950]:  notice: Transition 1397
> (Complete=0, 
> Pending=0, Fired=0, Skipped=0, Incomplete=0, 
> Source=/var/lib/pacemaker/pengine/pe-input-184.bz2): Complete
> Nov 26 08:22:02 c6mds1 crmd[26950]:  notice: State transition 
> S_TRANSITION_ENGINE -> S_IDLE
> 
> 
> Can anyone advise what this is about and how to stop the
> messages?  pcs 
> resource cleanup doesn't help.
> 
> We're on centos7.6.
> 
> Thanks,
> Alastair.

Hi,

Those are routine messages indicating that the cluster is rechecking
whether anything needs to be done. The frequency is controlled by the
cluster-recheck-interval property.

In the version you have, any value for failure-timeout or time-based
rules is not guaranteed to be checked more often than the recheck
interval. The recheck interval also serves as a fail-safe against
certain types of policy engine bugs (those that do not schedule all
needed actions in a single transition).

In 2.0.3 (released yesterday), the recheck interval is calculated
dynamically for failure-timeout and rules, so it has less importance.

More generally, log messages at "notice" level are just informational
and do not indicate anything wrong. Problems are logged at "warning",
"error", or "critical" level.

I could see an argument for lowering the "calculated"/"complete"
messages to "info" level (which doesn't go into the system log) when no
actions are needed. The state transition messages should stay at
notice, though.
-- 
Ken Gaillot <kgaillot at redhat.com>