[ClusterLabs Developers] Master scores ignored during cluster startup

[Ken Gaillot]

On Tue, 2017-08-29 at 22:55 +0200, Jehan-Guillaume de Rorthais wrote:
> Hi all,
> 
> We discussed this issue with Ken Gaillot and Lars Ellenberg today on IRC. This
> is just a sum up of the problem.
> 
> Some users reported me that the PAF RA was not promoting the master after a
> cluster startup. The problem is that PAF is using master score with "-l
> forever", not transient ones disappearing on cluster reboot ("-l reboot"). It
> expects the master scores to be present on cluster start.
> 
> So basically, on cluster startup, the master score is just ignored and no slave
> is being promoted...before the next cluster recheck (default to 15min later).
> 
> You could reproduce the issue using the RA attached to this email and the
> following scenario:
> 
>   pcs cluster setup --name cluster_demo srv1 srv2
>   pcs cluster start --all
>   pcs cluster cib cluster1.xml
>   pcs -f cluster1.xml property set stonith-enabled=false
>   pcs -f cluster1.xml resource create stateful stateful-custom \
>     op monitor interval=10s role="Master" \
>     op monitor interval=11s role="Slave"
>   pcs -f cluster1.xml resource master stateful-ha stateful
>   pcs cluster cib-push cluster1.xml
>   crm_master -r stateful -l forever -v 1
>   # the master is elected and set its score to 10
>   pcs cluster stop --all
>   pcs cluster start --all
>   # no master elected despite the master score
> 
> Our discussion led to the theory that this might be correlated to this
> commit where master score is ignored for a resource if it is not started yet:
> https://github.com/beekhof/pacemaker/commit/65f1a22a4b66581159d8b747dbd49fa5e2ef34e1
> 
> If this is the actual issue, I suppose this should be improved to detect if
> another master is existing or not before ignoring the master score. Or maybe to
> detect if we are in a cluster startup or enabling the stateful resource in the
> cluster.
> 
> Thoughts?

The issue has turned out to be more complicated, and unrelated to that
commit. I see two problems: clone_name is NULL when master_score() is
called, so the code looks for master-whatever:0 rather than
master-whatever; and startup triggers the "has been filtered" block of
code in master_score(), so we don't even get as far as checking
clone_name.

My instinct for the second issue is that we should be able to skip the
"has been filtered" short-circuit if the resource isn't running or
probed *anywhere* (as opposed to the particular node being checked). I
haven't tracked down the first issue yet.
-- 
Ken Gaillot <kgaillot at redhat.com>