[ClusterLabs] Maintenance & Pacemaker Restart Demotes MS Resources

[Ken Gaillot]

On Wed, 2019-06-05 at 07:40 -0700, Dirk Gassen wrote:
> Hi,
> 
> I have the following CIB:
> > primitive AppserverIP IPaddr \
> >         params ip=10.1.8.70 cidr_netmask=255.255.255.192 nic=eth0 \
> >         op monitor interval=30s
> > primitive MariaDB mysql \
> >         params binary="/usr/bin/mysqld_safe"
> pid="/var/run/mysqld/mysqld.pid" socket="/var/run/mysqld/mysqld.sock"
> replication_user=repl replication_passwd="r3plic at tion"
> max_slave_lag=15 evict_outdated_slaves=false test_user=repl
> test_passwd="r3plic at tion" config="/etc/mysql/my.cnf" user=mysql
> group=mysql datadir="/opt/mysql" \
> >         op monitor interval=27s role=Master OCF_CHECK_LEVEL=1 \
> >         op monitor interval=35s timeout=30 role=Slave
> OCF_CHECK_LEVEL=1 \
> >         op start interval=0 timeout=130 \
> >         op stop interval=0 timeout=130
> > ms ms_MariaDB MariaDB \
> >         meta master-max=1 master-node-max=1 clone-node-max=1
> notify=true globally-unique=false target-role=Started is-managed=true
> > colocation colo_sm_aip inf: AppserverIP:Started ms_MariaDB:Master
> 
> When I do "crm node testras3 maintenance && systemctl stop pacemaker
> && systemctl start pacemaker && crm node testras3 ready" the cluster
> decides to demote ms_MariaDB and (because of the colocation) to stop
> AppserverIP. it then follows up immediately with promoting ms_MariaDB
> and starting AppserverIP again.
> 
> If I leave out restarting pacemaker the cluster does not demote
> ms_MariaDB and AppserverIP is left running.
> 
> Why is the demotion happening and is there a way to avoid this?

It looks like there isn't enough time between starting pacemaker and
taking the node out of maintenance for pacemaker to re-detect the state
of all resources. It's best to do that manually, i.e. wait for the
status output to show all the resources again, but you could automate
it with a fixed sleep or maybe a brief sleep plus crm_resource --wait.

> Corosync 2.3.5-3ubuntu2.3 and Pacemaker 1.1.14-2ubuntu1.6
> 
> Sincerely,
> Dirk
> -- 
> Dirk Gassen
> Senior Software Engineer | GetWellNetwork
> o: 240.482.3146
> e: dgassen at getwellnetwork.com
> To help people take an active role in their health journey
-- 
Ken Gaillot <kgaillot at redhat.com>