[ClusterLabs] MySQL Cluster: Strange behaviour when forcing movement of resources

Thu Feb 16 10:34:49 EST 2017

On 02/16/2017 02:26 AM, Félix Díaz de Rada wrote:
> 
> Hi all,
> 
> We are currently setting up a MySQL cluster (Master-Slave) over this
> platform:
> - Two nodes, on RHEL 7.0
> - pacemaker-1.1.10-29.el7.x86_64
> - corosync-2.3.3-2.el7.x86_64
> - pcs-0.9.115-32.el7.x86_64
> There is a IP address resource to be used as a "virtual IP".
> 
> This is configuration of cluster:
> 
> Cluster Name: webmobbdprep
> Corosync Nodes:
>  webmob1bdprep-ges webmob2bdprep-ges
> Pacemaker Nodes:
>  webmob1bdprep-ges webmob2bdprep-ges
> 
> Resources:
>  Group: G_MySQL_M
>   Meta Attrs: priority=100
>   Resource: MySQL_M (class=ocf provider=heartbeat type=mysql_m)
>    Attributes:
> binary=/opt/mysql/mysql-5.7.17-linux-glibc2.5-x86_64/bin/mysqld_safe
> config=/data/webmob_prep/webmob_prep.cnf datadir=/data/webmob_prep
> log=/data/webmob_prep/webmob_prep.err
> pid=/data/webmob_prep/webmob_rep.pid
> socket=/data/webmob_prep/webmob_prep.sock user=mysql group=mysql
> test_table=replica.pacemaker_test test_user=root
>    Meta Attrs: resource-stickiness=1000
>    Operations: promote interval=0s timeout=120 (MySQL_M-promote-timeout-120)
>                demote interval=0s timeout=120 (MySQL_M-demote-timeout-120)
>                start interval=0s timeout=120s on-fail=restart
> (MySQL_M-start-timeout-120s-on-fail-restart)
>                stop interval=0s timeout=120s (MySQL_M-stop-timeout-120s)
>                monitor interval=60s timeout=30s OCF_CHECK_LEVEL=1
> (MySQL_M-monitor-interval-60s-timeout-30s)
>   Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
>    Attributes: ip=172.18.64.44 nic=ens160:1 cidr_netmask=32
>    Meta Attrs: target-role=Started migration-threshold=3
> failure-timeout=60s
>    Operations: start interval=0s timeout=20s (ClusterIP-start-timeout-20s)
>                stop interval=0s timeout=20s (ClusterIP-stop-timeout-20s)
>                monitor interval=60s (ClusterIP-monitor-interval-60s)
>  Resource: MySQL_S (class=ocf provider=heartbeat type=mysql_s)
>   Attributes:
> binary=/opt/mysql/mysql-5.7.17-linux-glibc2.5-x86_64/bin/mysqld_safe
> config=/data/webmob_prep/webmob_prep.cnf datadir=/data/webmob_prep
> log=/data/webmob_prep/webmob_prep.err
> pid=/data/webmob_prep/webmob_rep.pid
> socket=/data/webmob_prep/webmob_prep.sock user=mysql group=mysql
> test_table=replica.pacemaker_test test_user=root
>   Meta Attrs: resource-stickiness=0
>   Operations: promote interval=0s timeout=120 (MySQL_S-promote-timeout-120)
>               demote interval=0s timeout=120 (MySQL_S-demote-timeout-120)
>               start interval=0s timeout=120s on-fail=restart
> (MySQL_S-start-timeout-120s-on-fail-restart)
>               stop interval=0s timeout=120s (MySQL_S-stop-timeout-120s)
>               monitor interval=60s timeout=30s OCF_CHECK_LEVEL=1
> (MySQL_S-monitor-interval-60s-timeout-30s)
> 
> Stonith Devices:
> Fencing Levels:
> 
> Location Constraints:
> Ordering Constraints:
>   start MySQL_M then start ClusterIP (Mandatory)
> (id:order-MySQL_M-ClusterIP-mandatory)
>   start G_MySQL_M then start MySQL_S (Mandatory)
> (id:order-G_MySQL_M-MySQL_S-mandatory)
> Colocation Constraints:
>   G_MySQL_M with MySQL_S (-100) (id:colocation-G_MySQL_M-MySQL_S-INFINITY)
> 
> Cluster Properties:
>  cluster-infrastructure: corosync
>  dc-version: 1.1.10-29.el7-368c726
>  last-lrm-refresh: 1487148812
>  no-quorum-policy: ignore
>  stonith-enabled: false
> 
> Pacemaker works as expected under most of situations, but there is one
> scenario that is really not understable to us. I will try to describe it:
> 
> a - Master resource (and Cluster IP address) are active on node 1 and
> Slave resource is active on node 2.
> b - We force movement of Master resource to node 2.
> c - Pacemaker stops all resources: Master, Slave and Cluster IP.
> d - Master resource and Cluster IP are started on node 2 (this is OK),
> but Slave also tries to start (??). It fails (logically, because Master
> resource has been started on the same node), it logs an "unknown error"
> and its state is marked as "failed". This is a capture of 'pcs status'
> at that point:
> 
> OFFLINE: [ webmob1bdprep-ges ]
> Online: [ webmob2bdprep-ges ]
> 
> Full list of resources:
> 
> Resource Group: G_MySQL_M
>     MySQL_M (ocf::heartbeat:mysql_m): Started webmob2bdprep-ges
>     ClusterIP (ocf::heartbeat:IPaddr2): Started webmob2bdprep-ges
> MySQL_S (ocf::heartbeat:mysql_s): FAILED webmob2bdprep-ges
> 
> Failed actions:
>     MySQL_M_monitor_60000 on webmob2bdprep-ges 'master' (8): call=62,
> status=complete, last-rc-change='Wed Feb 15 11:54:08 2017', queued=0ms,
> exec=0ms
>     MySQL_S_start_0 on webmob2bdprep-ges 'unknown error' (1): call=78,
> status=complete, last-rc-change='Wed Feb 15 11:54:17 2017', queued=40ms,
> exec=0ms
> 
> PCSD Status:
> webmob1bdprep-ges: Offline
> webmob2bdprep-ges: Online
> 
> e - Pacemaker moves Slave resource to node 1 and starts it. Now we have
> both resources started again, Master on node 2 and Slave on node 1.
> f - One minute later, Pacemaker restarts both resources (???).
> 
> So we are wondering:
> - After the migration of the Master resource, why Pacemaker tries to
> start Slave resource on the same node where Master resource has been
> started previously? Why is trying to do it even with a colocation
> restriction as everyone can see on our configuration?

In the configuration here, your colocation has two issues:

1. Colocating "G_MySQL_M with MySQL_S" means that MySQL_S must be placed
first, and only then can G_MySQL_M be placed in relation to it. Since
G_MySQL_M is more important, the colocation should be the other way
around ("MySQL_S with G_MySQL_M").

2. The score of -100 indicates a preference, not a requirement. If only
one node is available to host resources, the cluster is allowed to start
both resources on the same node. Using -INFINITY would make it a
requirement.

> - Once Slave resource has been pushed out to the other node (and
> started), why both resources are restarted again one minute later?

I'm guessing that the other node was initially unavailable, and then
came back online. So first, Pacemaker started both resources on the only
available node. Then once the other node became available, it
re-distributed the resources.

Disregarding the above, however, the biggest thing that jumps out at me
is that this is not how master/slave relationships are usually modeled
in Pacemaker. Pacemaker has built-in support for master/slave clones, so
that a single resource agent can be used for both the master and the
slave, and Pacemaker takes care of promoting/demoting, keeping the
instances on different nodes, etc. See:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#s-resource-clone

> Maybe there is something on our configuration that is the cause for it,
> but we can not figure out what could be. Also, we have been working with
> other Pacemaker cluster (also for MySQL) for the last three years, using
> a very similar configuration and we have not ever seen a behaviour as
> described before. Only difference between former cluster and this new
> one is that the older one is on a RHEL 6 platform and it runs cman and
> not corosync.
> 
> So we would thank any help or remark about why our cluster is doing
> this. Although it is not such a critical problem, it could be an issue
> on a production environment.
> 
> Best Regards,
> 
> 
> 
> 
> 
> -- 
> /*Félix Díaz de Rada*
> fdiaz at gfi.es <mailto:email at gfi.es>
> //—/
> *Gfi Norte* *| * *www.gfi.es* <http://www.gfi.es>
> /Calle Licenciado Poza 55, Planta 2
> 48013 Bilbao Bizkaia
> Teléfono: +34 94 424 18 25/
> 
> Síguenos: Blog <http://blog.gfi.es/> | Facebook
> <https://www.facebook.com/gfiinformatica> | LinkedIn
> <http://www.linkedin.com/company/gfi-inform-tica> | Twitter
> <https://twitter.com/GFI_Informatica> | YouTube
> <http://www.youtube.com/user/GFIInformatica>
> Buscamos continuamente talento: http://www.gfi.es/carreras //