[ClusterLabs] MySQL Cluster: Strange behaviour when forcing movement of resources

Félix Díaz de Rada fdiaz at gfi.es
Thu Feb 16 03:26:20 EST 2017


Hi all,

We are currently setting up a MySQL cluster (Master-Slave) over this 
platform:
- Two nodes, on RHEL 7.0
- pacemaker-1.1.10-29.el7.x86_64
- corosync-2.3.3-2.el7.x86_64
- pcs-0.9.115-32.el7.x86_64
There is a IP address resource to be used as a "virtual IP".

This is configuration of cluster:

Cluster Name: webmobbdprep
Corosync Nodes:
  webmob1bdprep-ges webmob2bdprep-ges
Pacemaker Nodes:
  webmob1bdprep-ges webmob2bdprep-ges

Resources:
  Group: G_MySQL_M
   Meta Attrs: priority=100
   Resource: MySQL_M (class=ocf provider=heartbeat type=mysql_m)
    Attributes: 
binary=/opt/mysql/mysql-5.7.17-linux-glibc2.5-x86_64/bin/mysqld_safe 
config=/data/webmob_prep/webmob_prep.cnf datadir=/data/webmob_prep 
log=/data/webmob_prep/webmob_prep.err 
pid=/data/webmob_prep/webmob_rep.pid 
socket=/data/webmob_prep/webmob_prep.sock user=mysql group=mysql 
test_table=replica.pacemaker_test test_user=root
    Meta Attrs: resource-stickiness=1000
    Operations: promote interval=0s timeout=120 
(MySQL_M-promote-timeout-120)
                demote interval=0s timeout=120 (MySQL_M-demote-timeout-120)
                start interval=0s timeout=120s on-fail=restart 
(MySQL_M-start-timeout-120s-on-fail-restart)
                stop interval=0s timeout=120s (MySQL_M-stop-timeout-120s)
                monitor interval=60s timeout=30s OCF_CHECK_LEVEL=1 
(MySQL_M-monitor-interval-60s-timeout-30s)
   Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
    Attributes: ip=172.18.64.44 nic=ens160:1 cidr_netmask=32
    Meta Attrs: target-role=Started migration-threshold=3 
failure-timeout=60s
    Operations: start interval=0s timeout=20s (ClusterIP-start-timeout-20s)
                stop interval=0s timeout=20s (ClusterIP-stop-timeout-20s)
                monitor interval=60s (ClusterIP-monitor-interval-60s)
  Resource: MySQL_S (class=ocf provider=heartbeat type=mysql_s)
   Attributes: 
binary=/opt/mysql/mysql-5.7.17-linux-glibc2.5-x86_64/bin/mysqld_safe 
config=/data/webmob_prep/webmob_prep.cnf datadir=/data/webmob_prep 
log=/data/webmob_prep/webmob_prep.err 
pid=/data/webmob_prep/webmob_rep.pid 
socket=/data/webmob_prep/webmob_prep.sock user=mysql group=mysql 
test_table=replica.pacemaker_test test_user=root
   Meta Attrs: resource-stickiness=0
   Operations: promote interval=0s timeout=120 (MySQL_S-promote-timeout-120)
               demote interval=0s timeout=120 (MySQL_S-demote-timeout-120)
               start interval=0s timeout=120s on-fail=restart 
(MySQL_S-start-timeout-120s-on-fail-restart)
               stop interval=0s timeout=120s (MySQL_S-stop-timeout-120s)
               monitor interval=60s timeout=30s OCF_CHECK_LEVEL=1 
(MySQL_S-monitor-interval-60s-timeout-30s)

Stonith Devices:
Fencing Levels:

Location Constraints:
Ordering Constraints:
   start MySQL_M then start ClusterIP (Mandatory) 
(id:order-MySQL_M-ClusterIP-mandatory)
   start G_MySQL_M then start MySQL_S (Mandatory) 
(id:order-G_MySQL_M-MySQL_S-mandatory)
Colocation Constraints:
   G_MySQL_M with MySQL_S (-100) (id:colocation-G_MySQL_M-MySQL_S-INFINITY)

Cluster Properties:
  cluster-infrastructure: corosync
  dc-version: 1.1.10-29.el7-368c726
  last-lrm-refresh: 1487148812
  no-quorum-policy: ignore
  stonith-enabled: false

Pacemaker works as expected under most of situations, but there is one 
scenario that is really not understable to us. I will try to describe it:

a - Master resource (and Cluster IP address) are active on node 1 and 
Slave resource is active on node 2.
b - We force movement of Master resource to node 2.
c - Pacemaker stops all resources: Master, Slave and Cluster IP.
d - Master resource and Cluster IP are started on node 2 (this is OK), 
but Slave also tries to start (??). It fails (logically, because Master 
resource has been started on the same node), it logs an "unknown error" 
and its state is marked as "failed". This is a capture of 'pcs status' 
at that point:

OFFLINE: [ webmob1bdprep-ges ]
Online: [ webmob2bdprep-ges ]

Full list of resources:

Resource Group: G_MySQL_M
     MySQL_M (ocf::heartbeat:mysql_m): Started webmob2bdprep-ges
     ClusterIP (ocf::heartbeat:IPaddr2): Started webmob2bdprep-ges
MySQL_S (ocf::heartbeat:mysql_s): FAILED webmob2bdprep-ges

Failed actions:
     MySQL_M_monitor_60000 on webmob2bdprep-ges 'master' (8): call=62, 
status=complete, last-rc-change='Wed Feb 15 11:54:08 2017', queued=0ms, 
exec=0ms
     MySQL_S_start_0 on webmob2bdprep-ges 'unknown error' (1): call=78, 
status=complete, last-rc-change='Wed Feb 15 11:54:17 2017', queued=40ms, 
exec=0ms

PCSD Status:
webmob1bdprep-ges: Offline
webmob2bdprep-ges: Online

e - Pacemaker moves Slave resource to node 1 and starts it. Now we have 
both resources started again, Master on node 2 and Slave on node 1.
f - One minute later, Pacemaker restarts both resources (???).

So we are wondering:
- After the migration of the Master resource, why Pacemaker tries to 
start Slave resource on the same node where Master resource has been 
started previously? Why is trying to do it even with a colocation 
restriction as everyone can see on our configuration?
- Once Slave resource has been pushed out to the other node (and 
started), why both resources are restarted again one minute later?

Maybe there is something on our configuration that is the cause for it, 
but we can not figure out what could be. Also, we have been working with 
other Pacemaker cluster (also for MySQL) for the last three years, using 
a very similar configuration and we have not ever seen a behaviour as 
described before. Only difference between former cluster and this new 
one is that the older one is on a RHEL 6 platform and it runs cman and 
not corosync.

So we would thank any help or remark about why our cluster is doing 
this. Although it is not such a critical problem, it could be an issue 
on a production environment.

Best Regards,





-- 
/*Félix Díaz de Rada*
fdiaz at gfi.es <mailto:email at gfi.es>
//—/
*Gfi Norte* *| * *www.gfi.es* <http://www.gfi.es>
/Calle Licenciado Poza 55, Planta 2
48013 Bilbao Bizkaia
Teléfono: +34 94 424 18 25/

Síguenos: Blog <http://blog.gfi.es/> | Facebook 
<https://www.facebook.com/gfiinformatica> | LinkedIn 
<http://www.linkedin.com/company/gfi-inform-tica> | Twitter 
<https://twitter.com/GFI_Informatica> | YouTube 
<http://www.youtube.com/user/GFIInformatica>
Buscamos continuamente talento: http://www.gfi.es/carreras //
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20170216/15a9f675/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gfi.jpg
Type: image/jpeg
Size: 27844 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20170216/15a9f675/attachment-0002.jpg>


More information about the Users mailing list