[ClusterLabs] MySQL Cluster: Strange behaviour when forcing movement of resources
Félix Díaz de Rada
fdiaz at gfi.es
Fri Feb 17 10:03:00 EST 2017
Hello Ken,
Thank you so much for your remarks. We have just applied them and will
perform some tests to see changes.
Best Regards,
El 16-02-2017 a las 16:34, Ken Gaillot escribió:
> On 02/16/2017 02:26 AM, Félix Díaz de Rada wrote:
>> Hi all,
>>
>> We are currently setting up a MySQL cluster (Master-Slave) over this
>> platform:
>> - Two nodes, on RHEL 7.0
>> - pacemaker-1.1.10-29.el7.x86_64
>> - corosync-2.3.3-2.el7.x86_64
>> - pcs-0.9.115-32.el7.x86_64
>> There is a IP address resource to be used as a "virtual IP".
>>
>> This is configuration of cluster:
>>
>> Cluster Name: webmobbdprep
>> Corosync Nodes:
>> webmob1bdprep-ges webmob2bdprep-ges
>> Pacemaker Nodes:
>> webmob1bdprep-ges webmob2bdprep-ges
>>
>> Resources:
>> Group: G_MySQL_M
>> Meta Attrs: priority=100
>> Resource: MySQL_M (class=ocf provider=heartbeat type=mysql_m)
>> Attributes:
>> binary=/opt/mysql/mysql-5.7.17-linux-glibc2.5-x86_64/bin/mysqld_safe
>> config=/data/webmob_prep/webmob_prep.cnf datadir=/data/webmob_prep
>> log=/data/webmob_prep/webmob_prep.err
>> pid=/data/webmob_prep/webmob_rep.pid
>> socket=/data/webmob_prep/webmob_prep.sock user=mysql group=mysql
>> test_table=replica.pacemaker_test test_user=root
>> Meta Attrs: resource-stickiness=1000
>> Operations: promote interval=0s timeout=120 (MySQL_M-promote-timeout-120)
>> demote interval=0s timeout=120 (MySQL_M-demote-timeout-120)
>> start interval=0s timeout=120s on-fail=restart
>> (MySQL_M-start-timeout-120s-on-fail-restart)
>> stop interval=0s timeout=120s (MySQL_M-stop-timeout-120s)
>> monitor interval=60s timeout=30s OCF_CHECK_LEVEL=1
>> (MySQL_M-monitor-interval-60s-timeout-30s)
>> Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
>> Attributes: ip=172.18.64.44 nic=ens160:1 cidr_netmask=32
>> Meta Attrs: target-role=Started migration-threshold=3
>> failure-timeout=60s
>> Operations: start interval=0s timeout=20s (ClusterIP-start-timeout-20s)
>> stop interval=0s timeout=20s (ClusterIP-stop-timeout-20s)
>> monitor interval=60s (ClusterIP-monitor-interval-60s)
>> Resource: MySQL_S (class=ocf provider=heartbeat type=mysql_s)
>> Attributes:
>> binary=/opt/mysql/mysql-5.7.17-linux-glibc2.5-x86_64/bin/mysqld_safe
>> config=/data/webmob_prep/webmob_prep.cnf datadir=/data/webmob_prep
>> log=/data/webmob_prep/webmob_prep.err
>> pid=/data/webmob_prep/webmob_rep.pid
>> socket=/data/webmob_prep/webmob_prep.sock user=mysql group=mysql
>> test_table=replica.pacemaker_test test_user=root
>> Meta Attrs: resource-stickiness=0
>> Operations: promote interval=0s timeout=120 (MySQL_S-promote-timeout-120)
>> demote interval=0s timeout=120 (MySQL_S-demote-timeout-120)
>> start interval=0s timeout=120s on-fail=restart
>> (MySQL_S-start-timeout-120s-on-fail-restart)
>> stop interval=0s timeout=120s (MySQL_S-stop-timeout-120s)
>> monitor interval=60s timeout=30s OCF_CHECK_LEVEL=1
>> (MySQL_S-monitor-interval-60s-timeout-30s)
>>
>> Stonith Devices:
>> Fencing Levels:
>>
>> Location Constraints:
>> Ordering Constraints:
>> start MySQL_M then start ClusterIP (Mandatory)
>> (id:order-MySQL_M-ClusterIP-mandatory)
>> start G_MySQL_M then start MySQL_S (Mandatory)
>> (id:order-G_MySQL_M-MySQL_S-mandatory)
>> Colocation Constraints:
>> G_MySQL_M with MySQL_S (-100) (id:colocation-G_MySQL_M-MySQL_S-INFINITY)
>>
>> Cluster Properties:
>> cluster-infrastructure: corosync
>> dc-version: 1.1.10-29.el7-368c726
>> last-lrm-refresh: 1487148812
>> no-quorum-policy: ignore
>> stonith-enabled: false
>>
>> Pacemaker works as expected under most of situations, but there is one
>> scenario that is really not understable to us. I will try to describe it:
>>
>> a - Master resource (and Cluster IP address) are active on node 1 and
>> Slave resource is active on node 2.
>> b - We force movement of Master resource to node 2.
>> c - Pacemaker stops all resources: Master, Slave and Cluster IP.
>> d - Master resource and Cluster IP are started on node 2 (this is OK),
>> but Slave also tries to start (??). It fails (logically, because Master
>> resource has been started on the same node), it logs an "unknown error"
>> and its state is marked as "failed". This is a capture of 'pcs status'
>> at that point:
>>
>> OFFLINE: [ webmob1bdprep-ges ]
>> Online: [ webmob2bdprep-ges ]
>>
>> Full list of resources:
>>
>> Resource Group: G_MySQL_M
>> MySQL_M (ocf::heartbeat:mysql_m): Started webmob2bdprep-ges
>> ClusterIP (ocf::heartbeat:IPaddr2): Started webmob2bdprep-ges
>> MySQL_S (ocf::heartbeat:mysql_s): FAILED webmob2bdprep-ges
>>
>> Failed actions:
>> MySQL_M_monitor_60000 on webmob2bdprep-ges 'master' (8): call=62,
>> status=complete, last-rc-change='Wed Feb 15 11:54:08 2017', queued=0ms,
>> exec=0ms
>> MySQL_S_start_0 on webmob2bdprep-ges 'unknown error' (1): call=78,
>> status=complete, last-rc-change='Wed Feb 15 11:54:17 2017', queued=40ms,
>> exec=0ms
>>
>> PCSD Status:
>> webmob1bdprep-ges: Offline
>> webmob2bdprep-ges: Online
>>
>> e - Pacemaker moves Slave resource to node 1 and starts it. Now we have
>> both resources started again, Master on node 2 and Slave on node 1.
>> f - One minute later, Pacemaker restarts both resources (???).
>>
>> So we are wondering:
>> - After the migration of the Master resource, why Pacemaker tries to
>> start Slave resource on the same node where Master resource has been
>> started previously? Why is trying to do it even with a colocation
>> restriction as everyone can see on our configuration?
> In the configuration here, your colocation has two issues:
>
> 1. Colocating "G_MySQL_M with MySQL_S" means that MySQL_S must be placed
> first, and only then can G_MySQL_M be placed in relation to it. Since
> G_MySQL_M is more important, the colocation should be the other way
> around ("MySQL_S with G_MySQL_M").
>
> 2. The score of -100 indicates a preference, not a requirement. If only
> one node is available to host resources, the cluster is allowed to start
> both resources on the same node. Using -INFINITY would make it a
> requirement.
>
>> - Once Slave resource has been pushed out to the other node (and
>> started), why both resources are restarted again one minute later?
> I'm guessing that the other node was initially unavailable, and then
> came back online. So first, Pacemaker started both resources on the only
> available node. Then once the other node became available, it
> re-distributed the resources.
>
>
> Disregarding the above, however, the biggest thing that jumps out at me
> is that this is not how master/slave relationships are usually modeled
> in Pacemaker. Pacemaker has built-in support for master/slave clones, so
> that a single resource agent can be used for both the master and the
> slave, and Pacemaker takes care of promoting/demoting, keeping the
> instances on different nodes, etc. See:
>
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#s-resource-clone
>
>
>> Maybe there is something on our configuration that is the cause for it,
>> but we can not figure out what could be. Also, we have been working with
>> other Pacemaker cluster (also for MySQL) for the last three years, using
>> a very similar configuration and we have not ever seen a behaviour as
>> described before. Only difference between former cluster and this new
>> one is that the older one is on a RHEL 6 platform and it runs cman and
>> not corosync.
>>
>> So we would thank any help or remark about why our cluster is doing
>> this. Although it is not such a critical problem, it could be an issue
>> on a production environment.
>>
>> Best Regards,
>>
>>
>>
>>
>>
>> --
>> /*Félix Díaz de Rada*
>> fdiaz at gfi.es <mailto:email at gfi.es>
>> //—/
>> *Gfi Norte* *| * *www.gfi.es* <http://www.gfi.es>
>> /Calle Licenciado Poza 55, Planta 2
>> 48013 Bilbao Bizkaia
>> Teléfono: +34 94 424 18 25/
>>
>> Síguenos: Blog <http://blog.gfi.es/> | Facebook
>> <https://www.facebook.com/gfiinformatica> | LinkedIn
>> <http://www.linkedin.com/company/gfi-inform-tica> | Twitter
>> <https://twitter.com/GFI_Informatica> | YouTube
>> <http://www.youtube.com/user/GFIInformatica>
>> Buscamos continuamente talento: http://www.gfi.es/carreras //
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
--
--
/*Félix Díaz de Rada*
fdiaz at gfi.es <mailto:email at gfi.es>
//—/
*Gfi Norte* *| * *www.gfi.es* <http://www.gfi.es>
/Calle Licenciado Poza 55, Planta 2
48013 Bilbao Bizkaia
Teléfono: +34 94 424 18 25/
Síguenos: Blog <http://blog.gfi.es/> | Facebook
<https://www.facebook.com/gfiinformatica> | LinkedIn
<http://www.linkedin.com/company/gfi-inform-tica> | Twitter
<https://twitter.com/GFI_Informatica> | YouTube
<http://www.youtube.com/user/GFIInformatica>
Buscamos continuamente talento: http://www.gfi.es/carreras //
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20170217/f747133e/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gfi.jpg
Type: image/jpeg
Size: 27844 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20170217/f747133e/attachment-0003.jpg>
More information about the Users
mailing list