<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hello Ken,</p>
<p>Thank you so much for your remarks. We have just applied them and
will perform some tests to see changes.<br>
</p>
<p>Best Regards,<br>
</p>
<br>
<div class="moz-cite-prefix">El 16-02-2017 a las 16:34, Ken Gaillot
escribió:<br>
</div>
<blockquote
cite="mid:b4eeb836-8728-7e5e-83c7-b1c494e21b1e@redhat.com"
type="cite">
<pre wrap="">On 02/16/2017 02:26 AM, Félix Díaz de Rada wrote:
</pre>
<blockquote type="cite">
<pre wrap="">
Hi all,
We are currently setting up a MySQL cluster (Master-Slave) over this
platform:
- Two nodes, on RHEL 7.0
- pacemaker-1.1.10-29.el7.x86_64
- corosync-2.3.3-2.el7.x86_64
- pcs-0.9.115-32.el7.x86_64
There is a IP address resource to be used as a "virtual IP".
This is configuration of cluster:
Cluster Name: webmobbdprep
Corosync Nodes:
webmob1bdprep-ges webmob2bdprep-ges
Pacemaker Nodes:
webmob1bdprep-ges webmob2bdprep-ges
Resources:
Group: G_MySQL_M
Meta Attrs: priority=100
Resource: MySQL_M (class=ocf provider=heartbeat type=mysql_m)
Attributes:
binary=/opt/mysql/mysql-5.7.17-linux-glibc2.5-x86_64/bin/mysqld_safe
config=/data/webmob_prep/webmob_prep.cnf datadir=/data/webmob_prep
log=/data/webmob_prep/webmob_prep.err
pid=/data/webmob_prep/webmob_rep.pid
socket=/data/webmob_prep/webmob_prep.sock user=mysql group=mysql
test_table=replica.pacemaker_test test_user=root
Meta Attrs: resource-stickiness=1000
Operations: promote interval=0s timeout=120 (MySQL_M-promote-timeout-120)
demote interval=0s timeout=120 (MySQL_M-demote-timeout-120)
start interval=0s timeout=120s on-fail=restart
(MySQL_M-start-timeout-120s-on-fail-restart)
stop interval=0s timeout=120s (MySQL_M-stop-timeout-120s)
monitor interval=60s timeout=30s OCF_CHECK_LEVEL=1
(MySQL_M-monitor-interval-60s-timeout-30s)
Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=172.18.64.44 nic=ens160:1 cidr_netmask=32
Meta Attrs: target-role=Started migration-threshold=3
failure-timeout=60s
Operations: start interval=0s timeout=20s (ClusterIP-start-timeout-20s)
stop interval=0s timeout=20s (ClusterIP-stop-timeout-20s)
monitor interval=60s (ClusterIP-monitor-interval-60s)
Resource: MySQL_S (class=ocf provider=heartbeat type=mysql_s)
Attributes:
binary=/opt/mysql/mysql-5.7.17-linux-glibc2.5-x86_64/bin/mysqld_safe
config=/data/webmob_prep/webmob_prep.cnf datadir=/data/webmob_prep
log=/data/webmob_prep/webmob_prep.err
pid=/data/webmob_prep/webmob_rep.pid
socket=/data/webmob_prep/webmob_prep.sock user=mysql group=mysql
test_table=replica.pacemaker_test test_user=root
Meta Attrs: resource-stickiness=0
Operations: promote interval=0s timeout=120 (MySQL_S-promote-timeout-120)
demote interval=0s timeout=120 (MySQL_S-demote-timeout-120)
start interval=0s timeout=120s on-fail=restart
(MySQL_S-start-timeout-120s-on-fail-restart)
stop interval=0s timeout=120s (MySQL_S-stop-timeout-120s)
monitor interval=60s timeout=30s OCF_CHECK_LEVEL=1
(MySQL_S-monitor-interval-60s-timeout-30s)
Stonith Devices:
Fencing Levels:
Location Constraints:
Ordering Constraints:
start MySQL_M then start ClusterIP (Mandatory)
(id:order-MySQL_M-ClusterIP-mandatory)
start G_MySQL_M then start MySQL_S (Mandatory)
(id:order-G_MySQL_M-MySQL_S-mandatory)
Colocation Constraints:
G_MySQL_M with MySQL_S (-100) (id:colocation-G_MySQL_M-MySQL_S-INFINITY)
Cluster Properties:
cluster-infrastructure: corosync
dc-version: 1.1.10-29.el7-368c726
last-lrm-refresh: 1487148812
no-quorum-policy: ignore
stonith-enabled: false
Pacemaker works as expected under most of situations, but there is one
scenario that is really not understable to us. I will try to describe it:
a - Master resource (and Cluster IP address) are active on node 1 and
Slave resource is active on node 2.
b - We force movement of Master resource to node 2.
c - Pacemaker stops all resources: Master, Slave and Cluster IP.
d - Master resource and Cluster IP are started on node 2 (this is OK),
but Slave also tries to start (??). It fails (logically, because Master
resource has been started on the same node), it logs an "unknown error"
and its state is marked as "failed". This is a capture of 'pcs status'
at that point:
OFFLINE: [ webmob1bdprep-ges ]
Online: [ webmob2bdprep-ges ]
Full list of resources:
Resource Group: G_MySQL_M
MySQL_M (ocf::heartbeat:mysql_m): Started webmob2bdprep-ges
ClusterIP (ocf::heartbeat:IPaddr2): Started webmob2bdprep-ges
MySQL_S (ocf::heartbeat:mysql_s): FAILED webmob2bdprep-ges
Failed actions:
MySQL_M_monitor_60000 on webmob2bdprep-ges 'master' (8): call=62,
status=complete, last-rc-change='Wed Feb 15 11:54:08 2017', queued=0ms,
exec=0ms
MySQL_S_start_0 on webmob2bdprep-ges 'unknown error' (1): call=78,
status=complete, last-rc-change='Wed Feb 15 11:54:17 2017', queued=40ms,
exec=0ms
PCSD Status:
webmob1bdprep-ges: Offline
webmob2bdprep-ges: Online
e - Pacemaker moves Slave resource to node 1 and starts it. Now we have
both resources started again, Master on node 2 and Slave on node 1.
f - One minute later, Pacemaker restarts both resources (???).
So we are wondering:
- After the migration of the Master resource, why Pacemaker tries to
start Slave resource on the same node where Master resource has been
started previously? Why is trying to do it even with a colocation
restriction as everyone can see on our configuration?
</pre>
</blockquote>
<pre wrap="">
In the configuration here, your colocation has two issues:
1. Colocating "G_MySQL_M with MySQL_S" means that MySQL_S must be placed
first, and only then can G_MySQL_M be placed in relation to it. Since
G_MySQL_M is more important, the colocation should be the other way
around ("MySQL_S with G_MySQL_M").
2. The score of -100 indicates a preference, not a requirement. If only
one node is available to host resources, the cluster is allowed to start
both resources on the same node. Using -INFINITY would make it a
requirement.
</pre>
<blockquote type="cite">
<pre wrap="">- Once Slave resource has been pushed out to the other node (and
started), why both resources are restarted again one minute later?
</pre>
</blockquote>
<pre wrap="">
I'm guessing that the other node was initially unavailable, and then
came back online. So first, Pacemaker started both resources on the only
available node. Then once the other node became available, it
re-distributed the resources.
Disregarding the above, however, the biggest thing that jumps out at me
is that this is not how master/slave relationships are usually modeled
in Pacemaker. Pacemaker has built-in support for master/slave clones, so
that a single resource agent can be used for both the master and the
slave, and Pacemaker takes care of promoting/demoting, keeping the
instances on different nodes, etc. See:
<a class="moz-txt-link-freetext" href="http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#s-resource-clone">http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#s-resource-clone</a>
</pre>
<blockquote type="cite">
<pre wrap="">Maybe there is something on our configuration that is the cause for it,
but we can not figure out what could be. Also, we have been working with
other Pacemaker cluster (also for MySQL) for the last three years, using
a very similar configuration and we have not ever seen a behaviour as
described before. Only difference between former cluster and this new
one is that the older one is on a RHEL 6 platform and it runs cman and
not corosync.
So we would thank any help or remark about why our cluster is doing
this. Although it is not such a critical problem, it could be an issue
on a production environment.
Best Regards,
--
/*Félix Díaz de Rada*
<a class="moz-txt-link-abbreviated" href="mailto:fdiaz@gfi.es">fdiaz@gfi.es</a> <a class="moz-txt-link-rfc2396E" href="mailto:email@gfi.es"><mailto:email@gfi.es></a>
//—/
*Gfi Norte* *| * *www.gfi.es* <a class="moz-txt-link-rfc2396E" href="http://www.gfi.es"><http://www.gfi.es></a>
/Calle Licenciado Poza 55, Planta 2
48013 Bilbao Bizkaia
Teléfono: +34 94 424 18 25/
Síguenos: Blog <a class="moz-txt-link-rfc2396E" href="http://blog.gfi.es/"><http://blog.gfi.es/></a> | Facebook
<a class="moz-txt-link-rfc2396E" href="https://www.facebook.com/gfiinformatica"><https://www.facebook.com/gfiinformatica></a> | LinkedIn
<a class="moz-txt-link-rfc2396E" href="http://www.linkedin.com/company/gfi-inform-tica"><http://www.linkedin.com/company/gfi-inform-tica></a> | Twitter
<a class="moz-txt-link-rfc2396E" href="https://twitter.com/GFI_Informatica"><https://twitter.com/GFI_Informatica></a> | YouTube
<a class="moz-txt-link-rfc2396E" href="http://www.youtube.com/user/GFIInformatica"><http://www.youtube.com/user/GFIInformatica></a>
Buscamos continuamente talento: <a class="moz-txt-link-freetext" href="http://www.gfi.es/carreras">http://www.gfi.es/carreras</a> //
</pre>
</blockquote>
<pre wrap="">
_______________________________________________
Users mailing list: <a class="moz-txt-link-abbreviated" href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a>
<a class="moz-txt-link-freetext" href="http://lists.clusterlabs.org/mailman/listinfo/users">http://lists.clusterlabs.org/mailman/listinfo/users</a>
Project Home: <a class="moz-txt-link-freetext" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a>
Getting started: <a class="moz-txt-link-freetext" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a>
Bugs: <a class="moz-txt-link-freetext" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a>
</pre>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<meta http-equiv="content-type" content="text/html;
charset=windows-1252">
<br>
<div class="moz-signature">-- <br>
<title></title>
<style type="text/css">
<!--
EM
{
font-size:10pt;
font-family:Calibri;
font-style:normal;
color:#303F48;
}
A
{
font-family:Calibri;
font-style:normal;
color:#303F48;
font-size:9pt;
}
H5
{
font-size:9pt;
font-family:Calibri;
font-style:normal;
color:#F79646;
}
H6
{
font-size:10pt;
font-family:Calibri;
font-style:normal;
color:green;
}
--> </style><em><b>Félix Díaz de Rada</b><br>
fdiaz<a style="color:#F79646" href="mailto:email@gfi.es">@gfi.es</a><br>
<em><em>—</em><br>
<b>Gfi Norte</b> <span style="color:#F79646"><b> | </b></span>
<a style="color:#F79646" href="http://www.gfi.es"><b>www.gfi.es</b></a><br>
<em>Calle Licenciado Poza 55, Planta 2<br>
48013 Bilbao Bizkaia<br>
Teléfono: +34 94 424 18 25</em><br>
<img alt="" src="cid:part3.C9F04699.0EAD1EED@gfi.es"
height="115" width="279"><br>
<span>Síguenos: </span><a style="color:orange"
href="http://blog.gfi.es/">Blog</a> <span
style="color:#F79646"> | </span><a style="color:#F79646"
href="https://www.facebook.com/gfiinformatica">Facebook</a>
<span style="color:#F79646"> | </span><a
style="color:#F79646"
href="http://www.linkedin.com/company/gfi-inform-tica"><span>LinkedIn</span></a>
<span style="color:#F79646"> | </span><a
style="color:#F79646"
href="https://twitter.com/GFI_Informatica">Twitter</a> <span
style="color:#F79646"> | </span><a style="color:#F79646"
href="http://www.youtube.com/user/GFIInformatica">YouTube</a>
<br>
<a>Buscamos continuamente talento: </a><a
style="color:#F79646" href="http://www.gfi.es/carreras">http://www.gfi.es/carreras</a>
</em></em></div>
</div>
</body>
</html>