<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-signature">Hi all,<br>
<br>
We are currently setting up a MySQL cluster (Master-Slave) over
this platform:<br>
- Two nodes, on RHEL 7.0<br>
- pacemaker-1.1.10-29.el7.x86_64<br>
- corosync-2.3.3-2.el7.x86_64<br>
- pcs-0.9.115-32.el7.x86_64<br>
There is a IP address resource to be used as a "virtual IP".<br>
<br>
This is configuration of cluster:<br>
<br>
<tt>Cluster Name: webmobbdprep</tt><tt><br>
</tt><tt>Corosync Nodes:</tt><tt><br>
</tt><tt> webmob1bdprep-ges webmob2bdprep-ges </tt><tt><br>
</tt><tt>Pacemaker Nodes:</tt><tt><br>
</tt><tt> webmob1bdprep-ges webmob2bdprep-ges </tt><tt><br>
</tt><tt><br>
</tt><tt>Resources: </tt><tt><br>
</tt><tt> Group: G_MySQL_M</tt><tt><br>
</tt><tt> Meta Attrs: priority=100 </tt><tt><br>
</tt><tt> Resource: MySQL_M (class=ocf provider=heartbeat
type=mysql_m)</tt><tt><br>
</tt><tt> Attributes:
binary=/opt/mysql/mysql-5.7.17-linux-glibc2.5-x86_64/bin/mysqld_safe
config=/data/webmob_prep/webmob_prep.cnf
datadir=/data/webmob_prep log=/data/webmob_prep/webmob_prep.err
pid=/data/webmob_prep/webmob_rep.pid
socket=/data/webmob_prep/webmob_prep.sock user=mysql group=mysql
test_table=replica.pacemaker_test test_user=root </tt><tt><br>
</tt><tt> Meta Attrs: resource-stickiness=1000 </tt><tt><br>
</tt><tt> Operations: promote interval=0s timeout=120
(MySQL_M-promote-timeout-120)</tt><tt><br>
</tt><tt> demote interval=0s timeout=120
(MySQL_M-demote-timeout-120)</tt><tt><br>
</tt><tt> start interval=0s timeout=120s
on-fail=restart (MySQL_M-start-timeout-120s-on-fail-restart)</tt><tt><br>
</tt><tt> stop interval=0s timeout=120s
(MySQL_M-stop-timeout-120s)</tt><tt><br>
</tt><tt> monitor interval=60s timeout=30s
OCF_CHECK_LEVEL=1 (MySQL_M-monitor-interval-60s-timeout-30s)</tt><tt><br>
</tt><tt> Resource: ClusterIP (class=ocf provider=heartbeat
type=IPaddr2)</tt><tt><br>
</tt><tt> Attributes: ip=172.18.64.44 nic=ens160:1
cidr_netmask=32 </tt><tt><br>
</tt><tt> Meta Attrs: target-role=Started migration-threshold=3
failure-timeout=60s </tt><tt><br>
</tt><tt> Operations: start interval=0s timeout=20s
(ClusterIP-start-timeout-20s)</tt><tt><br>
</tt><tt> stop interval=0s timeout=20s
(ClusterIP-stop-timeout-20s)</tt><tt><br>
</tt><tt> monitor interval=60s
(ClusterIP-monitor-interval-60s)</tt><tt><br>
</tt><tt> Resource: MySQL_S (class=ocf provider=heartbeat
type=mysql_s)</tt><tt><br>
</tt><tt> Attributes:
binary=/opt/mysql/mysql-5.7.17-linux-glibc2.5-x86_64/bin/mysqld_safe
config=/data/webmob_prep/webmob_prep.cnf
datadir=/data/webmob_prep log=/data/webmob_prep/webmob_prep.err
pid=/data/webmob_prep/webmob_rep.pid
socket=/data/webmob_prep/webmob_prep.sock user=mysql group=mysql
test_table=replica.pacemaker_test test_user=root </tt><tt><br>
</tt><tt> Meta Attrs: resource-stickiness=0 </tt><tt><br>
</tt><tt> Operations: promote interval=0s timeout=120
(MySQL_S-promote-timeout-120)</tt><tt><br>
</tt><tt> demote interval=0s timeout=120
(MySQL_S-demote-timeout-120)</tt><tt><br>
</tt><tt> start interval=0s timeout=120s
on-fail=restart (MySQL_S-start-timeout-120s-on-fail-restart)</tt><tt><br>
</tt><tt> stop interval=0s timeout=120s
(MySQL_S-stop-timeout-120s)</tt><tt><br>
</tt><tt> monitor interval=60s timeout=30s
OCF_CHECK_LEVEL=1 (MySQL_S-monitor-interval-60s-timeout-30s)</tt><tt><br>
</tt><tt><br>
</tt><tt>Stonith Devices: </tt><tt><br>
</tt><tt>Fencing Levels: </tt><tt><br>
</tt><tt><br>
</tt><tt>Location Constraints:</tt><tt><br>
</tt><tt>Ordering Constraints:</tt><tt><br>
</tt><tt> start MySQL_M then start ClusterIP (Mandatory)
(id:order-MySQL_M-ClusterIP-mandatory)</tt><tt><br>
</tt><tt> start G_MySQL_M then start MySQL_S (Mandatory)
(id:order-G_MySQL_M-MySQL_S-mandatory)</tt><tt><br>
</tt><tt>Colocation Constraints:</tt><tt><br>
</tt><tt> G_MySQL_M with MySQL_S (-100)
(id:colocation-G_MySQL_M-MySQL_S-INFINITY)</tt><tt><br>
</tt><tt><br>
</tt><tt>Cluster Properties:</tt><tt><br>
</tt><tt> cluster-infrastructure: corosync</tt><tt><br>
</tt><tt> dc-version: 1.1.10-29.el7-368c726</tt><tt><br>
</tt><tt> last-lrm-refresh: 1487148812</tt><tt><br>
</tt><tt> no-quorum-policy: ignore</tt><tt><br>
</tt><tt> stonith-enabled: false</tt><br>
<br>
Pacemaker works as expected under most of situations, but there is
one scenario that is really not understable to us. I will try to
describe it:<br>
<br>
a - Master resource (and Cluster IP address) are active on node 1
and Slave resource is active on node 2.<br>
b - We force movement of Master resource to node 2.<br>
c - Pacemaker stops all resources: Master, Slave and Cluster IP.<br>
d - Master resource and Cluster IP are started on node 2 (this is
OK), but Slave also tries to start (??). It fails (logically,
because Master resource has been started on the same node), it
logs an "unknown error" and its state is marked as "failed". This
is a capture of 'pcs status' at that point:<br>
<br>
<tt><tt>OFFLINE</tt>: [ webmob1bdprep-ges ] <br>
</tt><tt><tt>Online</tt>: [ webmob2bdprep-ges ] <br>
<br>
Full list of resources: <br>
<br>
Resource Group: G_MySQL_M <br>
MySQL_M (ocf::heartbeat:mysql_m): Started webmob2bdprep-ges
<br>
ClusterIP (ocf::heartbeat:IPaddr2): Started webmob2bdprep-ges
<br>
MySQL_S (ocf::heartbeat:mysql_s): FAILED webmob2bdprep-ges <br>
<br>
Failed actions: <br>
MySQL_M_monitor_60000 on webmob2bdprep-ges 'master' (8):
call=62, status=complete, last-rc-change='Wed Feb 15 11:54:08
2017', queued=0ms, exec=0ms <br>
MySQL_S_start_0 on webmob2bdprep-ges 'unknown error' (1):
call=78, status=complete, last-rc-change='Wed Feb 15 11:54:17
2017', queued=40ms, exec=0ms <br>
<br>
PCSD Status: <br>
webmob1bdprep-ges: </tt><tt><tt>Offline</tt><br>
webmob2bdprep-ges: </tt><tt><tt>Online </tt><br>
<br>
</tt>e - Pacemaker moves Slave resource to node 1 and starts it.
Now we have both resources started again, Master on node 2 and
Slave on node 1.<br>
f - One minute later, Pacemaker restarts both resources (???).<br>
<br>
So we are wondering:<br>
- After the migration of the Master resource, why Pacemaker tries
to start Slave resource on the same node where Master resource has
been started previously? Why is trying to do it even with a
colocation restriction as everyone can see on our configuration?<br>
- Once Slave resource has been pushed out to the other node (and
started), why both resources are restarted again one minute later?<br>
<br>
Maybe there is something on our configuration that is the cause
for it, but we can not figure out what could be. Also, we have
been working with other Pacemaker cluster (also for MySQL) for the
last three years, using a very similar configuration and we have
not ever seen a behaviour as described before. Only difference
between former cluster and this new one is that the older one is
on a RHEL 6 platform and it runs cman and not corosync.<br>
<br>
So we would thank any help or remark about why our cluster is
doing this. Although it is not such a critical problem, it could
be an issue on a production environment.<br>
<br>
Best Regards,<br>
<br>
<br>
<br>
<br>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<br>
<div class="moz-signature">-- <br>
<title></title>
<style type="text/css">
<!--
EM
{
font-size:10pt;
font-family:Calibri;
font-style:normal;
color:#303F48;
}
A
{
font-family:Calibri;
font-style:normal;
color:#303F48;
font-size:9pt;
}
H5
{
font-size:9pt;
font-family:Calibri;
font-style:normal;
color:#F79646;
}
H6
{
font-size:10pt;
font-family:Calibri;
font-style:normal;
color:green;
}
--> </style><em><b>Félix Díaz de Rada</b><br>
fdiaz<a style="color:#F79646" href="mailto:email@gfi.es">@gfi.es</a><br>
<em><em>—</em><br>
<b>Gfi Norte</b> <span style="color:#F79646"><b> | </b></span>
<a style="color:#F79646" href="http://www.gfi.es"><b>www.gfi.es</b></a><br>
<em>Calle Licenciado Poza 55, Planta 2<br>
48013 Bilbao Bizkaia<br>
Teléfono: +34 94 424 18 25</em><br>
<img alt="" src="cid:part3.B0AE7C0C.F3394DC0@gfi.es"
height="115" width="279"><br>
<span>Síguenos: </span><a style="color:orange"
href="http://blog.gfi.es/">Blog</a> <span
style="color:#F79646"> | </span><a style="color:#F79646"
href="https://www.facebook.com/gfiinformatica">Facebook</a>
<span style="color:#F79646"> | </span><a
style="color:#F79646"
href="http://www.linkedin.com/company/gfi-inform-tica"><span>LinkedIn</span></a>
<span style="color:#F79646"> | </span><a
style="color:#F79646"
href="https://twitter.com/GFI_Informatica">Twitter</a> <span
style="color:#F79646"> | </span><a style="color:#F79646"
href="http://www.youtube.com/user/GFIInformatica">YouTube</a>
<br>
<a>Buscamos continuamente talento: </a><a
style="color:#F79646" href="http://www.gfi.es/carreras">http://www.gfi.es/carreras</a>
</em></em></div>
</div>
</body>
</html>