<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">Le sam. 6 oct. 2018 à 06:13, Andrei Borzenkov <<a href="mailto:arvidjaar@gmail.com" target="_blank">arvidjaar@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">05.10.2018 15:00, Simon Bomm пишет:<br>
> Hi all,<br>
> <br>
> Using pacemaker 1.1.18-11 and mysql resource agent (<br>
> <a href="https://github.com/ClusterLabs/resource-agents/blob/RHEL6/heartbeat/mysql" rel="noreferrer" target="_blank">https://github.com/ClusterLabs/resource-agents/blob/RHEL6/heartbeat/mysql</a>),<br>
> I run into an unwanted behaviour. My point of view of course, maybe it's<br>
> expected to be as it is that's why I ask.<br>
> <br>
> # My test case is the following :<br>
> <br>
> Everything is OK on my cluster, crm_mon output is as below (no failed<br>
> actions)<br>
> <br>
> Master/Slave Set: ms_mysql-master [ms_mysql]<br>
> Masters: [ db-master ]<br>
> Slaves: [ db-slave ]<br>
> <br>
> 1. I insert in a table on master, no issue data is replicated.<br>
> 2. I shut down net int on the master (vm),<br>
<br></blockquote><div><br></div><div>First, thanks for taking time to answer me</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
What exactly does it mean? How do you shut down net?<br>
<br></blockquote><div><br></div><div>Disconnect the network card from VMWare vSphere Console</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> pacemaker correctly start on the<br>
> other node. Master is seen as offline, and db-slave is now master<br>
> <br>
> Master/Slave Set: ms_mysql-master [ms_mysql]<br>
> Masters: [ db-slave ]<br>
> <br>
> 3. I bring back my net int up, pacemaker see the node online and set the<br>
> old-master as a the new slave :<br>
> <br>
> Master/Slave Set: ms_mysql-master [ms_mysql]<br>
> Masters: [ db-slave ]<br>
> Slaves: [ db-master ]<br>
> <br>
> 4. From this point, my external monitoring bash script shows that SQL and<br>
> IO thread are not running, but I can't see any error in the pcs<br>
> status/crm_mon outputs.<br>
<br>
Pacemaker just shows what resource agents claim. If resource agent<br>
claims resource is started, there is nothing pacemaker can do. You need<br>
to debug what resource agent does.<br>
<br></blockquote><div><br></div><div>I've debugged it quite a lot, and that's what drove me to isolate error below : </div><div><br></div><div>> mysql -h localhost -u user-repl -pmysqlreplpw -e "START SLAVE"<br>> ERROR 1200 (HY000) at line 1: Misconfigured slave: MASTER_HOST was not set;<br>> Fix in config file or with CHANGE MASTER TO <br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Consequence is that I continue inserting on my new<br>
> promoted master but the data is never consumed by my former master computer.<br>
> <br>
> # Questions :<br>
> <br>
> - Is this some kind of safety behaviour to avoid data corruption when a<br>
> node is back online ?<br>
> - When I want to manually start it like ocf does it returns this error :<br>
> <br>
> mysql -h localhost -u user-repl -pmysqlreplpw -e "START SLAVE"<br>
> ERROR 1200 (HY000) at line 1: Misconfigured slave: MASTER_HOST was not set;<br>
> Fix in config file or with CHANGE MASTER TO<br>
> <br>
> - I would expect the cluster to stop the slave and show a failed action, am<br>
> I wrong here ?<br>
> <br>
<br>
I am not familiar with specific application and its structure. From<br>
quick browsing monitor action does mostly check for running process. Is<br>
mySQL process running?<br></blockquote><div><br></div><div>Yes it is, as you mentionned previously the config wants pacemaker to start mysql resource so no problems.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> # Other details (not sure it matters a lot)<br>
> <br>
> No stonith enabled, no fencing or auto-failback.<br>
<br>
How are you going to resolve split-brain without stonith? "Stopping net"<br>
sounds exactly like split brain, in which case further investigation is<br>
rather pointless.<br>
<br></blockquote><div><br></div><div>You make the point, as I'm not very familiar with stonithd, I first disable this to avoid unwanted behaviour but I'll definitely follow your advise and dig around.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Anyway, to give some non-hypothetical answer full configuration and logs<br>
from both systems are needed.<br>
<br></blockquote><div><br></div><div>Sure, please find the full configuration </div><div><br></div><div><div>Cluster Name: app_cluster</div><div>Corosync Nodes:</div><div> app-central-master app-central-slave app-db-master app-db-slave app-quorum</div><div>Pacemaker Nodes:</div><div> app-central-master app-central-slave app-db-master app-db-slave app-quorum</div><div><br></div><div>Resources:</div><div> Master: ms_mysql-master</div><div> Meta Attrs: master-node-max=1 clone_max=2 globally-unique=false clone-node-max=1 notify=true</div><div> Resource: ms_mysql (class=ocf provider=heartbeat type=mysql-app)</div><div> Attributes: binary=/usr/bin/mysqld_safe config=/etc/my.cnf.d/server.cnf datadir=/var/lib/mysql evict_outdated_slaves=false max_slave_lag=15 pid=/var/lib/mysql/mysql.pid replication_passwd=mysqlreplpw replication_user=app-repl socket=/var/lib/mysql/mysql.sock test_passwd=mysqlrootpw test_user=root</div><div> Operations: demote interval=0s timeout=120 (ms_mysql-demote-interval-0s)</div><div> monitor interval=20 timeout=30 (ms_mysql-monitor-interval-20)</div><div> monitor interval=10 role=Master timeout=30 (ms_mysql-monitor-interval-10)</div><div> monitor interval=30 role=Slave timeout=30 (ms_mysql-monitor-interval-30)</div><div> notify interval=0s timeout=90 (ms_mysql-notify-interval-0s)</div><div> promote interval=0s timeout=120 (ms_mysql-promote-interval-0s)</div><div> start interval=0s timeout=120 (ms_mysql-start-interval-0s)</div><div> stop interval=0s timeout=120 (ms_mysql-stop-interval-0s)</div><div> Resource: vip_mysql (class=ocf provider=heartbeat type=IPaddr2-app)</div><div> Attributes: broadcast=10.30.255.255 cidr_netmask=16 flush_routes=true ip=10.30.3.229 nic=ens160</div><div> Operations: monitor interval=10s timeout=20s (vip_mysql-monitor-interval-10s)</div><div> start interval=0s timeout=20s (vip_mysql-start-interval-0s)</div><div> stop interval=0s timeout=20s (vip_mysql-stop-interval-0s)</div><div> Group: app</div><div> Resource: misc_app (class=ocf provider=heartbeat type=misc-app)</div><div> Attributes: crondir=/etc/app-failover/resources/cron/,/etc/cron.d/</div><div> Meta Attrs: target-role=started</div><div> Operations: monitor interval=5s timeout=20s (misc_app-monitor-interval-5s)</div><div> start interval=0s timeout=20s (misc_app-start-interval-0s)</div><div> stop interval=0s timeout=20s (misc_app-stop-interval-0s)</div><div> Resource: cbd_central_broker (class=ocf provider=heartbeat type=cbd-central-broker)</div><div> Meta Attrs: target-role=started</div><div> Operations: monitor interval=5s timeout=20s (cbd_central_broker-monitor-interval-5s)</div><div> start interval=0s timeout=90s (cbd_central_broker-start-interval-0s)</div><div> stop interval=0s timeout=90s (cbd_central_broker-stop-interval-0s)</div><div> Resource: centcore (class=ocf provider=heartbeat type=centcore)</div><div> Meta Attrs: target-role=started</div><div> Operations: monitor interval=5s timeout=20s (centcore-monitor-interval-5s)</div><div> start interval=0s timeout=90s (centcore-start-interval-0s)</div><div> stop interval=0s timeout=90s (centcore-stop-interval-0s)</div><div> Resource: apptrapd (class=ocf provider=heartbeat type=apptrapd)</div><div> Meta Attrs: target-role=started</div><div> Operations: monitor interval=5s timeout=20s (apptrapd-monitor-interval-5s)</div><div> start interval=0s timeout=90s (apptrapd-start-interval-0s)</div><div> stop interval=0s timeout=90s (apptrapd-stop-interval-0s)</div><div> Resource: app_central_sync (class=ocf provider=heartbeat type=app-central-sync)</div><div> Meta Attrs: target-role=started</div><div> Operations: monitor interval=5s timeout=20s (app_central_sync-monitor-interval-5s)</div><div> start interval=0s timeout=90s (app_central_sync-start-interval-0s)</div><div> stop interval=0s timeout=90s (app_central_sync-stop-interval-0s)</div><div> Resource: snmptrapd (class=ocf provider=heartbeat type=snmptrapd)</div><div> Meta Attrs: target-role=started</div><div> Operations: monitor interval=5s timeout=20s (snmptrapd-monitor-interval-5s)</div><div> start interval=0s timeout=90s (snmptrapd-start-interval-0s)</div><div> stop interval=0s timeout=90s (snmptrapd-stop-interval-0s)</div><div> Resource: http (class=ocf provider=heartbeat type=apacheapp)</div><div> Meta Attrs: target-role=started</div><div> Operations: monitor interval=5s timeout=20s (http-monitor-interval-5s)</div><div> start interval=0s timeout=40s (http-start-interval-0s)</div><div> stop interval=0s timeout=60s (http-stop-interval-0s)</div><div> Resource: vip_app (class=ocf provider=heartbeat type=IPaddr2-app)</div><div> Attributes: broadcast=10.30.255.255 cidr_netmask=16 flush_routes=true ip=10.30.3.230 nic=ens160</div><div> Meta Attrs: target-role=started</div><div> Operations: monitor interval=10s timeout=20s (vip_app-monitor-interval-10s)</div><div> start interval=0s timeout=20s (vip_app-start-interval-0s)</div><div> stop interval=0s timeout=20s (vip_app-stop-interval-0s)</div><div> Resource: centengine (class=ocf provider=heartbeat type=centengine)</div><div> Meta Attrs: multiple-active=stop_start target-role=started</div><div> Operations: monitor interval=5s timeout=20s (centengine-monitor-interval-5s)</div><div> start interval=0s timeout=90s (centengine-start-interval-0s)</div><div> stop interval=0s timeout=90s (centengine-stop-interval-0s)</div><div><br></div><div>Stonith Devices:</div><div>Fencing Levels:</div><div><br></div><div>Location Constraints:</div><div> Resource: app</div><div> Disabled on: app-db-master (score:-INFINITY) (id:location-app-app-db-master--INFINITY)</div><div> Disabled on: app-db-slave (score:-INFINITY) (id:location-app-app-db-slave--INFINITY)</div><div> Resource: ms_mysql</div><div> Disabled on: app-central-master (score:-INFINITY) (id:location-ms_mysql-app-central-master--INFINITY)</div><div> Disabled on: app-central-slave (score:-INFINITY) (id:location-ms_mysql-app-central-slave--INFINITY)</div><div> Resource: vip_mysql</div><div> Disabled on: app-central-master (score:-INFINITY) (id:location-vip_mysql-app-central-master--INFINITY)</div><div> Disabled on: app-central-slave (score:-INFINITY) (id:location-vip_mysql-app-central-slave--INFINITY)</div><div>Ordering Constraints:</div><div>Colocation Constraints:</div><div> vip_mysql with ms_mysql-master (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master)</div><div> ms_mysql-master with vip_mysql (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started)</div><div>Ticket Constraints:</div><div><br></div><div>Alerts:</div><div> No alerts defined</div><div><br></div><div>Resources Defaults:</div><div> resource-stickiness: INFINITY</div><div>Operations Defaults:</div><div> No defaults set</div><div><br></div><div>Cluster Properties:</div><div> cluster-infrastructure: corosync</div><div> cluster-name: app_cluster</div><div> dc-version: 1.1.18-11.el7_5.3-2b07d5c5a9</div><div> have-watchdog: false</div><div> last-lrm-refresh: 1538740285</div><div> ms_mysql_REPL_INFO: app-db-master|mysql-bin.000012|327</div><div> stonith-enabled: false</div><div> symmetric-cluster: true</div><div>Node Attributes:</div><div> app-quorum: standby=on</div><div><br></div><div>Quorum:</div><div> Options:</div><div> Device:</div><div> votes: 1</div><div> Model: net</div><div> algorithm: ffsplit</div><div> host: app-quorum</div></div><div><br></div><div><br></div><div>Logs are below </div><div><br></div><div>SLAVE when I disconnect interface (node is isolated), and associated crm_mon, lgtm and can get the behaviour : </div><div><br></div><div><div>Oct 10 09:20:07 app-db-slave corosync[1055]: [TOTEM ] A processor failed, forming new configuration.</div><div>Oct 10 09:20:11 app-db-slave corosync[1055]: [TOTEM ] A new membership (<a href="http://10.30.3.245:196" target="_blank">10.30.3.245:196</a>) was formed. Members left: 3</div><div>Oct 10 09:20:11 app-db-slave corosync[1055]: [TOTEM ] Failed to receive the leave message. failed: 3</div><div>Oct 10 09:20:11 app-db-slave corosync[1055]: [QUORUM] Members[4]: 1 2 4 5</div><div>Oct 10 09:20:11 app-db-slave corosync[1055]: [MAIN ] Completed service synchronization, ready to provide service.</div><div>Oct 10 09:20:11 app-db-slave cib[1168]: notice: Node app-db-master state is now lost</div><div>Oct 10 09:20:11 app-db-slave attrd[1172]: notice: Node app-db-master state is now lost</div><div>Oct 10 09:20:11 app-db-slave attrd[1172]: notice: Removing all app-db-master attributes for peer loss</div><div>Oct 10 09:20:11 app-db-slave stonith-ng[1170]: notice: Node app-db-master state is now lost</div><div>Oct 10 09:20:11 app-db-slave pacemakerd[1084]: notice: Node app-db-master state is now lost</div><div>Oct 10 09:20:11 app-db-slave crmd[1175]: notice: Node app-db-master state is now lost</div><div>Oct 10 09:20:11 app-db-slave cib[1168]: notice: Purged 1 peer with id=3 and/or uname=app-db-master from the membership cache</div><div>Oct 10 09:20:11 app-db-slave stonith-ng[1170]: notice: Purged 1 peer with id=3 and/or uname=app-db-master from the membership cache</div><div>Oct 10 09:20:11 app-db-slave attrd[1172]: notice: Purged 1 peer with id=3 and/or uname=app-db-master from the membership cache</div><div>Oct 10 09:20:11 app-db-slave crmd[1175]: notice: Result of notify operation for ms_mysql on app-db-slave: 0 (ok)</div><div>Oct 10 09:20:12 app-db-slave mysql-app(ms_mysql)[21165]: INFO: app-db-slave promote is starting</div><div>Oct 10 09:20:12 app-db-slave IPaddr2-app(vip_mysql)[21134]: INFO: Adding inet address <a href="http://10.30.3.229/16" target="_blank">10.30.3.229/16</a> with broadcast address 10.30.255.255 to device ens160</div><div>Oct 10 09:20:12 app-db-slave IPaddr2-app(vip_mysql)[21134]: INFO: Bringing device ens160 up</div><div>Oct 10 09:20:12 app-db-slave IPaddr2-app(vip_mysql)[21134]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -c 5 -I ens160 -s 10.30.3.229 10.30.255.255</div><div>Oct 10 09:20:12 app-db-slave crmd[1175]: notice: Result of start operation for vip_mysql on app-db-slave: 0 (ok)</div><div>Oct 10 09:20:12 app-db-slave lrmd[1171]: notice: ms_mysql_promote_0:21165:stderr [ Error performing operation: No such device or address ]</div><div>Oct 10 09:20:12 app-db-slave crmd[1175]: notice: Result of promote operation for ms_mysql on app-db-slave: 0 (ok)</div><div>Oct 10 09:20:12 app-db-slave mysql-app(ms_mysql)[21285]: INFO: app-db-slave This will be the new master, ignoring post-promote notification.</div><div>Oct 10 09:20:12 app-db-slave crmd[1175]: notice: Result of notify operation for ms_mysql on app-db-slave: 0 (ok)</div></div><div><br></div><div><br></div><div><div>Node app-quorum: standby</div><div>Online: [ app-central-master app-central-slave app-db-slave ]</div><div>OFFLINE: [ app-db-master ]</div><div><br></div><div>Active resources:</div><div><br></div><div> Master/Slave Set: ms_mysql-master [ms_mysql]</div><div> Masters: [ app-db-slave ]</div><div>vip_mysql (ocf::heartbeat:IPaddr2-app): Started app-db-slave</div></div><div><br></div><div>And logs from the master during its isolation : </div><div><br></div><div><div>Oct 10 09:23:10 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:11 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:13 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:14 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:16 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:17 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:19 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:20 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:22 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:23 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:25 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:26 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:28 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:29 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:31 app-db-master kernel: vmxnet3 0000:03:00.0 ens160: NIC Link is Up 10000 Mbps</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.1436] device (ens160): carrier: link connected</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.1444] device (ens160): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.1456] policy: auto-activating connection 'ens160'</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.1470] device (ens160): Activation: starting connection 'ens160' (9fe36e64-13ca-40cb-a174-5b4e16b826f4)</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.1473] device (ens160): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.1474] manager: NetworkManager state is now CONNECTING</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.1479] device (ens160): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.1485] device (ens160): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.2214] device (ens160): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.2235] device (ens160): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.2238] device (ens160): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.2240] manager: NetworkManager state is now CONNECTED_LOCAL</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.2554] manager: NetworkManager state is now CONNECTED_SITE</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.2555] policy: set 'ens160' (ens160) as default for IPv4 routing and DNS</div><div>Oct 10 09:23:31 app-db-master systemd: Starting Network Manager Script Dispatcher Service...</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.2556] device (ens160): Activation: successful, device activated.</div><div>Oct 10 09:23:31 app-db-master NetworkManager[692]: <info> [1539156211.2564] manager: NetworkManager state is now CONNECTED_GLOBAL</div><div>Oct 10 09:23:31 app-db-master dbus[686]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'</div><div>Oct 10 09:23:31 app-db-master dbus[686]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'</div><div>Oct 10 09:23:31 app-db-master systemd: Started Network Manager Script Dispatcher Service.</div><div>Oct 10 09:23:31 app-db-master nm-dispatcher: req:1 'up' [ens160]: new request (3 scripts)</div><div>Oct 10 09:23:31 app-db-master nm-dispatcher: req:1 'up' [ens160]: start running ordered scripts...</div><div>Oct 10 09:23:31 app-db-master nm-dispatcher: req:2 'connectivity-change': new request (3 scripts)</div><div>Oct 10 09:23:31 app-db-master nm-dispatcher: req:2 'connectivity-change': start running ordered scripts...</div><div>Oct 10 09:23:31 app-db-master corosync[1029]: [MAIN ] Totem is unable to form a cluster because of an operating system or network fault (reason: totem is continuously in gather state). The most common cause of this message is that the local firewall is configured improperly.</div><div>Oct 10 09:23:31 app-db-master corosync[1029]: [TOTEM ] The network interface [10.30.3.247] is now up.</div><div>Oct 10 09:23:31 app-db-master corosync[1029]: [TOTEM ] adding new UDPU member {10.30.3.245}</div><div>Oct 10 09:23:31 app-db-master corosync[1029]: [TOTEM ] adding new UDPU member {10.30.3.246}</div><div>Oct 10 09:23:31 app-db-master corosync[1029]: [TOTEM ] adding new UDPU member {10.30.3.247}</div><div>Oct 10 09:23:31 app-db-master corosync[1029]: [TOTEM ] adding new UDPU member {10.30.3.248}</div><div>Oct 10 09:23:31 app-db-master corosync[1029]: [TOTEM ] adding new UDPU member {10.30.3.249}</div></div><div><br></div><div>As you can see, node is back online and can communicate again with other nodes, so pacemaker start mysql as expected and promote it as slave : </div><div><br></div><div><div><div><div>Node aoo-quorum: standby</div><div>Online: [ app-central-master app-central-slave app-db-master app-db-slave ]</div><div><br></div><div>Active resources:</div><div><br></div><div> Master/Slave Set: ms_mysql-master [ms_mysql]</div><div> Masters: [ app-db-slave ]</div><div> Slaves: [ app-db-master ]</div></div><br class="m_8699240398962666832m_-919811478140281985gmail-Apple-interchange-newline"></div></div><div>Resource-agents oriented logs are below : </div><div><br></div><div>Master : </div><div><div>Oct 10 09:24:01 app-db-master crmd[5177]: notice: Result of demote operation for ms_mysql on app-db-master: 0 (ok)</div><div>Oct 10 09:24:02 app-db-master mysql-app(ms_mysql)[5592]: INFO: app-db-master Ignoring post-demote notification for my own demotion.</div><div>Oct 10 09:24:02 app-db-master crmd[5177]: notice: Result of notify operation for ms_mysql on app-db-master: 0 (ok)</div></div><div><br></div><div>Slave: </div><div><br></div><div><div>Oct 10 09:24:01 app-db-slave crmd[1175]: notice: Result of notify operation for ms_mysql on app-db-slave: 0 (ok)</div><div>Oct 10 09:24:02 app-db-slave mysql-app(ms_mysql)[22969]: INFO: app-db-slave Ignoring pre-demote notification execpt for my own demotion.</div><div>Oct 10 09:24:02 app-db-slave crmd[1175]: notice: Result of notify operation for ms_mysql on app-db-slave: 0 (ok)</div><div>Oct 10 09:24:03 app-db-slave mysql-app(ms_mysql)[22999]: INFO: app-db-slave post-demote notification for app-db-master.</div><div>Oct 10 09:24:03 app-db-slave mysql-app(ms_mysql)[22999]: WARNING: Attempted to unset the replication master on an instance that is not configured as a replication slave</div><div>Oct 10 09:24:03 app-db-slave crmd[1175]: notice: Result of notify operation for ms_mysql on app-db-slave: 0 (ok)</div></div><div><br></div><div>So I expect to have a running replication at this point, but when I perform SHOW SLAVE STATUS on my <i>new</i> slave, I get an empty response : </div><div><br></div><div><div>MariaDB [(none)]> SHOW SLAVE STATUS \G</div><div>Empty set (0.00 sec)</div><div><br></div><div>MariaDB [(none)]> Ctrl-C -- exit!</div><div>Aborted</div><div>(reverse-i-search)`ab': systemctl en^Cle corosync</div><div>[root@app-db-master ~]# bash /etc/app-failover/mysql-exploit/mysql-check-status.sh</div><div>Connection Status 'app-db-master' [OK]</div><div>Connection Status 'app-db-slave' [OK]</div><div>Slave Thread Status [KO]</div><div>Error reports:</div><div> No slave (maybe because we cannot check a server).</div><div>Position Status [SKIP]</div><div>Error reports:</div><div> Skip because we can't identify a unique slave.</div></div><div><br></div><div>From what I understand the is_slave function from <a href="https://github.com/ClusterLabs/resource-agents/blob/RHEL6/heartbeat/mysql" target="_blank">https://github.com/ClusterLabs/resource-agents/blob/RHEL6/heartbeat/mysql</a> works as expected, because as it gets an empty set when performing the monitor action it does not consider it as a replication slave, so I guess there is an issue from the issue already presented above and the CHANGE_MASTER_TO query that failed because of error "ERROR 1200 (HY000) at line 1: Misconfigured slave: MASTER_HOST was not set;" </div><div><br></div><div>I may miss something obvious .. Please tell me if I can bring more information around my issue. </div><div><br></div><div>Rgds</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Symetric cluster<br>
> configured.<br>
> <br>
> Details of my pacemaker resource configuration is<br>
> <br>
> Master: ms_mysql-master<br>
> Meta Attrs: master-node-max=1 clone_max=2 globally-unique=false<br>
> clone-node-max=1 notify=true<br>
> Resource: ms_mysql (class=ocf provider=heartbeat type=mysql)<br>
> Attributes: binary=/usr/bin/mysqld_safe config=/etc/my.cnf.d/server.cnf<br>
> datadir=/var/lib/mysql evict_outdated_slaves=false max_slave_lag=15<br>
> pid=/var/lib/mysql/mysql.pid replication_passwd=mysqlreplpw<br>
> replication_user=user-repl socket=/var/lib/mysql/mysql.sock<br>
> test_passwd=mysqlrootpw test_user=root<br>
> Operations: demote interval=0s timeout=120 (ms_mysql-demote-interval-0s)<br>
> monitor interval=20 timeout=30 (ms_mysql-monitor-interval-20)<br>
> monitor interval=10 role=Master timeout=30<br>
> (ms_mysql-monitor-interval-10)<br>
> monitor interval=30 role=Slave timeout=30<br>
> (ms_mysql-monitor-interval-30)<br>
> notify interval=0s timeout=90 (ms_mysql-notify-interval-0s)<br>
> promote interval=0s timeout=120<br>
> (ms_mysql-promote-interval-0s)<br>
> start interval=0s timeout=120 (ms_mysql-start-interval-0s)<br>
> stop interval=0s timeout=120 (ms_mysql-stop-interval-0s)<br>
> <br>
> Any things I'm missing on this ? Did not find a clearly similar usecase<br>
> when googling around network outage and pacemaker.<br>
> <br>
> Thanks<br>
> <br>
> <br>
> <br>
> _______________________________________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
> <br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
> <br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div></div></div></div></div></div></div></div></div></div></div></div></div></div>