<div dir="ltr"><br><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Thanks,<br>Jaspal Singla<br></div></div></div>
<br><div class="gmail_quote">On Mon, Aug 22, 2016 at 7:42 PM, <span dir="ltr"><<a href="mailto:users-request@clusterlabs.org" target="_blank">users-request@clusterlabs.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send Users mailing list submissions to<br>
<a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:users-request@clusterlabs.org">users-request@clusterlabs.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:users-owner@clusterlabs.org">users-owner@clusterlabs.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Users digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Mysql slave did not start replication after failure, and<br>
read-only IP also remained active on the much outdated slave<br>
(Attila Megyeri)<br>
2. Re: Entire Group stop on stopping of single Resource (Jan Pokorn?)<br>
3. Re: Mysql slave did not start replication after failure, and<br>
read-only IP also remained active on the much outdated slave<br>
(Ken Gaillot)<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Mon, 22 Aug 2016 14:24:28 +0200<br>
From: Attila Megyeri <<a href="mailto:amegyeri@minerva-soft.com">amegyeri@minerva-soft.com</a>><br>
To: Cluster Labs - All topics related to open-source clustering<br>
welcomed <<a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a>><br>
Subject: Re: [ClusterLabs] Mysql slave did not start replication after<br>
failure, and read-only IP also remained active on the much outdated<br>
slave<br>
Message-ID:<br>
<<wbr>DA9AC973EEA03848B46F36E07B947E<wbr>0403E5393E9922@DESRV05.<wbr>minerva-soft.local><br>
<br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi Andrei,<br>
<br>
I waited several hours, and nothing happened.<br>
<br>
I assume that the RA does not treat this case properly. Mysql was running, but the "show slave status" command returned something that the RA was not prepared to parse, and instead of reporting a non-readable attribute, it returned some generic error, that did not stop the server.<br>
<br>
Rgds,<br>
Attila<br>
<br>
<br>
-----Original Message-----<br>
From: Andrei Borzenkov [mailto:<a href="mailto:arvidjaar@gmail.com">arvidjaar@gmail.com</a>]<br>
Sent: Monday, August 22, 2016 11:42 AM<br>
To: Cluster Labs - All topics related to open-source clustering welcomed <<a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a>><br>
Subject: Re: [ClusterLabs] Mysql slave did not start replication after failure, and read-only IP also remained active on the much outdated slave<br>
<br>
On Mon, Aug 22, 2016 at 12:18 PM, Attila Megyeri<br>
<<a href="mailto:amegyeri@minerva-soft.com">amegyeri@minerva-soft.com</a>> wrote:<br>
> Dear community,<br>
><br>
><br>
><br>
> A few days ago we had an issue in our Mysql M/S replication cluster.<br>
><br>
> We have a one R/W Master, and a one RO Slave setup. RO VIP is supposed to be<br>
> running on the slave if it is not too much behind the master, and if any<br>
> error occurs, RO VIP is moved to the master.<br>
><br>
><br>
><br>
> Something happened with the slave Mysql (some disk issue, still<br>
> investigating), but the problem is, that the slave VIP remained on the slave<br>
> device, even though the slave process was not running, and the server was<br>
> much outdated.<br>
><br>
><br>
><br>
> During the issue the following log entries appeared (just an extract as it<br>
> would be too long):<br>
><br>
><br>
><br>
><br>
><br>
> Aug 20 02:04:07 ctdb1 corosync[1056]: [MAIN ] Corosync main process was<br>
> not scheduled for 14088.5488 ms (threshold is 4000.0000 ms). Consider token<br>
> timeout increase.<br>
><br>
> Aug 20 02:04:07 ctdb1 corosync[1056]: [TOTEM ] A processor failed, forming<br>
> new configuration.<br>
><br>
> Aug 20 02:04:34 ctdb1 corosync[1056]: [MAIN ] Corosync main process was<br>
> not scheduled for 27065.2559 ms (threshold is 4000.0000 ms). Consider token<br>
> timeout increase.<br>
><br>
> Aug 20 02:04:34 ctdb1 corosync[1056]: [TOTEM ] A new membership (xxx:6720)<br>
> was formed. Members left: 168362243 168362281 168362282 168362301 168362302<br>
> 168362311 168362312 1<br>
><br>
> Aug 20 02:04:34 ctdb1 corosync[1056]: [TOTEM ] A new membership (xxx:6724)<br>
> was formed. Members<br>
><br>
> ..<br>
><br>
> Aug 20 02:13:28 ctdb1 corosync[1056]: [MAIN ] Completed service<br>
> synchronization, ready to provide service.<br>
><br>
> ..<br>
><br>
> Aug 20 02:13:29 ctdb1 attrd[1584]: notice: attrd_trigger_update: Sending<br>
> flush op to all hosts for: readable (1)<br>
><br>
> ?<br>
><br>
> Aug 20 02:13:32 ctdb1 mysql(db-mysql)[10492]: INFO: post-demote notification<br>
> for ctdb1<br>
><br>
> Aug 20 02:13:32 ctdb1 IPaddr2(db-ip-master)[10490]: INFO: IP status = ok,<br>
> IP_CIP=<br>
><br>
> Aug 20 02:13:32 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
> db-ip-master_stop_0 (call=371, rc=0, cib-update=179, confirmed=true) ok<br>
><br>
> Aug 20 02:13:32 ctdb1 IPaddr2(db-ip-slave)[10620]: INFO: Adding inet address<br>
> xxx/24 with broadcast address xxxx to device eth0<br>
><br>
> Aug 20 02:13:32 ctdb1 IPaddr2(db-ip-slave)[10620]: INFO: Bringing device<br>
> eth0 up<br>
><br>
> Aug 20 02:13:32 ctdb1 IPaddr2(db-ip-slave)[10620]: INFO:<br>
> /usr/lib/heartbeat/send_arp -i 200 -r 5 -p<br>
> /usr/var/run/resource-agents/<wbr>send_arp-xxx eth0 xxx auto not_used not_used<br>
><br>
> Aug 20 02:13:32 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
> db-ip-slave_start_0 (call=377, rc=0, cib-update=180, confirmed=true) ok<br>
><br>
> Aug 20 02:13:32 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
> db-ip-slave_monitor_20000 (call=380, rc=0, cib-update=181, confirmed=false)<br>
> ok<br>
><br>
> Aug 20 02:13:32 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
> db-mysql_notify_0 (call=374, rc=0, cib-update=0, confirmed=true) ok<br>
><br>
> Aug 20 02:13:32 ctdb1 attrd[1584]: notice: attrd_trigger_update: Sending<br>
> flush op to all hosts for: master-db-mysql (1)<br>
><br>
> Aug 20 02:13:32 ctdb1 attrd[1584]: notice: attrd_perform_update: Sent<br>
> update 1622: master-db-mysql=1<br>
><br>
> Aug 20 02:13:32 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
> db-mysql_demote_0 (call=384, rc=0, cib-update=182, confirmed=true) ok<br>
><br>
> Aug 20 02:13:33 ctdb1 mysql(db-mysql)[11160]: INFO: Ignoring post-demote<br>
> notification for my own demotion.<br>
><br>
> Aug 20 02:13:33 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
> db-mysql_notify_0 (call=387, rc=0, cib-update=0, confirmed=true) ok<br>
><br>
> Aug 20 02:13:33 ctdb1 mysql(db-mysql)[11185]: ERROR: check_slave invoked on<br>
> an instance that is not a replication slave.<br>
><br>
> Aug 20 02:13:33 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
> db-mysql_monitor_7000 (call=390, rc=0, cib-update=183, confirmed=false) ok<br>
><br>
> Aug 20 02:13:33 ctdb1 ntpd[1560]: Listen normally on 16 eth0 xxxx. UDP 123<br>
><br>
> Aug 20 02:13:33 ctdb1 ntpd[1560]: Deleting interface #12 eth0, xxx#123,<br>
> interface stats: received=0, sent=0, dropped=0, active_time=2637334 secs<br>
><br>
> Aug 20 02:13:33 ctdb1 ntpd[1560]: peers refreshed<br>
><br>
> Aug 20 02:13:33 ctdb1 ntpd[1560]: new interface(s) found: waking up resolver<br>
><br>
> Aug 20 02:13:40 ctdb1 mysql(db-mysql)[11224]: ERROR: check_slave invoked on<br>
> an instance that is not a replication slave.<br>
><br>
> Aug 20 02:13:47 ctdb1 mysql(db-mysql)[11263]: ERROR: check_slave invoked on<br>
> an instance that is not a replication slave.<br>
><br>
><br>
><br>
> And from this time, the last two lines repeat every 7 seconds (mysql<br>
> monitoring interval)<br>
><br>
><br>
><br>
><br>
><br>
> The expected behavior was that the slave (RO) VIP should have been moved to<br>
> the master, as the secondary db was outdated.<br>
><br>
> Unfortunately I cannot recall what crm_mon was showing when the issue was<br>
> present, but I am sure that the RA did not handle the situation properly.<br>
><br>
><br>
><br>
> Placing the slave node into standby and the online resolved the issue<br>
> immediately (Slave started to sync, and in a few minutes it catched up the<br>
> master).<br>
><br>
><br>
><br>
><br>
><br>
> Here is the relevant config from the configuration:<br>
><br>
><br>
><br>
><br>
><br>
> primitive db-ip-master ocf:heartbeat:IPaddr2 \<br>
><br>
> params lvs_support="true" ip="XXX" cidr_netmask="24"<br>
> broadcast="XXX" \<br>
><br>
> op start interval="0" timeout="20s" on-fail="restart" \<br>
><br>
> op monitor interval="20s" timeout="20s" \<br>
><br>
> op stop interval="0" timeout="20s" on-fail="block"<br>
><br>
><br>
><br>
> primitive db-ip-slave ocf:heartbeat:IPaddr2 \<br>
><br>
> params lvs_support="true" ip="XXX" cidr_netmask="24"<br>
> broadcast="XXX" \<br>
><br>
> op start interval="0" timeout="20s" on-fail="restart" \<br>
><br>
> op monitor interval="20s" timeout="20s" \<br>
><br>
> op stop interval="0" timeout="20s" on-fail="block" \<br>
><br>
> meta target-role="Started"<br>
><br>
><br>
><br>
> primitive db-mysql ocf:heartbeat:mysql \<br>
><br>
> params binary="/usr/bin/mysqld_safe"<br>
> config="/etc/mysql/my.cnf" datadir="/var/lib/mysql" user="mysql"<br>
> pid="/var/run/mysqld/mysqld.<wbr>pid" socket="/var/run/mysqld/<wbr>mysqld.sock"<br>
> test_passwd="XXX" test_table="XXX" test_user="XXX" replication_user="XXX"<br>
> replication_passwd="XXX" additional_parameters="--skip-<wbr>slave-start" \<br>
><br>
> op start interval="0" timeout="240s" on-fail="restart" \<br>
><br>
> op stop interval="0" timeout="120s" on-fail="block" \<br>
><br>
> op monitor interval="7" timeout="30s" on-fail="restart"<br>
> OCF_CHECK_LEVEL="1" \<br>
><br>
> op promote interval="0" timeout="120" on-fail="restart" \<br>
><br>
> op demote interval="0" timeout="120" on-fail="block"<br>
><br>
><br>
><br>
> ms mysql db-mysql \<br>
><br>
> meta notify="true" master-max="1" clone-max="2"<br>
> target-role="Started" is-managed="true"<br>
><br>
><br>
><br>
> location db-ip-m-1 db-ip-master 0: ctdb1<br>
><br>
> location db-ip-m-2 db-ip-master 0: ctdb2<br>
><br>
> location db-ip-s-1 db-ip-slave 0: ctdb1<br>
><br>
> location db-ip-s-2 db-ip-slave 0: ctdb2<br>
><br>
> location db-ip-s-readable db-ip-slave \<br>
><br>
> rule $id="rule-no-reader-slave" -inf: readable lt 1<br>
><br>
<br>
How long did you wait? Conditions are reevaluated every<br>
cluster-recheck-interval which is 15 minutes by default.<br>
<br>
> location db-mysql-loc-1 mysql 100: ctdb1<br>
><br>
> location db-mysql-loc-2 mysql 100: ctdb2<br>
><br>
><br>
><br>
> colocation db-ip-slave-master -50: db-ip-slave db-ip-master<br>
><br>
> colocation db-ip-with-master inf: db-ip-master mysql:Master<br>
><br>
> colocation db-slave-on-db inf: db-ip-slave mysql<br>
><br>
> order master-after-db inf: mysql db-ip-master<br>
><br>
> order slave-after-db inf: mysql db-ip-slave<br>
><br>
><br>
><br>
> property $id="cib-bootstrap-options" \<br>
><br>
> dc-version="1.1.10-42f2063" \<br>
><br>
> cluster-infrastructure="<wbr>corosync" \<br>
><br>
> symmetric-cluster="false" \<br>
><br>
> cluster-recheck-interval="2m" \<br>
><br>
> no-quorum-policy="stop" \<br>
><br>
> stop-orphan-resources="false" \<br>
><br>
> start-failure-is-fatal="false" \<br>
><br>
> maintenance-mode="false"<br>
><br>
> property $id="mysql_replication" \<br>
><br>
> db-mysql_REPL_INFO="ctdb2|<wbr>mysql-bin.002928|107"<br>
><br>
> rsc_defaults $id="rsc-options" \<br>
><br>
> resource-stickiness="0"<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> Do you have any hints what could have gone wrong, and how we could avoid<br>
> such issues in the future?<br>
><br>
><br>
><br>
> Versions:<br>
><br>
> Ubuntu Trusty Tahr<br>
><br>
> Pacemaker 1.1.10<br>
><br>
> Corosync 2.3.3<br>
><br>
> Resource agents 3.9.3<br>
><br>
><br>
><br>
><br>
><br>
> Thanks a lot in advance,<br>
><br>
><br>
><br>
> Attila<br>
><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> <a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
><br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Mon, 22 Aug 2016 15:32:03 +0200<br>
From: Jan Pokorn? <<a href="mailto:jpokorny@redhat.com">jpokorny@redhat.com</a>><br>
To: Cluster Labs - All topics related to open-source clustering<br>
welcomed <<a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a>><br>
Subject: Re: [ClusterLabs] Entire Group stop on stopping of single<br>
Resource<br>
Message-ID: <<a href="mailto:20160822133203.GO15835@redhat.com">20160822133203.GO15835@<wbr>redhat.com</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br></blockquote><div><br></div><div>Hi John,</div><div><br></div><div>Thanks for replying. Please find the in-line answers:</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
On 19/08/16 23:09 +0530, jaspal singla wrote:<br>
> I have an resource group (ctm_service) comprise of various resources. Now<br>
> the requirement is when one of its resource stops for soem time (10-20)<br>
> seconds, I want entire group will be stopped.<br>
<br>
Note that if resource is stopped _just_ for this period (in seconds)<br>
while monitor is set to a bigger value (30 s), pacemaker may miss the<br>
resource being intermittently stopped.<br></blockquote><div><br></div><div>original config has stop interval of 5s and the resource does get stopped but here I want to achieve that - If any of the resource stops, all other resources configured along with that resource would also be stopped. Is their a way for such setting? Can I use on-fail=standby on group so that if any of the resource gets stop, rest of the resources would also gets stopped?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> Is it possible to achieve this in pacemaker. Please help!<br>
<br>
Just for clarification, do you mean stopped completely within the<br>
cluster and not just on the node the group was running when one of<br>
its resources stopped?<br></blockquote><div><br></div><div>Just stopped completely on that node. We have separate scripts which takes care of switching over to other node but for that entire resources on the serving nodes has to be stopped. Is thier any way? Please suggest.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> Resource Group: ctm_service<br>
> FSCheck<br>
> (lsb:../../..//cisco/<wbr>PrimeOpticalServer/HA/bin/<wbr>FsCheckAgent.py):<br>
> (target-role:Stopped) Stopped<br>
> NTW_IF (lsb:../../..//cisco/<wbr>PrimeOpticalServer/HA/bin/<wbr>NtwIFAgent.py):<br>
> (target-role:Stopped) Stopped<br>
> CTM_RSYNC (lsb:../../..//cisco/<wbr>PrimeOpticalServer/HA/bin/<wbr>RsyncAgent.py):<br>
> (target-role:Stopped) Stopped<br>
> REPL_IF (lsb:../../..//cisco/<wbr>PrimeOpticalServer/HA/bin/ODG_<wbr>IFAgent.py):<br>
> (target-role:Stopped) Stopped<br>
> ORACLE_REPLICATOR<br>
> (lsb:../../..//cisco/<wbr>PrimeOpticalServer/HA/bin/ODG_<wbr>ReplicatorAgent.py):<br>
> (target-role:Stopped) Stopped<br>
> CTM_SID (lsb:../../..//cisco/<wbr>PrimeOpticalServer/HA/bin/<wbr>OracleAgent.py):<br>
> (target-role:Stopped) Stopped<br>
> CTM_SRV (lsb:../../..//cisco/<wbr>PrimeOpticalServer/HA/bin/<wbr>CtmAgent.py):<br>
> (target-role:Stopped) Stopped<br>
> CTM_APACHE (lsb:../../..//cisco/<wbr>PrimeOpticalServer/HA/bin/<wbr>ApacheAgent.py):<br>
> (target-role:Stopped) Stopped<br>
> ______________________________<wbr>______________________________<br>
> _________________<br>
><br>
><br>
> This is resource and resource group properties:<br>
><br>
> ______________________________<wbr>______________________________<br>
> ___________________<br>
><br>
> pcs -f cib.xml.geo resource create FSCheck lsb:../../..//cisco/<br>
> PrimeOpticalServer/HA/bin/<wbr>FsCheckAgent.py op monitor id=FSCheck-OP-monitor<br>
> name=monitor interval=30s<br>
> pcs -f cib.xml.geo resource create NTW_IF lsb:../../..//cisco/<br>
> PrimeOpticalServer/HA/bin/<wbr>NtwIFAgent.py op monitor id=NtwIFAgent-OP-monitor<br>
> name=monitor interval=30s<br>
> pcs -f cib.xml.geo resource create CTM_RSYNC lsb:../../..//cisco/<br>
> PrimeOpticalServer/HA/bin/<wbr>RsyncAgent.py op monitor id=CTM_RSYNC-OP-monitor<br>
> name=monitor interval=30s on-fail=ignore stop id=CTM_RSYNC-OP-stop<br>
> interval=0 on-fail=stop<br>
> pcs -f cib.xml.geo resource create REPL_IF lsb:../../..//cisco/<br>
> PrimeOpticalServer/HA/bin/ODG_<wbr>IFAgent.py op monitor id=REPL_IF-OP-monitor<br>
> name=monitor interval=30 on-fail=ignore stop id=REPL_IF-OP-stop interval=0<br>
> on-fail=stop<br>
> pcs -f cib.xml.geo resource create ORACLE_REPLICATOR lsb:../../..//cisco/<br>
> PrimeOpticalServer/HA/bin/ODG_<wbr>ReplicatorAgent.py op monitor<br>
> id=ORACLE_REPLICATOR-OP-<wbr>monitor name=monitor interval=30s on-fail=ignore<br>
> stop id=ORACLE_REPLICATOR-OP-stop interval=0 on-fail=stop<br>
> pcs -f cib.xml.geo resource create CTM_SID lsb:../../..//cisco/<br>
> PrimeOpticalServer/HA/bin/<wbr>OracleAgent.py op monitor id=CTM_SID-OP-monitor<br>
> name=monitor interval=30s<br>
> pcs -f cib.xml.geo resource create CTM_SRV lsb:../../..//cisco/<br>
> PrimeOpticalServer/HA/bin/<wbr>CtmAgent.py op monitor id=CTM_SRV-OP-monitor<br>
> name=monitor interval=30s<br>
> pcs -f cib.xml.geo resource create CTM_APACHE lsb:../../..//cisco/<br>
> PrimeOpticalServer/HA/bin/<wbr>ApacheAgent.py op monitor<br>
> id=CTM_APACHE-OP-monitor name=monitor interval=30s<br>
> pcs -f cib.xml.geo resource create CTM_HEARTBEAT lsb:../../..//cisco/<br>
> PrimeOpticalServer/HA/bin/<wbr>HeartBeat.py op monitor<br>
> id=CTM_HEARTBEAT-OP-monitor name=monitor interval=30s<br>
> pcs -f cib.xml.geo resource create FLASHBACK lsb:../../..//cisco/<br>
> PrimeOpticalServer/HA/bin/<wbr>FlashBackMonitor.py op monitor<br>
> id=FLASHBACK-OP-monitor name=monitor interval=30s<br>
><br>
><br>
> pcs -f cib.xml.geo resource group add ctm_service FSCheck NTW_IF CTM_RSYNC<br>
> REPL_IF ORACLE_REPLICATOR CTM_SID CTM_SRV CTM_APACHE<br>
><br>
> pcs -f cib.xml.geo resource meta ctm_service migration-threshold=1<br>
> failure-timeout=10 target-role=stopped<br>
<br>
Why do you have target-role=stopped (should preferably be title-cased<br>
"Stopped") here/is that only for the test purposes? I ask as it may<br>
intefere with any subsequent modifications.<br>
<br></blockquote><div><br></div><div>I don't want pacemaker should be able to start the resources by their won configured under "ctm_service" group. We manually perform that operation.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
P.S. The presented configuration resembles output of clufter, so any<br>
feedback to be turned into its improvements welcome.<br>
<br></blockquote><div>Yea, I am aware that and you helped in the past for clufter setup.</div><div><br></div><div>One thing I noticed that - Whenever I add any resource operation through script (add to temp file and then import to CIB main database), it won't work.</div><div><br></div><div>I am able to see the added resource operation through pcs resource ctm_service --full command but these are of no use.</div><div><br></div><div>I have to manually add all of the resource operations one by one on running CIB config like this:</div><div><br></div><div>pcs resource op add CTM_ARV monitor interval=10s timeout=20s<br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
--<br>
Jan (Poki)<br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: application/pgp-signature<br>
Size: 819 bytes<br>
Desc: not available<br>
URL: <<a href="http://clusterlabs.org/pipermail/users/attachments/20160822/f99b8eff/attachment-0001.sig" rel="noreferrer" target="_blank">http://clusterlabs.org/<wbr>pipermail/users/attachments/<wbr>20160822/f99b8eff/attachment-<wbr>0001.sig</a>><br>
<br></blockquote><div><br></div><div>Please help to let me know if there is any option to put the entire node on standby, if any of the resource gets stopped.</div><div><br></div><div><br></div><div>BR,</div><div>JAspal</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
------------------------------<br>
<br>
Message: 3<br>
Date: Mon, 22 Aug 2016 09:11:34 -0500<br>
From: Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
To: <a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a><br>
Subject: Re: [ClusterLabs] Mysql slave did not start replication after<br>
failure, and read-only IP also remained active on the much outdated<br>
slave<br>
Message-ID: <<a href="mailto:4186e373-38d4-af62-c2b1-589f968c9463@redhat.com">4186e373-38d4-af62-c2b1-<wbr>589f968c9463@redhat.com</a>><br>
Content-Type: text/plain; charset=utf-8<br>
<br>
On 08/22/2016 07:24 AM, Attila Megyeri wrote:<br>
> Hi Andrei,<br>
><br>
> I waited several hours, and nothing happened.<br>
<br>
And actually, we can see from the configuration you provided that<br>
cluster-recheck-interval is 2 minutes.<br>
<br>
I don't see anything about stonith; is it enabled and tested? This looks<br>
like a situation where stonith would come into play. I know that power<br>
fencing can be rough on a MySQL database, but perhaps intelligent<br>
switches with network fencing would be appropriate.<br>
<br>
The "Corosync main process was not scheduled" message is the start of<br>
the trouble. It means the system was overloaded and corosync didn't get<br>
any CPU time, so it couldn't maintain cluster communication.<br>
<br>
Probably the most useful thing would be to upgrade to a recent version<br>
of corosync+pacemaker+resource-<wbr>agents. Recent corosync versions run with<br>
realtime priority, which makes this much less likely.<br>
<br>
Other than that, figure out what the load issue was, and try to prevent<br>
it from recurring.<br>
<br>
I'm not familiar enough with the RA to comment on its behavior. If you<br>
think it's suspect, check the logs during the incident for messages from<br>
the RA.<br>
<br>
> I assume that the RA does not treat this case properly. Mysql was running, but the "show slave status" command returned something that the RA was not prepared to parse, and instead of reporting a non-readable attribute, it returned some generic error, that did not stop the server.<br>
><br>
> Rgds,<br>
> Attila<br>
><br>
><br>
> -----Original Message-----<br>
> From: Andrei Borzenkov [mailto:<a href="mailto:arvidjaar@gmail.com">arvidjaar@gmail.com</a>]<br>
> Sent: Monday, August 22, 2016 11:42 AM<br>
> To: Cluster Labs - All topics related to open-source clustering welcomed <<a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a>><br>
> Subject: Re: [ClusterLabs] Mysql slave did not start replication after failure, and read-only IP also remained active on the much outdated slave<br>
><br>
> On Mon, Aug 22, 2016 at 12:18 PM, Attila Megyeri<br>
> <<a href="mailto:amegyeri@minerva-soft.com">amegyeri@minerva-soft.com</a>> wrote:<br>
>> Dear community,<br>
>><br>
>><br>
>><br>
>> A few days ago we had an issue in our Mysql M/S replication cluster.<br>
>><br>
>> We have a one R/W Master, and a one RO Slave setup. RO VIP is supposed to be<br>
>> running on the slave if it is not too much behind the master, and if any<br>
>> error occurs, RO VIP is moved to the master.<br>
>><br>
>><br>
>><br>
>> Something happened with the slave Mysql (some disk issue, still<br>
>> investigating), but the problem is, that the slave VIP remained on the slave<br>
>> device, even though the slave process was not running, and the server was<br>
>> much outdated.<br>
>><br>
>><br>
>><br>
>> During the issue the following log entries appeared (just an extract as it<br>
>> would be too long):<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> Aug 20 02:04:07 ctdb1 corosync[1056]: [MAIN ] Corosync main process was<br>
>> not scheduled for 14088.5488 ms (threshold is 4000.0000 ms). Consider token<br>
>> timeout increase.<br>
>><br>
>> Aug 20 02:04:07 ctdb1 corosync[1056]: [TOTEM ] A processor failed, forming<br>
>> new configuration.<br>
>><br>
>> Aug 20 02:04:34 ctdb1 corosync[1056]: [MAIN ] Corosync main process was<br>
>> not scheduled for 27065.2559 ms (threshold is 4000.0000 ms). Consider token<br>
>> timeout increase.<br>
>><br>
>> Aug 20 02:04:34 ctdb1 corosync[1056]: [TOTEM ] A new membership (xxx:6720)<br>
>> was formed. Members left: 168362243 168362281 168362282 168362301 168362302<br>
>> 168362311 168362312 1<br>
>><br>
>> Aug 20 02:04:34 ctdb1 corosync[1056]: [TOTEM ] A new membership (xxx:6724)<br>
>> was formed. Members<br>
>><br>
>> ..<br>
>><br>
>> Aug 20 02:13:28 ctdb1 corosync[1056]: [MAIN ] Completed service<br>
>> synchronization, ready to provide service.<br>
>><br>
>> ..<br>
>><br>
>> Aug 20 02:13:29 ctdb1 attrd[1584]: notice: attrd_trigger_update: Sending<br>
>> flush op to all hosts for: readable (1)<br>
>><br>
>> ?<br>
>><br>
>> Aug 20 02:13:32 ctdb1 mysql(db-mysql)[10492]: INFO: post-demote notification<br>
>> for ctdb1<br>
>><br>
>> Aug 20 02:13:32 ctdb1 IPaddr2(db-ip-master)[10490]: INFO: IP status = ok,<br>
>> IP_CIP=<br>
>><br>
>> Aug 20 02:13:32 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
>> db-ip-master_stop_0 (call=371, rc=0, cib-update=179, confirmed=true) ok<br>
>><br>
>> Aug 20 02:13:32 ctdb1 IPaddr2(db-ip-slave)[10620]: INFO: Adding inet address<br>
>> xxx/24 with broadcast address xxxx to device eth0<br>
>><br>
>> Aug 20 02:13:32 ctdb1 IPaddr2(db-ip-slave)[10620]: INFO: Bringing device<br>
>> eth0 up<br>
>><br>
>> Aug 20 02:13:32 ctdb1 IPaddr2(db-ip-slave)[10620]: INFO:<br>
>> /usr/lib/heartbeat/send_arp -i 200 -r 5 -p<br>
>> /usr/var/run/resource-agents/<wbr>send_arp-xxx eth0 xxx auto not_used not_used<br>
>><br>
>> Aug 20 02:13:32 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
>> db-ip-slave_start_0 (call=377, rc=0, cib-update=180, confirmed=true) ok<br>
>><br>
>> Aug 20 02:13:32 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
>> db-ip-slave_monitor_20000 (call=380, rc=0, cib-update=181, confirmed=false)<br>
>> ok<br>
>><br>
>> Aug 20 02:13:32 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
>> db-mysql_notify_0 (call=374, rc=0, cib-update=0, confirmed=true) ok<br>
>><br>
>> Aug 20 02:13:32 ctdb1 attrd[1584]: notice: attrd_trigger_update: Sending<br>
>> flush op to all hosts for: master-db-mysql (1)<br>
>><br>
>> Aug 20 02:13:32 ctdb1 attrd[1584]: notice: attrd_perform_update: Sent<br>
>> update 1622: master-db-mysql=1<br>
>><br>
>> Aug 20 02:13:32 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
>> db-mysql_demote_0 (call=384, rc=0, cib-update=182, confirmed=true) ok<br>
>><br>
>> Aug 20 02:13:33 ctdb1 mysql(db-mysql)[11160]: INFO: Ignoring post-demote<br>
>> notification for my own demotion.<br>
>><br>
>> Aug 20 02:13:33 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
>> db-mysql_notify_0 (call=387, rc=0, cib-update=0, confirmed=true) ok<br>
>><br>
>> Aug 20 02:13:33 ctdb1 mysql(db-mysql)[11185]: ERROR: check_slave invoked on<br>
>> an instance that is not a replication slave.<br>
>><br>
>> Aug 20 02:13:33 ctdb1 crmd[1586]: notice: process_lrm_event: LRM operation<br>
>> db-mysql_monitor_7000 (call=390, rc=0, cib-update=183, confirmed=false) ok<br>
>><br>
>> Aug 20 02:13:33 ctdb1 ntpd[1560]: Listen normally on 16 eth0 xxxx. UDP 123<br>
>><br>
>> Aug 20 02:13:33 ctdb1 ntpd[1560]: Deleting interface #12 eth0, xxx#123,<br>
>> interface stats: received=0, sent=0, dropped=0, active_time=2637334 secs<br>
>><br>
>> Aug 20 02:13:33 ctdb1 ntpd[1560]: peers refreshed<br>
>><br>
>> Aug 20 02:13:33 ctdb1 ntpd[1560]: new interface(s) found: waking up resolver<br>
>><br>
>> Aug 20 02:13:40 ctdb1 mysql(db-mysql)[11224]: ERROR: check_slave invoked on<br>
>> an instance that is not a replication slave.<br>
>><br>
>> Aug 20 02:13:47 ctdb1 mysql(db-mysql)[11263]: ERROR: check_slave invoked on<br>
>> an instance that is not a replication slave.<br>
>><br>
>><br>
>><br>
>> And from this time, the last two lines repeat every 7 seconds (mysql<br>
>> monitoring interval)<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> The expected behavior was that the slave (RO) VIP should have been moved to<br>
>> the master, as the secondary db was outdated.<br>
>><br>
>> Unfortunately I cannot recall what crm_mon was showing when the issue was<br>
>> present, but I am sure that the RA did not handle the situation properly.<br>
>><br>
>><br>
>><br>
>> Placing the slave node into standby and the online resolved the issue<br>
>> immediately (Slave started to sync, and in a few minutes it catched up the<br>
>> master).<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> Here is the relevant config from the configuration:<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> primitive db-ip-master ocf:heartbeat:IPaddr2 \<br>
>><br>
>> params lvs_support="true" ip="XXX" cidr_netmask="24"<br>
>> broadcast="XXX" \<br>
>><br>
>> op start interval="0" timeout="20s" on-fail="restart" \<br>
>><br>
>> op monitor interval="20s" timeout="20s" \<br>
>><br>
>> op stop interval="0" timeout="20s" on-fail="block"<br>
>><br>
>><br>
>><br>
>> primitive db-ip-slave ocf:heartbeat:IPaddr2 \<br>
>><br>
>> params lvs_support="true" ip="XXX" cidr_netmask="24"<br>
>> broadcast="XXX" \<br>
>><br>
>> op start interval="0" timeout="20s" on-fail="restart" \<br>
>><br>
>> op monitor interval="20s" timeout="20s" \<br>
>><br>
>> op stop interval="0" timeout="20s" on-fail="block" \<br>
>><br>
>> meta target-role="Started"<br>
>><br>
>><br>
>><br>
>> primitive db-mysql ocf:heartbeat:mysql \<br>
>><br>
>> params binary="/usr/bin/mysqld_safe"<br>
>> config="/etc/mysql/my.cnf" datadir="/var/lib/mysql" user="mysql"<br>
>> pid="/var/run/mysqld/mysqld.<wbr>pid" socket="/var/run/mysqld/<wbr>mysqld.sock"<br>
>> test_passwd="XXX" test_table="XXX" test_user="XXX" replication_user="XXX"<br>
>> replication_passwd="XXX" additional_parameters="--skip-<wbr>slave-start" \<br>
>><br>
>> op start interval="0" timeout="240s" on-fail="restart" \<br>
>><br>
>> op stop interval="0" timeout="120s" on-fail="block" \<br>
>><br>
>> op monitor interval="7" timeout="30s" on-fail="restart"<br>
>> OCF_CHECK_LEVEL="1" \<br>
>><br>
>> op promote interval="0" timeout="120" on-fail="restart" \<br>
>><br>
>> op demote interval="0" timeout="120" on-fail="block"<br>
>><br>
>><br>
>><br>
>> ms mysql db-mysql \<br>
>><br>
>> meta notify="true" master-max="1" clone-max="2"<br>
>> target-role="Started" is-managed="true"<br>
>><br>
>><br>
>><br>
>> location db-ip-m-1 db-ip-master 0: ctdb1<br>
>><br>
>> location db-ip-m-2 db-ip-master 0: ctdb2<br>
>><br>
>> location db-ip-s-1 db-ip-slave 0: ctdb1<br>
>><br>
>> location db-ip-s-2 db-ip-slave 0: ctdb2<br>
>><br>
>> location db-ip-s-readable db-ip-slave \<br>
>><br>
>> rule $id="rule-no-reader-slave" -inf: readable lt 1<br>
>><br>
><br>
> How long did you wait? Conditions are reevaluated every<br>
> cluster-recheck-interval which is 15 minutes by default.<br>
><br>
>> location db-mysql-loc-1 mysql 100: ctdb1<br>
>><br>
>> location db-mysql-loc-2 mysql 100: ctdb2<br>
>><br>
>><br>
>><br>
>> colocation db-ip-slave-master -50: db-ip-slave db-ip-master<br>
>><br>
>> colocation db-ip-with-master inf: db-ip-master mysql:Master<br>
>><br>
>> colocation db-slave-on-db inf: db-ip-slave mysql<br>
>><br>
>> order master-after-db inf: mysql db-ip-master<br>
>><br>
>> order slave-after-db inf: mysql db-ip-slave<br>
>><br>
>><br>
>><br>
>> property $id="cib-bootstrap-options" \<br>
>><br>
>> dc-version="1.1.10-42f2063" \<br>
>><br>
>> cluster-infrastructure="<wbr>corosync" \<br>
>><br>
>> symmetric-cluster="false" \<br>
>><br>
>> cluster-recheck-interval="2m" \<br>
>><br>
>> no-quorum-policy="stop" \<br>
>><br>
>> stop-orphan-resources="false" \<br>
>><br>
>> start-failure-is-fatal="false" \<br>
>><br>
>> maintenance-mode="false"<br>
>><br>
>> property $id="mysql_replication" \<br>
>><br>
>> db-mysql_REPL_INFO="ctdb2|<wbr>mysql-bin.002928|107"<br>
>><br>
>> rsc_defaults $id="rsc-options" \<br>
>><br>
>> resource-stickiness="0"<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> Do you have any hints what could have gone wrong, and how we could avoid<br>
>> such issues in the future?<br>
>><br>
>><br>
>><br>
>> Versions:<br>
>><br>
>> Ubuntu Trusty Tahr<br>
>><br>
>> Pacemaker 1.1.10<br>
>><br>
>> Corosync 2.3.3<br>
>><br>
>> Resource agents 3.9.3<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> Thanks a lot in advance,<br>
>><br>
>><br>
>><br>
>> Attila<br>
<br>
<br>
<br>
------------------------------<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
<br>
End of Users Digest, Vol 19, Issue 41<br>
******************************<wbr>*******<br>
</blockquote></div><br></div></div>