<div dir="ltr">Hello there again, people!<div><br></div><div>After upgrading both nodes to such SW versions:</div><div><div><br></div><div>pacemaker.x86_64 1.1.10-14.el6_5.3</div><div>pacemaker-cli.x86_64 1.1.10-14.el6_5.3</div>
<div>pacemaker-cluster-libs.x86_64</div><div>pacemaker-libs.x86_64 1.1.10-14.el6_5.3</div></div><div><div>corosync.x86_64 1.4.1-17.el6_5.1 @updates </div><div>corosynclib.x86_64 1.4.1-17.el6_5.1 @updates</div>
</div><div><br></div><div>I am still facing the same problem: slave in Master/Slave set of MySQL won't start.</div><div>Master actually works correctly.</div><div>Output of cibadmin -Q on both nodes is identical.</div>
<div><br></div><div>And here's the log of what happens when i try to do "cleanup MySQL_MasterSlave": <a href="http://pastebin.com/J90NuyEX">http://pastebin.com/J90NuyEX</a>. </div><div>By now i have MySQL slave running in manual mode, but this definitely is not what i'm trying to achieve using Pacemaker.</div>
<div>Can anyone help with this?</div><div>Again, my pacemaker configuration:</div><div><br></div><div><div>node wb-db1 \</div><div> attributes standby=off</div><div>node wb-db2 \</div><div> attributes standby=off</div>
<div>primitive ClusterIP IPaddr2 \</div><div> params ip=10.0.1.68 cidr_netmask=32 nic=bond0.100 \</div><div> op monitor interval=30s \</div><div> meta target-role=Started</div><div>primitive MySQL mysql \</div>
<div> params binary="/usr/bin/mysqld_safe" enable_creation=1 replication_user=slave_user replication_passwd=here_goes_the_password datadir="/var/lib/mysql/db" socket="/var/run/mysqld/mysqld.sock" config="/etc/my.cnf" reader_attribute=readerOK evict_outdated_slaves=false max_slave_lag=600 \</div>
<div> op monitor interval=30s \</div><div> op monitor interval=35s role=Master OCF_CHECK_LEVEL=1 \</div><div> op monitor interval=60s role=Slave timeout=60s OCF_CHECK_LEVEL=1 \</div><div> op notify interval=0 timeout=90 \</div>
<div> op start interval=0 timeout=120 \</div><div> op stop interval=0 timeout=120</div><div>primitive MySQL_Reader_VIP IPaddr2 \</div><div> params ip=10.0.1.66 cidr_netmask=32 nic=bond0.100 \</div><div>
meta target-role=Started</div><div>primitive ping-gateway ocf:pacemaker:ping \</div><div> params host_list=10.0.1.1 multiplier=100 timeout=1 \</div><div> op monitor interval=10s timeout=20s</div><div>
primitive resMON ocf:pacemaker:ClusterMon \</div><div> op start interval=0 timeout=90s \</div><div> op stop interval=0 timeout=100s \</div><div> op monitor interval=10s timeout=30s \</div><div> params extra_options="--mail-prefix MainDB_Cluster_Notification --mail-from <a href="mailto:cluster-alarm@gmsu.ua">cluster-alarm@gmsu.ua</a> --mail-to <a href="mailto:cluster-alarm@gmsu.ua">cluster-alarm@gmsu.ua</a> --mail-host <a href="http://mx.gmsu.ua">mx.gmsu.ua</a>"</div>
<div>ms MySQL_MasterSlave MySQL \</div><div> meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true globally-unique=false target-role=Started is-managed=true</div><div>clone pingclone ping-gateway \</div>
<div> meta target-role=Started</div><div>location No-MySQL_Reader_VIP MySQL_Reader_VIP \</div><div> rule $id="No-MySQL_Reader_VIP-rule" -inf: readerOK eq 0 or not_defined readerOK</div><div>location cli-prefer-ClusterIP ClusterIP \</div>
<div> rule $id="cli-prefer-rule-ClusterIP" inf: #uname eq wb-db1</div><div>location cli-standby-MySQL_MasterSlave MySQL_MasterSlave \</div><div> rule $id="cli-standby-rule-MySQL_MasterSlave" -inf: #uname eq wb-db1</div>
<div>location resourceClusterIPwithping ClusterIP \</div><div> rule $id="resourceClusterIPwithping-rule" -inf: not_defined pingd or pingd lte 0</div><div>colocation MySQL_Reader_VIP_dislike_ClusterIP -200: MySQL_Reader_VIP ClusterIP</div>
<div>colocation MysqlMaster-with-ClusterIP inf: MySQL_MasterSlave:Master ClusterIP</div><div>order MysqlMaster-after-ClusterIP inf: ClusterIP MySQL_MasterSlave:promote</div><div>property cib-bootstrap-options: \</div><div>
dc-version=1.1.10-14.el6_5.3-368c726 \</div></div><div><div> cluster-infrastructure="classic openais (with plugin)" \</div><div> expected-quorum-votes=2 \</div><div> no-quorum-policy=ignore \</div>
<div> stonith-enabled=false \</div><div> last-lrm-refresh=1402318675</div><div>property mysql_replication: \</div><div> MySQL_REPL_INFO="wb-db2|mysql-bin.000126|107"</div><div>rsc_defaults rsc-options: \</div>
<div> resource-stickiness=200</div></div><div><br></div><div>Thank you!</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-06-05 3:17 GMT+03:00 Andrew Beekhof <span dir="ltr"><<a href="mailto:andrew@beekhof.net" target="_blank">andrew@beekhof.net</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
On 30 May 2014, at 6:32 pm, Виталий Туровец <<a href="mailto:corebug@corebug.net">corebug@corebug.net</a>> wrote:<br>
<br>
> Hello there, people!<br>
> I am new to this list, so please excuse me if i'm posting to the wrong place.<br>
><br>
> I've got a pacemaker cluster with such a configuration: <a href="http://pastebin.com/1SbWWh4n" target="_blank">http://pastebin.com/1SbWWh4n</a>.<br>
><br>
> Output of "crm status":<br>
> ============<br>
> Last updated: Fri May 30 11:22:59 2014<br>
> Last change: Thu May 29 03:22:38 2014 via crmd on wb-db2<br>
> Stack: openais<br>
> Current DC: wb-db2 - partition with quorum<br>
> Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14<br>
> 2 Nodes configured, 2 expected votes<br>
> 7 Resources configured.<br>
> ============<br>
><br>
> Online: [ wb-db2 wb-db1 ]<br>
><br>
> ClusterIP (ocf::heartbeat:IPaddr2): Started wb-db2<br>
> MySQL_Reader_VIP (ocf::heartbeat:IPaddr2): Started wb-db2<br>
> resMON (ocf::pacemaker:ClusterMon): Started wb-db2<br>
> Master/Slave Set: MySQL_MasterSlave [MySQL]<br>
> Masters: [ wb-db2 ]<br>
> Stopped: [ MySQL:1 ]<br>
> Clone Set: pingclone [ping-gateway]<br>
> Started: [ wb-db1 wb-db2 ]<br>
><br>
> There was an unclean shutdown of a cluster and after that i've faced a problem that a slave of MySQL_MasterSlave resource does not come up.<br>
> When i try to do a "cleanup MySQL_MasterSlave" i see such thing in logs:<br>
<br>
</div></div>Most of those errors are cosmetic and fixed in later versions.<br>
<br>
> Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14<br>
<br>
It you can get to rhel 6.5 you'll have access to 1.1.10 where these are fixed.<br>
<div><div class="h5"><br>
><br>
> May 29 03:22:22 [4423] wb-db1 crmd: warning: decode_transition_key: Bad UUID (crm-resource-4819) in sscanf result (3) for 0:0:crm-resource-4819<br>
> May 29 03:22:22 [4423] wb-db1 crmd: warning: decode_transition_key: Bad UUID (crm-resource-4819) in sscanf result (3) for 0:0:crm-resource-4819<br>
> May 29 03:22:22 [4423] wb-db1 crmd: info: ais_dispatch_message: Membership 408: quorum retained<br>
> May 29 03:22:22 [4418] wb-db1 cib: info: set_crm_log_level: New log level: 3 0<br>
> May 29 03:22:38 [4421] wb-db1 attrd: notice: attrd_ais_dispatch: Update relayed from wb-db2<br>
> May 29 03:22:38 [4421] wb-db1 attrd: notice: attrd_ais_dispatch: Update relayed from wb-db2<br>
> May 29 03:22:38 [4418] wb-db1 cib: info: apply_xml_diff: Digest mis-match: expected 2f5bc3d7f673df3cf37f774211976d69, calculated b8a7adf0e34966242551556aab605286<br>
> May 29 03:22:38 [4418] wb-db1 cib: notice: cib_process_diff: Diff 0.243.4 -> 0.243.5 not applied to 0.243.4: Failed application of an update diff<br>
> May 29 03:22:38 [4418] wb-db1 cib: info: cib_server_process_diff: Requesting re-sync from peer<br>
> May 29 03:22:38 [4418] wb-db1 cib: notice: cib_server_process_diff: Not applying diff 0.243.4 -> 0.243.5 (sync in progress)<br>
> May 29 03:22:38 [4418] wb-db1 cib: info: cib_replace_notify: Replaced: -1.-1.-1 -> 0.243.5 from wb-db2<br>
> May 29 03:22:38 [4421] wb-db1 attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: pingd (100)<br>
> May 29 03:22:38 [4421] wb-db1 attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<br>
> May 29 03:22:38 [4418] wb-db1 cib: info: set_crm_log_level: New log level: 3 0<br>
> May 29 03:22:38 [4418] wb-db1 cib: info: apply_xml_diff: Digest mis-match: expected 754ed3b1d999e34d93e0835b310fd98a, calculated c322686deb255936ab54e064c696b6b8<br>
> May 29 03:22:38 [4418] wb-db1 cib: notice: cib_process_diff: Diff 0.244.5 -> 0.244.6 not applied to 0.244.5: Failed application of an update diff<br>
> May 29 03:22:38 [4418] wb-db1 cib: info: cib_server_process_diff: Requesting re-sync from peer<br>
> May 29 03:22:38 [4423] wb-db1 crmd: info: delete_resource: Removing resource MySQL:0 for 4996_crm_resource (internal) on wb-db2<br>
> May 29 03:22:38 [4423] wb-db1 crmd: info: notify_deleted: Notifying 4996_crm_resource on wb-db2 that MySQL:0 was deleted<br>
> May 29 03:22:38 [4418] wb-db1 cib: notice: cib_server_process_diff: Not applying diff 0.244.5 -> 0.244.6 (sync in progress)<br>
> May 29 03:22:38 [4423] wb-db1 crmd: warning: decode_transition_key: Bad UUID (crm-resource-4996) in sscanf result (3) for 0:0:crm-resource-4996<br>
> May 29 03:22:38 [4418] wb-db1 cib: notice: cib_server_process_diff: Not applying diff 0.244.6 -> 0.244.7 (sync in progress)<br>
> May 29 03:22:38 [4418] wb-db1 cib: notice: cib_server_process_diff: Not applying diff 0.244.7 -> 0.244.8 (sync in progress)<br>
> May 29 03:22:38 [4418] wb-db1 cib: info: cib_replace_notify: Replaced: -1.-1.-1 -> 0.244.8 from wb-db2<br>
> May 29 03:22:38 [4421] wb-db1 attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: pingd (100)<br>
> May 29 03:22:38 [4421] wb-db1 attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<br>
> May 29 03:22:38 [4423] wb-db1 crmd: notice: do_lrm_invoke: Not creating resource for a delete event: (null)<br>
> May 29 03:22:38 [4423] wb-db1 crmd: info: notify_deleted: Notifying 4996_crm_resource on wb-db2 that MySQL:1 was deleted<br>
> May 29 03:22:38 [4423] wb-db1 crmd: warning: decode_transition_key: Bad UUID (crm-resource-4996) in sscanf result (3) for 0:0:crm-resource-4996<br>
> May 29 03:22:38 [4423] wb-db1 crmd: warning: decode_transition_key: Bad UUID (crm-resource-4996) in sscanf result (3) for 0:0:crm-resource-4996<br>
> May 29 03:22:38 [4418] wb-db1 cib: info: set_crm_log_level: New log level: 3 0<br>
> May 29 03:22:38 [4423] wb-db1 crmd: info: ais_dispatch_message: Membership 408: quorum retained<br>
><br>
> Here's the cibadmin -Q output from node that is alive: <a href="http://pastebin.com/aeqfTaCe" target="_blank">http://pastebin.com/aeqfTaCe</a><br>
> And here's the one from failed node: <a href="http://pastebin.com/ME2U5vjK" target="_blank">http://pastebin.com/ME2U5vjK</a><br>
> The question is: how do i somehow cleanup the things for master/slave resource MySQL_MasterSlave to start working properly?<br>
><br>
> Thank you!<br>
><br>
> --<br>
><br>
><br>
><br>
><br>
> ~~~<br>
> WBR,<br>
> Vitaliy Turovets<br>
> Lead Operations Engineer<br>
> Global Message Services Ukraine<br>
> <a href="tel:%2B38%28093%29265-70-55" value="+380932657055">+38(093)265-70-55</a><br>
> VITU-RIPE<br>
><br>
</div></div>> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><br><br><br><br>~~~<br>WBR,<br>Vitaliy Turovets<br>Lead Operations Engineer<div>Global Message Services Ukraine<br>+38(093)265-70-55<br>VITU-RIPE<br>
<br></div></div>
</div>