<html><head></head><body><div class="ydp2d20f1f6yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:10px;"><div dir="ltr" data-setdir="false">I've been testing an implementation of a HA mysql cluster for a few months now. I came to this project with no preior knoweldge of what was copncerned/needed and have learned orgainscally via various online how-tos and web sites which many cases wrere slightly out-of-date to missing large chunks of perinent information. Thats not a criticism at all of those still helpful aids, but more an indication of how there are huge holes in my knowledge..<br><br>So with that background ...<br><br>The cluster consits of 2 centos7 servers (esterla and rafeiro) running <br>DRBD90<br><div>corosync 2.4.5</div><div dir="ltr" data-setdir="false">pacemaker 0.9.169</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">On the whole its all running fine with some squeaks that we are hoping are down to underlying SAN issues.<br><br> <div dir="ltr" data-setdir="false"><span>However...</span></div><div dir="ltr" data-setdir="false"><span><br></span></div><div dir="ltr" data-setdir="false"><div><span>earlier this week we had some split-brain issues - some of which seem to have fixed themselves, others not. What we</span></div><div dir="ltr" data-setdir="false"> did notice that whilst the split-brain was being reported the overall cluster remained up (of course?) in that the VIP remained up, abnd the mysql instance remained abvailavle via the VIP on port 3306. The underlying coincern being of course that had a "flip" occurred from previous master to the previous slave, the new master's drbd device (moun ted on /var/lib/mysql) may well be out of sync and thus contain "old" data.<br><br><div dir="ltr" data-setdir="false">So - system logs recently show this<br><br><div dir="ltr" data-setdir="false">ESTRELA</div>Oct 18th<br><div dir="ltr" data-setdir="false">Oct 18 04:04:28 wp-vldyn-estrela kernel: [584651.491139] drbd mysql01/0 drbd0: Split-Brain detected, 1 primaries, automatically solved. Sync from peer node<br>Oct 18 04:04:28 wp-vldyn-estrela kernel: [584651.491139] drbd mysql01/0 drbd0: Split-Brain detected, 1 primaries, automatically solved. Sync from peer node<br><br>Oct 19th<br><div dir="ltr" data-setdir="false">Oct 19 03:45:43 wp-vldyn-estrela kernel: [47892.092191] drbd mysql01/0 drbd0: Split-Brain detected but unresolved, dropping connection!<br>Oct 19 03:45:43 wp-vldyn-estrela kernel: [47892.092191] drbd mysql01/0 drbd0: Split-Brain detected but unresolved, dropping connection!<br><br><br><div dir="ltr" data-setdir="false">RAFEIRO<br><div dir="ltr" data-setdir="false"><div>Oct 18<br></div>Oct 18 04:04:28 wp-vldyn-rafeiro kernel: [584652.907126] drbd mysql01/0 drbd0: Split-Brain detected, 1 primaries, automatically solved. Sync from this node<br>Oct 18 04:04:28 wp-vldyn-rafeiro kernel: [584652.907126] drbd mysql01/0 drbd0: Split-Brain detected, 1 primaries, automatically solved. Sync from this node<br><br><div dir="ltr" data-setdir="false">Oct 19<br><div>Oct 19 03:45:43 wp-vldyn-rafeiro kernel: [47864.401284] drbd mysql01/0 drbd0: Split-Brain detected but unresolved, dropping connection!<br>Oct 19 03:45:43 wp-vldyn-rafeiro kernel: [47864.401284] drbd mysql01/0 drbd0: Split-Brain detected but unresolved, dropping connection!<br><br></div><div><br></div><br></div></div></div>So on the 18th the split-brain issues was detected but (automatically?) fixed.<br>But on the 19th it wasnt...<br><br>Any ideas how to investigate why it worked on the 18th and not the 19th? I am presuming the drbd config is set up to automatically fix stuff but maybe we just got lucky and it isnt? (Ive googled automatic fixes but I am afarid I cant follow what Im being told/reading :-( )<br><br><div>drbd config below</div><div><br></div><div dir="ltr" data-setdir="false">ta</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">ian<br><br>==================<br></div></div></div><div dir="ltr" data-setdir="false"><div dir="ltr" data-setdir="false"><div dir="ltr" data-setdir="false">ESTRELA</div>resource mysql01 {<br> protocol C;<br> meta-disk internal;<br> device /dev/drbd0;<br> disk /dev/vg_mysql/lv_mysql;<br> handlers {<br> split-brain "/usr/lib/drbd/notify-split-brain.sh root";<br> }<br> net {<br> allow-two-primaries no;<br> after-sb-0pri discard-zero-changes;<br> after-sb-1pri discard-secondary;<br> after-sb-2pri disconnect;<br> rr-conflict disconnect;<br> }<br> disk {<br> on-io-error detach;<br> }<br> syncer {<br> verify-alg sha1;<br> }<br> on estrela {<br> address 10.108.248.165:7789;<br> }<br> on rafeiro {<br> address 10.108.248.166:7789;<br> }<br>}<br><br></div><div><br></div></div><br>RAFEIRO<br><div>resource mysql01 {<br> protocol C;<br> meta-disk internal;<br> device /dev/drbd0;<br> disk /dev/vg_mysql/lv_mysql;<br> handlers {<br> split-brain "/usr/lib/drbd/notify-split-brain.sh root";<br> }<br> net {<br> allow-two-primaries no;<br> after-sb-0pri discard-zero-changes;<br> after-sb-1pri discard-secondary;<br> after-sb-2pri disconnect;<br> rr-conflict disconnect;<br> }<br> disk {<br> on-io-error detach;<br> }<br> syncer {<br> verify-alg sha1;<br> }<br> on estrela {<br> address 10.108.248.165:7789;<br> }<br> on rafeiro {<br> address 10.108.248.166:7789;<br> }<br>}<br><br></div><div><br></div></div><br><span></span></div><span><br><br></span></div></div><br></div></div></body></html>