<div dir="ltr">I faced with the same problem a few years ago - we needed to make a two-node cluster working in a "split-brain" situation. We were looking at a resource agent called SFEX which is disk based - <a href="http://www.linux-ha.org/wiki/Sfex_(resource_agent)" target="_blank">http://www.linux-ha.org/wiki/<wbr>Sfex_(resource_agent)</a> . At the end we rejected SFEX because, if I am not mistaken, it uses only one drive. Therefore it becomes a single point of failure, if the drive fails - all resources stop. You will probably face with the same problem inventing this kind of resource (not disk, but network based).<div><br></div><div>We ended up not using SFEX, but instead made good fencing and redundant links (BTW, we used "teamd"). The only problem we've got is stonith ping-pong. We solved that by making a "stonith ping-pong detector" logic.</div><div><br></div><div>I am not sure this is really helpful for you, but just in case I decided to share my experience. </div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Thank you,<div>Kostia</div></div></div></div></div></div>
<br><div class="gmail_quote">On Sun, Oct 9, 2016 at 11:33 PM, Eric Robinson <span dir="ltr"><<a href="mailto:eric.robinson@psmnv.com" target="_blank">eric.robinson@psmnv.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<font face="Calibri" size="2"><span style="font-size:11pt">
<div>I’ve been working on a script for preventing split-brain in 2-node clusters and I would appreciate comments from everyone. If someone already has a solution like this, let me know!</div>
<div> </div>
<div>Most of my database clusters are 2-nodes, with each node in a geographically separate data center. Our layout looks like the following diagram. Each server node has three physical connections to the world. LANs A, B , C, D are all physically separate cable
plants and cross-connects between the data centers (using different switches, routers, power, fiber paths, etc.). This is to ensure maximum cluster communication intelligence. LANs A and B (Corosync ring 0) are bonded at the NICs, as are LANs C and D (Corosync
ring 1). </div>
<div> </div>
<div>Hopefully this diagram will come through intact…</div>
<div> </div>
<div> </div>
<div> </div>
<div><font face="Consolas"> +----------------+</font></div>
<div><font face="Consolas"> | |</font></div>
<div><font face="Consolas"> | Third party |</font></div>
<div><font face="Consolas"> | Web Hosting |</font></div>
<div><font face="Consolas"> +---+--------+---+</font></div>
<div><font face="Consolas"> | |</font></div>
<div><font face="Consolas"> | |</font></div>
<div><font face="Consolas"> | |</font></div>
<div><font face="Consolas"> | |</font></div>
<div><font face="Consolas"> | |</font></div>
<div><font face="Consolas"> | |</font></div>
<div><font face="Consolas"> ++XX |</font></div>
<div><font face="Consolas"> XXX XXXXXX+-+XXX</font></div>
<div><font face="Consolas"> XX XX XXX</font></div>
<div><font face="Consolas"> XXXXXXX XX</font></div>
<div><font face="Consolas"> XXXX XX X</font></div>
<div><font face="Consolas"> X XXX</font></div>
<div><font face="Consolas"> +--------+ The Interwebs XXX+-----+</font></div>
<div><font face="Consolas"> | XXX X |</font></div>
<div><font face="Consolas"> | XX XX |</font></div>
<div><font face="Consolas"> | X XX |</font></div>
<div><font face="Consolas"> | X XXXX XXXXXXXXXXX |</font></div>
<div><font face="Consolas"> | XXXXXX XX XX |</font></div>
<div><font face="Consolas"> | XXXXXXX |</font></div>
<div><font face="Consolas"> | <wbr> |</font></div>
<div><font face="Consolas"> | Internet <wbr> | Internet</font></div>
<div><font face="Consolas"> | <wbr> |</font></div>
<div><font face="Consolas"> | <wbr> |</font></div>
<div><font face="Consolas"> | <wbr> |</font></div>
<div><font face="Consolas"> | LAN A |</font></div>
<div><font face="Consolas"> | +-----------------------------<wbr>------+ |</font></div>
<div><font face="Consolas"> | | LAN B | |</font></div>
<div><font face="Consolas"> | | +---------------------------+ <wbr> | |</font></div>
<div><font face="Consolas"> | | | | | |</font></div>
<div><font face="Consolas">+---+---+---+----+ <wbr> +-----+---+---+--+</font></div>
<div><font face="Consolas">| | | |</font></div>
<div><font face="Consolas">| Node 1 | | Node 2 |</font></div>
<div><font face="Consolas">| | | |</font></div>
<div><font face="Consolas">+------+---+-----+ <wbr> +-----+---+------+</font></div>
<div><font face="Consolas"> | | LAN C | |</font></div>
<div><font face="Consolas"> | +----------------------------+<wbr> |</font></div>
<div><font face="Consolas"> | LAN D |</font></div>
<div><font face="Consolas"> +-----------------------------<wbr>-------+</font></div>
<div> </div>
<div> </div>
<div> </div>
<div>Even with all that connectivity it is possible that something could happen to interrupt communication between the 2 data centers, or the connectivity been 1 of the data centers and the Internet, and split brain would result. I have been working on a way
to prevent this using a concept I call a “dead drop.” This idea takes its name from the spy world, where spies cannot communicate directly, but they are able to pass simple information and status messages to each other by using a blind drop in a previously
agreed location. Spy X makes a mark on a tree. Later, spy Y comes by and sees the mark, and knows that spy X is okay. He leaves a mark of his own on the tree, and later spy X sees it and knows that spy Y is okay. Neither spy owns the tree or the land it is
on. </div>
<div> </div>
<div>The same idea applies here. Suppose all direct TCP/IP connectivity were to be severed between Nodes 1 and 2, but both of them are still able to reach the Internet. Normally, split brain would result. But SUPPOSE they were both running scripts that use
curl requests to post and retrieve simple status messages to and from a third party web host. In other words, even though the nodes cannot talk to each other directly, they can still leave messages at a dead drop location for each other to read. If Node 2 was
in standby mode, normally it would switch to primary. However, if it checks the dead drop and sees a message from Node 1 that says, “I’m still okay and communicating with customers.” Then Node 2 knows not to become cluster primary. This script could possibly
be implemented as a cluster resource, with most other resources dependent on it.</div>
<div> </div>
<div>The dead drop needs no intelligence other than the ability to read and write simple text files, and it can run on any third-party web host (or on multiple web sites). It does not fill the role of a quorum or arbitrator. The 2 Nodes themselves remain in
control of their own failover decisions. </div>
<div> </div>
<div>I’m SURE this has been attempted already and I don’t want to re-invent the wheel, but I have not seen this approach anywhere. Maybe there’s a good reason for that because it simply won’t work? The arbitration solutions I have seen all rely on a third machine
that plays a complex role in arbitration. </div>
<div> </div>
<div>Thoughts?</div>
<div> </div>
<div>--</div>
<div>Eric Robinson</div>
<div> </div>
<div> </div>
<div> </div>
</span></font>
</div>
<br>______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br></blockquote></div><br></div>