[ClusterLabs] I've been working on a split-brain prevention strategy for 2-node clusters.
Kostiantyn Ponomarenko
konstantin.ponomarenko at gmail.com
Sun Oct 9 23:16:18 CEST 2016
I faced with the same problem a few years ago - we needed to make a
two-node cluster working in a "split-brain" situation. We were looking at a
resource agent called SFEX which is disk based -
http://www.linux-ha.org/wiki/Sfex_(resource_agent) . At the end we rejected
SFEX because, if I am not mistaken, it uses only one drive. Therefore it
becomes a single point of failure, if the drive fails - all resources stop.
You will probably face with the same problem inventing this kind of
resource (not disk, but network based).
We ended up not using SFEX, but instead made good fencing and redundant
links (BTW, we used "teamd"). The only problem we've got is stonith
ping-pong. We solved that by making a "stonith ping-pong detector" logic.
I am not sure this is really helpful for you, but just in case I decided to
share my experience.
Thank you,
Kostia
On Sun, Oct 9, 2016 at 11:33 PM, Eric Robinson <eric.robinson at psmnv.com>
wrote:
> I’ve been working on a script for preventing split-brain in 2-node
> clusters and I would appreciate comments from everyone. If someone already
> has a solution like this, let me know!
>
> Most of my database clusters are 2-nodes, with each node in a
> geographically separate data center. Our layout looks like the following
> diagram. Each server node has three physical connections to the world. LANs
> A, B , C, D are all physically separate cable plants and cross-connects
> between the data centers (using different switches, routers, power, fiber
> paths, etc.). This is to ensure maximum cluster communication intelligence.
> LANs A and B (Corosync ring 0) are bonded at the NICs, as are LANs C and D
> (Corosync ring 1).
>
> Hopefully this diagram will come through intact…
>
>
>
> +----------------+
> | |
> | Third party |
> | Web Hosting |
> +---+--------+---+
> | |
> | |
> | |
> | |
> | |
> | |
> ++XX |
> XXX XXXXXX+-+XXX
> XX XX XXX
> XXXXXXX XX
> XXXX XX X
> X XXX
> +--------+ The Interwebs XXX+-----+
> | XXX X |
> | XX XX |
> | X XX |
> | X XXXX XXXXXXXXXXX |
> | XXXXXX XX XX |
> | XXXXXXX |
> | |
> | Internet | Internet
> | |
> | |
> | |
> | LAN A |
> | +-----------------------------------+ |
> | | LAN B | |
> | | +---------------------------+ | |
> | | | | | |
> +---+---+---+----+ +-----+---+---+--+
> | | | |
> | Node 1 | | Node 2 |
> | | | |
> +------+---+-----+ +-----+---+------+
> | | LAN C | |
> | +----------------------------+ |
> | LAN D |
> +------------------------------------+
>
>
>
> Even with all that connectivity it is possible that something could happen
> to interrupt communication between the 2 data centers, or the connectivity
> been 1 of the data centers and the Internet, and split brain would result.
> I have been working on a way to prevent this using a concept I call a “dead
> drop.” This idea takes its name from the spy world, where spies cannot
> communicate directly, but they are able to pass simple information and
> status messages to each other by using a blind drop in a previously agreed
> location. Spy X makes a mark on a tree. Later, spy Y comes by and sees the
> mark, and knows that spy X is okay. He leaves a mark of his own on the
> tree, and later spy X sees it and knows that spy Y is okay. Neither spy
> owns the tree or the land it is on.
>
> The same idea applies here. Suppose all direct TCP/IP connectivity were to
> be severed between Nodes 1 and 2, but both of them are still able to reach
> the Internet. Normally, split brain would result. But SUPPOSE they were
> both running scripts that use curl requests to post and retrieve simple
> status messages to and from a third party web host. In other words, even
> though the nodes cannot talk to each other directly, they can still leave
> messages at a dead drop location for each other to read. If Node 2 was in
> standby mode, normally it would switch to primary. However, if it checks
> the dead drop and sees a message from Node 1 that says, “I’m still okay and
> communicating with customers.” Then Node 2 knows not to become cluster
> primary. This script could possibly be implemented as a cluster resource,
> with most other resources dependent on it.
>
> The dead drop needs no intelligence other than the ability to read and
> write simple text files, and it can run on any third-party web host (or on
> multiple web sites). It does not fill the role of a quorum or arbitrator.
> The 2 Nodes themselves remain in control of their own failover decisions.
>
> I’m SURE this has been attempted already and I don’t want to re-invent the
> wheel, but I have not seen this approach anywhere. Maybe there’s a good
> reason for that because it simply won’t work? The arbitration solutions I
> have seen all rely on a third machine that plays a complex role in
> arbitration.
>
> Thoughts?
>
> --
> Eric Robinson
>
>
>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://clusterlabs.org/pipermail/users/attachments/20161010/b7ed4897/attachment.html>
More information about the Users
mailing list