[ClusterLabs] Recovering after split-brain

[Ken Gaillot]

On 06/20/2016 11:33 PM, Nikhil Utane wrote:
> Let me give the full picture about our solution. It will then make it
> easy to have the discussion.
> 
> We are looking at providing N + 1 Redundancy to our application servers,
> i.e. 1 standby for upto N active (currently N<=5). Each server will have
> some unique configuration. The standby will store the configuration of
> all the active servers such that whichever server goes down, the standby
> loads that particular configuration and becomes active. The server that
> went down will now become standby. 
> We have bundled all the configuration that every server has into a
> resource such that during failover the resource is moved to the newly
> active server, and that way it takes up the personality of the server
> that went down. To put it differently, every active server has a
> 'unique' resource that is started by Pacemaker whereas standby has none.
> 
> Our servers do not write anything to an external database, all the
> writing is done to the CIB file under the resource that it is currently
> managing. We also have some clients that connect to the active servers
> (1 client can connect to only 1 server, 1 server can have multiple
> clients) and provide service to end-users. Now the reason I say that
> split-brain is not an issue for us, is coz the clients can only connect
> to 1 of the active servers at any given time (we have to handle the case
> that all clients move together and do not get distributed). So even if
> two servers become active with same personality, the clients can only
> connect to 1 of them. (Initial plan was to go configure quorum but later
> I was told that service availability is of utmost importance and since
> impact of split-brain is limited, we are thinking of doing away with it).
> 
> Now the concern I have is, once the split is resolved, I would have 2
> actives, each having its own view of the resource, trying to synchronize
> the CIB. At this point I want the one that has the clients attached to
> it win.
> I am thinking I can implement a monitor function that can bring down the
> resource if it doesn't find any clients attached to it within a given
> period of time. But to understand the Pacemaker behavior, what exactly
> would happen if the same resource is found to be active on two nodes
> after recovery?
> 
> -Thanks
> Nikhil

In general, monitor actions should not change the state of the service
in any way.

Pacemaker's behavior when finding multiple instances of a resource
running when there should be only one is configurable via the
multiple-active property:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_resource_meta_attributes

By default, it stops all the instances, and then starts one instance.
The alternatives are to stop all the instances and leave them stopped,
or to unmanage the resource (i.e. refuse to stop or start it).