[ClusterLabs] Antw: Re: questions about startup fencing

[Jehan-Guillaume de Rorthais]

On Tue, 5 Dec 2017 10:05:03 +0100
Tomas Jelinek <tojeline at redhat.com> wrote:

> Dne 4.12.2017 v 17:21 Jehan-Guillaume de Rorthais napsal(a):
> > On Mon, 4 Dec 2017 16:50:47 +0100
> > Tomas Jelinek <tojeline at redhat.com> wrote:
> >   
> >> Dne 4.12.2017 v 14:21 Jehan-Guillaume de Rorthais napsal(a):  
> >>> On Mon, 4 Dec 2017 12:31:06 +0100
> >>> Tomas Jelinek <tojeline at redhat.com> wrote:
> >>>      
> >>>> Dne 4.12.2017 v 10:36 Jehan-Guillaume de Rorthais napsal(a):  
> >>>>> On Fri, 01 Dec 2017 16:34:08 -0600
> >>>>> Ken Gaillot <kgaillot at redhat.com> wrote:
> >>>>>         
> >>>>>> On Thu, 2017-11-30 at 07:55 +0100, Ulrich Windl wrote:  
> >>>>>>>
> >>>>>>>            
> >>>>>>>> Kristoffer Gronlund <kgronlund at suse.com> wrote:  
> >>>>>>>>> Adam Spiers <aspiers at suse.com> writes:
> >>>>>>>>>            
> >>>>>>>>>> - The whole cluster is shut down cleanly.
> >>>>>>>>>>
> >>>>>>>>>> - The whole cluster is then started up again.  (Side question:
> >>>>>>>>>> what
> >>>>>>>>>>      happens if the last node to shut down is not the first to
> >>>>>>>>>> start up?
> >>>>>>>>>>      How will the cluster ensure it has the most recent version of
> >>>>>>>>>> the
> >>>>>>>>>>      CIB?  Without that, how would it know whether the last man
> >>>>>>>>>> standing
> >>>>>>>>>>      was shut down cleanly or not?)  
> >>>>>>>>>
> >>>>>>>>> This is my opinion, I don't really know what the "official"
> >>>>>>>>> pacemaker
> >>>>>>>>> stance is: There is no such thing as shutting down a cluster
> >>>>>>>>> cleanly. A
> >>>>>>>>> cluster is a process stretching over multiple nodes - if they all
> >>>>>>>>> shut
> >>>>>>>>> down, the process is gone. When you start up again, you
> >>>>>>>>> effectively have
> >>>>>>>>> a completely new cluster.  
> >>>>>>>>
> >>>>>>>> Sorry, I don't follow you at all here.  When you start the cluster
> >>>>>>>> up
> >>>>>>>> again, the cluster config from before the shutdown is still there.
> >>>>>>>> That's very far from being a completely new cluster :-)  
> >>>>>>>
> >>>>>>> The problem is you cannot "start the cluster" in pacemaker; you can
> >>>>>>> only "start nodes". The nodes will come up one by one. As opposed (as
> >>>>>>> I had said) to HP Sertvice Guard, where there is a "cluster formation
> >>>>>>> timeout". That is, the nodes wait for the specified time for the
> >>>>>>> cluster to "form". Then the cluster starts as a whole. Of course that
> >>>>>>> only applies if the whole cluster was down, not if a single node was
> >>>>>>> down.  
> >>>>>>
> >>>>>> I'm not sure what that would specifically entail, but I'm guessing we
> >>>>>> have some of the pieces already:
> >>>>>>
> >>>>>> - Corosync has a wait_for_all option if you want the cluster to be
> >>>>>> unable to have quorum at start-up until every node has joined. I don't
> >>>>>> think you can set a timeout that cancels it, though.
> >>>>>>
> >>>>>> - Pacemaker will wait dc-deadtime for the first DC election to
> >>>>>> complete. (if I understand it correctly ...)
> >>>>>>
> >>>>>> - Higher-level tools can start or stop all nodes together (e.g. pcs has
> >>>>>> pcs cluster start/stop --all).  
> >>>>>
> >>>>> Based on this discussion, I have some questions about pcs:
> >>>>>
> >>>>> * how is it shutting down the cluster when issuing "pcs cluster stop
> >>>>> --all"?  
> >>>>
> >>>> First, it sends a request to each node to stop pacemaker. The requests
> >>>> are sent in parallel which prevents resources from being moved from node
> >>>> to node. Once pacemaker stops on all nodes, corosync is stopped on all
> >>>> nodes in the same manner.  
> >>>
> >>> What if for some external reasons one node is slower (load, network,
> >>> whatever) than the others and start reacting ? Sending queries in parallel
> >>> doesn't feels safe enough in regard with all the race conditions that can
> >>> occurs in the same time.
> >>>
> >>> Am I missing something ?
> >>>      
> >>
> >> If a node gets the request later than others, some resources may be
> >> moved to it before it starts shutting down pacemaker as well. Pcs waits
> >> for all nodes to shutdown pacemaker before it moves to shutting down
> >> corosync. This way, quorum is maintained the whole time pacemaker is
> >> shutting down and therefore no services are blocked from stopping due to
> >> lack of quorum.  
> > 
> > OK, so if admins or RA expect to start in, the same conditions the cluster
> > was shut downed, we have to take care of the shutdown ourselves by hands.
> > Considering disabling the resource before shutting down might be the best
> > option in the situation as the CRM will take care of switching off things
> > correctly in a proper transition.  
> 
> My understanding is that pacemaker takes care of switching off things 
> correctly in a proper transition on its shutdown. So there should be no 
> extra care needed. Pacemaker developers, however, need to confirm that.

Sure, but then, the resource would move away from the node if some other
node(s) (with appropriate constraints) are up when the transition is computed.

If you are looking at master resources, this could raise a lot of master score
changes during the cluster shutdown for no reasons.