[ClusterLabs] Antw: Salvaging aborted resource migration

[Ken Gaillot]

On Thu, 2018-09-27 at 18:00 +0200, Ferenc Wágner wrote:
> Ken Gaillot <kgaillot at redhat.com> writes:
> 
> > On Thu, 2018-09-27 at 09:36 +0200, Ulrich Windl wrote:
> > 
> > > Obviously you violated the most important cluster rule that is
> > > "be
> > > patient".  Maybe the next important is "Don't change the
> > > configuration while the cluster is not in IDLE state" ;-)
> > 
> > Agreed -- although even idle, removing a ban can result in a
> > migration
> > back (if something like stickiness doesn't prevent it).
> 
> I've got no problem with that in general.  However, I can't gurantee
> that every configuration change happens in idle state, certain
> operations (mostly resource additions) are done by several
> administrators without synchronization, and of course asynchronous
> cluster events can also happen any time.  So I have to ask: what are
> the
> consequences of breaking this "impossible" rule?

It's not truly a rule, just a "better safe than sorry" approach. In
general the cluster is very forgiving about frequent config changes
from any node. The only non-obvious consequence is that if the cluster
is still making changes based on the previous config, then it will wait
for any action already in progress to complete, then abandon that and
recalculate based on the new config, which might reverse actions just
taken.

> > There's currently no way to tell pacemaker that an operation (i.e.
> > migrate_from) is a no-op and can be ignored. If a migration is only
> > partially completed, it has to be considered a failure and
> > reverted.
> 
> OK.  Are there other complex operations which can "partially
> complete"
> if a transition is aborted by some event?

I believe migration is the only one in that category. Perhaps a restart
could be considered similar, as it involves a separate stop and start,
but a completed stop doesn't have to be reversed in that case, so that
wouldn't cause any similar issues.

> Now let's suppose a pull migration scenario: migrate_to does nothing,
> but in this tiny window a configuration change aborts the transition.
> The resources would go through a full recovery (stop+start), right?

Yes

> Now let's suppose migrate_from gets scheduled and starts performing
> the
> migration.  Before it finishes, a configuration change aborts the
> transition.  The cluster waits for the outstanding operation to
> finish,
> doesn't it?  And if it finishes successfully, is the migration
> considered complete requiring no recovery?

Correct. If an agent has actually been executed, the cluster will wait
for that operation to complete or timeout before recalculating.

(As an aside, that can cause problems of a different sort: if an
operation in progress has a very long timeout and takes that whole
time, it can delay recovery of other resources that newly fail, even if
their recovery would not depend on the outcome of that operation.
That's a complicated problem to solve because that last clause is not
obvious to a computer program without simulating all possible results,
and even then, it can't be sure that the operation won't do something
like change a node attribute that might affect other resources.)

> > I'm not sure why the reload was scheduled; I suspect it's a bug due
> > to
> > a restart being needed but no parameters having changed. There
> > should
> > be special handling for a partial migration to make the stop
> > required.
> 
> Probably CLBZ#5309 again...  You debugged a pe-input file for me with
> a
> similar issue almost exactly a year ago (thread subject "Pacemaker
> resource parameter reload confusion").  Time to upgrade this cluster,
> I
> guess.
-- 
Ken Gaillot <kgaillot at redhat.com>