[ClusterLabs] Proper procedure for pacemaker RPM upgrades in active cluster

[Ken Gaillot]

On Wed, 2018-01-17 at 11:20 -0500, Doug Cahill wrote:
> Thank you Ken for the feedback on this.  That doc was exactly what I
> was looking for but missed it in my searches.  I did some testing
> with
> using "crm configure property maintenance-mode=true" and my upgrade
> from 1.1.17 to 1.1.18 worked in my test environment.
> 
> Good info on the 1.1.18 regression fix, thanks.  Will the fixes for
> the regression eventually be put into a specific release under the
> 1.1.x version or will I need to wait for a 2.0 release?  I'm hesitant
> to release a build of the 1.1 upstream into my production.

There will be another 1.1 release eventually. However, there is no more
active development in the 1.1 branch. The only things that will go in
there now are backports of selected bugfixes from the active branches,
so it's at least as safe to use as an official 1.1 release.

> On Mon, Jan 15, 2018 at 6:10 PM, Ken Gaillot <kgaillot at redhat.com>
> wrote:
> > On Mon, 2018-01-15 at 15:42 -0500, Doug Cahill wrote:
> > > Hello,
> > > 
> > > I'm looking for some guidance on pacemaker RPM upgrades in a
> > > running
> > > cluster environment.  I'm looking to automate the process of
> > > upgrading
> > > the RPMs when we decide to plan an upgrade cycle for our
> > > clusters.
> > > 
> > > What I found is that during the RPM upgrade process the
> > > pacemaker.x86_64 RPM will shutdown the pacemaker service.  My
> > > question
> > > regarding this is...is it possible to upgrade the RPM component
> > > but
> > > delay the restart part of the pacemaker service to a later
> > > time?  If
> > > delaying the restart isn't possible, what is the preferred
> > > process
> > > for
> > > people with existing clusters that require package
> > > upgrades?  Should
> > > I
> > > upgrade the passive side first and then fail over to it and then
> > > upgrade the other node which is now passive?  Does pacemaker
> > > support
> > > running two nodes at different version levels during the upgrade
> > > process?  Would enabling maintenance mode be appropriate/ideal
> > > for
> > > this?
> > 
> > Yes to most of those :)
> > 
> > Detailed information about upgrade techniques:
> > 
> > http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-singl
> > e/Pa
> > cemaker_Explained/index.html#_upgrading
> > 
> > Basically, the failover scenario you mentioned is the "rolling
> > upgrade"
> > technique, and the maintenance mode scenario you mentioned is the
> > "detach and reattach" technique.
> > 
> > Each has advantages and disadvantages. A rolling upgrade lets you
> > keep
> > on node on a known working setup as long as possible, while a
> > detach-
> > and-reattach gives you zero downtime (as long the upgrade has no
> > problems ...).
> > 
> > > 
> > > I last experienced this situation when I upgraded from 1.1.15 to
> > > 1.1.17.  Now that pacemaker 1.1.18 is available I'm looking to
> > > plan
> > > this process a little better and would like to know what others
> > > use
> > > as
> > > a procedure.
> > > 
> > > Basic software config:
> > > CentOS 6.x (2.6.32-696.13.2.el6.x86_64)
> > > pacemaker.x86_64       1.1.17-1.el6
> > > corosync.x86_64        2.4.2-1.el6
> > > crmsh.noarch           3.0.1_283-0
> > > Two-node Cluster resources are configured for active/passive
> > > operation.
> > > 
> > > Thanks,
> > > -Doug
> > 
> > On a side note, if you're building 1.1.18 packages yourself, it's a
> > good idea to use the latest upstream 1.1 branch, because it fixes
> > an
> > important regression in 1.1.18.
> > --
> > Ken Gaillot <kgaillot at redhat.com>

-- 
Ken Gaillot <kgaillot at redhat.com>