[ClusterLabs] mayhem when exiting maintenance mode

[Jehan-Guillaume de Rorthais]

On Tue, 4 Feb 2020 08:39:04 -0600
Derek Viljoen <derekv at infinite.io> wrote:

> We have a three-node postgres cluster running on Ubuntu 14.04, currently at
> Postgres 9.5 with Corosync 2.4.2 and Pacemaker 1.1.18.

Are you using the default pgsql resource agent?

> I'm trying to automate upgrading the database to 11.4.  (Our product is a
> network appliance, so it needs to be automated for our customers)
> 
> I first put the cluster into maintenance mode, perform the upgrade, update
> the resource paths in the crm config to point to the new db instance,

I suppose this is what requires all your pgsql the resource to restart. 

The default pgsql resource agent doesn't seem to support the reload action, so
the cluster must restart the resource:

https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Pacemaker_Explained/#_reloading_services_after_a_definition_change

After your upgrade:

* make sure standbys are up to date
* gracefully stop old and new instances everywhere
* switch paths
* start your new pgsql instances
* make sure the new primary is on the same node than the previous one
* check with crm_simulate -SL what would happen in normal situation
* exit maintenance mode.

Regards,