[ClusterLabs] mayhem when exiting maintenance mode

[Strahil Nikolov]

 Hi Derek,
have you run a simulation via crm_simulate before that ? Usually it indicates what will happen ,when you remove the maintenance.
What comes first to my mind is:1. Are you abe to do a rolling upgrade?2. When you remove the maintenance , do you have a postgres DB in master mode ? Is it on the same node it was before the maintenance ?
I have the feeling that you are starting the postgres afterwards, but there is no DB in master mode.
3. Have you checked the logs about any indication ? Usually lrmd indicates local (for the node ) resource issue, crmd - global one.
Keep in mind that the current DC keeps info for all nodes - so you should start from there.
Best Regards,Strahil Nikolov
    В вторник, 4 февруари 2020 г., 16:53:49 ч. Гринуич+2, Derek Viljoen <derekv at infinite.io> написа:  
 
 We have a three-node postgres cluster running on Ubuntu 14.04, currently at Postgres 9.5 with Corosync 2.4.2 and Pacemaker 1.1.18.
I'm trying to automate upgrading the database to 11.4.  (Our product is a network appliance, so it needs to be automated for our customers)
I first put the cluster into maintenance mode, perform the upgrade, update the resource paths in the crm config to point to the new db instance, restore the db from the old version (required by postgres to do major version upgrades).  At the end of all these steps everything looks good.
But when I turn off maintenance mode all of my db nodes suddenly go down and all three appear to be in slave mode, with no master.  If I wait a few minutes it appears that node 2 takes over as master, but it has an empty database, because apparently it wasn't able to replicate the restored db from the original master yet.  Can anyone tell me what is causing this?
Derek Viljoenderekv at infinite.io_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users

ClusterLabs home: https://www.clusterlabs.org/  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20200204/bdba9631/attachment.html>