[ClusterLabs] Mysql upgrade in DRBD setup

[Ken Gaillot]

On Fri, 2017-10-13 at 17:35 +0200, Attila Megyeri wrote:
> Hi Ken, Kristián,
> 
> 
> Thanks - I am familiar with the native replication, and we use that
> as well.
> But in this scenario I have to use DRBD. (There is a DRBD Mysql
> cluster that is a central site, which is replicated to many sites
> using native replication, and all sites have DRBD clusters as well -
> In this setup I have to use DRBD for high availability).
> 
> 
> Anyway - I thought there is a better approach for the DRBD-replicated 
> Mysql than what I outlined.
> What I am concerned about, is what will happen if I upgrade the
> active node (let's say I'm okay with the downtime) - when I fail over
> to the other node, where the program files and the data files are on
> different versions...And when I start upgrading that.
> 
> Any experience anyone?
> 
> @Kristián: my experience shows that If I try to update mysql without
> a mounted data fs - it will fail terribly... So the only option is to
> upgrade the mounted, and active instance - but the issue is the
> version difference (prog vs. data)

Exactly -- which is why I'd still go with native replication for this,
too. It just adds a step in the upgrade process I outlined earlier:
repoint all the other sites' mysql instances to the second central
server after it is upgraded (before or after it is made master, doesn't
matter). I'm assuming only the master is allowed to write.

Another alternative would be to use galera for multi-master (at least
for the two servers at the central site).

Also, it's still possible to use DRBD beneath a native replication
setup, but you'd have to replicate both the master and slave data
(using only one at a time on any given server). This makes more sense
if the mysql servers are running inside VMs or containers that can
migrate between the physical machines.

> 
> Thanks!
> 
> 
> 
> -----Original Message-----
> From: Ken Gaillot [mailto:kgaillot at redhat.com] 
> Sent: Thursday, October 12, 2017 9:22 PM
> To: Cluster Labs - All topics related to open-source clustering
> welcomed <users at clusterlabs.org>
> Subject: Re: [ClusterLabs] Mysql upgrade in DRBD setup
> 
> On Thu, 2017-10-12 at 18:51 +0200, Attila Megyeri wrote:
> > Hi all,
> >  
> > What is the recommended mysql server upgrade methodology in case of
> > an 
> > active/passive DRBD storage?
> > (Ubuntu is the platform)
> 
> If you want to minimize downtime in a MySQL upgrade, your best bet is
> to use MySQL native replication rather than replicate the storage.
> 
> 1. starting point: node1 = master, node2 = slave 2. stop mysql on
> node2, upgrade, start mysql again, ensure OK 3. switch master to
> node2 and slave to node1, ensure OK 4. stop mysql on node1, upgrade,
> start mysql again, ensure OK
> 
> You might have a small window where the database is read-only while
> you switch masters (you can keep it to a few seconds if you arrange
> things well), but other than that, you won't have any downtime, even
> if some part of the upgrade gives you trouble.
> 
> >  
> > 1)      On the passive node the mysql data directory is not
> > mounted, 
> > so the backup fails (some postinstall jobs will attempt to perform 
> > manipulations on certain files in the data directory).
> > 2)      If the upgrade is done on the active node, it will restart
> > the 
> > service (with the service restart, not in a crm managed fassion…), 
> > which is not a very good option (downtime in a HA solution). Not
> > to 
> > mention, that it will update some files in the mysql data
> > directory, 
> > which can cause strange issues if the A/P pair is changed – since
> > on 
> > the other node the program code will still be the old one, while
> > the 
> > data dir is already upgraded.
> >  
> > Any hints are welcome!
> >  
> > Thanks,
> > Attila
> >  
> > _______________________________________________
> > Users mailing list: Users at clusterlabs.org 
> > http://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org Getting started: 
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> > pdf
> > Bugs: http://bugs.clusterlabs.org
> 
> --
> Ken Gaillot <kgaillot at redhat.com>
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org http://lists.clusterlabs.or
> g/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot <kgaillot at redhat.com>