[ClusterLabs] Can't See Why This Cluster Failed Over

[Eric Robinson]

>> Apr 03 08:50:40 [22767] ha14a       crmd:   notice: process_lrm_event:  ha14a-p_mysql_746_start_0:148 [ starting mysqld... \ngave up waiting\n ]

> This is where your trouble starts. p_mysql_746 fails to start; the init
> script returned exit code 1, and showed the output in the above line.

A second set of eyes does wonders. I totally missed that!

> I can't see the full configuration, but I'm guessing the colocation
> added with the new resource means other resources now depend on it, so
> they get stopped. Perhaps you got the sense of the colocation backwards,
> i.e. you wanted the new resource to be placed with the old ones, rather
> than vice versa?

I think I may have something wrong in the config. My goal is:

VIP --> depends on --> Filesystem --> depends on --> LVM --> depends on --> drbd

p_mysql_057  --> depends on --> VIP
p_mysql_092 --> depends on --> VIP 
p_mysql_187 --> depends on --> VIP
etc...

Moving VIP, Filesystem, LVP, or drbd moves the whole stack. 

Each MySQL instance should be able to stop or start without impacting the others.

Here's the config. I don't know why the CRM put in the parenthesis where it did. That's not the way I typed it. I usually have all my mysql instances between parenthesis and everything else outside.


node ha14a
node ha14b
primitive p_drbd0 ocf:linbit:drbd \
        params drbd_resource=ha01_mysql \
        op monitor interval=31s role=Slave \
        op monitor interval=30s role=Master
primitive p_drbd1 ocf:linbit:drbd \
        params drbd_resource=ha02_mysql \
        op monitor interval=29s role=Slave \
        op monitor interval=28s role=Master
primitive p_fs_clust19 Filesystem \
        params device="/dev/vg_on_drbd0/lv_on_drbd0" directory="/ha01_mysql" fstype=ext4 options=noatime \
        meta target-role=Started
primitive p_fs_clust20 Filesystem \
        params device="/dev/vg_on_drbd1/lv_on_drbd1" directory="/ha02_mysql" fstype=ext4 options=noatime
primitive p_lv_on_drbd0 LVM \
        params volgrpname=vg_on_drbd0 \
        meta target-role=Started
primitive p_lv_on_drbd1 LVM \
        params volgrpname=vg_on_drbd1
primitive p_mysql_057 lsb:mysql_057
primitive p_mysql_092 lsb:mysql_092
primitive p_mysql_187 lsb:mysql_187
primitive p_mysql_213 lsb:mysql_213
primitive p_mysql_250 lsb:mysql_250
primitive p_mysql_289 lsb:mysql_289
primitive p_mysql_312 lsb:mysql_312
primitive p_mysql_702 lsb:mysql_702
primitive p_mysql_743 lsb:mysql_743
primitive p_mysql_745 lsb:mysql_745
primitive p_mysql_746 lsb:mysql_746
primitive p_vip_clust19 IPaddr2 \
        params ip=192.168.9.106 cidr_netmask=32 \
        op monitor interval=30s \
        meta target-role=Started
primitive p_vip_clust20 IPaddr2 \
        params ip=192.168.9.107 cidr_netmask=32 \
        op monitor interval=29s
ms ms_drbd0 p_drbd0 \
        meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true target-role=Master
ms ms_drbd1 p_drbd1 \
        meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true target-role=Master
colocation c_clust19 inf: ( p_mysql_057 p_mysql_092 p_mysql_187 p_mysql_213 p_mysql_250 p_mysql_289 p_mysql_312 p_vip_clust19 p_mysql_702 p_mysql_743 p_mysql_745 p_mysql_746 p_fs_clust19 p_lv_on_drbd0 ) ( ms_drbd0:Master )
colocation c_clust20 inf: p_vip_clust20 p_fs_clust20 p_lv_on_drbd1 ms_drbd1:Master
order o_clust19 inf: ms_drbd0:promote ( p_lv_on_drbd0:start ) ( p_fs_clust19 p_vip_clust19 ) ( p_mysql_057 p_mysql_092 p_mysql_187 p_mysql_213 p_mysql_250 p_mysql_289 p_mysql_312 p_mysql_702 p_mysql_743 p_mysql_745 p_mysql_746 )
order o_clust20 inf: ms_drbd1:promote p_lv_on_drbd1 p_fs_clust20 p_vip_clust20
property cib-bootstrap-options: \
        dc-version=1.1.11-97629de \
        cluster-infrastructure="classic openais (with plugin)" \
        expected-quorum-votes=2 \
        no-quorum-policy=ignore \
        stonith-enabled=false \
        maintenance-mode=false \
        last-lrm-refresh=1484558149