[ClusterLabs] unable to start mysql as a clustered service, OK stand-alone
bergman at merctech.com
bergman at merctech.com
Tue Aug 9 20:20:55 UTC 2016
I've got a 3-node CentOS6 cluster and I'm trying to add mysql 5.1 as a new service. Other cluster
services (IP addresses, Postgresql, applications) work fine.
The mysql config file and data files are located on shared, cluster-wide storage (GPFS), as are
config/data files for other services which work correctly.
On each node, I can successfully start mysql locally via:
service mysqld start
and via:
rg_test test /etc/cluster/cluster.conf start service mysql
(in each case, the corresponding command with the 'stop' option will also successfully shut down mysql).
However, attempting to start the mysql service with clusvcadm results in the service failing over
from one node to the next, and being marked as "stopped" after the last node.
Each failover happens very quickly, in about 5 seconds. I suspect that rgmanager isn't waiting long
enough for mysql to start before checking if it is running and I have added startup delays in
cluster.conf, but they don't seem to be honored. Nothing is written into the mysql log file at this
time -- no startup or failure messages, which implies that the mysqld never begins to run. The only
log entries (/var/log/messages, /var/log/cluster/*, etc) reference rgmanager, not the mysql process
itself.
Any suggestions?
RHCS components:
cman-3.0.12.1-78.el6.x86_64
luci-0.26.0-78.el6.centos.x86_64
rgmanager-3.0.12.1-26.el6_8.3.x86_64
ricci-0.16.2-86.el6.x86_64
corosync-1.4.7-5.el6.x86_64
--------------------- /etc/cluster/cluster.conf (edited subset) -----------------
<cluster config_version="63" name="example-rhcs">
<rm>
<resources>
<postgres-8 config_file="/var/lib/pgsql/data/postgresql.conf" name="PostgreSQL8" postmaster_user="postgres" startup_wait="25"/>
<ip address="192.168.169.173" sleeptime="10"/>
<mysql config_file="/cluster_shared/mysql_centos6/etc/my.cnf" listen_address="192.168.169.173" name="mysql" shutdown_wait="10" startup_wait="30"/>
</resources>
<service max_restarts="3" name="mysql" recovery="restart" restart_expire_time="180">
<ip ref="192.168.169.173">
<mysql ref="mysql"/>
</ip>
</service>
</rm>
</cluster>
--------------------------------------------------------------------------
--------------------- /var/log/cluster/rgmanager.log from attempt to start mysql with clusvcadm -----------------------
Aug 08 11:58:16 rgmanager Recovering failed service service:mysql
Aug 08 11:58:16 rgmanager [ip] Link for eth2: Detected
Aug 08 11:58:16 rgmanager [ip] Adding IPv4 address 192.168.169.173/24 to eth2
Aug 08 11:58:16 rgmanager [ip] Pinging addr 192.168.169.173 from dev eth2
Aug 08 11:58:18 rgmanager [ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:34 brd ff:ff:ff:ff:ff:ff
Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql
Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed
Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql
Aug 08 11:58:19 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running
Aug 08 11:58:19 rgmanager [mysql] Starting Service mysql:mysql
Aug 08 11:58:19 rgmanager [mysql] Looking For IP Address > Succeed - IP Address Found
Aug 08 11:58:20 rgmanager [mysql] Starting Service mysql:mysql > Succeed
Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql
Aug 08 11:58:21 rgmanager 1 events processed
Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running
Aug 08 11:58:21 rgmanager start on mysql "mysql" returned 7 (unspecified)
Aug 08 11:58:21 rgmanager #68: Failed to start service:mysql; return value: 1
Aug 08 11:58:21 rgmanager Stopping service service:mysql
Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql
Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed
Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql
Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed - File Doesn't Exist
Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql > Succeed
--------------------------------------------------------------------------------
------------------------ rg_test output -----------------------------------------
Running in test mode.
<debug> Link for eth2: Detected
[ip] Link for eth2: Detected
<info> Adding IPv4 address 192.168.169.173/24 to eth2
[ip] Adding IPv4 address 192.168.169.173/24 to eth2
<debug> Pinging addr 192.168.169.173 from dev eth2
[ip] Pinging addr 192.168.169.173 from dev eth2
<debug> Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff
[ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff
<debug> Verifying Configuration Of mysql:mysql
[mysql] Verifying Configuration Of mysql:mysql
<debug> Verifying Configuration Of mysql:mysql > Succeed
[mysql] Verifying Configuration Of mysql:mysql > Succeed
<debug> Monitoring Service mysql:mysql
[mysql] Monitoring Service mysql:mysql
<error> Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
[mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
<error> Monitoring Service mysql:mysql > Service Is Not Running
[mysql] Monitoring Service mysql:mysql > Service Is Not Running
<info> Starting Service mysql:mysql
[mysql] Starting Service mysql:mysql
<debug> Looking For IP Address > Succeed - IP Address Found
[mysql] Looking For IP Address > Succeed - IP Address Found
<debug> Starting Service mysql:mysql > Succeed
[mysql] Starting Service mysql:mysql > Succeed
<debug> Monitoring Service mysql:mysql
[mysql] Monitoring Service mysql:mysql
<debug> Monitoring Service mysql:mysql > Service Is Running
[mysql] Monitoring Service mysql:mysql > Service Is Running
Loading resource rule from /usr/share/cluster/checkquorum
Loading resource rule from /usr/share/cluster/mysql.sh
Loading resource rule from /usr/share/cluster/ip.sh
Loading resource rule from /usr/share/cluster/SAPDatabase
Loading resource rule from /usr/share/cluster/fs.sh
Loading resource rule from /usr/share/cluster/fence_scsi_check.pl
Loading resource rule from /usr/share/cluster/script.sh
Loading resource rule from /usr/share/cluster/fence_scsi_check_hardreboot.pl
Loading resource rule from /usr/share/cluster/samba.sh
Loading resource rule from /usr/share/cluster/oradg.sh
Loading resource rule from /usr/share/cluster/nfsexport.sh
Loading resource rule from /usr/share/cluster/orainstance.sh
Loading resource rule from /usr/share/cluster/service.sh
Loading resource rule from /usr/share/cluster/postgres-8.sh
Loading resource rule from /usr/share/cluster/named.sh
Loading resource rule from /usr/share/cluster/bind-mount.sh
Loading resource rule from /usr/share/cluster/svclib_nfslock
Loading resource rule from /usr/share/cluster/lvm_by_lv.sh
Loading resource rule from /usr/share/cluster/ASEHAagent.sh
Loading resource rule from /usr/share/cluster/netfs.sh
Loading resource rule from /usr/share/cluster/tomcat-6.sh
Loading resource rule from /usr/share/cluster/clusterfs.sh
Loading resource rule from /usr/share/cluster/lvm.sh
Loading resource rule from /usr/share/cluster/openldap.sh
Loading resource rule from /usr/share/cluster/vm.sh
Loading resource rule from /usr/share/cluster/ocf-shellfuncs
Loading resource rule from /usr/share/cluster/lvm_by_vg.sh
Loading resource rule from /usr/share/cluster/SAPInstance
Loading resource rule from /usr/share/cluster/nfsserver.sh
Loading resource rule from /usr/share/cluster/nfsclient.sh
Loading resource rule from /usr/share/cluster/apache.sh
Loading resource rule from /usr/share/cluster/oracledb.sh
Loading resource rule from /usr/share/cluster/oralistener.sh
Starting mysql...
Start of mysql complete
---------------------------------------------------------------------------------
--
Mark Bergman
More information about the Users
mailing list