Hi,<br><br>I have two nodes rspa and rspa2 (both Centos 5.3 32bits) with the following packages:<br><br>drbd83-8.3.8-1.el5.centos<br>heartbeat-3.0.3-2.3.el5<br>pacemaker-1.0.10-1.4.el5<br><br>rspa is stopped, and rspa2 has all the resources (IP, FileSystem, Mysql, Apache and DRBD Master)<br>
When I start heatbeat on rspa, for some reason (I don't have any resource_location specified) it tries to move all resources to that node, but when trying to demote drbd on rspa2 (node2) and promote drbd on rspa (node1) something must go wrong as my DRBD partition (being used by MySQL) gets unresponsive.<br>
<br>Next it stops Apache (works), and tries to stop MySQL which fails because it uses the unresponsive partition.<br>As a result of this my high availability cluster ends up in the limbo; it doesn't migrate to node1, neither to node2.<br>
<br>Any help is welcome here...<br><br> [root@rspa2 ~]# crm status<br>============<br>Last updated: Tue Dec 21 18:12:47 2010<br>Stack: Heartbeat<br>Current DC: <a href="http://rspa2.sadiel.es">rspa2.sadiel.es</a> (2680c85b-7e6c-4610-88b2-510feb60c4b4) - partition with quorum<br>
Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3<br>2 Nodes configured, 2 expected votes<br>2 Resources configured.<br>============<br><br>Online: [ rspa2.domain rspa.domain ]<br><br> Resource Group: mysql<br> fs_mysql (ocf::heartbeat:Filesystem): Started rspa2.domain<br>
ip_mysql (ocf::heartbeat:IPaddr2): Started rspa2.domain<br> mysqld (lsb:mysqld): Started rspa2.domain (unmanaged) FAILED<br> apache (lsb:httpd): Stopped <br> Master/Slave Set: ms_drbd_mysql<br>
Masters: [ rspa2.domain ]<br> Slaves: [ rspa.domain ]<br><br>Failed actions:<br> mysqld_stop_0 (node=rspa2.domain, call=18, rc=-2, status=Timed Out): unknown exec error<br><br>Please see my Pacemaker config:<br>
<br>node $id="2680c85b-7e6c-4610-88b2-510feb60c4b4" rspa2.domain \<br> attributes standby="off"<br>node $id="f9be4a80-ec2a-42e3-8d86-62dd050b437b" rspa.domain \<br> attributes standby="off"<br>
primitive apache lsb:httpd<br>primitive drbd_mysql ocf:linbit:drbd \<br> params drbd_resource="r0" \<br> op monitor interval="15s" \<br> op monitor interval="16s" role="Master"<br>
primitive fs_mysql ocf:heartbeat:Filesystem \<br> params device="/dev/drbd0" directory="/opt/drbd/" fstype="xfs"<br>primitive ip_mysql ocf:heartbeat:IPaddr2 \<br> params ip="172.18.2.150" nic="eth0:1"<br>
primitive mysqld lsb:mysqld<br>group mysql fs_mysql ip_mysql mysqld apache<br>ms ms_drbd_mysql drbd_mysql \<br> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" target-role="Started" is-managed="true"<br>
colocation mysql_on_drbd inf: mysql ms_drbd_mysql:Master<br>order mysql_after_drbd inf: ms_drbd_mysql:promote mysql:start<br>property $id="cib-bootstrap-options" \<br> no-quorum-policy="ignore" \<br>
stonith-enabled="false" \<br> expected-quorum-votes="2" \<br> dc-version="1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3" \<br> cluster-infrastructure="Heartbeat"<br><br>
This is what's printed in /var/log/messages: <a href="http://pastebin.com/W68jPQKJ">http://pastebin.com/W68jPQKJ</a><br>And /var/log/ha.log : <a href="http://pastebin.com/SBQz1gU3">http://pastebin.com/SBQz1gU3</a><br>
<br>My DRBD partition (dev/drbd0) is mounted on /opt/drbd and when I do "ls" it just hangs.<br>In case it's useful, please see here lsof output:<br><br>[root@rspa2 ~]# lsof | grep drbd<br>drbd0_wor 3422 root cwd DIR 8,2 4096 2 /<br>
drbd0_wor 3422 root rtd DIR 8,2 4096 2 /<br>drbd0_wor 3422 root txt unknown /proc/3422/exe<br>drbd0_rec 3425 root cwd DIR 8,2 4096 2 /<br>
drbd0_rec 3425 root rtd DIR 8,2 4096 2 /<br>drbd0_rec 3425 root txt unknown /proc/3425/exe<br>drbd0_ase 4876 root cwd DIR 8,2 4096 2 /<br>
drbd0_ase 4876 root rtd DIR 8,2 4096 2 /<br>drbd0_ase 4876 root txt unknown /proc/4876/exe<br>mysqld 12322 mysql cwd DIR 147,0 96 131 /opt/drbd/mysql<br>
mysqld 12322 mysql 3uW REG 147,0 10485760 135 /opt/drbd/mysql/ibdata1<br>mysqld 12322 mysql 8uW REG 147,0 5242880 133 /opt/drbd/mysql/ib_logfile0<br>mysqld 12322 mysql 9uW REG 147,0 5242880 134 /opt/drbd/mysql/ib_logfile1<br>
ls 12729 root 3r DIR 147,0 51 128 /opt/drbd<br>bash 12889 root 3r DIR 147,0 51 128 /opt/drbd<br>ls 13117 root 3r DIR 147,0 51 128 /opt/drbd<br>
<br>Heartbeat configuration file:<br>[root@rspa2 ~]# cat /etc/ha.d/<a href="http://ha.cf">ha.cf</a> <br>use_logd no<br>logfile /var/log/ha.log<br>autojoin none<br>warntime 5<br>deadtime 15<br>initdead 30<br>ucast eth0 172.18.2.137<br>
node rspa.domain rspa2.domain<br>crm yes<br><br>And last but not least, my DRBD configuration on both nodes:<br><br>global {<br> usage-count yes;<br>}<br>common {<br> protocol C;<br> syncer {<br> rate 10M;<br> }<br>
}<br>resource r0 {<br> net {<br> data-integrity-alg md5;<br> }<br> on rspa.domain {<br> device /dev/drbd0;<br> disk /dev/sda4;<br> address IP:7789;<br> meta-disk internal;<br> }<br> on rspa2.domain {<br>
device /dev/drbd0;<br> disk /dev/sda4;<br> address IP:7789;<br> meta-disk internal;<br> }<br>}<br><br><br><br><br><br>