<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; "><div><div>Hi, </div><div><br></div><div>I am having this issue where it appears that everything is working correctly, but when I simulate failure the failover fails to work correctly. the Migrate command works fine, I can transfer the service, and the error I get when a node is put into standby or a server goes down is </div><div><br></div><div>Any help would be greatly appreciated</div><div><br></div><div>Brian Cavanagh</div><div><br></div><div>PS Disregard if this double posted</div><div><br></div><div><br></div><div>Working fine …. </div><div><div>============</div><div>Last updated: Fri Jan 28 12:17:24 2011</div><div>Stack: Heartbeat</div><div>Current DC: mdb4 (050fc65c-29ad-4333-93c4-34d98405b952)<span class="Apple-tab-span" style="white-space: pre; "> </span>- partition with quorum</div><div>Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3</div><div>2 Nodes configured, 1 expected votes</div><div>2 Resources configured.</div><div>============</div><div><br></div><div>Online: [ mdb4 mdb3 ]</div><div><br></div><div> Master/Slave Set: ms_drbd_mysql</div><div> Masters: [ mdb4 ]</div><div> Slaves: [ mdb3 ]</div><div> Resource Group: mysql</div><div> ip1 (ocf::heartbeat:IPaddr2):<span class="Apple-tab-span" style="white-space: pre; "> </span>Started mdb4</div><div> ip1arp (ocf::heartbeat:SendArp):<span class="Apple-tab-span" style="white-space: pre; "> </span>Started mdb4</div><div> ip2 (ocf::heartbeat:IPaddr2):<span class="Apple-tab-span" style="white-space: pre; "> </span>Started mdb4</div><div> ip2arp (ocf::heartbeat:SendArp):<span class="Apple-tab-span" style="white-space: pre; "> </span>Started mdb4</div><div> fs_mysql (ocf::heartbeat:Filesystem): Started mdb4</div><div> mysqld (ocf::heartbeat:mysql): Started mdb4</div></div><div><br></div><div>Crm resource migrate mysql</div><div><br></div><div><div>============</div><div>Last updated: Fri Jan 28 12:18:58 2011</div><div>Stack: Heartbeat</div><div>Current DC: mdb4 (050fc65c-29ad-4333-93c4-34d98405b952)<span class="Apple-tab-span" style="white-space: pre; "> </span>- partition with quorum</div><div>Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3</div><div>2 Nodes configured, 1 expected votes</div><div>2 Resources configured.</div><div>============</div><div><br></div><div>Online: [ mdb4 mdb3 ]</div><div><br></div><div> Master/Slave Set: ms_drbd_mysql</div><div> Masters: [ mdb3 ]</div><div> Slaves: [ mdb4 ]</div><div> Resource Group: mysql</div><div> ip1 (ocf::heartbeat:IPaddr2):<span class="Apple-tab-span" style="white-space: pre; "> </span>Started mdb3</div><div> ip1arp (ocf::heartbeat:SendArp):<span class="Apple-tab-span" style="white-space: pre; "> </span>Started mdb3</div><div> ip2 (ocf::heartbeat:IPaddr2):<span class="Apple-tab-span" style="white-space: pre; "> </span>Started mdb3</div><div> ip2arp (ocf::heartbeat:SendArp):<span class="Apple-tab-span" style="white-space: pre; "> </span>Started mdb3</div><div> fs_mysql (ocf::heartbeat:Filesystem): Started mdb3</div><div> mysqld (ocf::heartbeat:mysql): Started mdb3</div></div><div><br></div><div>Crm resource unmove mysql</div><div>Crm node standby mdb3</div><div><br></div><div><div>============</div><div>Last updated: Fri Jan 28 12:20:40 2011</div><div>Stack: Heartbeat</div><div>Current DC: mdb4 (050fc65c-29ad-4333-93c4-34d98405b952)<span class="Apple-tab-span" style="white-space: pre; "> </span>- partition with quorum</div><div>Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3</div><div>2 Nodes configured, 1 expected votes</div><div>2 Resources configured.</div><div>============</div><div><br></div><div>Node mdb3 (5f4014cd-472e-4ab3-95e3-759152f16f52): standby</div><div>Online: [ mdb4 ]</div><div><br></div><div> Master/Slave Set: ms_drbd_mysql</div><div> drbd_mysql:0<span class="Apple-tab-span" style="white-space: pre; "> </span>(ocf::linbit:drbd): Slave mdb4 (unmanaged) FAILED</div><div> drbd_mysql:1<span class="Apple-tab-span" style="white-space: pre; "> </span>(ocf::linbit:drbd): Slave mdb3 (unmanaged) FAILED</div><div><br></div><div>Failed actions:</div><div> drbd_mysql:0_stop_0 (node=mdb4, call=67, rc=6, status=complete): not configured</div><div> drbd_mysql:1_stop_0 (node=mdb3, call=65, rc=6, status=complete): not configured</div><div><br></div></div><div><br></div><div>Error logs don't say much</div><div>Tail –n 30 /var/log/messages</div><div><br></div><div><div>Jan 28 12:20:31 mdb3 IPaddr2[9506]: INFO: ip -f inet addr delete 192.168.162.12/17 dev eth0</div><div>Jan 28 12:20:31 mdb3 crmd: [2781]: info: process_lrm_event: LRM operation ip1_stop_0 (call=61, rc=0, cib-update=69, confirmed=true) ok</div><div>Jan 28 12:20:32 mdb3 crmd: [2781]: info: do_lrm_rsc_op: Performing key=13:8:0:dc2c6518-0d45-4ecc-ac70-c7044d59c1c8 op=drbd_mysql:1_demote_0 )</div><div>Jan 28 12:20:32 mdb3 lrmd: [2778]: info: rsc:drbd_mysql:1:62: demote</div><div>Jan 28 12:20:32 mdb3 kernel: block drbd0: role( Primary -> Secondary ) </div><div>Jan 28 12:20:32 mdb3 lrmd: [2778]: info: RA output: (drbd_mysql:1:demote:stdout) </div><div>Jan 28 12:20:32 mdb3 crmd: [2781]: info: process_lrm_event: LRM operation drbd_mysql:1_demote_0 (call=62, rc=0, cib-update=70, confirmed=true) ok</div><div>Jan 28 12:20:34 mdb3 crmd: [2781]: info: do_lrm_rsc_op: Performing key=69:8:0:dc2c6518-0d45-4ecc-ac70-c7044d59c1c8 op=drbd_mysql:1_notify_0 )</div><div>Jan 28 12:20:34 mdb3 lrmd: [2778]: info: rsc:drbd_mysql:1:63: notify</div><div>Jan 28 12:20:34 mdb3 lrmd: [2778]: info: RA output: (drbd_mysql:1:notify:stdout) </div><div>Jan 28 12:20:34 mdb3 crmd: [2781]: info: process_lrm_event: LRM operation drbd_mysql:1_notify_0 (call=63, rc=0, cib-update=71, confirmed=true) ok</div><div>Jan 28 12:20:36 mdb3 crmd: [2781]: info: do_lrm_rsc_op: Performing key=63:8:0:dc2c6518-0d45-4ecc-ac70-c7044d59c1c8 op=drbd_mysql:1_notify_0 )</div><div>Jan 28 12:20:36 mdb3 lrmd: [2778]: info: rsc:drbd_mysql:1:64: notify</div><div>Jan 28 12:20:36 mdb3 crmd: [2781]: info: process_lrm_event: LRM operation drbd_mysql:1_notify_0 (call=64, rc=0, cib-update=72, confirmed=true) ok</div><div>Jan 28 12:20:37 mdb3 crmd: [2781]: info: do_lrm_rsc_op: Performing key=14:8:0:dc2c6518-0d45-4ecc-ac70-c7044d59c1c8 op=drbd_mysql:1_stop_0 )</div><div>Jan 28 12:20:37 mdb3 lrmd: [2778]: info: rsc:drbd_mysql:1:65: stop</div><div>Jan 28 12:20:37 mdb3 drbd[9631]: ERROR: you really should enable notify when using this RA</div><div>Jan 28 12:20:37 mdb3 crmd: [2781]: info: process_lrm_event: LRM operation drbd_mysql:1_stop_0 (call=65, rc=6, cib-update=73, confirmed=true) not configured</div><div>Jan 28 12:20:39 mdb3 attrd: [2780]: info: attrd_ha_callback: Update relayed from mdb4</div><div>Jan 28 12:20:39 mdb3 attrd: [2780]: info: find_hash_entry: Creating hash entry for fail-count-drbd_mysql:1</div><div>Jan 28 12:20:39 mdb3 attrd: [2780]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_mysql:1 (INFINITY)</div><div>Jan 28 12:20:40 mdb3 attrd: [2780]: info: attrd_perform_update: Sent update 21: fail-count-drbd_mysql:1=INFINITY</div><div>Jan 28 12:20:40 mdb3 attrd: [2780]: info: attrd_ha_callback: Update relayed from mdb4</div><div>Jan 28 12:20:40 mdb3 attrd: [2780]: info: find_hash_entry: Creating hash entry for last-failure-drbd_mysql:1</div><div>Jan 28 12:20:40 mdb3 attrd: [2780]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_mysql:1 (1296235239)</div><div>Jan 28 12:20:40 mdb3 attrd: [2780]: info: attrd_perform_update: Sent update 24: last-failure-drbd_mysql:1=1296235239</div><div>Jan 28 12:20:40 mdb3 attrd: [2780]: info: attrd_ha_callback: flush message from mdb4</div><div>Jan 28 12:20:40 mdb3 attrd: [2780]: info: find_hash_entry: Creating hash entry for fail-count-drbd_mysql:0</div><div>Jan 28 12:20:40 mdb3 attrd: [2780]: info: attrd_ha_callback: flush message from mdb4</div><div>Jan 28 12:20:40 mdb3 attrd: [2780]: info: find_hash_entry: Creating hash entry for last-failure-drbd_mysql:0</div></div><div><br></div><div>/* configurations */</div><div>Crm configure </div><div><div>node $id="050fc65c-29ad-4333-93c4-34d98405b952" mdb4 \</div><div> attributes standby="off"</div><div>node $id="5f4014cd-472e-4ab3-95e3-759152f16f52" mdb3 \</div><div> attributes standby="on"</div><div>primitive drbd_mysql ocf:linbit:drbd \</div><div> params drbd_resource="r0" \</div><div> op monitor interval="15s"</div><div>primitive fs_mysql ocf:heartbeat:Filesystem \</div><div> params device="/dev/drbd/by-res/r0" directory="/var/lib/mysql" fstype="ext3" \</div><div> op start interval="0" timeout="60" \</div><div> op stop interval="0" timeout="120"</div><div>primitive ip1 ocf:heartbeat:IPaddr2 \</div><div> params ip="192.168.162.12" nic="eth0:0" cidr_netmask="17" \</div><div> op monitor interval="5s"</div><div>primitive ip1arp ocf:heartbeat:SendArp \</div><div> params ip="192.168.162.12" nic="eth0:0"</div><div>primitive ip2 ocf:heartbeat:IPaddr2 \</div><div> params ip="97.107.136.62" nic="eth0:2" cidr_netmask="24" \</div><div> op monitor interval="5s"</div><div>primitive ip2arp ocf:heartbeat:SendArp \</div><div> params ip="97.107.136.62" nic="eth0:2"</div><div>primitive mysqld ocf:heartbeat:mysql \</div><div> params binary="/usr/sbin/mysqld" config="/etc/mysql/my.cnf" user="mysql" group="mysql" log="/var/log/mysql_safe.log" pid="/var/lib/mysql/mysqld.pid" datadir="/var/lib/mysql" \ </div><div> op monitor interval="30s" timeout="30s" \</div><div> op start interval="0" timeout="120" \</div><div> op stop interval="0" timeout="120"</div><div>group mysql ip1 ip1arp ip2 ip2arp fs_mysql mysqld \</div><div> meta target-role="Started"</div><div>ms ms_drbd_mysql drbd_mysql \</div><div> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" target-role="Started"</div><div>location cli-standby-mysql mysql \</div><div> rule $id="cli-standby-rule-mysql" -inf: #uname eq mdb4</div><div>colocation mysql_on_drbd inf: mysql ms_drbd_mysql:Master</div><div>order mysql_after_drbd inf: ms_drbd_mysql:promote mysql:start</div><div>property $id="cib-bootstrap-options" \</div><div> dc-version="1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3" \</div><div> cluster-infrastructure="Heartbeat" \</div><div> expected-quorum-votes="1" \</div><div> stonith-enabled="false" \</div><div> no-quorum-policy="ignore"</div><div>rsc_defaults $id="rsc-options" \</div><div> resource-stickiness="100"</div></div><div><br></div><div>/etc/drbd.conf </div><div><div>global {</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>usage-count yes;</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span># minor-count dialog-refresh disable-ip-verification</div><div>}</div><div><br></div><div>common {</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>protocol C;</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>handlers {</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>fence-peer "/usr/lib/drbd/crm-fence-peer.sh";</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>split-brain "/usr/lib/drbd/notify-split-brain.sh root";</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>}</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>startup {</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span># wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>}</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>disk {</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span># on-io-error fencing use-bmbv no-disk-barrier no-disk-flushes</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span># no-disk-drain no-md-flushes max-bio-bvecs</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>}</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>net {</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span># sndbuf-size rcvbuf-size timeout connect-int ping-int ping-timeout max-buffers</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span># max-epoch-size ko-count allow-two-primaries cram-hmac-alg shared-secret</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span># after-sb-0pri after-sb-1pri after-sb-2pri data-integrity-alg no-tcp-cork</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>}</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>syncer {</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span># rate after al-extents use-rle cpu-mask verify-alg csums-alg</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>}</div><div>}</div><div><br></div></div><div><div><br></div><div>resource r0 {</div><div> protocol C;</div><div> syncer {</div><div> rate 4M;</div><div> }</div><div> startup {</div><div> wfc-timeout 15;</div><div> degr-wfc-timeout 60;</div><div> }</div><div> net {</div><div> cram-hmac-alg sha1;</div><div> shared-secret "[snip]";</div><div> }</div><div> on mdb3 {</div><div> device /dev/drbd0;</div><div> disk /dev/xvdc;</div><div> address 192.168.156.171:7788;</div><div> meta-disk internal;</div><div> }</div><div> on mdb4 {</div><div> device /dev/drbd0;</div><div> disk /dev/xvdc;</div><div> address 192.168.140.133:7788;</div><div> meta-disk internal;</div><div> }</div><div>}</div></div><div><br></div><div>/etc/ha.d/ha.cf mdb3</div><div><div><div>logfile /var/log/heartbeat.log</div><div>logfacility local0</div><div>keepalive 2</div><div>deadtime 15</div><div>warntime 5</div><div>initdead 120</div><div>udpport 694</div><div>ucast eth0 173.255.238.128</div><div>auto_failback on</div><div>node mdb3</div><div>node mdb4</div><div>use_logd no</div><div>crm respawn</div></div></div><div><br></div><div>/etc/ha.d/ha.cf mdb4</div><div><div>logfile /var/log/heartbeat.log</div><div>logfacility local0</div><div>keepalive 2</div><div>deadtime 15</div><div>warntime 5</div><div>initdead 120</div><div>udpport 694</div><div>ucast eth0 173.255.238.191</div><div>auto_failback on</div><div>node mdb3</div><div>node mdb4</div><div>use_logd no</div><div>crm respawn</div></div><div><br></div><div><br></div></div></body></html>