<div dir="ltr"><div><strong>hi,<br><br><br>i wants to testing the fail-over capabilities of my cluster.<br>i run pkill -9 corosync on 2nd node and i saw on the 1node that he wants to stonith the node2 but he "giving up after too many failures to fence node"<br>
<br><br><br>via commandline it works without any problems<br>fence_virsh -a host2 -l root -x -k /root/.ssh/id_rsa -o reboot -v -n zarafa02<br><br><br></strong><strong>setup<br>
2x kvm guest (zarafa01=node1 / zarafa02=node2)<br>
2x kvm host <br>
rhel 6.4<br>
pacemaker,corosync,drbd</strong><br><strong><br></strong></div><div><strong>hopefully somebody can help me with the issue and the 2nd issue after run the fence_virsh via commandline the pacemaker service isn´t up on the 2nd node.<br>
</strong></div><div><strong><br><br>node1/var/log/messages<br>Oct 23 09:35:28 zarafa01 pengine[2866]: warning: stage6: Scheduling Node zarafa02for STONITH<br>Oct 23 09:35:28 zarafa01 pengine[2866]: notice: LogActions: Stop drbd_mysql:1#011(zarafa02)<br>
Oct 23 09:35:28 zarafa01 pengine[2866]: notice: LogActions: Stop drbd_zarafa:1#011(zarafa02)<br>Oct 23 09:35:28 zarafa01 pengine[2866]: notice: LogActions: Stop apache:1#011(zarafa02)<br>Oct 23 09:35:28 zarafa01 pengine[2866]: notice: LogActions: Stop stonith-zarafa01#011(zarafa02)<br>
Oct 23 09:35:28 zarafa01 pengine[2866]: warning: process_pe_message: Calculated Transition 183: (null)<br>Oct 23 09:35:28 zarafa01 crmd[29263]: notice: te_fence_node: Executing reboot fencing operation (124) on zarafa02 (timeout=60000)<br>
Oct 23 09:35:28 zarafa01 stonith-ng[2863]: notice: handle_request: Client crmd.29263.8f8f06d0 wants to fence (reboot) 'zarafa02' with device '(any)'<br>Oct 23 09:35:28 zarafa01 stonith-ng[2863]: notice: initiate_remote_stonith_op: Initiating remote operation reboot for zarafa02: 88604a94-8e2e-4ce4-9d08-85559e339f8e (0)<br>
Oct 23 09:35:28 zarafa01 crmd[29263]: notice: process_lrm_event: LRM operation drbd_mysql_notify_0 (call=710, rc=0, cib-update=0, confirmed=true) ok<br>Oct 23 09:35:28 zarafa01 crmd[29263]: notice: process_lrm_event: LRM operation drbd_zarafa_notify_0 (call=712, rc=0, cib-update=0, confirmed=true) ok<br>
Oct 23 09:36:40 zarafa01 stonith-ng[2863]: error: remote_op_done: Operation reboot of zarafa02 by zarafa01 for crmd.29263@zarafa01.88604a94: Timer expired<br>Oct 23 09:36:40 zarafa01 crmd[29263]: notice: tengine_stonith_callback: Stonith operation 5/124:183:0:cf74ef64-3995-414e-8ebd-ebacc89ace85: Timer expired (-62)<br>
Oct 23 09:36:40 zarafa01 crmd[29263]: notice: tengine_stonith_callback: Stonith operation 5 for zarafa02 failed (Timer expired): aborting transition.<br>Oct 23 09:36:40 zarafa01 crmd[29263]: notice: tengine_stonith_notify: Peer zarafa02 was not terminated (st_notify_fence) by zarafa01 for zarafa01: Timer expired (ref=88604a94-8e2e-4ce4-9d08-85559e339f8e) by client crmd.29263<br>
Oct 23 09:36:40 zarafa01 crmd[29263]: notice: run_graph: Transition 183 (Complete=9, Pending=0, Fired=0, Skipped=9, Incomplete=11, Source=unknown): Stopped<br>Oct 23 09:36:40 zarafa01 pengine[2866]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Oct 23 09:36:40 zarafa01 pengine[2866]: warning: pe_fence_node: Node zarafa02 will be fenced because the node is no longer part of the cluster<br>Oct 23 09:36:40 zarafa01 pengine[2866]: warning: determine_online_status: Node zarafa02 is unclean<br>
Oct 23 09:37:52 zarafa01 crmd[29263]: notice: tengine_stonith_callback: Stonith operation 6 for zarafa02 failed (Timer expired): aborting transition.<br>Oct 23 09:37:52 zarafa01 crmd[29263]: notice: tengine_stonith_notify: Peer zarafa02 was not terminated (st_notify_fence) by zarafa01 for zarafa01: Timer expired (ref=b13b2562-4124-4e6c-acca-e1114f7d9b98) by client crmd.29263<br>
Oct 23 09:37:52 zarafa01 crmd[29263]: notice: run_graph: Transition 184 (Complete=9, Pending=0, Fired=0, Skipped=9, Incomplete=11, Source=unknown): Stopped<br>Oct 23 09:37:52 zarafa01 pengine[2866]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Oct 23 09:37:52 zarafa01 pengine[2866]: warning: pe_fence_node: Node zarafa02 will be fenced because the node is no longer part of the cluster<br>Oct 23 09:37:52 zarafa01 pengine[2866]: warning: determine_online_status: Node zarafa02 is unclean<br>
Oct 23 09:39:04 zarafa01 pengine[2866]: warning: determine_online_status: Node zarafa02 is unclean<br>Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action drbd_mysql:1_stop_0 on zarafa02 is unrunnable (offline)<br>
Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action drbd_mysql:1_stop_0 on zarafa02 is unrunnable (offline)<br>Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action drbd_mysql:1_stop_0 on zarafa02 is unrunnable (offline)<br>
Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action drbd_mysql:1_stop_0 on zarafa02 is unrunnable (offline)<br>Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action drbd_zarafa:1_stop_0 on zarafa02 is unrunnable (offline)<br>
Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action drbd_zarafa:1_stop_0 on zarafa02 is unrunnable (offline)<br>Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action drbd_zarafa:1_stop_0 on zarafa02 is unrunnable (offline)<br>
Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action drbd_zarafa:1_stop_0 on zarafa02 is unrunnable (offline)<br>Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action apache:1_stop_0 on zarafa02 is unrunnable (offline)<br>
Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action apache:1_stop_0 on zarafa02 is unrunnable (offline)<br>Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action stonith-zarafa01_stop_0 on zarafa02 is unrunnable (offline)<br>
Oct 23 09:39:04 zarafa01 pengine[2866]: warning: custom_action: Action stonith-zarafa01_stop_0 on zarafa02 is unrunnable (offline)<br>Oct 23 09:43:52 zarafa01 pengine[2866]: notice: LogActions: Stop apache:1#011(zarafa02)<br>
Oct 23 09:43:52 zarafa01 pengine[2866]: notice: LogActions: Stop stonith-zarafa01#011(zarafa02)<br>Oct 23 09:43:52 zarafa01 crmd[29263]: notice: te_fence_node: Executing reboot fencing operation (124) on zarafa02 (timeout=60000)<br>
Oct 23 09:43:52 zarafa01 pengine[2866]: warning: process_pe_message: Calculated Transition 190: (null)<br>Oct 23 09:43:52 zarafa01 stonith-ng[2863]: notice: handle_request: Client crmd.29263.8f8f06d0 wants to fence (reboot) 'zarafa02' with device '(any)'<br>
Oct 23 09:43:52 zarafa01 stonith-ng[2863]: notice: initiate_remote_stonith_op: Initiating remote operation reboot for zarafa02: de24f595-81e3-49f5-8886-07c8c1b22ec7 (0)<br>Oct 23 09:43:52 zarafa01 crmd[29263]: notice: process_lrm_event: LRM operation drbd_mysql_notify_0 (call=752, rc=0, cib-update=0, confirmed=true) ok<br>
Oct 23 09:43:52 zarafa01 crmd[29263]: notice: process_lrm_event: LRM operation drbd_zarafa_notify_0 (call=754, rc=0, cib-update=0, confirmed=true) ok<br>Oct 23 09:44:04 zarafa01 rsyslogd-2177: imuxsock lost 92458 messages from pid 1927 due to rate-limiting<br>
Oct 23 09:44:04 zarafa01 rsyslogd-2177: imuxsock begins to drop messages from pid 1927 due to rate-limiting<br>Oct 23 09:45:02 zarafa01 rsyslogd-2177: imuxsock lost 13836 messages from pid 1927 due to rate-limiting<br>Oct 23 09:45:03 zarafa01 rsyslogd-2177: imuxsock begins to drop messages from pid 1927 due to rate-limiting<br>
Oct 23 09:45:04 zarafa01 stonith-ng[2863]: error: remote_op_done: Operation reboot of zarafa02 by zarafa01 for crmd.29263@zarafa01.de24f595: Timer expired<br>Oct 23 09:45:04 zarafa01 crmd[29263]: notice: tengine_stonith_callback: Stonith operation 12/124:190:0:cf74ef64-3995-414e-8ebd-ebacc89ace85: Timer expired (-62)<br>
Oct 23 09:45:04 zarafa01 crmd[29263]: notice: tengine_stonith_callback: Stonith operation 12 for zarafa02 failed (Timer expired): aborting transition.<br>Oct 23 09:45:04 zarafa01 crmd[29263]: notice: tengine_stonith_notify: Peer zarafa02 was not terminated (st_notify_fence) by zarafa01 for zarafa01: Timer expired (ref=de24f595-81e3-49f5-8886-07c8c1b22ec7) by client crmd.29263<br>
Oct 23 09:45:04 zarafa01 crmd[29263]: notice: run_graph: Transition 190 (Complete=9, Pending=0, Fired=0, Skipped=9, Incomplete=11, Source=unknown): Stopped<br>Oct 23 09:45:04 zarafa01 crmd[29263]: notice: too_many_st_failures: Too many failures to fence zarafa02 (11), giving up<br>
Oct 23 09:45:08 zarafa01 rsyslogd-2177: imuxsock lost 178501 messages from pid 1927 due to rate-limiting<br><br><br>node zarafa01\<br> attributes standby="off"<br>node zarafa02 \<br> attributes standby="off"<br>
primitive apache ocf:heartbeat:apache \<br> params configfile="/etc/httpd/conf/httpd.conf" \<br> op monitor interval="60s" \<br> op start interval="0" timeout="40s" \<br>
op stop interval="0" timeout="60s"<br>primitive drbd_mysql ocf:linbit:drbd \<br> params drbd_resource="mysql" \<br> op start interval="0" timeout="240" \<br>
op stop interval="0" timeout="100" \<br> op monitor interval="59s" role="Master" timeout="30s" \<br> op monitor interval="60s" role="Slave" timeout="30s"<br>
primitive drbd_zarafa ocf:linbit:drbd \<br> params drbd_resource="zarafa" \<br> op start interval="0" timeout="240" \<br> op stop interval="0" timeout="240" \<br>
op monitor interval="59s" role="Master" timeout="30s" \<br> op monitor interval="60s" role="Slave" timeout="30s"<br>primitive mysql_fs ocf:heartbeat:Filesystem \<br>
params device="/dev/drbd0" directory="/data/mysql" fstype="ext4" options="noatime" \<br> op start interval="0" timeout="240" \<br> op stop interval="0" timeout="100" \<br>
op monitor interval="30s" timeout="40s"<br>primitive mysql_ip ocf:heartbeat:IPaddr2 \<br> params ip="0.0.0.0" iflabel="MYSQL" cidr_netmask="20" nic="eth0" \<br>
op monitor interval="30s"<br>primitive mysqld lsb:mysqld \<br> op monitor interval="10" timeout="30" \<br> op start interval="0" timeout="500" \<br> op stop interval="0" timeout="500"<br>
primitive stonith-zarafa01 stonith:fence_virsh \<br> params pcmk_host_list="zarafa01" pcmk_host_check="static-list" action="reboot" ipaddr="host01" secure="true" login="root" identity_file="/root/.ssh/id_rsa" \<br>
op monitor interval="300s" \<br> op start interval="0" timeout="60s" \<br> meta failure-timeout="180s"<br>primitive stonith-zarafa02 stonith:fence_virsh \<br> params pcmk_host_list="zarafa02" pcmk_host_check="static-list" action="reboot" ipaddr="host02" secure="true" delay="5" login="root" identity_file="/root/.ssh/id_rsa" \<br>
op monitor interval="300s" \<br> op start interval="0" timeout="60s" \<br> meta failure-timeout="180s"<br>primitive zarafa-dagent lsb:zarafa-dagent \<br> op monitor interval="30" timeout="30" \<br>
meta target-role="Started"<br>primitive zarafa-gateway lsb:zarafa-gateway \<br> op monitor interval="30" timeout="30"<br>primitive zarafa-ical lsb:zarafa-ical \<br> op monitor interval="30" timeout="30"<br>
primitive zarafa-indexer lsb:zarafa-indexer \<br> op monitor interval="60" timeout="60" \<br> op start interval="0" timeout="120" \<br> op stop interval="0" timeout="120"<br>
primitive zarafa-licensed lsb:zarafa-licensed \<br> op monitor interval="30" timeout="30"<br>primitive zarafa-monitor lsb:zarafa-monitor \<br> op monitor interval="30" timeout="30"<br>
primitive zarafa-server lsb:zarafa-server \<br> op monitor interval="30" timeout="90" \<br> meta target-role="Started"<br>primitive zarafa-spooler lsb:zarafa-spooler \<br> op monitor interval="30" timeout="30"<br>
primitive zarafa_fs ocf:heartbeat:Filesystem \<br> params device="/dev/drbd1" directory="/data/zarafa" fstype="ext4" \<br> op start interval="0" timeout="240" \<br>
op stop interval="0" timeout="100" \<br> op monitor interval="30s" timeout="40s" \<br> meta target-role="Started"<br>primitive zarafa_ip ocf:heartbeat:IPaddr2 \<br>
params ip="0.0.0.1" iflabel="ZARAFA" cidr_netmask="20" nic="eth0" \<br> op monitor interval="30s" \<br> meta target-role="Started"<br>group mysql mysql_fs mysql_ip mysqld \<br>
meta target-role="Started"<br>group zarafa zarafa_fs zarafa_ip zarafa-server zarafa-spooler zarafa-dagent zarafa-licensed zarafa-monitor zarafa-gateway zarafa-ical zarafa-indexer \<br> meta target-role="Started"<br>
ms ms_drbd_mysql drbd_mysql \<br> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" target-role="Started"<br>ms ms_drbd_zarafa drbd_zarafa \<br>
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" target-role="Started"<br>clone apache_clone apache<br>location cli-prefer-mysql mysql \<br>
rule $id="cli-prefer-rule-mysql" inf: #uname eq zarafa01<br>location drbd-fence-by-handler-mysql-ms_drbd_mysql ms_drbd_mysql \<br> rule $id="drbd-fence-by-handler-mysql-rule-ms_drbd_mysql" $role="Master" -inf: #uname ne zarafa01<br>
location drbd-fence-by-handler-zarafa-ms_drbd_zarafa ms_drbd_zarafa \<br> rule $id="drbd-fence-by-handler-zarafa-rule-ms_drbd_zarafa" $role="Master" -inf: #uname ne zarafa01<br>location preferred_on_mysql mysql 100: zarafa01<br>
location preferred_on_zarafa zarafa 100: zarafa01<br>location stonith-by-zarafa01 stonith-zarafa02 -inf: zarafa02<br>location stonith-by-zarafa02 stonith-zarafa01 -inf: zarafa01<br>colocation mysql_on_drbd inf: mysql ms_drbd_mysql:Master<br>
colocation zarafa_on_drbd inf: zarafa ms_drbd_zarafa:Master<br>order mysql_after_drbd inf: ms_drbd_mysql:promote mysql:start<br>order zarafa_after_drbd inf: ms_drbd_zarafa:promote zarafa:start<br>order zarafa_after_mysql inf: mysql:start zarafa:start<br>
property $id="cib-bootstrap-options" \<br> dc-version="1.1.8-7.el6-394e906" \<br> cluster-infrastructure="classic openais (with plugin)" \<br> expected-quorum-votes="2" \<br>
stonith-enabled="true" \<br> cluster-recheck-interval="5min" \<br> no-quorum-policy="ignore" \<br> last-lrm-refresh="1382443560" \<br> maintenance-mode="off"<br>
rsc_defaults $id="rsc-options" \<br> resource-stickiness="200" \<br> failure-timeout="10min" \<br> migration-threshold="3"<br><br><br>crm status<br>Last updated: Wed Oct 23 10:51:51 2013<br>
Last change: Wed Oct 23 10:12:17 2013 via cibadmin on zarafa01<br>Stack: classic openais (with plugin)<br>Current DC: zarafa01 - partition with quorum<br>Version: 1.1.8-7.el6-394e906<br>2 Nodes configured, 2 expected votes<br>
21 Resources configured.<br><br><br>Online: [ zarafa01 zarafa02]<br><br> Resource Group: mysql<br> mysql_fs (ocf::heartbeat:Filesystem): Started zarafa01<br> mysql_ip (ocf::heartbeat:IPaddr2): Started zarafa01<br>
mysqld (lsb:mysqld): Started zarafa01<br> Master/Slave Set: ms_drbd_mysql [drbd_mysql]<br> Masters: [ zarafa01 ]<br> Stopped: [ drbd_mysql:1 ]<br> Resource Group: zarafa<br> zarafa_fs (ocf::heartbeat:Filesystem): Started zarafa01<br>
zarafa_ip (ocf::heartbeat:IPaddr2): Started zarafa01<br> zarafa-server (lsb:zarafa-server): Started zarafa01<br> zarafa-spooler (lsb:zarafa-spooler): Started zarafa01<br> zarafa-dagent (lsb:zarafa-dagent): Started zarafa01<br>
zarafa-licensed (lsb:zarafa-licensed): Started zarafa01<br> zarafa-monitor (lsb:zarafa-monitor): Started zarafa01<br> zarafa-gateway (lsb:zarafa-gateway): Started zarafa01<br> zarafa-ical (lsb:zarafa-ical): Started zarafa01<br>
zarafa-indexer (lsb:zarafa-indexer): Started zarafa01<br> Master/Slave Set: ms_drbd_zarafa [drbd_zarafa]<br> Masters: [ zarafa01 ]<br> Stopped: [ drbd_zarafa:1 ]<br> Clone Set: apache_clone [apache]<br>
Started: [ zarafa01 ]<br> Stopped: [ apache:1 ]<br> stonith-zarafa02 (stonith:fence_virsh): Started zarafa01<br><br><br><br></strong></div><div><strong>thanks<br>beo<br></strong></div><div><strong><br><br><br>
<br></strong></div><strong></strong><div><div><strong><br></strong></div></div></div>