<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='color:#1F497D'>After further testing even with stonith enabled the cluster still gets stuck in this state, presumably waiting on IO. I can get around it by setting “on-fail=fence” on the LVM resources but shouldn’t Pacemaker be smart enough to realise the host is effectively offline? Or am I missing some timeout value that would fix this situation?<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>pacemaker-1.1.2-7.el6.x86_64<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>corosync-1.2.3-21.el6.x86_64<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>RHEL 6.0<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Config:<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>node host001.domain \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> attributes standby="off"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>node host002.domain \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> attributes standby="off"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>primitive MyApp_IP ocf:heartbeat:IPaddr \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> params ip="192.168.104.26" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> op monitor interval="10s"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>primitive MyApp_fs_graph ocf:heartbeat:Filesystem \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> params device="/dev/VolGroupB00/AppLV2" directory="/naab1" fstype="ext4" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> op monitor interval="10" timeout="10"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>primitive MyApp_fs_landing ocf:heartbeat:Filesystem \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> params device="/dev/VolGroupB01/AppLV1" directory="/naab2" fstype="ext4" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> op monitor interval="10" timeout="10"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>primitive MyApp_lvm_graph ocf:heartbeat:LVM \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> params volgrpname="VolGroupB00" exclusive="yes" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> op monitor interval="10" timeout="10" on-fail="fence" depth="0"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>primitive MyApp_lvm_landing ocf:heartbeat:LVM \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> params volgrpname="VolGroupB01" exclusive="yes" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> op monitor interval="10" timeout="10" on-fail="fence" depth="0"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>primitive MyApp_scsi_reservation ocf:heartbeat:sg_persist \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> params sg_persist_resource="scsi_reservation0" devs="/dev/dm-6 /dev/dm-7" required_devs_nof="2" reservation_type="1"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>primitive MyApp_init_script lsb:AppInitScript \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> op monitor interval="10" timeout="10"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>primitive fence_host001.domain stonith:fence_ipmilan \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> params ipaddr="192.168.16.148" passwd="password" login="root" pcmk_host_list="host001.domain" pcmk_host_check="static-list" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> meta target-role="Started"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>primitive fence_host002.domain stonith:fence_ipmilan \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> params ipaddr="192.168.16.149" passwd="password" login="root" pcmk_host_list="host002.domain" pcmk_host_check="static-list" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> meta target-role="Started"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>group MyApp_group MyApp_lvm_graph MyApp_lvm_landing MyApp_fs_graph MyApp_fs_landing MyApp_IP MyApp_init_script \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> meta target-role="Started" migration-threshold="2" on-fail="restart" failure-timeout="300s"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>ms ms_MyApp_scsi_reservation MyApp_scsi_reservation \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>colocation MyApp_group_on_scsi_reservation inf: MyApp_group ms_MyApp_scsi_reservation:Master<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>order MyApp_group_after_scsi_reservation inf: ms_MyApp_scsi_reservation:promote MyApp_group:start<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>property $id="cib-bootstrap-options" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> dc-version="1.1.2-f059ec7ced7a86f18e5490b67ebf4a0b963bccfe" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> cluster-infrastructure="openais" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> expected-quorum-votes="2" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> no-quorum-policy="ignore" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> stonith-enabled="true" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> last-lrm-refresh="1305129673"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>rsc_defaults $id="rsc-options" \<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'> resource-stickiness="1"<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-GB'> Max Williams [mailto:Max.Williams@betfair.com] <br><b>Sent:</b> 11 May 2011 13:55<br><b>To:</b> The Pacemaker cluster resource manager (pacemaker@oss.clusterlabs.org)<br><b>Subject:</b> [Pacemaker] Failover when storage fails<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi,<o:p></o:p></p><p class=MsoNormal>I want to configure pacemaker to failover a group of resources and sg_persist (master/slave) when there is a problem with the storage but when I cause the iSCSI LUN to disappear simulating a failure, the cluster always gets stuck in this state:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-left:36.0pt'>Last updated: Wed May 11 10:52:43 2011<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>Stack: openais<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>Current DC: host001.domain - partition with quorum<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>Version: 1.1.2-f059ec7ced7a86f18e5490b67ebf4a0b963bccfe<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>2 Nodes configured, 2 expected votes<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>4 Resources configured.<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>============<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p><p class=MsoNormal style='margin-left:36.0pt'>Online: [ host002.domain host001.domain ]<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p><p class=MsoNormal style='margin-left:36.0pt'>fence_host002.domain (stonith:fence_ipmilan): Started host001.domain<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>fence_host001.domain (stonith:fence_ipmilan): Started host001.domain<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>Resource Group: MyApp_group<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'> MyApp_lvm_graph (ocf::heartbeat:LVM): Started host002.domain FAILED<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'> MyApp_lvm_landing (ocf::heartbeat:LVM): Started host002.domain FAILED<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'> MyApp_fs_graph (ocf::heartbeat:Filesystem): Started host002.domain<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'> MyApp_fs_landing (ocf::heartbeat:Filesystem): Started host002.domain<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'> MyApp_IP (ocf::heartbeat:IPaddr): Stopped<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'> MyApp_init_script (lsb:abworkload): Stopped<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>Master/Slave Set: ms_MyApp_scsi_reservation<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'> Masters: [ host002.domain ]<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'> Slaves: [ host001.domain ]<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><o:p> </o:p></p><p class=MsoNormal style='margin-left:36.0pt'>Failed actions:<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'> MyApp_lvm_graph_monitor_10000 (node=host002.domain, call=129, rc=-2, status=Timed Out): unknown exec error<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'> MyApp_lvm_landing_monitor_10000 (node=host002.domain, call=130, rc=-2, status=Timed Out): unknown exec error<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>This is printed over and over in the logs:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:56 host002 lrmd: [2561]: info: perform_op:2884: operation stop[202] on ocf::Filesystem::MyApp_fs_graph for client 31850, its parameters: fstype=[ext4] crm_feature_set=[3.0.2] device=[/dev/VolGroupB00/abb_graph] CRM_meta_timeout=[20000] directory=[/naab1] for rsc is already running.<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:56 host002 lrmd: [2561]: info: perform_op:2894: postponing all ops on resource MyApp_fs_graph by 1000 ms<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:57 host002 lrmd: [2561]: info: perform_op:2884: operation stop[202] on ocf::Filesystem::MyApp_fs_graph for client 31850, its parameters: fstype=[ext4] crm_feature_set=[3.0.2] device=[/dev/VolGroupB00/abb_graph] CRM_meta_timeout=[20000] directory=[/naab1] for rsc is already running.<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:57 host002 lrmd: [2561]: info: perform_op:2894: postponing all ops on resource MyApp_fs_graph by 1000 ms<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:58 host002 lrmd: [2561]: info: perform_op:2884: operation stop[202] on ocf::Filesystem::MyApp_fs_graph for client 31850, its parameters: fstype=[ext4] crm_feature_set=[3.0.2] device=[/dev/VolGroupB00/abb_graph] CRM_meta_timeout=[20000] directory=[/naab1] for rsc is already running.<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:58 host002 lrmd: [2561]: info: perform_op:2894: postponing all ops on resource MyApp_fs_graph by 1000 ms<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:58 host002 lrmd: [2561]: WARN: MyApp_lvm_graph:monitor process (PID 1938) timed out (try 1). Killing with signal SIGTERM (15).<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:58 host002 lrmd: [2561]: WARN: MyApp_lvm_landing:monitor process (PID 1939) timed out (try 1). Killing with signal SIGTERM (15).<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:58 host002 lrmd: [2561]: WARN: operation monitor[190] on ocf::LVM::MyApp_lvm_graph for client 31850, its parameters: CRM_meta_depth=[0] depth=[0] exclusive=[yes] crm_feature_set=[3.0.2] volgrpname=[VolGroupB00] CRM_meta_on_fail=[standby] CRM_meta_name=[monitor] CRM_meta_interval=[10000] CRM_meta_timeout=[10000] : pid [1938] timed out<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:58 host002 lrmd: [2561]: WARN: operation monitor[191] on ocf::LVM::MyApp_lvm_landing for client 31850, its parameters: CRM_meta_depth=[0] depth=[0] exclusive=[yes] crm_feature_set=[3.0.2] volgrpname=[VolGroupB01] CRM_meta_on_fail=[standby] CRM_meta_name=[monitor] CRM_meta_interval=[10000] CRM_meta_timeout=[10000] : pid [1939] timed out<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:58 host002 crmd: [31850]: ERROR: process_lrm_event: LRM operation MyApp_lvm_graph_monitor_10000 (190) Timed Out (timeout=10000ms)<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:58 host002 crmd: [31850]: ERROR: process_lrm_event: LRM operation MyApp_lvm_landing_monitor_10000 (191) Timed Out (timeout=10000ms)<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:59 host002 lrmd: [2561]: info: perform_op:2884: operation stop[202] on ocf::Filesystem::MyApp_fs_graph for client 31850, its parameters: fstype=[ext4] crm_feature_set=[3.0.2] device=[/dev/VolGroupB00/abb_graph] CRM_meta_timeout=[20000] directory=[/naab1] for rsc is already running.<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>May 11 12:34:59 host002 lrmd: [2561]: info: perform_op:2894: postponing all ops on resource MyApp_fs_graph by 1000 ms<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>And I noticed there are about 100 vgdisplay processes running in D state.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>How can I configure Pacemaker so the other host forces sg_persist to be a master and then just takes the whole resource group without fencing?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’ve tried “on-fail=standby” or “migration-threshold=0” but it just always gets stuck in this state. If I reconnect the LUN everything resumes and it instantly fails over but this is less than ideal.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks,<o:p></o:p></p><p class=MsoNormal>Max<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";mso-fareast-language:EN-GB'><br>________________________________________________________________________<br>In order to protect our email recipients, Betfair Group use SkyScan from <br>MessageLabs to scan all Incoming and Outgoing mail for viruses.<br><br>________________________________________________________________________<o:p></o:p></span></p></div><BR>
________________________________________________________________________<BR>
In order to protect our email recipients, Betfair Group use SkyScan from <BR>
MessageLabs to scan all Incoming and Outgoing mail for viruses.<BR>
<BR>
________________________________________________________________________<BR>
</body></html>