<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:#0070C0;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:#0070C0;
mso-fareast-language:EN-US;}
p.Courier, li.Courier, div.Courier
{mso-style-name:Courier;
mso-style-link:"Courier Char";
margin:0cm;
margin-bottom:.0001pt;
line-height:105%;
font-size:8.0pt;
font-family:"Courier New";
color:black;
mso-fareast-language:EN-US;}
span.CourierChar
{mso-style-name:"Courier Char";
mso-style-link:Courier;
font-family:"Courier New";
color:black;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;
color:#0070C0;}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE-AT" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText"><span lang="EN-US">Hi again<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Sorry for missing the omission of the master role within the colocation constraint.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">I added it - but unfortunately still no success.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">(In the meantime I added 2 additional filesystem resources on top of the NFSServer, but that should not change anything regarding the root problem that I miss the demote of DRBDClone.)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">I again started with all resources located at ventsi-clst1 and issued a 'pcs resource move DRBD_global_clst' (the resource next collocated next to the DRBDClone).<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">With that I end up with all primitive resources stopped and the DRBDClone resource still being master at ventsi-clst1.<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Here is what pacemaker pretends has to be done:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">==================================================================</span><span lang="EN-US" style="color:black"><o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">[root@ventsi-clst2 ~]# crm_simulate -Ls<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Current cluster status:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Online: [ ventsi-clst1-sync ventsi-clst2-sync ]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">ipmi-fence-clst1 (stonith:fence_ipmilan): Started ventsi-clst2-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">ipmi-fence-clst2 (stonith:fence_ipmilan): Started ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">IPaddrNFS (ocf::heartbeat:IPaddr2): Stopped<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">NFSServer (ocf::heartbeat:nfsserver): Stopped<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Master/Slave Set: DRBDClone [DRBD]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> <span style="background:yellow;mso-highlight:yellow">
Masters: [ ventsi-clst1-sync ]</span> </span><span lang="EN-US" style="font-size:11.0pt;line-height:105%;font-family:"Calibri",sans-serif;color:#0070C0"><=== still not demoted</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Slaves: [ ventsi-clst2-sync ]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">DRBD_global_clst (ocf::heartbeat:Filesystem): Stopped<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">NFS_global_clst (ocf::heartbeat:Filesystem): Stopped<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">BIND_global_clst (ocf::heartbeat:Filesystem): Stopped<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Allocation scores:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: ipmi-fence-clst1 allocation score on ventsi-clst1-sync: -INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: ipmi-fence-clst1 allocation score on ventsi-clst2-sync: INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: ipmi-fence-clst2 allocation score on ventsi-clst1-sync: INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: ipmi-fence-clst2 allocation score on ventsi-clst2-sync: -INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">clone_color: DRBDClone allocation score on ventsi-clst1-sync: 0<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">clone_color: DRBDClone allocation score on ventsi-clst2-sync: 0<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">clone_color: DRBD:0 allocation score on ventsi-clst1-sync: INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">clone_color: DRBD:0 allocation score on ventsi-clst2-sync: 0<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">clone_color: DRBD:1 allocation score on ventsi-clst1-sync: 0<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">clone_color: DRBD:1 allocation score on ventsi-clst2-sync: INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: DRBD:0 allocation score on ventsi-clst1-sync: INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: DRBD:0 allocation score on ventsi-clst2-sync: 0<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: DRBD:1 allocation score on ventsi-clst1-sync: -INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: DRBD:1 allocation score on ventsi-clst2-sync: INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">DRBD:1 promotion score on ventsi-clst2-sync: 10000<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">DRBD:0 promotion score on ventsi-clst1-sync: 1<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: DRBD_global_clst allocation score on ventsi-clst1-sync: -INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: DRBD_global_clst allocation score on ventsi-clst2-sync: INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: IPaddrNFS allocation score on ventsi-clst1-sync: -INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: IPaddrNFS allocation score on ventsi-clst2-sync: 0<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: NFSServer allocation score on ventsi-clst1-sync: -INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: NFSServer allocation score on ventsi-clst2-sync: 0<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: NFS_global_clst allocation score on ventsi-clst1-sync: 0<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: NFS_global_clst allocation score on ventsi-clst2-sync: -INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: BIND_global_clst allocation score on ventsi-clst1-sync: -INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">native_color: BIND_global_clst allocation score on ventsi-clst2-sync: 0<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Transition Summary:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Start IPaddrNFS (ventsi-clst2-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Start NFSServer (ventsi-clst2-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* <span style="background:yellow;mso-highlight:yellow">
Demote DRBD:0 (Master -> Slave ventsi-clst1-sync)</span> </span><span lang="EN-US" style="font-size:11.0pt;line-height:105%;font-family:"Calibri",sans-serif;color:#0070C0"><=== this demote never happens</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Promote DRBD:1 (Slave -> Master ventsi-clst2-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Start DRBD_global_clst (ventsi-clst2-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Start NFS_global_clst (ventsi-clst1-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Start BIND_global_clst (ventsi-clst2-sync)<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">And this is the executed transaction:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">==================================================================</span><span lang="EN-US" style="color:black"><o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">[root@ventsi-clst2 ~]# crm_simulate --xml-file /var/lib/pacemaker/pengine/pe-input-1157.bz2 --save-graph problem5.graph --save-dotfile problem5.dot -V --simulate<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Using the original execution date of: 2016-11-09 17:54:10Z<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Current cluster status:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Online: [ ventsi-clst1-sync ventsi-clst2-sync ]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">ipmi-fence-clst1 (stonith:fence_ipmilan): Started ventsi-clst2-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">ipmi-fence-clst2 (stonith:fence_ipmilan): Started ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">IPaddrNFS (ocf::heartbeat:IPaddr2): Started ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">NFSServer (ocf::heartbeat:nfsserver): Started ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Master/Slave Set: DRBDClone [DRBD]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Masters: [ ventsi-clst1-sync ]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Slaves: [ ventsi-clst2-sync ]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">DRBD_global_clst (ocf::heartbeat:Filesystem): Started ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">NFS_global_clst (ocf::heartbeat:Filesystem): Started ventsi-clst2-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">BIND_global_clst (ocf::heartbeat:Filesystem): Started ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Transition Summary:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Stop IPaddrNFS (ventsi-clst1-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Stop NFSServer (ventsi-clst1-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Stop DRBD_global_clst (ventsi-clst1-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Stop NFS_global_clst (Started ventsi-clst2-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Stop BIND_global_clst (ventsi-clst1-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Executing cluster transition:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Resource action: NFS_global_clst stop on ventsi-clst2-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Resource action: BIND_global_clst stop on ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Resource action: NFSServer stop on ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Resource action: IPaddrNFS stop on ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Resource action: DRBD_global_clst stop on ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">* Pseudo action: all_stopped </span><span lang="EN-US" style="font-size:11.0pt;line-height:105%;font-family:"Calibri",sans-serif;color:#0070C0"><=== no demote</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Using the original execution date of: 2016-11-09 17:54:10Z<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Revised cluster status:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Online: [ ventsi-clst1-sync ventsi-clst2-sync ]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">ipmi-fence-clst1 (stonith:fence_ipmilan): Started ventsi-clst2-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">ipmi-fence-clst2 (stonith:fence_ipmilan): Started ventsi-clst1-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">IPaddrNFS (ocf::heartbeat:IPaddr2): Stopped<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">NFSServer (ocf::heartbeat:nfsserver): Stopped<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Master/Slave Set: DRBDClone [DRBD]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Masters: [ ventsi-clst1-sync ]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Slaves: [ ventsi-clst2-sync ]<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">DRBD_global_clst (ocf::heartbeat:Filesystem): Stopped<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">NFS_global_clst (ocf::heartbeat:Filesystem): Stopped<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">BIND_global_clst (ocf::heartbeat:Filesystem): Stopped<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">And finally here the updated config:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">==================================================================</span><span lang="EN-US" style="color:black"><o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">[root@ventsi-clst1 ~]# pcs config<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Cluster Name: clst1<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Corosync Nodes:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">ventsi-clst1-sync ventsi-clst2-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Pacemaker Nodes:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">ventsi-clst1-sync ventsi-clst2-sync<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Resources:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Resource: IPaddrNFS (class=ocf provider=heartbeat type=IPaddr2)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Attributes: ip=xxx.xxx.xxx.xxx cidr_netmask=24<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Operations: start interval=0 timeout=20 (IPaddrNFS-start-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> stop interval=0 timeout=20 (IPaddrNFS-stop-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> monitor interval=10 timeout=20 (IPaddrNFS-monitor-interval-10)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Resource: NFSServer (class=ocf provider=heartbeat type=nfsserver)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Attributes: nfs_shared_infodir=/drbdmnts/global_clst/nfsserversettings/ nfs_ip=xxx.xxx.xxx.xxx nfsd_args="-H xxx.xxx.xxx.xxx"<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Operations: start interval=0 timeout=40 (NFSServer-start-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> stop interval=0 timeout=20 (NFSServer-stop-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> monitor interval=10 timeout=20 (NFSServer-monitor-interval-10)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Master: DRBDClone<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Meta Attrs: master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Resource: DRBD (class=ocf provider=linbit type=drbd)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Attributes: drbd_resource=nfsdata<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Operations: start interval=0 timeout=240 (DRBD-start-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> promote interval=0 timeout=90 (DRBD-promote-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> demote interval=0 timeout=90 (DRBD-demote-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> stop interval=0 timeout=100 (DRBD-stop-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> monitor interval=9 role=Master timeout=5 (DRBD-monitor-interval-9)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> monitor interval=10 role=Slave timeout=5 (DRBD-monitor-interval-10)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Resource: DRBD_global_clst (class=ocf provider=heartbeat type=Filesystem)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Attributes: device=/dev/drbd1 directory=/drbdmnts/global_clst fstype=ext4<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Operations: start interval=0 timeout=60 (DRBD_global_clst-start-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> stop interval=0 timeout=60 (DRBD_global_clst-stop-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> monitor interval=20 timeout=40 (DRBD_global_clst-monitor-interval-20)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Resource: NFS_global_clst (class=ocf provider=heartbeat type=Filesystem)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Attributes: device=xxx.xxx.xxx.xxx:/drbdmnts/global_clst/nfs directory=/global/nfs fstype=nfs<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Operations: start interval=0 timeout=60 (NFS_global_clst-start-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> stop interval=0 timeout=60 (NFS_global_clst-stop-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> monitor interval=20 timeout=40 (NFS_global_clst-monitor-interval-20)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Resource: BIND_global_clst (class=ocf provider=heartbeat type=Filesystem)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Attributes: device=/drbdmnts/global_clst/nfs directory=/global/nfs fstype=none options=bind<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Operations: start interval=0 timeout=60 (BIND_global_clst-start-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> stop interval=0 timeout=60 (BIND_global_clst-stop-interval-0)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> monitor interval=20 timeout=40 (BIND_global_clst-monitor-interval-20)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Stonith Devices:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Resource: ipmi-fence-clst1 (class=stonith type=fence_ipmilan)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Attributes: lanplus=1 login=foo passwd=bar action=reboot ipaddr=yyy.yyy.yyy.yyy pcmk_host_check=static-list pcmk_host_list=ventsi-clst1-sync auth=password timeout=30 cipher=1<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Operations: monitor interval=60 (ipmi-fence-clst1-monitor-interval-60)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Resource: ipmi-fence-clst2 (class=stonith type=fence_ipmilan)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Attributes: lanplus=1 login=foo passwd=bar action=reboot ipaddr=zzz.zzz.zzz.zzz pcmk_host_check=static-list pcmk_host_list=ventsi-clst2-sync auth=password timeout=30 cipher=1<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Operations: monitor interval=60 (ipmi-fence-clst2-monitor-interval-60)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Fencing Levels:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Location Constraints:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Resource: DRBD_global_clst<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Disabled on: ventsi-clst1-sync (score:-INFINITY) (role: Started) (id:cli-ban-DRBD_global_clst-on-ventsi-clst1-sync)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Resource: ipmi-fence-clst1<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Disabled on: ventsi-clst1-sync (score:-INFINITY) (id:location-ipmi-fence-clst1-ventsi-clst1-sync--INFINITY)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Resource: ipmi-fence-clst2<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> Disabled on: ventsi-clst2-sync (score:-INFINITY) (id:location-ipmi-fence-clst2-ventsi-clst2-sync--INFINITY)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Ordering Constraints:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> start IPaddrNFS then start NFSServer (kind:Mandatory) (id:order-IPaddrNFS-NFSServer-mandatory)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> promote DRBDClone then start DRBD_global_clst (kind:Mandatory) (id:order-DRBDClone-DRBD_global_clst-mandatory)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> start DRBD_global_clst then start IPaddrNFS (kind:Mandatory) (id:order-DRBD_global_clst-IPaddrNFS-mandatory)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> start NFSServer then start NFS_global_clst (kind:Mandatory) (id:order-NFSServer-NFS_global_clst-mandatory)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> start NFSServer then start BIND_global_clst (kind:Mandatory) (id:order-NFSServer-BIND_global_clst-mandatory)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Colocation Constraints:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> NFSServer with IPaddrNFS (score:INFINITY) (id:colocation-NFSServer-IPaddrNFS-INFINITY)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> IPaddrNFS with DRBD_global_clst (score:INFINITY) (id:colocation-IPaddrNFS-DRBD_global_clst-INFINITY)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> NFS_global_clst with NFSServer (score:-INFINITY) (id:colocation-NFS_global_clst-NFSServer--INFINITY)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> BIND_global_clst with NFSServer (score:INFINITY) (id:colocation-BIND_global_clst-NFSServer-INFINITY)<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"> <span style="background:yellow;mso-highlight:yellow">
DRBD_global_clst with DRBDClone (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-DRBD_global_clst-DRBDClone-INFINITY)</span><o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Resources Defaults:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">resource-stickiness: INFINITY<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Operations Defaults:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">timeout: 10s<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="Courier"><span lang="EN-US">Cluster Properties:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">cluster-infrastructure: cman<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">dc-version: 1.1.14-8.el6-70404b0<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">have-watchdog: false<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">last-lrm-refresh: 1478703150<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">no-quorum-policy: ignore<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">stonith-enabled: true<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">symmetric-cluster: true<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">Node Attributes:<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">ventsi-clst1-sync: PostgresSon-data-status=DISCONNECT<o:p></o:p></span></p>
<p class="Courier"><span lang="EN-US">ventsi-clst2-sync: PostgresSon-data-status=DISCONNECT<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Kind regards<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Andi<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US" style="mso-fareast-language:DE-AT">-----Original Message-----<br>
From: Ken Gaillot [mailto:kgaillot@redhat.com] <br>
Sent: Dienstag, 8. November 2016 22:29<br>
To: users@clusterlabs.org<br>
Subject: Re: [ClusterLabs] DRBD demote/promote not called - Why? How to fix?</span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">On 11/04/2016 01:57 PM, CART Andreas wrote:<o:p></o:p></p>
<p class="MsoPlainText">> Hi<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> I have a basic 2 node active/passive cluster with Pacemaker (1.1.14 ,<o:p></o:p></p>
<p class="MsoPlainText">> pcs: 0.9.148) / CMAN (3.0.12.1) / Corosync (1.4.7) on RHEL 6.8.<o:p></o:p></p>
<p class="MsoPlainText">> This cluster runs NFS on top of DRBD (8.4.4).<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Basically the system is working on both nodes and I can switch the<o:p></o:p></p>
<p class="MsoPlainText">> resources from one node to the other.<o:p></o:p></p>
<p class="MsoPlainText">> But switching resources to the other node does not work, if I try to<o:p></o:p></p>
<p class="MsoPlainText">> move just one resource and have the others follow due to the location<o:p></o:p></p>
<p class="MsoPlainText">> constraints.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> From the logged messages I see that in this “failure case” there is NO<o:p></o:p></p>
<p class="MsoPlainText">> attempt to demote/promote the DRBD clone resource.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Here is my setup:<o:p></o:p></p>
<p class="MsoPlainText">> ==================================================================<o:p></o:p></p>
<p class="MsoPlainText">> Cluster Name: clst1<o:p></o:p></p>
<p class="MsoPlainText">> Corosync Nodes:<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst1-sync ventsi-clst2-sync<o:p></o:p></p>
<p class="MsoPlainText">> Pacemaker Nodes:<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst1-sync ventsi-clst2-sync<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Resources:<o:p></o:p></p>
<p class="MsoPlainText">> Resource: IPaddrNFS (class=ocf provider=heartbeat type=IPaddr2)<o:p></o:p></p>
<p class="MsoPlainText">> Attributes: ip=xxx.xxx.xxx.xxx cidr_netmask=24<o:p></o:p></p>
<p class="MsoPlainText">> Operations: start interval=0s timeout=20s (IPaddrNFS-start-interval-0s)<o:p></o:p></p>
<p class="MsoPlainText">> stop interval=0s timeout=20s (IPaddrNFS-stop-interval-0s)<o:p></o:p></p>
<p class="MsoPlainText">> monitor interval=5s (IPaddrNFS-monitor-interval-5s)<o:p></o:p></p>
<p class="MsoPlainText">> Resource: NFSServer (class=ocf provider=heartbeat type=nfsserver)<o:p></o:p></p>
<p class="MsoPlainText">> Attributes: nfs_shared_infodir=/var/lib/nfsserversettings/<o:p></o:p></p>
<p class="MsoPlainText">> nfs_ip=xxx.xxx.xxx.xxx nfsd_args="-H xxx.xxx.xxx.xxx"<o:p></o:p></p>
<p class="MsoPlainText">> Operations: start interval=0s timeout=40 (NFSServer-start-interval-0s)<o:p></o:p></p>
<p class="MsoPlainText">> stop interval=0s timeout=20s (NFSServer-stop-interval-0s)<o:p></o:p></p>
<p class="MsoPlainText">> monitor interval=10s timeout=20s<o:p></o:p></p>
<p class="MsoPlainText">> (NFSServer-monitor-interval-10s)<o:p></o:p></p>
<p class="MsoPlainText">> Master: DRBDClone<o:p></o:p></p>
<p class="MsoPlainText">> Meta Attrs: master-max=1 master-node-max=1 clone-max=2<o:p></o:p></p>
<p class="MsoPlainText">> clone-node-max=1 notify=true<o:p></o:p></p>
<p class="MsoPlainText">> Resource: DRBD (class=ocf provider=linbit type=drbd)<o:p></o:p></p>
<p class="MsoPlainText">> Attributes: drbd_resource=nfsdata<o:p></o:p></p>
<p class="MsoPlainText">> Operations: start interval=0s timeout=240 (DRBD-start-interval-0s)<o:p></o:p></p>
<p class="MsoPlainText">> promote interval=0s timeout=90 (DRBD-promote-interval-0s)<o:p></o:p></p>
<p class="MsoPlainText">> demote interval=0s timeout=90 (DRBD-demote-interval-0s)<o:p></o:p></p>
<p class="MsoPlainText">> stop interval=0s timeout=100 (DRBD-stop-interval-0s)<o:p></o:p></p>
<p class="MsoPlainText">> monitor interval=1s timeout=5 (DRBD-monitor-interval-1s)<o:p></o:p></p>
<p class="MsoPlainText">> Resource: DRBD_global_clst (class=ocf provider=heartbeat type=Filesystem)<o:p></o:p></p>
<p class="MsoPlainText">> Attributes: device=/dev/drbd1 directory=/drbdmnts/global_clst fstype=ext4<o:p></o:p></p>
<p class="MsoPlainText">> Operations: start interval=0s timeout=60<o:p></o:p></p>
<p class="MsoPlainText">> (DRBD_global_clst-start-interval-0s)<o:p></o:p></p>
<p class="MsoPlainText">> stop interval=0s timeout=60<o:p></o:p></p>
<p class="MsoPlainText">> (DRBD_global_clst-stop-interval-0s)<o:p></o:p></p>
<p class="MsoPlainText">> monitor interval=20 timeout=40<o:p></o:p></p>
<p class="MsoPlainText">> (DRBD_global_clst-monitor-interval-20)<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Stonith Devices:<o:p></o:p></p>
<p class="MsoPlainText">> Resource: ipmi-fence-clst1 (class=stonith type=fence_ipmilan)<o:p></o:p></p>
<p class="MsoPlainText">> Attributes: lanplus=1 login=foo passwd=bar action=reboot<o:p></o:p></p>
<p class="MsoPlainText">> ipaddr=yyy.yyy.yyy.yyy pcmk_host_check=static-list<o:p></o:p></p>
<p class="MsoPlainText">> pcmk_host_list=ventsi-clst1-sync auth=password timeout=30 cipher=1<o:p></o:p></p>
<p class="MsoPlainText">> Operations: monitor interval=60s (ipmi-fence-clst1-monitor-interval-60s)<o:p></o:p></p>
<p class="MsoPlainText">> Resource: ipmi-fence-clst2 (class=stonith type=fence_ipmilan)<o:p></o:p></p>
<p class="MsoPlainText">> Attributes: lanplus=1 login=foo passwd=bar action=reboot<o:p></o:p></p>
<p class="MsoPlainText">> ipaddr=zzz.zzz.zzz.zzz pcmk_host_check=static-list<o:p></o:p></p>
<p class="MsoPlainText">> pcmk_host_list=ventsi-clst2-sync auth=password timeout=30 cipher=1<o:p></o:p></p>
<p class="MsoPlainText">> Operations: monitor interval=60s (ipmi-fence-clst2-monitor-interval-60s)<o:p></o:p></p>
<p class="MsoPlainText">> Fencing Levels:<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Location Constraints:<o:p></o:p></p>
<p class="MsoPlainText">> Resource: ipmi-fence-clst1<o:p></o:p></p>
<p class="MsoPlainText">> Disabled on: ventsi-clst1-sync (score:-INFINITY)<o:p></o:p></p>
<p class="MsoPlainText">> (id:location-ipmi-fence-clst1-ventsi-clst1-sync--INFINITY)<o:p></o:p></p>
<p class="MsoPlainText">> Resource: ipmi-fence-clst2<o:p></o:p></p>
<p class="MsoPlainText">> Disabled on: ventsi-clst2-sync (score:-INFINITY)<o:p></o:p></p>
<p class="MsoPlainText">> (id:location-ipmi-fence-clst2-ventsi-clst2-sync--INFINITY)<o:p></o:p></p>
<p class="MsoPlainText">> Ordering Constraints:<o:p></o:p></p>
<p class="MsoPlainText">> start IPaddrNFS then start NFSServer (kind:Mandatory)<o:p></o:p></p>
<p class="MsoPlainText">> (id:order-IPaddrNFS-NFSServer-mandatory)<o:p></o:p></p>
<p class="MsoPlainText">> promote DRBDClone then start DRBD_global_clst (kind:Mandatory)<o:p></o:p></p>
<p class="MsoPlainText">> (id:order-DRBDClone-DRBD_global_clst-mandatory)<o:p></o:p></p>
<p class="MsoPlainText">> start DRBD_global_clst then start IPaddrNFS (kind:Mandatory)<o:p></o:p></p>
<p class="MsoPlainText">> (id:order-DRBD_global_clst-IPaddrNFS-mandatory)<o:p></o:p></p>
<p class="MsoPlainText">> Colocation Constraints:<o:p></o:p></p>
<p class="MsoPlainText">> NFSServer with IPaddrNFS (score:INFINITY)<o:p></o:p></p>
<p class="MsoPlainText">> (id:colocation-NFSServer-IPaddrNFS-INFINITY)<o:p></o:p></p>
<p class="MsoPlainText">> DRBD_global_clst with DRBDClone (score:INFINITY)<o:p></o:p></p>
<p class="MsoPlainText">> (id:colocation-DRBD_global_clst-DRBDClone-INFINITY)<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">It took me a while to notice it, it's easily overlooked, but the above<o:p></o:p></p>
<p class="MsoPlainText">constraint is the problem. It says DRBD_global_clst must be located<o:p></o:p></p>
<p class="MsoPlainText">where DRBDClone is running ... not necessarily where DRBDClone is<o:p></o:p></p>
<p class="MsoPlainText">master. This constraint should be created like this:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">pcs constraint colocation add DRBD_global_clst with master DBRDClone<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> IPaddrNFS with DRBD_global_clst (score:INFINITY)<o:p></o:p></p>
<p class="MsoPlainText">> (id:colocation-IPaddrNFS-DRBD_global_clst-INFINITY)<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Resources Defaults:<o:p></o:p></p>
<p class="MsoPlainText">> resource-stickiness: INFINITY<o:p></o:p></p>
<p class="MsoPlainText">> Operations Defaults:<o:p></o:p></p>
<p class="MsoPlainText">> timeout: 10s<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Cluster Properties:<o:p></o:p></p>
<p class="MsoPlainText">> cluster-infrastructure: cman<o:p></o:p></p>
<p class="MsoPlainText">> dc-version: 1.1.14-8.el6-70404b0<o:p></o:p></p>
<p class="MsoPlainText">> have-watchdog: false<o:p></o:p></p>
<p class="MsoPlainText">> last-lrm-refresh: 1478277432<o:p></o:p></p>
<p class="MsoPlainText">> no-quorum-policy: ignore<o:p></o:p></p>
<p class="MsoPlainText">> stonith-enabled: true<o:p></o:p></p>
<p class="MsoPlainText">> symmetric-cluster: true<o:p></o:p></p>
<p class="MsoPlainText">> ==================================================================<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Initial state is e.g. this (all resources at node1):<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Online: [ ventsi-clst1-sync ventsi-clst2-sync ]<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Full list of resources:<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> ipmi-fence-clst1 (stonith:fence_ipmilan): Started<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst2-sync<o:p></o:p></p>
<p class="MsoPlainText">> ipmi-fence-clst2 (stonith:fence_ipmilan): Started<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst1-sync<o:p></o:p></p>
<p class="MsoPlainText">> IPaddrNFS (ocf::heartbeat:IPaddr2): Started ventsi-clst1-sync<o:p></o:p></p>
<p class="MsoPlainText">> NFSServer (ocf::heartbeat:nfsserver): Started ventsi-clst1-sync<o:p></o:p></p>
<p class="MsoPlainText">> Master/Slave Set: DRBDClone [DRBD]<o:p></o:p></p>
<p class="MsoPlainText">> Masters: [ ventsi-clst1-sync ]<o:p></o:p></p>
<p class="MsoPlainText">> Slaves: [ ventsi-clst2-sync ]<o:p></o:p></p>
<p class="MsoPlainText">> DRBD_global_clst (ocf::heartbeat:Filesystem): Started<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst1-sync<o:p></o:p></p>
<p class="MsoPlainText">> ==================================================================<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> If I shutdown the cluster at node 1 (‘pcs cluster stop’) or if I move<o:p></o:p></p>
<p class="MsoPlainText">> the DRBD clone resource (‘pcs resource move DRBDClone’) all resources<o:p></o:p></p>
<p class="MsoPlainText">> switch successfully to node2.<o:p></o:p></p>
<p class="MsoPlainText">> I.e. the demote/promote of the DRBD clone resource is working in these<o:p></o:p></p>
<p class="MsoPlainText">> cases.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> But if I try to move any other resource (e.g. ‘pcs resource move<o:p></o:p></p>
<p class="MsoPlainText">> NFSServer’) the resources NFSServer, IPaddrNFS and DRBD_global_clst are<o:p></o:p></p>
<p class="MsoPlainText">> stopped at node 1, but then already follows starting of the<o:p></o:p></p>
<p class="MsoPlainText">> DRBD_global_clst resource at node2, which fails due to the missing<o:p></o:p></p>
<p class="MsoPlainText">> demote/promote.<o:p></o:p></p>
<p class="MsoPlainText">> As far as I can see there is some follow-up attempt to repair things<o:p></o:p></p>
<p class="MsoPlainText">> partially as the resources are started again at node1 exclusive the<o:p></o:p></p>
<p class="MsoPlainText">> resource which I moved due to my move command.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Final state is like this:<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Online: [ ventsi-clst1-sync ventsi-clst2-sync ]<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Full list of resources:<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> ipmi-fence-clst1 (stonith:fence_ipmilan): Started<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst2-sync<o:p></o:p></p>
<p class="MsoPlainText">> ipmi-fence-clst2 (stonith:fence_ipmilan): Started<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst1-sync<o:p></o:p></p>
<p class="MsoPlainText">> IPaddrNFS (ocf::heartbeat:IPaddr2): Started ventsi-clst1-sync<o:p></o:p></p>
<p class="MsoPlainText">> NFSServer (ocf::heartbeat:nfsserver): Stopped<o:p></o:p></p>
<p class="MsoPlainText">> Master/Slave Set: DRBDClone [DRBD]<o:p></o:p></p>
<p class="MsoPlainText">> Masters: [ ventsi-clst1-sync ]<o:p></o:p></p>
<p class="MsoPlainText">> Slaves: [ ventsi-clst2-sync ]<o:p></o:p></p>
<p class="MsoPlainText">> DRBD_global_clst (ocf::heartbeat:Filesystem): Started<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst1-sync<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Failed Actions:<o:p></o:p></p>
<p class="MsoPlainText">> * DRBD_global_clst_start_0 on ventsi-clst2-sync 'unknown error' (1):<o:p></o:p></p>
<p class="MsoPlainText">> call=778, status=complete, exitreason='none',<o:p></o:p></p>
<p class="MsoPlainText">> last-rc-change='Fri Nov 4 19:32:56 2016', queued=0ms, exec=43ms<o:p></o:p></p>
<p class="MsoPlainText">> ==================================================================<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Here are the logged messages for this “failure case”:<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.163982+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC<o:p></o:p></p>
<p class="MsoPlainText">> cause=C_FSA_INTERNAL origin=abort_transition_graph ]<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.168100+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> On loss of CCM Quorum: Ignore<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.181252+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Move IPaddrNFS#011(Started ventsi-clst1-sync -> ventsi-clst2-sync)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.181260+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Move NFSServer#011(Started ventsi-clst1-sync -> ventsi-clst2-sync)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.181278+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Move DRBD_global_clst#011(Started ventsi-clst1-sync -><o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst2-sync) <=== here no demote/promote is listed<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.182385+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Calculated Transition 202: /var/lib/pacemaker/pengine/pe-input-766.bz2<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.182998+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Initiating action 15: stop NFSServer_stop_0 on ventsi-clst1-sync (local)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.196265+01:00 ventsi-clst1<o:p></o:p></p>
<p class="MsoPlainText">> nfsserver(NFSServer)[15978]: INFO: Stopping NFS server ...<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.249137+01:00 ventsi-clst1 kernel: nfsd: last server<o:p></o:p></p>
<p class="MsoPlainText">> has exited, flushing export cache<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.252241+01:00 ventsi-clst1 rpc.mountd[15282]: Caught<o:p></o:p></p>
<p class="MsoPlainText">> signal 15, un-registering and exiting.<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.632708+01:00 ventsi-clst1<o:p></o:p></p>
<p class="MsoPlainText">> nfsserver(NFSServer)[15978]: INFO: Stopping sm-notify<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.650552+01:00 ventsi-clst1<o:p></o:p></p>
<p class="MsoPlainText">> nfsserver(NFSServer)[15978]: INFO: Stopping rpc.statd<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:55.666777+01:00 ventsi-clst1 rpc.statd[15243]: Caught<o:p></o:p></p>
<p class="MsoPlainText">> signal 15, un-registering and exiting<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.692819+01:00 ventsi-clst1<o:p></o:p></p>
<p class="MsoPlainText">> nfsserver(NFSServer)[15978]: INFO: NFS server stopped<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.695523+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Operation NFSServer_stop_0: ok (node=ventsi-clst1-sync, call=1220, rc=0,<o:p></o:p></p>
<p class="MsoPlainText">> cib-update=1695, confirmed=true)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.696243+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Initiating action 12: stop IPaddrNFS_stop_0 on ventsi-clst1-sync (local)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.727882+01:00 ventsi-clst1 IPaddr2(IPaddrNFS)[16108]:<o:p></o:p></p>
<p class="MsoPlainText">> INFO: IP status = ok, IP_CIP=<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.733383+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Operation IPaddrNFS_stop_0: ok (node=ventsi-clst1-sync, call=1222, rc=0,<o:p></o:p></p>
<p class="MsoPlainText">> cib-update=1696, confirmed=true)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.733917+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Initiating action 48: stop DRBD_global_clst_stop_0 on ventsi-clst1-sync<o:p></o:p></p>
<p class="MsoPlainText">> (local)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.757181+01:00 ventsi-clst1<o:p></o:p></p>
<p class="MsoPlainText">> Filesystem(DRBD_global_clst)[16163]: INFO: Running stop for /dev/drbd1<o:p></o:p></p>
<p class="MsoPlainText">> on /drbdmnts/global_clst<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.764684+01:00 ventsi-clst1<o:p></o:p></p>
<p class="MsoPlainText">> Filesystem(DRBD_global_clst)[16163]: INFO: Trying to unmount<o:p></o:p></p>
<p class="MsoPlainText">> /drbdmnts/global_clst<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.771260+01:00 ventsi-clst1<o:p></o:p></p>
<p class="MsoPlainText">> Filesystem(DRBD_global_clst)[16163]: INFO: unmounted<o:p></o:p></p>
<p class="MsoPlainText">> /drbdmnts/global_clst successfully<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.776640+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Operation DRBD_global_clst_stop_0: ok (node=ventsi-clst1-sync,<o:p></o:p></p>
<p class="MsoPlainText">> call=1224, rc=0, cib-update=1697, confirmed=true)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.777140+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Initiating action 49: start DRBD_global_clst_start_0 on<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst2-sync <=== hereis the attempt to start the filesystem at<o:p></o:p></p>
<p class="MsoPlainText">> the other node, although DRBD has not yet been promoted<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.840137+01:00 ventsi-clst1 crmd[6116]: warning:<o:p></o:p></p>
<p class="MsoPlainText">> Action 49 (DRBD_global_clst_start_0) on ventsi-clst2-sync failed<o:p></o:p></p>
<p class="MsoPlainText">> (target: 0 vs. rc: 1): Error<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.840158+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Transition aborted by DRBD_global_clst_start_0 'modify' on<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst2-sync: Event failed<o:p></o:p></p>
<p class="MsoPlainText">> (magic=0:1;49:202:0:b7941532-c74b-40cc-a8ad-27b5502b8fba, cib=0.649.4,<o:p></o:p></p>
<p class="MsoPlainText">> source=match_graph_event:381, 0)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.840232+01:00 ventsi-clst1 crmd[6116]: warning:<o:p></o:p></p>
<p class="MsoPlainText">> Action 49 (DRBD_global_clst_start_0) on ventsi-clst2-sync failed<o:p></o:p></p>
<p class="MsoPlainText">> (target: 0 vs. rc: 1): Error<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.840328+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Transition 202 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=5,<o:p></o:p></p>
<p class="MsoPlainText">> Source=/var/lib/pacemaker/pengine/pe-input-766.bz2): Complete<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.843693+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> On loss of CCM Quorum: Ignore<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.844072+01:00 ventsi-clst1 pengine[6115]: warning:<o:p></o:p></p>
<p class="MsoPlainText">> Processing failed op start for DRBD_global_clst on ventsi-clst2-sync:<o:p></o:p></p>
<p class="MsoPlainText">> unknown error (1)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.844102+01:00 ventsi-clst1 pengine[6115]: warning:<o:p></o:p></p>
<p class="MsoPlainText">> Processing failed op start for DRBD_global_clst on ventsi-clst2-sync:<o:p></o:p></p>
<p class="MsoPlainText">> unknown error (1)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.845071+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Start IPaddrNFS#011(ventsi-clst2-sync)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.845078+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Start NFSServer#011(ventsi-clst2-sync)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.845081+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Demote DRBD:0#011(Master -> Slave ventsi-clst1-sync) <=== here there<o:p></o:p></p>
<p class="MsoPlainText">> would be the necessarydemote/promote … but it’s too late; the start of<o:p></o:p></p>
<p class="MsoPlainText">> the filesystem already failed…<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.845083+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Promote DRBD:1#011(Slave -> Master ventsi-clst2-sync)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.845084+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Recover DRBD_global_clst#011(Started ventsi-clst2-sync)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.847986+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Calculated Transition 203: /var/lib/pacemaker/pengine/pe-input-767.bz2
<o:p></o:p></p>
<p class="MsoPlainText">> <=== … so the above transition gets caught by thefollowing attempt to<o:p></o:p></p>
<p class="MsoPlainText">> repair things partially<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.867679+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> On loss of CCM Quorum: Ignore<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.868074+01:00 ventsi-clst1 pengine[6115]: warning:<o:p></o:p></p>
<p class="MsoPlainText">> Processing failed op start for DRBD_global_clst on ventsi-clst2-sync:<o:p></o:p></p>
<p class="MsoPlainText">> unknown error (1)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.868101+01:00 ventsi-clst1 pengine[6115]: warning:<o:p></o:p></p>
<p class="MsoPlainText">> Processing failed op start for DRBD_global_clst on ventsi-clst2-sync:<o:p></o:p></p>
<p class="MsoPlainText">> unknown error (1)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.868287+01:00 ventsi-clst1 pengine[6115]: warning:<o:p></o:p></p>
<p class="MsoPlainText">> Forcing DRBD_global_clst away from ventsi-clst2-sync after 1000000<o:p></o:p></p>
<p class="MsoPlainText">> failures (max=1000000)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.869011+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Start IPaddrNFS#011(ventsi-clst1-sync)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.869023+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Recover DRBD_global_clst#011(Started ventsi-clst2-sync -> ventsi-clst1-sync)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.869770+01:00 ventsi-clst1 pengine[6115]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Calculated Transition 204: /var/lib/pacemaker/pengine/pe-input-768.bz2<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.870065+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Initiating action 3: stop DRBD_global_clst_stop_0 on ventsi-clst2-sync<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.908075+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Initiating action 42: start DRBD_global_clst_start_0 on<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst1-sync (local)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.931072+01:00 ventsi-clst1<o:p></o:p></p>
<p class="MsoPlainText">> Filesystem(DRBD_global_clst)[16242]: INFO: Running start for /dev/drbd1<o:p></o:p></p>
<p class="MsoPlainText">> on /drbdmnts/global_clst<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.943250+01:00 ventsi-clst1 kernel: EXT4-fs (drbd1):<o:p></o:p></p>
<p class="MsoPlainText">> warning: maximal mount count reached, running e2fsck is recommended<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.953253+01:00 ventsi-clst1 kernel: EXT4-fs (drbd1):<o:p></o:p></p>
<p class="MsoPlainText">> mounted filesystem with ordered data mode. Opts:<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.964284+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Operation DRBD_global_clst_start_0: ok (node=ventsi-clst1-sync,<o:p></o:p></p>
<p class="MsoPlainText">> call=1225, rc=0, cib-update=1701, confirmed=true)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.965104+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Initiating action 10: start IPaddrNFS_start_0 on ventsi-clst1-sync (local)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.965325+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Initiating action 43: monitor DRBD_global_clst_monitor_20000 on<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst1-sync (local)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:56.996235+01:00 ventsi-clst1 IPaddr2(IPaddrNFS)[16308]:<o:p></o:p></p>
<p class="MsoPlainText">> INFO: Adding inet address xxx.xxx.xxx.xxx/24 with broadcast address<o:p></o:p></p>
<p class="MsoPlainText">> xxx.xxx.xxx.255 to device bond0<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:57.002059+01:00 ventsi-clst1 IPaddr2(IPaddrNFS)[16308]:<o:p></o:p></p>
<p class="MsoPlainText">> INFO: Bringing device bond0 up<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:57.008128+01:00 ventsi-clst1 IPaddr2(IPaddrNFS)[16308]:<o:p></o:p></p>
<p class="MsoPlainText">> INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p<o:p></o:p></p>
<p class="MsoPlainText">> /var/run/resource-agents/send_arp-xxx.xxx.xxx.xxx bond0 xxx.xxx.xxx.xxx<o:p></o:p></p>
<p class="MsoPlainText">> auto not_used not_used<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:57.020159+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Operation IPaddrNFS_start_0: ok (node=ventsi-clst1-sync, call=1226,<o:p></o:p></p>
<p class="MsoPlainText">> rc=0, cib-update=1703, confirmed=true)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:57.020901+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Initiating action 11: monitor IPaddrNFS_monitor_5000 on<o:p></o:p></p>
<p class="MsoPlainText">> ventsi-clst1-sync (local)<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:57.052231+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> Transition 204 (Complete=6, Pending=0, Fired=0, Skipped=0, Incomplete=0,<o:p></o:p></p>
<p class="MsoPlainText">> Source=/var/lib/pacemaker/pengine/pe-input-768.bz2): Complete<o:p></o:p></p>
<p class="MsoPlainText">> 2016-11-04T19:32:57.052251+01:00 ventsi-clst1 crmd[6116]: notice:<o:p></o:p></p>
<p class="MsoPlainText">> State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS<o:p></o:p></p>
<p class="MsoPlainText">> cause=C_FSA_INTERNAL origin=notify_crmd ]<o:p></o:p></p>
<p class="MsoPlainText">> ==================================================================<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Any ideas what could be the reason for this behavior?<o:p></o:p></p>
<p class="MsoPlainText">> And how could this be fixed?<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> (I already found several articles on the internet with the<o:p></o:p></p>
<p class="MsoPlainText">> recommendation to have two separately configured monitor operations for<o:p></o:p></p>
<p class="MsoPlainText">> the DRBD resource configured one for the master role and another one for<o:p></o:p></p>
<p class="MsoPlainText">> the slave role.<o:p></o:p></p>
<p class="MsoPlainText">> Already tried this to no avail.)<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Regards<o:p></o:p></p>
<p class="MsoPlainText">> Andi<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">_______________________________________________<o:p></o:p></p>
<p class="MsoPlainText">Users mailing list: Users@clusterlabs.org<o:p></o:p></p>
<p class="MsoPlainText">http://clusterlabs.org/mailman/listinfo/users<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Project Home: http://www.clusterlabs.org<o:p></o:p></p>
<p class="MsoPlainText">Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf<o:p></o:p></p>
<p class="MsoPlainText">Bugs: http://bugs.clusterlabs.org<o:p></o:p></p>
</div>
</body>
</html>