<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Baskerville;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle26
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:windowtext'>></span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:windowtext'> I don't think that having a hook that bypass stonith is the right way….<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The intention is NOT to bypass STONITH. STONITH shall always remain active, and an integral part of the cluster. The discussion is about bailing out of situations when the STONITH itself fails due to fencing agent failures, and how one can automate the process of bailing out.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>All that the surviving nodes in the cluster need to be informed is that the failed node has indeed failed, and therefore the suggestion for a hook.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The hook (lets’ say: STONITH-Failure-Recovery-Hook) under discussion will only be fired when Fencing Agent fails. STONITH-Failure-Recovery-Hook is realized via a script. The "${CRM_alert_rsc}" , "${CRM_alert_task}", "${CRM_alert_desc}" "${CRM_alert_node}" in the Pacemaker Alert can use used to match up with STONITH resource and its failures, and invoke the STONITH-Failure-Recovery-Hook as appropriate.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I also agree with Klaus that a quorum device is a good strategy.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>That needs 3<sup>rd</sup> node in the cluster. If such an option can be exercised, it should be.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanx.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'> Tomer Azran [mailto:tomer.azran@edp.co.il] <br><b>Sent:</b> Tuesday, July 25, 2017 3:00 AM<br><b>To:</b> kwenning@redhat.com; Cluster Labs - All topics related to open-source clustering welcomed; Prasad, Shashank<br><b>Subject:</b> RE: [ClusterLabs] Two nodes cluster issue<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I tend to agree with Klaus – I don't think that having a hook that bypass stonith is the right way. It is better to not use stonith at all.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I think I will try to use an iScsi target on my qdevice and set SBD to use it.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I still don't understand why qdevice can't take the place SBD with shared storage; correct me if I'm wrong, but it looks like both of them are there for the same reason.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:windowtext'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:windowtext'> Klaus Wenninger [<a href="mailto:kwenning@redhat.com">mailto:kwenning@redhat.com</a>] <br><b>Sent:</b> Monday, July 24, 2017 9:01 PM<br><b>To:</b> Cluster Labs - All topics related to open-source clustering welcomed <<a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a>>; Prasad, Shashank <<a href="mailto:ssprasad@vanu.com">ssprasad@vanu.com</a>><br><b>Subject:</b> Re: [ClusterLabs] Two nodes cluster issue<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On 07/24/2017 07:32 PM, Prasad, Shashank wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Sometimes IPMI fence devices use shared power of the node, and it cannot be avoided.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>In such scenarios the HA cluster is NOT able to handle the power failure of a node, since the power is shared with its own fence device.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The failure of IPMI based fencing can also exist due to other reasons also.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>A failure to fence the failed node will cause cluster to be marked UNCLEAN.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>To get over it, the following command needs to be invoked on the surviving node.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>pcs stonith confirm <failed_node_name> --force</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>This can be automated by hooking a recovery script, when the the Stonith resource ‘Timed Out’ event.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>To be more specific, the Pacemaker Alerts can be used for watch for Stonith timeouts and failures.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>In that script, all that’s essentially to be executed is the aforementioned command.</span><o:p></o:p></p></blockquote><p class=MsoNormal style='margin-bottom:12.0pt'><br>If I get you right here you can disable fencing then in the first place.<br>Actually quorum-based-watchdog-fencing is the way to do this in a<br>safe manner. This of course assumes you have a proper source for<br>quorum in your 2-node-setup with e.g. qdevice or using a shared<br>disk with sbd (not directly pacemaker quorum here but similar thing<br>handled inside sbd).<br><br><o:p></o:p></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Since the alerts are issued from ‘hacluster’ login, sudo permissions for ‘hacluster’ needs to be configured.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanx.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'> Klaus Wenninger [</span><a href="mailto:kwenning@redhat.com"><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>mailto:kwenning@redhat.com</span></a><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>] <br><b>Sent:</b> Monday, July 24, 2017 9:24 PM<br><b>To:</b> Kristián Feldsam; Cluster Labs - All topics related to open-source clustering welcomed<br><b>Subject:</b> Re: [ClusterLabs] Two nodes cluster issue</span><o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><div><p class=MsoNormal>On 07/24/2017 05:37 PM, Kristián Feldsam wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal>I personally think that power off node by switched pdu is more safe, or not?<o:p></o:p></p></blockquote><p class=MsoNormal style='margin-bottom:12.0pt'><br>True if that is working in you environment. If you can't do a physical setup<br>where you aren't simultaneously loosing connection to both your node and<br>the switch-device (or you just want to cover cases where that happens)<br>you have to come up with something else.<br><br><br><o:p></o:p></p><div><p class=MsoNormal><br><span style='font-family:Baskerville'>S pozdravem Kristián Feldsam<br>Tel.: +420 773 303 353, +421 944 137 535<br>E-mail.: </span><a href="mailto:support@feldhost.cz"><span style='font-family:Baskerville'>support@feldhost.cz</span></a><span style='font-family:Baskerville'><br><br></span><a href="http://www.feldhost.cz"><span style='font-family:Baskerville'>www.feldhost.cz</span></a><span style='font-family:Baskerville'> -<span class=apple-converted-space> </span><b>Feld</b>Host</span>™<span class=apple-converted-space><span style='font-family:Baskerville'> </span></span><span style='font-family:Baskerville'>– profesionální hostingové a serverové služby za adekvátní ceny.<br><br>FELDSAM s.r.o.<br>V rohu 434/3<br>Praha 4 – Libuš, PSČ 142 00<br>IČ: 290 60 958, DIČ: CZ290 60 958<br>C 200350 vedená u Městského soudu v Praze<br><br>Banka: Fio banka a.s.<br>Číslo účtu: 2400330446/2010<br>BIC: FIOBCZPPXX<br>IBAN: CZ82 2010 0000 0024 0033 0446</span> <o:p></o:p></p></div><p class=MsoNormal> <o:p></o:p></p><div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>On 24 Jul 2017, at 17:27, Klaus Wenninger <<a href="mailto:kwenning@redhat.com">kwenning@redhat.com</a>> wrote:<o:p></o:p></p></div><p class=MsoNormal> <o:p></o:p></p><div><div><p class=MsoNormal style='background:white'><span style='font-family:Baskerville'>On 07/24/2017 05:15 PM, Tomer Azran wrote:</span><o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt;font-variant-caps:
normal;orphans: auto;text-align:start;widows:
auto;-webkit-text-size-adjust:
auto;-webkit-text-stroke-width:
0px;background-color:rgb(255,
255,
255);word-spacing:0px'><div><p class=MsoNormal style='background:white'><span style='font-size:11.0pt;font-family:"Arial","sans-serif"'>I still don't understand why the qdevice concept doesn't help on this situation. Since the master node is down, I would expect the quorum to declare it as dead.</span><o:p></o:p></p></div><div><p class=MsoNormal style='background:white'><span style='font-size:11.0pt;font-family:"Arial","sans-serif"'>Why doesn't it happens?</span><o:p></o:p></p></div></blockquote><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:Baskerville'><br>That is not how quorum works. It just limits the decision-making to the quorate subset of the cluster.<br>Still the unknown nodes are not sure to be down.<br>That is why I suggested to have quorum-based watchdog-fencing with sbd.<br>That would assure that within a certain time all nodes of the non-quorate part<br>of the cluster are down.<br><br><br></span><o:p></o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:Baskerville'><br><br></span><o:p></o:p></p><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:Baskerville'>On Mon, Jul 24, 2017 at 4:15 PM +0300, "Dmitri Maziuk"<span class=apple-converted-space> </span><</span><a href="mailto:dmitri.maziuk@gmail.com" target="_blank"><span style='font-family:Baskerville'>dmitri.maziuk@gmail.com</span></a><span style='font-family:Baskerville'>><span class=apple-converted-space> </span>wrote:</span><o:p></o:p></p><div><pre>On 2017-07-24 07:51, Tomer Azran wrote:<o:p></o:p></pre><pre>> We don't have the ability to use it.<o:p></o:p></pre><pre>> Is that the only solution?<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>No, but I'd recommend thinking about it first. Are you sure you will <o:p></o:p></pre><pre>care about your cluster working when your server room is on fire? 'Cause <o:p></o:p></pre><pre>unless you have halon suppression, your server room is a complete <o:p></o:p></pre><pre>write-off anyway. (Think water from sprinklers hitting rich chunky volts <o:p></o:p></pre><pre>in the servers.)<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Dima<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>_______________________________________________<o:p></o:p></pre><pre>Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><o:p></o:p></pre><pre><a href="http://lists.clusterlabs.org/mailman/listinfo/users">http://lists.clusterlabs.org/mailman/listinfo/users</a><o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Project Home: <a href="http://www.clusterlabs.org/">http://www.clusterlabs.org</a><o:p></o:p></pre><pre>Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><o:p></o:p></pre><pre>Bugs: <a href="http://bugs.clusterlabs.org/">http://bugs.clusterlabs.org</a><o:p></o:p></pre></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:Baskerville'><br><br><br></span><o:p></o:p></p><pre>_______________________________________________<o:p></o:p></pre><pre>Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><o:p></o:p></pre><pre><a href="http://lists.clusterlabs.org/mailman/listinfo/users">http://lists.clusterlabs.org/mailman/listinfo/users</a><o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Project Home: <a href="http://www.clusterlabs.org/">http://www.clusterlabs.org</a><o:p></o:p></pre><pre>Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><o:p></o:p></pre><pre>Bugs: <a href="http://bugs.clusterlabs.org/">http://bugs.clusterlabs.org</a><o:p></o:p></pre><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;font-variant-caps:
normal;text-align:start;-webkit-text-stroke-width:
0px;background-color:rgb(255,
255,
255);word-spacing:0px'><span style='font-family:Baskerville'> </span><o:p></o:p></p><pre style='background:white;font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='font-size:12.0pt'>-- </span><o:p></o:p></pre><pre style='background:white'><span style='font-size:12.0pt'>Klaus Wenninger</span><o:p></o:p></pre><pre style='background:white'><span style='font-size:12.0pt'> </span><o:p></o:p></pre><pre style='background:white'><span style='font-size:12.0pt'>Senior Software Engineer, EMEA ENG Openstack Infrastructure</span><o:p></o:p></pre><pre style='background:white'><span style='font-size:12.0pt'> </span><o:p></o:p></pre><pre style='background:white'><span style='font-size:12.0pt'>Red Hat</span><o:p></o:p></pre><pre style='background:white'><span style='font-size:12.0pt'> </span><o:p></o:p></pre><pre style='background:white'><a href="mailto:kwenning@redhat.com"><span style='font-size:12.0pt'>kwenning@redhat.com</span></a><span style='font-size:12.0pt'> </span><o:p></o:p></pre><p class=MsoNormal><span style='font-family:Baskerville'>_______________________________________________<br>Users mailing list:<span class=apple-converted-space> </span></span><a href="mailto:Users@clusterlabs.org"><span style='font-family:Baskerville;background:white'>Users@clusterlabs.org</span></a><span style='font-family:Baskerville'><br></span><a href="http://lists.clusterlabs.org/mailman/listinfo/users"><span style='font-family:Baskerville'>http://lists.clusterlabs.org/mailman/listinfo/users</span></a><span style='font-family:Baskerville'><br><br>Project Home:<span class=apple-converted-space> </span></span><a href="http://www.clusterlabs.org/"><span style='font-family:Baskerville;background:white'>http://www.clusterlabs.org</span></a><span style='font-family:Baskerville'><br>Getting started:<span class=apple-converted-space> </span></span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf"><span style='font-family:Baskerville'>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</span></a><span style='font-family:Baskerville'><br>Bugs:<span class=apple-converted-space> </span></span><a href="http://bugs.clusterlabs.org/"><span style='font-family:Baskerville;background:white'>http://bugs.clusterlabs.org</span></a><o:p></o:p></p></div></blockquote></div><p class=MsoNormal> <o:p></o:p></p></div><p class=MsoNormal style='margin-bottom:12.0pt'><br><br><o:p></o:p></p><pre>_______________________________________________<o:p></o:p></pre><pre>Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><o:p></o:p></pre><pre><a href="http://lists.clusterlabs.org/mailman/listinfo/users">http://lists.clusterlabs.org/mailman/listinfo/users</a><o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Project Home: <a href="http://www.clusterlabs.org">http://www.clusterlabs.org</a><o:p></o:p></pre><pre>Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><o:p></o:p></pre><pre>Bugs: <a href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a><o:p></o:p></pre><pre> <o:p></o:p></pre></blockquote></div></div></body></html>