<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=windows-1252">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Hi Mark,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Thanks for your help! Indeed, you get at race condition... that’s
why an external quorum daemon (such as the one supplied with HP’s ServiceGuard)
would be nice. It looks like this is what Linux HA is heading for (<a
href="http://theclusterguy.clusterlabs.org/post/907043024/introducing-the-pacemaker-master-control-process-for">http://theclusterguy.clusterlabs.org/post/907043024/introducing-the-pacemaker-master-control-process-for</a>,
setup number 4) but it’s not there yet.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The only way to do it with SBD is to avoid auto startup of a
cluster node (e.g. disable corosync and pacemaker init scripts) --> avoids
fencing the other after being fenced. Or to use SBD on iSCSI storage... if you
have no network connection, you cannot fence and sdb will make watchdog timeout
--> the node which lost network connectivity is going to be reset. If network
connectivity is still missing at reboot, it cannot fence the other... otherwise
it’ll join the cluster. Any flaw here? That is +/- the same as with a quorum
server... if it cannot reach the quorum server, the node can startup but should
not fence the other (which has quorum).<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Anyway, I must admit: a quorum server or 3rd node seems safer by
design.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Rgds, Dirk<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> mark - pacemaker
list [mailto:m+pacemaker@nerdish.us] <br>
<b>Sent:</b> Friday, November 25, 2011 8:27 PM<br>
<b>To:</b> The Pacemaker cluster resource manager<br>
<b>Subject:</b> Re: [Pacemaker] 2 node cluster questions<o:p></o:p></span></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<div>
<p class=MsoNormal>Hi Dirk,<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<p class=MsoNormal>On Fri, Nov 25, 2011 at 6:05 AM, Hellemans Dirk D <<a
href="mailto:Dirk.Hellemans@hpcds.com">Dirk.Hellemans@hpcds.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hello
everyone,<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I’ve
been reading a lot lately about using Corosync/Openais in combination with
Pacemaker: SuSe Linux documentation, Pacemaker & Linux-ha website,
interesting blogs, mailinglists, etc. As I’m particularly interested in how
well two node clusters (located within the same server room) are handled, I was
a bit confused by the fact that quorum disks/ quorum servers are (not yet?)
supported/used. Some suggested to add a third node which is not actively
participating (e.g. only running corosync.... or with hearbeat but in standby
mode). That might be a solution but doesn’t “feel” right, especially if you
consider multiple two-node clusters... that would require a lot of extra
“quorum only nodes”. Somehow SBD (storage based death) in combination with a
hardware watchdog timer seemed to also provide a solution: run it on top of
iSCSI storage and you end up with a fencing device and some sort of “network
based quorum” as tiebreaker. If one node loses network connectivity, sbd +
watchdog will make sure it’s being fenced.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I’d
love to hear your ideas about 2 node cluster setups. What is the best way to do
it? Any chance we’ll get quorum disks/ quorum servers in the (near) future?<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
</div>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<p class=MsoNormal>Our experience with a two-node SBD-based cluster wasn't
good. After setup, we started on failure scenarios. The first test
was to drop network connectivity for one of the nodes while both could still
access storage. The nodes fenced each other (sort of like a STONITH
deathmatch you can read about), killing all services and leaving us waiting for
both nodes to boot back up. Obviously, a complete failure of testing, we
didn't even proceed with further checks. We took a standard PC and built
it out as a third node, giving the cluster true quorum, and now it's rock-solid
and absolutely correct in every failure scenario we throw at it. For
production use, the very real possibility of two nodes killing each other just
wasn't worth the risk to us.<o:p></o:p></p>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>If you go with two nodes and SBD, do a lot of testing.
No matter how much you test though, if they lose visibility to each other
on the network but can both still see the storage, you've got a race where the
node that *should* be fenced (the one that has its network cables disconnected)
can fence the node that is still 100% healthy and actively serving clients.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>Maybe there's a way to configure around that, I'd be
interested in hearing how if so.<o:p></o:p></p>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>Regards,<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Mark<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;
margin-left:4.8pt;margin-right:0in'>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>In
addition, say you’re not using sbd but an IPMI based fencing solution. You lose
network connectivity on one of the nodes (I know, they’re redundant but
still...sh*t happens ;) Does Pacemaker know which of both nodes lost network connectivity?
E.g.: node 1 runs Oracle database, node 2 nothing. Node 2 loses network
connectivity (e.g. both NICs without signal because unplugged by an errant
technician ;) )... => split brain situation occurs, but who’ll be fenced?
The one with Oracle running ?? I really hope not... cause in this case, the
cluster can “see” there’s no signal on the NICs of node2. Would be interesting
to know more about how Pacemaker/corosync makes such kind of decisions... how
to choose which one will be fenced in case of split brain. Is it randomly
chosen? Is it the DC which decides? Based on NIC state? I did some quick
testing with 2 VMs and at first, it looks like Pacemaker/corosync always fences
the correct node, or: the node where I unplugged the “virtual” cable. <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I’m
curious!<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks
a lot!<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Best
regards,<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Dirk<o:p></o:p></p>
</div>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a
href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><o:p></o:p></p>
</blockquote>
</div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>