<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-AU link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>When you suggest:<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>>>> </span>What about setting no-quorum-policy to
freeze and making the third node a full cluster member (that just doesn't run
any resources)?<o:p></o:p></p>
<p class=MsoNormal>That way, if you get a 1-1-1 split the nodes will leave all
services running where they were and while it waits for quorum.<o:p></o:p></p>
<p class=MsoNormal>And if it heals into a 1-2 split, then the majority will
terminate the rogue node and acquire all the services.<o:p></o:p></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>No-quorum-policy ‘Freeze’ rather than ‘Stop’
pretty much ASSURES me of getting a split brain for my fileserver cluster. Sounds
like the last thing I want to have. Any data local clients write to the cutoff node
(and its DRBD split-brain volume) cannot be later reconciled and will need to
be discarded. I’d rather not give the local clients that can still reach
that node a false sense of security of their data having been written to disk
(to a drbd volume that will be blown away and resynced with the quorum side
once connectivity is re-established). ‘Stop’ policy sounds safer. <o:p></o:p></span></p>
<p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Question 1:</span></b><span style='font-size:11.0pt;font-family:
"Calibri","sans-serif";color:#1F497D'> Am I missing anything re stop/ignore
no-quorum policy?<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Further, I’m having more trouble working out a list of
tabulated failure modes for this 3-way scenario, where 3-way outage-prone WAN
links get introduced. <o:p></o:p></span></p>
<p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Question 2:</span></b><span style='font-size:11.0pt;font-family:
"Calibri","sans-serif";color:#1F497D'> If one WAN link is broken – (A)
can speak to (B), (B) can speak to (C), but (A) CANNOT speak to (C), what
drives the quorum decision and what would happen? In particular, what would
happen if the node that can see both is the DC? <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Question 3:</span></b><span style='font-size:11.0pt;font-family:
"Calibri","sans-serif";color:#1F497D'> Just to verify I got this right - what
drives pacemaker’s STONITH events, <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>[a] RESOURCE monitoring failure, <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>or <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>[b] CRM’s crosstalk that establishes quorum-state / DC-election?<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> Andrew Beekhof [mailto:andrew@beekhof.net] <br>
<b>Sent:</b> Wednesday, 13 January 2010 7:24 PM<br>
<b>To:</b> pacemaker@oss.clusterlabs.org<br>
<b>Subject:</b> Re: [Pacemaker] Split Site 2-way clusters<o:p></o:p></span></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p>
<div>
<p class=MsoNormal>On Wed, Jan 13, 2010 at 8:19 AM, Miki Shapiro <<a
href="mailto:Miki.Shapiro@coles.com.au">Miki.Shapiro@coles.com.au</a>>
wrote:<o:p></o:p></p>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Separate
to my earlier post re CRM DC election in a 2-way cluster, I’m chasing up
the (separate) issue of making the cluster a CROSS-SITE one. <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>As
stated in yay other thread, I’m running a 2-way quorum-agnostic cluster
on a SLES11, openais, pacemaker, drbd (… clvm, ocfs2, ctdb, nfs, etc) on
HP Blades. <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>A
few old threads (with a rather elaborate answer from Lars) indicate that as of
March 2009 split-site wasn’t yet thoroughly supported as WAN connectivity
issues were not thoroughly addressed, and that as of then quorumd was not yet
sufficiently robust/tested/PROD-ready. <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What
we decided we want to do is rely on an extremely simple (and hopefully by inference
predictable and reliable) arbitrator - a THIRD linux server that lives at a
SEPARATE THIRD site altogether with no special HA-related daemons running on
it. <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I’ll
build a STONITH ocf script, configure it as a cloned STONITH resource running
on both nodes, and it will do roughly this when pinging the other node (via
either one or two redundant links) will fail:<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>ssh
arbitrator mkdir /tmp/$clustername && shoot-other-node ||
hard-suicide-NOW<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thus,
when split, the nodes race to the arbitrator. <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>First
to run the mkdir command on the arbitrator (and get rc=0) wins, gets the long
straw and lives. Loser gets shot (either by its peer if WAN allows peer
to communicate with soon-to-be-dead node’s iLO or by said node sealing
its own fate). <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Primary
failure modes not accounted for by a run-of-the-mill non-split-site cluster are
thus: <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p>1.<span style='font-size:7.0pt'> </span>One
node cut off – cutoff node will fail the race and suicide. Good node will
succeed and proceed to provide service.<o:p></o:p></p>
<p>2.<span style='font-size:7.0pt'> </span>Nodes
cut off from each other but can both access the arbitrator – slower node
will suicide. Faster node will succeed and proceed to provide service.<o:p></o:p></p>
<p>3.<span style='font-size:7.0pt'> </span>Both
nodes are cut off, or the comms issue affects both node1<->node2 comms
AND all ->arbitrator comms (double failure). – both nodes
suicide (and potentially leave me with two inconsistent and potentially corrupt
filesystems). Can’t see an easy way around this one (can anyone?)<o:p></o:p></p>
</div>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>Basically thats the part that the stuff we haven't written
yet is supposed to address.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>You want to avoid the "|| hard-suicide-NOW"
part of your logic, but you can't safely do that unless there is some way to
stop the services on the non-connected node(s) - preferably _really_ quickly.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>What about setting no-quorum-policy to freeze and making the
third node a full cluster member (that just doesn't run any resources)?<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>That way, if you get a 1-1-1 split the nodes will leave all
services running where they were and while it waits for quorum.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>And if it heals into a 1-2 split, then the majority will
terminate the rogue node and acquire all the services.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>The biggest problem is the reliability of your links and
stonith devices - give particular thought to how you'd fence _node_ A if comms
to _site_ A are down.... <o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-right:0cm'>
<div>
<div>
<p> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Looks
to me like this can easily be implemented without any fancy quorum servers (on
top of the required little ocf script and the existence of the arbitrator)<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Does
anyone have thoughts on this? Am I ignoring any major issues, or reinventing
the wheel, or should this this potentially work as I think it will?<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks!
<span style='font-family:Wingdings'>J</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>And
a little addendum which just occurred to me re transient WAN network issues: <o:p></o:p></p>
<p> <o:p></o:p></p>
<p>1.<span style='font-size:7.0pt'> </span>Transient
big (>2min) network issues will land me with a cluster that needs a human to
turn on one node on every time they happen. Bad. <o:p></o:p></p>
<p> <o:p></o:p></p>
<p>My proposed solution: classify a peer-failure as a WAN-problem by pinging
peer node’s core router when peer node appears dead, if router dead too
touch a WAN-problem-flagfile, and so long as the flag-file sits there the
survivor pings (done via ocf ping resource) other-side-router until it comes
online, then shooting a “check-power-status &&
O-GOD-IT-STILL-LIVES-KILL-IT-NOW || power-it-on” command to the
peer’s iLO (and promptly delete the flag). <o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Implementation cost: a wee bit of scripting and a wee bit of pacemaker
configuration. <o:p></o:p></p>
<p> <o:p></o:p></p>
<p>2.<span style='font-size:7.0pt'> </span>Transient
small network issues will require stretching pacemaker’s default timeouts
sufficiently to avoid (or end up in the item 1 bucket above)<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Am
very keen to know what the gurus think <span style='font-family:Wingdings'>J</span>
<span style='font-family:Wingdings'>J</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
text-autospace:none'><b><span style='color:#4F81BD'> </span></b><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
text-autospace:none'><b><span style='color:#4F81BD'>Miki Shapiro</span></b><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
text-autospace:none'><span style='font-size:9.0pt;color:#4F81BD'>Linux Systems
Engineer<br>
Infrastructure Services & Operations</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
text-autospace:none'><span style='font-size:9.5pt;color:#4F81BD'><br>
</span><img border=0 width=75 height=23 id="_x0000_i1025"
src="cid:image001.png@01CA950C.CC5921A0"><img border=0 width=3 height=3
id="_x0000_i1026" src="cid:image002.png@01CA950C.CC5921A0"><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:9.0pt;color:#4F81BD'>745 Springvale Road<br>
Mulgrave 3170 Australia<br>
Email</span><span style='font-size:9.5pt;color:#4F81BD'> <a
href="mailto:miki.shapiro@coles.com.au" target="_blank">miki.shapiro@coles.com.au</a><br>
</span><span style='font-size:9.0pt;color:#4F81BD'>Phone: 61 3 854 10520</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'><span
style='font-size:9.0pt;color:#4F81BD'>Fax: 61 3 854
10558</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
</div>
<p class=MsoNormal><br>
______________________________________________________________________<br>
This email and any attachments may contain privileged and confidential<br>
information and are intended for the named addressee only. If you have<br>
received this e-mail in error, please notify the sender and delete<br>
this e-mail immediately. Any confidentiality, privilege or copyright<br>
is not waived or lost because this e-mail has been sent to you in<br>
error. It is your responsibility to check this e-mail and any<br>
attachments for viruses. No warranty is made that this material is<br>
free from computer virus or any other defect or error. Any<br>
loss/damage incurred by using this material is not the sender's<br>
responsibility. The sender's entire liability will be limited to<br>
resupplying the material.<br>
______________________________________________________________________<o:p></o:p></p>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
_______________________________________________<br>
Pacemaker mailing list<br>
<a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><o:p></o:p></p>
</blockquote>
</div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<BR>
______________________________________________________________________<BR>
This email and any attachments may contain privileged and confidential<BR>
information and are intended for the named addressee only. If you have<BR>
received this e-mail in error, please notify the sender and delete<BR>
this e-mail immediately. Any confidentiality, privilege or copyright<BR>
is not waived or lost because this e-mail has been sent to you in<BR>
error. It is your responsibility to check this e-mail and any<BR>
attachments for viruses. No warranty is made that this material is<BR>
free from computer virus or any other defect or error. Any<BR>
loss/damage incurred by using this material is not the sender's<BR>
responsibility. The sender's entire liability will be limited to<BR>
resupplying the material.<BR>
______________________________________________________________________<BR>
</body>
</html>