<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:484585650;
mso-list-type:hybrid;
mso-list-template-ids:-1433884704 201916431 201916441 201916443 201916431 201916441 201916443 201916431 201916441 201916443;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1
{mso-list-id:1023895540;
mso-list-type:hybrid;
mso-list-template-ids:-16227430 201916431 201916441 201916443 201916431 201916441 201916443 201916431 201916441 201916443;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-AU link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Confused.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I *<b>am</b>* running DRBD in dual-master mode (apologies, I
should have mentioned that earlier), and there will be both WAN clients as well
as local-to-datacenter-clients writing to both nodes on both ends. It’s
safe to assume the clients will know not of the split. <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>In a WAN split I need to ensure that the node whose idea of drbd
volume will be kept once resync happens stays up, and node that’ll get blown
away and re-synced/overwritten becomes dead asap. <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>NodeX(Successfully) taking on data from clients while in quorumless-freeze-still-providing-service,
then discarding its hitherto collected client data when realizing other node
has quorum and discarding own data isn’t good.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>To recap what I understood so far:<o:p></o:p></span></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l1 level1 lfo2'><![if !supportLists]><span
style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><span
style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>CRM Availability on the multicast channel drives DC election,
but DC election is irrelevant to us here.<o:p></o:p></span></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l1 level1 lfo2'><![if !supportLists]><span
style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><span
style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>CRM Availability on the multicast channel (rather than resource
failure) drive who-is-in-quorum-and-who-is-not decisions [not sure here..
correct? Or does resource failure drive quorum? ]<o:p></o:p></span></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l1 level1 lfo2'><![if !supportLists]><span
style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><span
style='mso-list:Ignore'>3.<span style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Steve to clarify what happens quorum-wise if 1/3 nodes sees both
others, but the other two only see the first (“broken triangle”),
and whether this behaviour may differ based on whether the first node (which is
different as it sees both others) happens to be the DC at the time or not.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Given that anyone who goes about building a production cluster would
want to identify all likely failure modes and be able to predict how the
cluster behaves in each one, is there any user-targeted doco/rtfm material one
could read regarding how quorum establishment works in such scenarios? <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Setting up a 3-way with intermittent WAN links without getting a
clear understanding in advance of how the software will behave is … scary
</span><span style='font-size:11.0pt;font-family:Wingdings;color:#1F497D'>J</span><span
style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif"'><o:p> </o:p></span></b></p>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> Andrew Beekhof [mailto:andrew@beekhof.net] <br>
<b>Sent:</b> Thursday, 14 January 2010 7:56 PM<br>
<b>To:</b> pacemaker@oss.clusterlabs.org<br>
<b>Subject:</b> Re: [Pacemaker] Split Site 2-way clusters<o:p></o:p></span></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p>
<div>
<p class=MsoNormal>On Thu, Jan 14, 2010 at 1:40 AM, Miki Shapiro <<a
href="mailto:Miki.Shapiro@coles.com.au">Miki.Shapiro@coles.com.au</a>>
wrote:<o:p></o:p></p>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:11.0pt;color:#1F497D'>When you suggest:</span><o:p></o:p></p>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:11.0pt;color:#1F497D'>>>> </span>What about setting
no-quorum-policy to freeze and making the third node a full cluster member
(that just doesn't run any resources)?<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>That
way, if you get a 1-1-1 split the nodes will leave all services running where
they were and while it waits for quorum.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>And
if it heals into a 1-2 split, then the majority will terminate the rogue node
and acquire all the services.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:11.0pt;color:#1F497D'> </span><o:p></o:p></p>
</div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:11.0pt;color:#1F497D'>No-quorum-policy ‘Freeze’
rather than ‘Stop’ pretty much ASSURES me of getting a split brain
for my fileserver cluster. Sounds like the last thing I want to have. Any data
local clients write to the cutoff node (and its DRBD split-brain volume) cannot
be later reconciled and will need to be discarded. I’d rather not give
the local clients that can still reach that node a false sense of security of
their data having been written to disk (to a drbd volume that will be blown
away and resynced with the quorum side once connectivity is re-established).
‘Stop’ policy sounds safer.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class=MsoNormal>Are you using DRBD in dual-master mode?<o:p></o:p></p>
</div>
</div>
<div>
<p class=MsoNormal>Because freeze wont start anything new, so if you had one
writer before the split, you'll still only have one during.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Then when the split heals DRBD can resync as normal. Right?<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-right:0cm'>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span
style='font-size:11.0pt;color:#1F497D'>Question 1:</span></b><span
style='font-size:11.0pt;color:#1F497D'> Am I missing anything re stop/ignore
no-quorum policy?</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>Possibly for the freeze option.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-right:0cm'>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:11.0pt;color:#1F497D'>Further, I’m having more trouble
working out a list of tabulated failure modes for this 3-way scenario, where
3-way outage-prone WAN links get introduced. </span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span
style='font-size:11.0pt;color:#1F497D'>Question 2:</span></b><span
style='font-size:11.0pt;color:#1F497D'> If one WAN link is broken – (A)
can speak to (B), (B) can speak to (C), but (A) CANNOT speak to (C), what
drives the quorum decision and what would happen? In particular, what would happen
if the node that can see both is the DC? </span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>I'm not sure how totem works in this situation, but whether
the node that can see both is the DC is irrelevant.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Membership happens at a much lower level.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>I _think_ you'd end up with a 2-1 split, but I'm not sure
how it decides if its A-B or B-C<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Steve could probably tell you.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-right:0cm'>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span
style='font-size:11.0pt;color:#1F497D'>Question 3:</span></b><span
style='font-size:11.0pt;color:#1F497D'> Just to verify I got this right - what
drives pacemaker’s STONITH events, </span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:11.0pt;color:#1F497D'>[a] RESOURCE monitoring failure, </span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:11.0pt;color:#1F497D'>or </span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:11.0pt;color:#1F497D'>[b] CRM’s crosstalk that
establishes quorum-state / DC-election?</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>Its not an XOR condition.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal>Membership events (from openais) AND resource failures can
both result in fencing occurring.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>But please forget about DCs and DC elections - they are
really not relevant to any of this.<o:p></o:p></p>
</div>
</div>
</div>
<BR>
______________________________________________________________________<BR>
This email and any attachments may contain privileged and confidential<BR>
information and are intended for the named addressee only. If you have<BR>
received this e-mail in error, please notify the sender and delete<BR>
this e-mail immediately. Any confidentiality, privilege or copyright<BR>
is not waived or lost because this e-mail has been sent to you in<BR>
error. It is your responsibility to check this e-mail and any<BR>
attachments for viruses. No warranty is made that this material is<BR>
free from computer virus or any other defect or error. Any<BR>
loss/damage incurred by using this material is not the sender's<BR>
responsibility. The sender's entire liability will be limited to<BR>
resupplying the material.<BR>
______________________________________________________________________<BR>
</body>
</html>