<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Trebuchet MS";
panose-1:2 11 6 3 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:369915844;
mso-list-type:hybrid;
mso-list-template-ids:-1199922462 201916431 201916441 201916443 201916431 201916441 201916443 201916431 201916441 201916443;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1
{mso-list-id:1257397222;
mso-list-type:hybrid;
mso-list-template-ids:-1950210220 201916431 201916441 201916443 201916431 201916441 201916443 201916431 201916441 201916443;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l2
{mso-list-id:1698581911;
mso-list-type:hybrid;
mso-list-template-ids:436262604 1628981392 201916441 201916443 201916431 201916441 201916443 201916431 201916441 201916443;}
@list l2:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="2050" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-AU link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>Separate to my earlier post re CRM DC election in a 2-way cluster,
I’m chasing up the (separate) issue of making the cluster a CROSS-SITE
one. <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>As stated in yay other thread, I’m running a 2-way
quorum-agnostic cluster on a SLES11, openais, pacemaker, drbd (… clvm, ocfs2,
ctdb, nfs, etc) on HP Blades. <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>A few old threads (with a rather elaborate answer from Lars)
indicate that as of March 2009 split-site wasn’t yet thoroughly supported
as WAN connectivity issues were not thoroughly addressed, and that as of then
quorumd was not yet sufficiently robust/tested/PROD-ready. <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>What we decided we want to do is rely on an extremely simple
(and hopefully by inference predictable and reliable) arbitrator - a THIRD linux
server that lives at a SEPARATE THIRD site altogether with no special HA-related
daemons running on it. <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I’ll build a STONITH ocf script, configure it as a
cloned STONITH resource running on both nodes, and it will do roughly this when
pinging the other node (via either one or two redundant links) will fail:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>ssh arbitrator mkdir /tmp/$clustername && shoot-other-node
|| hard-suicide-NOW<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Thus, when split, the nodes race to the arbitrator. <o:p></o:p></p>
<p class=MsoNormal>First to run the mkdir command on the arbitrator (and get rc=0)
wins, gets the long straw and lives. Loser gets shot (either by its peer
if WAN allows peer to communicate with soon-to-be-dead node’s iLO or by said
node sealing its own fate). <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Primary failure modes not accounted for by a run-of-the-mill
non-split-site cluster are thus: <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span
style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>One node cut off – cutoff node will fail the race
and suicide. Good node will succeed and proceed to provide service.<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span
style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Nodes cut off from each other but can both access the
arbitrator – slower node will suicide. Faster node will succeed and
proceed to provide service.<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span
style='mso-list:Ignore'>3.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Both nodes are cut off, or the comms issue affects both
node1<->node2 comms AND all ->arbitrator comms (double failure).
– both nodes suicide (and potentially leave me with two inconsistent and
potentially corrupt filesystems). Can’t see an easy way around this one
(can anyone?)<o:p></o:p></p>
<p class=MsoListParagraph><o:p> </o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Looks to me like this can easily be implemented without any
fancy quorum servers (on top of the required little ocf script and the existence
of the arbitrator)<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Does anyone have thoughts on this? Am I ignoring any major
issues, or reinventing the wheel, or should this this potentially work as I
think it will?<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Thanks! <span style='font-family:Wingdings'>J</span><o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>And a little addendum which just occurred to me re transient
WAN network issues: <o:p></o:p></p>
<p class=MsoListParagraph><o:p> </o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l2 level1 lfo3'><![if !supportLists]><span
style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Transient big (>2min) network issues will land me
with a cluster that needs a human to turn on one node on every time they
happen. Bad. <o:p></o:p></p>
<p class=MsoListParagraph><o:p> </o:p></p>
<p class=MsoListParagraph>My proposed solution: classify a peer-failure as a WAN-problem
by pinging peer node’s core router when peer node appears dead, if router
dead too touch a WAN-problem-flagfile, and so long as the flag-file sits there the
survivor pings (done via ocf ping resource) other-side-router until it comes
online, then shooting a “check-power-status && O-GOD-IT-STILL-LIVES-KILL-IT-NOW
|| power-it-on” command to the peer’s iLO (and promptly delete the
flag). <o:p></o:p></p>
<p class=MsoListParagraph><o:p> </o:p></p>
<p class=MsoListParagraph>Implementation cost: a wee bit of scripting and a wee
bit of pacemaker configuration. <o:p></o:p></p>
<p class=MsoListParagraph><o:p> </o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l2 level1 lfo3'><![if !supportLists]><span
style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Transient small network issues will require stretching
pacemaker’s default timeouts sufficiently to avoid (or end up in the item
1 bucket above)<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Am very keen to know what the gurus think <span
style='font-family:Wingdings'>J</span> <span style='font-family:Wingdings'>J</span><o:p></o:p></p>
<p class=MsoNormal style='text-autospace:none'><b><span style='font-family:
"Trebuchet MS","sans-serif";color:#4F81BD'><o:p> </o:p></span></b></p>
<p class=MsoNormal style='text-autospace:none'><b><span style='font-family:
"Trebuchet MS","sans-serif";color:#4F81BD'>Miki Shapiro<o:p></o:p></span></b></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:9.0pt;
font-family:"Trebuchet MS","sans-serif";color:#4F81BD'>Linux Systems Engineer<br>
Infrastructure Services & Operations<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:9.5pt;
font-family:"Trebuchet MS","sans-serif";color:#4F81BD'><br>
</span><img width=75 height=23 id="Picture_x0020_1"
src="cid:image001.png@01CA9473.B8C182C0"><img width=3 height=3
id="Picture_x0020_2" src="cid:image002.png@01CA9473.B8C182C0"><span
style='font-size:12.0pt;font-family:"Times New Roman","serif";color:#4F81BD'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Trebuchet MS","sans-serif";
color:#4F81BD'>745 Springvale Road<br>
Mulgrave 3170 Australia<br>
Email</span><span style='font-size:9.5pt;font-family:"Trebuchet MS","sans-serif";
color:#4F81BD'> <a href="mailto:miki.shapiro@coles.com.au"><span
style='color:blue'>miki.shapiro@coles.com.au</span></a><br>
</span><span style='font-size:9.0pt;font-family:"Trebuchet MS","sans-serif";
color:#4F81BD'>Phone: 61 3 854 10520<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Trebuchet MS","sans-serif";
color:#4F81BD'>Fax: 61 3 854 10558<br>
<br>
</span><span style='color:#4F81BD'><o:p></o:p></span></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<BR>
______________________________________________________________________<BR>
This email and any attachments may contain privileged and confidential<BR>
information and are intended for the named addressee only. If you have<BR>
received this e-mail in error, please notify the sender and delete<BR>
this e-mail immediately. Any confidentiality, privilege or copyright<BR>
is not waived or lost because this e-mail has been sent to you in<BR>
error. It is your responsibility to check this e-mail and any<BR>
attachments for viruses. No warranty is made that this material is<BR>
free from computer virus or any other defect or error. Any<BR>
loss/damage incurred by using this material is not the sender's<BR>
responsibility. The sender's entire liability will be limited to<BR>
resupplying the material.<BR>
______________________________________________________________________<BR>
</body>
</html>