<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"ＭＳ 明朝";}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1625162330;
mso-list-type:hybrid;
mso-list-template-ids:-2069718410 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">If you don’t mind, please allow me to walk through my architecture just a bit. I know that I am far from an expert on this stuff, but I feel like I have a firm grasp on how this all works conceptually. That
said, I welcome your insights and advice on how to approach this problem—and any ready-made solutions to it you might have on hand. :)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We are deploying into two availability zones (AZs) in AWS. Our goal is to be able to absorb the loss of an entire AZ and continue to provide services to users. Our first problem comes with Microsoft SQL Server
running on Windows Server Failover Clustering. As you guys likely know, WSFC isn’t polite about staying up without a quorum. As such, I figured, oh, hey, I can build a two-node Pacemaker-based iSCSI Target cluster and expose luns from it via iSCSI so that
the WSFC nodes could have a witness filesystem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">So, I’ve managed to make that all happen. Yay! What I’m now trying to suss out is how I ensure that I’m covered for availability in the event of any kind of outage. As it turns out, I believe that I actually
have most of the covered.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">(I use “1o” to indicate “primary” and “2o” for “secondary”)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Planned Outage: effected node gracefully demoted from cluster, life goes on, everyone is happy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Unplanned Outage (NAS cluster node fails/unreachable): if 2o node failure, nothing happens; if 1o node failure, drbd promotes 2o to 1o, constrained vip, lvm, tgt, and lun resources automatically flip to 2o
node, life goes on, everyone is happy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">But still—the one thing we built this ridiculously complicated and overengineered thing for—I don’t feel like I have a good story when it comes to a severed AZ event (loss of perimeter communications, etc.)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Unplanned Outage (AZ connectivity severed): both nodes detect that the other node is gone so promote themselves to primary. The unsevered side would continue to work as expected, with the witness mounted by
the SQL servers in that AZ, life goes on and at least the USERS are happy… but the severed side is also sojourning on. Both sides of the SQL cluster would think they have quorum even if they can’t talk to their peer nodes, so they mark their peers as down
and keep on keeping on. No users would be connecting to the severed instances, but background and system tasks would proceed as normal, potentially writing new data to the databases making rejoining the nodes to the cluster a little bit tricky to say the least,
especially if the severed side’s network comes back up and both systems come to realize that they’re not consistent.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">So, my problem, I think, is two-fold:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt">What can I monitor from each of the NAS cluster instances (besides connectivity to one another) that would “ALWAYS” be available when things are working and NEVER available when they are broken?
Seems to me that if I can sort out something that meets these criteria (I was thinking, perhaps, a RESTful connection to the AWS API, but I’m not entirely sure you can’t get responses at API endpoints that may or may not be hosted inside the AZ), then I could
write a simple monitoring script that runs on both nodes that would act as a fencing and STONITH solution (if detect bad things, then shut down). Seems to me that this would prevent the data inconsistency since the severed side WSFC would lose its witness
file system, thus its quorum, and take itself offline. <o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt">Have I failed to account for another failure condition that could be potentially as/more harmful than anything I’ve thought of already?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Anyway, I’m hopeful, someone(s) here can share some of their own experiences from the trenches. Thank you for your time (and all the help you guys have been in getting this set up already).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F4E79">--</span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F4E79"> </span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:14.0pt;color:#27628C">[ jR ]</span></b><span style="font-size:14.0pt;color:#27628C"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F4E79"> @: <a href="mailto:jason@eramsey.org"><span style="color:blue">jason@eramsey.org</span></a></span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F4E79"> </span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F4E79"> </span><i><span style="font-size:9.0pt;color:#1F4E79">there is no path to greatness; greatness is the path</span></i><o:p></o:p></p>
</div>
</body>
</html>