<div><br></div>Ah, now those are the questions I'm asking now myself!<div>I hope someone else will step in now and offer some advice.<div><br></div><div>In a simple case, we want to prevent two nodes using a shared file-system without coordination. If they loose contact with each other and can't coordinate their access, one must die. (Even if you make it hard for them to loose contact by providing multiple communication channels, you have to assume it will happen and, so, we work with this case.)</div>
<div><br></div><div>The worst case is if each node thinks it is healthy but it can't coordinate with it's neighbor. It's really easy to accidentally implement gunfight stonith: whichever node can fence (STONITH) his neighbor fastest gets to live. In my lab, when I screwed this up, the machines would shoot each other over and over and over as they rebooted.</div>
<div><br></div><div>So, what are folks doing?</div><div><br></div><div><br></div><div><div class="gmail_quote">On Thu, Jan 6, 2011 at 10:45 AM, Michael Hittesdorf <span dir="ltr"><<a href="mailto:michael.hittesdorf@chicagotrading.com">michael.hittesdorf@chicagotrading.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-US" link="blue" vlink="blue">
<div>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10.0pt;font-family:Arial;color:navy">This is great information. Thanks. I was
wondering what criteria is used to determine that a ‘sick’ node should be
killed? If it can’t be contacted over the network for some length of time? If
the resources can’t be restarted on the box? What I’m most worried about is the
scenario where my backup loses contact with the primary due to a network failure
and the backup takes over even though the master is still running. This would
cause both nodes to mount my SAN attached storage and potentially corrupt it.
I’ve actually forced this to happen by disconnecting the master’s network
adapter on my test cluster. I wound up with a split brain situation where both
nodes were actively running. Would a STONITH device kill the master if the
master could not be contacted over the network? Or would the STONITH device indicate
that the master was ok and prevent the unwanted failover from occurring and
thus prevent the split brain scenario I just described?</span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10.0pt;font-family:Arial;color:navy">Thanks for all your help. It is much
appreciated!</span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10.0pt;font-family:Arial;color:navy">Mick</span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font></p>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<hr size="2" width="100%" align="center">
</span></font></div>
<p class="MsoNormal"><b><font size="2" face="Tahoma"><span style="font-size:10.0pt;font-family:Tahoma;font-weight:bold">From:</span></font></b><font size="2" face="Tahoma"><span style="font-size:10.0pt;font-family:Tahoma"> Mike Diehn
[mailto:<a href="mailto:mike.diehn@ansys.com" target="_blank">mike.diehn@ansys.com</a>] <br>
<b><span style="font-weight:bold">Sent:</span></b> Thursday, January 06, 2011
9:16 AM</span></font></p><div><div><font size="2" face="Tahoma"></font></div><div class="h5"><font size="2" face="Tahoma"><br>
<b><span style="font-weight:bold">To:</span></b> The
Pacemaker cluster resource manager<br>
<b><span style="font-weight:bold">Subject:</span></b> Re: [Pacemaker] how to
mount drive on SAN with pacemakerresourceagent?</font></div></div><p></p>
</div><div><div></div><div class="h5">
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">You want a STONITH tool that will let your nodes positively kill one
another without needing to rely on the "sick" node for anything.
So, the ideal solution is, yes, a networked power device. Something
that will let you power-off the sick node remotely.</span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Lacking that, you could use IPMI tool if your servers have BMCs.
Almost all server class machines do today. Things like Sun ILOM,
Dell DRAC, HP iLO.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The modules and scripts in /usr/lib64/stonith/plugins will give
you an idea of what's available already.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Do try to resist the temptation to use ssh to issue a shutdown command.
That's really just not useful and if you implement it, you check off
'implement stonith' on your list and move happily on thinking you're shared
file-system is now safe. When it isn't.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Does that help? Oh, one more thing, it took me an embarassingly
long time to discover that there is a "stonith" command and a bunch
of related "stuff." On my SLES 11 SP1 systems, with the HA
Extension Add-on, the stonith stuff came in as part of RPM package
cluster-glue-1.0.5-0.5.1.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Best,</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Mike</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">On Thu, Jan 6, 2011 at 9:53 AM, Michael Hittesdorf <<a href="mailto:michael.hittesdorf@chicagotrading.com" target="_blank">michael.hittesdorf@chicagotrading.com</a>>
wrote:</span></font></p>
<div link="blue" vlink="blue">
<div>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10.0pt;font-family:Arial;color:navy">Thanks for your reply. I now have the Filesystem resource working
on my test cluster. I’ve done some reading on STONITH as you suggested and am
now wondering how I determine what STONITH devices are actually available on my
servers and which one I should choose? The recommendation I’ve read
suggests the use of an external UPS that can be monitored over the network. Is
this the best approach? Are there other STONITH devices that are commonly used?
Why choose one over the other?</span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10.0pt;font-family:Arial;color:navy">Thanks in advance. Mick </span></font></p>
<p class="MsoNormal"><font size="2" color="navy" face="Arial"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font></p>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<hr size="2" width="100%" align="center">
</span></font></div>
<p class="MsoNormal"><b><font size="2" face="Tahoma"><span style="font-size:10.0pt;font-family:Tahoma;font-weight:bold">From:</span></font></b><font size="2" face="Tahoma"><span style="font-size:10.0pt;font-family:Tahoma"> Mike Diehn [mailto:<a href="mailto:mike.diehn@ansys.com" target="_blank">mike.diehn@ansys.com</a>] <br>
<b><span style="font-weight:bold">Sent:</span></b> Tuesday, January 04, 2011
2:54 PM<br>
<b><span style="font-weight:bold">To:</span></b> The
Pacemaker cluster resource manager<br>
<b><span style="font-weight:bold">Subject:</span></b> Re: [Pacemaker] how to
mount drive on SAN with pacemaker resourceagent?</span></font></p>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">To make sure the
failed server is actually dead, you want to use STONITH. So read about
that. Here are examples from our testing cluster. These are broken,
so don't use them as they are. That's why they are set to "Stopped"
right now. I probably have some timing stuff very wrong:</span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<blockquote style="margin-left:30.0pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><font size="3" face="Courier New"><span style="font-size:12.0pt;font-family:"Courier New"">primitive
ShootLebekmfs1 stonith:external/ipmi \</span></font></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Courier New"><span style="font-size:12.0pt;font-family:"Courier New"">
meta target-role="Stopped" \</span></font></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Courier New"><span style="font-size:12.0pt;font-family:"Courier New"">
params hostname="lebekmfs1"
ipaddr="10.1.1.59" userid="stonith"
passwd="ShootMeInTheHead" interface="lan"</span></font></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Courier New"><span style="font-size:12.0pt;font-family:"Courier New"">primitive
ShootLebekmfs2 stonith:external/ipmi \</span></font></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Courier New"><span style="font-size:12.0pt;font-family:"Courier New"">
meta target-role="Stopped" \</span></font></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Courier New"><span style="font-size:12.0pt;font-family:"Courier New"">
params hostname="lebekmfs2"
ipaddr="10.1.1.61" userid="stonith"
passwd="ShootMeInTheHead" interface="lan"</span></font></p>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">You can use the
ocf:heartbeat:Filesystem resource to mount any file system you can mount
manually. Here's one from a config in our test cluster. This works:</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<blockquote style="margin-left:30.0pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><font size="3" face="Courier New"><span style="font-size:12.0pt;font-family:"Courier New"">primitive
lvTest ocf:heartbeat:Filesystem \</span></font></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Courier New"><span style="font-size:12.0pt;font-family:"Courier New"">
params device="/dev/EkmCluVG/lvTest"
directory="/srv/test1" fstype="ocfs2" \</span></font></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Courier New"><span style="font-size:12.0pt;font-family:"Courier New"">
op monitor interval="10s" timeout="10s"</span></font></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Make sure you
remove the file system from your /etc/fstab if you're going to do it this way.
During testing, for my convenience, I leave it in, but add the noauto
option to prevent it being mounted on boot.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Best,</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Mike</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> </span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">On Tue, Jan 4, 2011
at 2:05 PM, Michael Hittesdorf <<a href="mailto:michael.hittesdorf@chicagotrading.com" target="_blank">michael.hittesdorf@chicagotrading.com</a>>
wrote:</span></font></p>
<div link="blue" vlink="navy">
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:Arial">Can I use
the Filesystem resource agent to mount a SAN drive in the event of a failover?
How do I ensure that the failed server no longer has the drive mounted so as to
prevent storage corruption? Having read several of the tutorials, I’m aware of
DRBD and the clustered file systems GFS2 and OCFS2. However, I don’t need
simultaneous access to the disk from both of my cluster nodes. I just want to
make the shared SAN storage available to the primary, active server only as my
cluster is active-passive. Is there a recommended way to accomplish this?</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:Arial"> </span></font></p>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:Arial">Thanks for
your help!</span></font></p>
</div>
<p><font size="1" face="Times New Roman"><span style="font-size:7.5pt">This
message is intended only for the personal and confidential use of the
recipients named above. If the reader of this email is not the intended
recipient, you have received this email in error and any review, dissemination,
distribution or copying is strictly prohibited. If you have received this email
in error, please notify the sender immediately by return email and permanently
delete the copy you received. This message is provided for informational
purposes and should not be construed as a solicitation or offer to buy or sell
any securities or related financial instruments. Neither CTC Holdings nor any
affiliates (CTC) are responsible for any recommendation, solicitation, offer or
agreement or any information about any transaction, customer account or account
activity that may be attached to or contained in this communication. CTC
accepts no liability for any content contained in the email, or any errors or
omissions arising as a result of e-mail transmission. Any opinions contained in
this email constitute the sender's best judgment at this time and are subject
to change without notice. CTC London Limited is authorized and regulated by the
Financial Services Authority.</span></font></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a></span></font></p>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><br>
<br clear="all">
<br>
-- <br>
Mike Diehn<br>
Senior Systems Administrator<br>
ANSYS, Inc - Lebanon, NH Office<br>
<a href="mailto:mike.diehn@ansys.com" target="_blank">mike.diehn@ansys.com</a>,
(603) 727-5492</span></font></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p><font size="1" face="Times New Roman"><span style="font-size:7.5pt">This
message is intended only for the personal and confidential use of the
recipients named above. If the reader of this email is not the intended
recipient, you have received this email in error and any review, dissemination,
distribution or copying is strictly prohibited. If you have received this email
in error, please notify the sender immediately by return email and permanently
delete the copy you received. This message is provided for informational
purposes and should not be construed as a solicitation or offer to buy or sell
any securities or related financial instruments. Neither CTC Holdings nor any
affiliates (CTC) are responsible for any recommendation, solicitation, offer or
agreement or any information about any transaction, customer account or account
activity that may be attached to or contained in this communication. CTC
accepts no liability for any content contained in the email, or any errors or
omissions arising as a result of e-mail transmission. Any opinions contained in
this email constitute the sender's best judgment at this time and are subject
to change without notice. CTC London Limited is authorized and regulated by the
Financial Services Authority.</span></font></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a></span></font></p>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><br>
<br clear="all">
<br>
-- <br>
Mike Diehn<br>
Senior Systems Administrator<br>
ANSYS, Inc - Lebanon, NH Office<br>
<a href="mailto:mike.diehn@ansys.com" target="_blank">mike.diehn@ansys.com</a>, (603) 727-5492</span></font></p>
</div>
</div></div></div><div><div></div><div class="h5">
<p></p><p><font size="1">This message is intended only for the personal and confidential use of the recipients named above. If the reader of this email is not the intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.</font><font size="1"> If you have received this email in error, please notify the sender immediately by return email and permanently delete the copy you received. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments.</font><font size="1"> Neither CTC Holdings nor any affiliates (CTC) are responsible for any recommendation, solicitation, offer or agreement or any information about any transaction, customer account or account activity that may be attached to or contained in this communication. CTC accepts no liability for any content contained in the email, or any errors or omissions arising as a result of e-mail transmission.</font><font size="1"> Any opinions contained in this email constitute the sender's best judgment at this time and are subject to change without notice. CTC London Limited is authorized and regulated by the Financial Services Authority.</font></p>
</div></div></div>
<br>_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Mike Diehn<br>Senior Systems Administrator<br>ANSYS, Inc - Lebanon, NH Office<br><a href="mailto:mike.diehn@ansys.com">mike.diehn@ansys.com</a>, (603) 727-5492<br>
</div></div>