<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 129.75pt 1.0in 129.7pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> -----Original Message-----</p>
<p class="MsoPlainText">> From: Users <users-bounces@clusterlabs.org> On Behalf Of Ulrich Windl</p>
<p class="MsoPlainText">> Sent: Tuesday, February 19, 2019 11:35 PM</p>
<p class="MsoPlainText">> To: users@clusterlabs.org</p>
<p class="MsoPlainText">> Subject: [ClusterLabs] Antw: Re: Why Do All The Services Go Down When</p>
<p class="MsoPlainText">> Just One Fails?</p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> >>> Eric Robinson <<a href="mailto:eric.robinson@psmnv.com"><span style="color:windowtext;text-decoration:none">eric.robinson@psmnv.com</span></a>> schrieb am 19.02.2019 um</p>
<p class="MsoPlainText">> >>> 21:06 in</p>
<p class="MsoPlainText">> Nachricht</p>
<p class="MsoPlainText">> <<a href="mailto:MN2PR03MB4845BE22FADA30B472174B79FA7C0@MN2PR03MB4845.namprd03.prod.outlook.com"><span style="color:windowtext;text-decoration:none">MN2PR03MB4845BE22FADA30B472174B79FA7C0@MN2PR03MB4845.nampr</span></a></p>
<p class="MsoPlainText"><a href="mailto:MN2PR03MB4845BE22FADA30B472174B79FA7C0@MN2PR03MB4845.namprd03.prod.outlook.com"><span style="color:windowtext;text-decoration:none">> d03.prod.outlook.com</span></a>></p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> >> -----Original Message-----</p>
<p class="MsoPlainText">> >> From: Users <<a href="mailto:users-bounces@clusterlabs.org"><span style="color:windowtext;text-decoration:none">users-bounces@clusterlabs.org</span></a>> On Behalf Of Ken Gaillot</p>
<p class="MsoPlainText">> >> Sent: Tuesday, February 19, 2019 10:31 AM</p>
<p class="MsoPlainText">> >> To: Cluster Labs - All topics related to open-source clustering</p>
<p class="MsoPlainText">> >> welcomed <<a href="mailto:users@clusterlabs.org"><span style="color:windowtext;text-decoration:none">users@clusterlabs.org</span></a>></p>
<p class="MsoPlainText">> >> Subject: Re: [ClusterLabs] Why Do All The Services Go Down When Just</p>
<p class="MsoPlainText">> >> One Fails?</p>
<p class="MsoPlainText">> >></p>
<p class="MsoPlainText">> >> On Tue, 2019-02-19 at 17:40 +0000, Eric Robinson wrote:</p>
<p class="MsoPlainText">> >> > > -----Original Message-----</p>
<p class="MsoPlainText">> >> > > From: Users <<a href="mailto:users-bounces@clusterlabs.org"><span style="color:windowtext;text-decoration:none">users-bounces@clusterlabs.org</span></a>> On Behalf Of Andrei</p>
<p class="MsoPlainText">> >> > > Borzenkov</p>
<p class="MsoPlainText">> >> > > Sent: Sunday, February 17, 2019 11:56 AM</p>
<p class="MsoPlainText">> >> > > To: <a href="mailto:users@clusterlabs.org"><span style="color:windowtext;text-decoration:none">users@clusterlabs.org</span></a></p>
<p class="MsoPlainText">> >> > > Subject: Re: [ClusterLabs] Why Do All The Services Go Down When</p>
<p class="MsoPlainText">> >> > > Just One Fails?</p>
<p class="MsoPlainText">> >> > ></p>
<p class="MsoPlainText">> >> > > 17.02.2019 0:44, Eric Robinson пишет:</p>
<p class="MsoPlainText">> >> > > > Thanks for the feedback, Andrei.</p>
<p class="MsoPlainText">> >> > > ></p>
<p class="MsoPlainText">> >> > > > I only want cluster failover to occur if the filesystem or drbd</p>
<p class="MsoPlainText">> >> > > > resources fail,</p>
<p class="MsoPlainText">> >> > ></p>
<p class="MsoPlainText">> >> > > or if the cluster messaging layer detects a complete node failure.</p>
<p class="MsoPlainText">> >> > > Is there a</p>
<p class="MsoPlainText">> >> > > way to tell PaceMaker not to trigger a cluster failover if any of</p>
<p class="MsoPlainText">> >> > > the p_mysql resources fail?</p>
<p class="MsoPlainText">> >> > > ></p>
<p class="MsoPlainText">> >> > ></p>
<p class="MsoPlainText">> >> > > Let's look at this differently. If all these applications depend</p>
<p class="MsoPlainText">> >> > > on each other, you should not be able to stop individual resource</p>
<p class="MsoPlainText">> >> > > in the first place - you need to group them or define dependency</p>
<p class="MsoPlainText">> >> > > so that stopping any resource would stop everything.</p>
<p class="MsoPlainText">> >> > ></p>
<p class="MsoPlainText">> >> > > If these applications are independent, they should not share</p>
<p class="MsoPlainText">> >> > > resources.</p>
<p class="MsoPlainText">> >> > > Each MySQL application should have own IP and own FS and own</p>
<p class="MsoPlainText">> >> > > block device for this FS so that they can be moved between</p>
<p class="MsoPlainText">> >> > > cluster nodes independently.</p>
<p class="MsoPlainText">> >> > ></p>
<p class="MsoPlainText">> >> > > Anything else will lead to troubles as you already observed.</p>
<p class="MsoPlainText">> >> ></p>
<p class="MsoPlainText">> >> > FYI, the MySQL services do not depend on each other. All of them</p>
<p class="MsoPlainText">> >> > depend on the floating IP, which depends on the filesystem, which</p>
<p class="MsoPlainText">> >> > depends on DRBD, but they do not depend on each other. Ideally, the</p>
<p class="MsoPlainText">> >> > failure of p_mysql_002 should not cause failure of other mysql</p>
<p class="MsoPlainText">> >> > resources, but now I understand why it happened. Pacemaker wanted</p>
<p class="MsoPlainText">> >> > to start it on the other node, so it needed to move the floating</p>
<p class="MsoPlainText">> >> > IP, filesystem, and DRBD primary, which had the cascade effect of</p>
<p class="MsoPlainText">> >> > stopping the other MySQL resources.</p>
<p class="MsoPlainText">> >> ></p>
<p class="MsoPlainText">> >> > I think I also understand why the p_vip_clust01 resource blocked.</p>
<p class="MsoPlainText">> >> ></p>
<p class="MsoPlainText">> >> > FWIW, we've been using Linux HA since 2006, originally Heartbeat,</p>
<p class="MsoPlainText">> >> > but then Corosync+Pacemaker. The past 12 years have been relatively</p>
<p class="MsoPlainText">> >> > problem free. This symptom is new for us, only within the past year.</p>
<p class="MsoPlainText">> >> > Our cluster nodes have many separate instances of MySQL running, so</p>
<p class="MsoPlainText">> >> > it is not practical to have that many filesystems, IPs, etc. We are</p>
<p class="MsoPlainText">> >> > content with the way things are, except for this new troubling</p>
<p class="MsoPlainText">> >> > behavior.</p>
<p class="MsoPlainText">> >> ></p>
<p class="MsoPlainText">> >> > If I understand the thread correctly, op-fail=stop will not work</p>
<p class="MsoPlainText">> >> > because the cluster will still try to stop the resources that are</p>
<p class="MsoPlainText">> >> > implied dependencies.</p>
<p class="MsoPlainText">> >> ></p>
<p class="MsoPlainText">> >> > Bottom line is, how do we configure the cluster in such a way that</p>
<p class="MsoPlainText">> >> > there are no cascading circumstances when a MySQL resource fails?</p>
<p class="MsoPlainText">> >> > Basically, if a MySQL resource fails, it fails. We'll deal with</p>
<p class="MsoPlainText">> >> > that on an ad-hoc basis. I don't want the whole cluster to barf.</p>
<p class="MsoPlainText">> >> > What about op-fail=ignore? Earlier, you suggested symmetrical=false</p>
<p class="MsoPlainText">> >> > might also do the trick, but you said it comes with its own can or worms.</p>
<p class="MsoPlainText">> >> > What are the downsides with op-fail=ignore or asymmetrical=false?</p>
<p class="MsoPlainText">> >> ></p>
<p class="MsoPlainText">> >> > --Eric</p>
<p class="MsoPlainText">> >></p>
<p class="MsoPlainText">> >> Even adding on-fail=ignore to the recurring monitors may not do what</p>
<p class="MsoPlainText">> >> you want, because I suspect that even an ignored failure will make</p>
<p class="MsoPlainText">> >> the node</p>
<p class="MsoPlainText">> less</p>
<p class="MsoPlainText">> >> preferable for all the other resources. But it's worth testing.</p>
<p class="MsoPlainText">> >></p>
<p class="MsoPlainText">> >> Otherwise, your best option is to remove all the recurring monitors</p>
<p class="MsoPlainText">> >> from</p>
<p class="MsoPlainText">> the</p>
<p class="MsoPlainText">> >> mysql resources, and rely on external monitoring (e.g. nagios,</p>
<p class="MsoPlainText">> >> icinga,</p>
<p class="MsoPlainText">> > monit,</p>
<p class="MsoPlainText">> >> ...) to detect problems.</p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > This is probably a dumb question, but can we remove just the monitor</p>
<p class="MsoPlainText">> > operation but leave the resource configured in the cluster? If a node</p>
<p class="MsoPlainText">> > fails</p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> > over, we do want the resources to start automatically on the new</p>
<p class="MsoPlainText">> > primary node.</p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> Actually I wonder whether this makes sense at all: IMHO a cluster ensures</p>
<p class="MsoPlainText">> that the phone does not ring at night to make me perform some recovery</p>
<p class="MsoPlainText">> operations after a failure. Once you move to manual start and stop of</p>
<p class="MsoPlainText">> resources, I fail to see the reason for a cluster.</p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> When done well, independent resources should be configured (and</p>
<p class="MsoPlainText">> managed) independently; otherwise they are dependent. There is no</p>
<p class="MsoPlainText">> "middle-way".</p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> Regards,</p>
<p class="MsoPlainText">> Ulrich</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">The following should show OK in a fixed font like Consolas, but the following setup is supposed to be possible, and is even referenced in the ClusterLabs documentation.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><span style="font-family:Consolas">+--------------+<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">| mysql001 +--+<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">+--------------+ |<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">+--------------+ |<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">| mysql002 +--+<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">+--------------+ |<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">+--------------+ | +-------------+ +------------+ +----------+<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">| mysql003 +----->+ floating ip +-->+ filesystem +-->+ blockdev |<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">+--------------+ | +-------------+ +------------+ +----------+<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">+--------------+ |<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">| mysql004 +--+<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">+--------------+ |<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">+--------------+ |<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">| mysql005 +--+<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">+--------------+<o:p></o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">In the layout above, the MySQL instances are dependent on the same underlying service stack, but they are not dependent on each other. Therefore, as I understand it, the failure of one MySQL instance should not cause the failure of other
MySQL instances if on-fail=ignore on-fail=stop. At least, that’s the way it seems to me, but based on the thread, I guess it does not behave that way.
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> >> --</p>
<p class="MsoPlainText">> >> Ken Gaillot <<a href="mailto:kgaillot@redhat.com"><span style="color:windowtext;text-decoration:none">kgaillot@redhat.com</span></a>></p>
<p class="MsoPlainText">> >></p>
<p class="MsoPlainText">> >> _______________________________________________</p>
<p class="MsoPlainText">> >> Users mailing list: <a href="mailto:Users@clusterlabs.org">
<span style="color:windowtext;text-decoration:none">Users@clusterlabs.org</span></a></p>
<p class="MsoPlainText">> >> <a href="https://lists.clusterlabs.org/mailman/listinfo/users">
<span style="color:windowtext;text-decoration:none">https://lists.clusterlabs.org/mailman/listinfo/users</span></a></p>
<p class="MsoPlainText">> >></p>
<p class="MsoPlainText">> >> Project Home: <a href="http://www.clusterlabs.org"><span style="color:windowtext;text-decoration:none">http://www.clusterlabs.org</span></a> Getting started:</p>
<p class="MsoPlainText">> >> <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">
<span style="color:windowtext;text-decoration:none">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</span></a></p>
<p class="MsoPlainText">> >> Bugs: <a href="http://bugs.clusterlabs.org"><span style="color:windowtext;text-decoration:none">http://bugs.clusterlabs.org</span></a></p>
<p class="MsoPlainText">> > _______________________________________________</p>
<p class="MsoPlainText">> > Users mailing list: <a href="mailto:Users@clusterlabs.org">
<span style="color:windowtext;text-decoration:none">Users@clusterlabs.org</span></a></p>
<p class="MsoPlainText">> > <a href="https://lists.clusterlabs.org/mailman/listinfo/users">
<span style="color:windowtext;text-decoration:none">https://lists.clusterlabs.org/mailman/listinfo/users</span></a></p>
<p class="MsoPlainText">> ></p>
<p class="MsoPlainText">> > Project Home: <a href="http://www.clusterlabs.org"><span style="color:windowtext;text-decoration:none">http://www.clusterlabs.org</span></a> Getting started:</p>
<p class="MsoPlainText">> > <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">
<span style="color:windowtext;text-decoration:none">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</span></a></p>
<p class="MsoPlainText">> > Bugs: <a href="http://bugs.clusterlabs.org"><span style="color:windowtext;text-decoration:none">http://bugs.clusterlabs.org</span></a></p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> _______________________________________________</p>
<p class="MsoPlainText">> Users mailing list: <a href="mailto:Users@clusterlabs.org">
<span style="color:windowtext;text-decoration:none">Users@clusterlabs.org</span></a></p>
<p class="MsoPlainText">> <a href="https://lists.clusterlabs.org/mailman/listinfo/users">
<span style="color:windowtext;text-decoration:none">https://lists.clusterlabs.org/mailman/listinfo/users</span></a></p>
<p class="MsoPlainText">> </p>
<p class="MsoPlainText">> Project Home: <a href="http://www.clusterlabs.org"><span style="color:windowtext;text-decoration:none">http://www.clusterlabs.org</span></a> Getting started:</p>
<p class="MsoPlainText">> <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">
<span style="color:windowtext;text-decoration:none">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</span></a></p>
<p class="MsoPlainText">> Bugs: <a href="http://bugs.clusterlabs.org"><span style="color:windowtext;text-decoration:none">http://bugs.clusterlabs.org</span></a></p>
</div>
</body>
</html>