<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>> Date: Mon, 7 Jul 2014 10:04:20 +0200<br><div>> From: lars.ellenberg@linbit.com<br>> To: drbd-user@lists.linbit.com; pacemaker@oss.clusterlabs.org<br>> Subject: Re: [Pacemaker] [DRBD-user] DRBD active/passive on Pacemaker+CMAN cluster unexpectedly performs STONITH when promoting<br>> <br>> On Fri, Jul 04, 2014 at 06:04:12PM +0200, Giuseppe Ragusa wrote:<br>> > > > The setup "almost" works (all seems ok with: "pcs status", "crm_mon<br>> > > > -Arf1", "corosync-cfgtool -s", "corosync-objctl | grep member") , but<br>> > > > every time it needs a resource promotion (to Master, i.e. becoming<br>> > > > primary) it either fails or fences the other node (the one supposed to<br>> > > > become Slave i.e. secondary) and only then succeeds.<br>> > > ><br>> > > > It happens, for example both on initial resource definition (when<br>> > > > attempting first start) and on node entering standby (when trying to<br>> > > > automatically move the resources by stopping then starting them).<br>> > > > <br>> > > > I collected a full "pcs cluster report" and I can provide a CIB dump,<br>> > > > but I will initially paste here an excerpt from my configuration just<br>> > > > in case it happens to be a simple configuration error that someone can<br>> > > > spot on the fly ;> (hoping...)<br>> > > > <br>> > > > Keep in mind that the setup has separated redundant network<br>> > > > connections for LAN (1 Gib/s LACP to switches), Corosync (1 Gib/s<br>> > > > roundrobin back-to-back) and DRBD (10 Gib/s roundrobin back-to-back)<br>> > > > and that FQDNs are correctly resolved through /etc/hosts<br>> > > <br>> > > Make sure youre DRBD are "Connected UpToDate/UpToDate"<br>> > > before you let the cluster take over control of who is master.<br>> > <br>> > Thanks for your important reminder.<br>> > <br>> > Actually they had been "Connected UpToDate/UpToDate", and I subsequently had all manually demoted to secondary<br>> > then down-ed before eventually stopping the (manually started) DRBD service.<br>> > <br>> > Only at the end did I start/configure the cluster.<br>> > <br>> > The problem is now resolved and it seems that my improper use of<br>> > rhcs_fence as fence-peer was the culprit (now switched to<br>> > crm-fence-peer.sh), but I still do not understand why rhcs_fence was<br>> > called at all in the beginning (once called, it may have caused<br>> > unforeseen consequences, I admit) since DRBD docs clearly state that<br>> > communication disruption must be involved in order to call fence-peer<br>> > into action.<br>> <br>> You likely managed to have data divergence<br>> between your instances of DRBD,<br>> likely caused by a cluster split-brain.<br><br>I'm quite positively sure that no communication disruption happened (both on the 2 RRP Corosync links and the separated DRBD link) up to the moment when I committed/started the first DRBD/KVM resources, but maybe something else interfered (I'm repeating myself, but I want to stress that after manually stopping the still-not-clusterd VM I manually made secondary the primary DRBD resource and then brought both sides "drbdadm down", but those seem innocent acts too).<br><br>> So DRBD would refuse to connect,<br>> and thus would be not connected when promoted.<br><br>The "not connected when promoted" seems the crucial part, whatever the reason behind.<br>As I said, I noticed an apparent "death" of cluster components (cluster not connected as output of "pcs status") on the victim node mere seconds before being shot.<br><br>> Just because you can shoot someone<br>> does not make your data any better,<br>> nor does it tell the victim node that his data is "bad"<br>> (from the shooting nodes point of view)<br>> so they would just keep killing each other then.<br><br>This is absolutely clear to me.<br>Thanks anyway for pointing it out.<br><br>> "Don't do that."<br><br>:)<br><br>> But tell the cluster to not even attempt to promote,<br>> unless the local data is known to be UpToDate *and*<br>> the remote data is either known (DRBD is connected)<br>> or the remote date is known to be bad (Outdated or worse).<br>> <br>> the ocf:linbit:drbd agent has an "adjust master scores"<br>> parameter for that. See there.<br><br>so maybe setting it to "0 10 1000 10000" (from the default of "5 10 1000 10000") could be enough, if I understood it right, given that I already have resource-and-stonith + crm-fence-peer.sh (and the stonith part is tested), but what would I gain from this (maybe spare me from unnecessary stonith?), given that no actual split brain happened and everything resolved "by itself"? <br>After restarting the shot node, it came up allright and eventually (by simply changing from rhcs_fence to crm-fence-peer.sh and reloading configuration) everything worked as expected (by me), with new resources brought up without stonith involved and resource moving working as expected.<br><br>If you think that what happened could reveal some bugs/misbehaviour, I can privately send you full unedited logs/reports.<br><br>Many thanks again for all the help and explanations.<br><br>Regards,<br>Giuseppe Ragusa<br><br></div> </div></body>
</html>