Hello,<br><br>thanks for the information. <br>I was looking at this page <a href="http://www.drbd.org/users-guide/s-pacemaker-fencing.html">http://www.drbd.org/users-guide/s-pacemaker-fencing.html</a><br>I did specify the following handlers :<br>
handlers {<br> fence-peer "/usr/lib/drbd/crm-fence-peer.sh";<br> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";<br><br> }<br><br>I disconnected the network cable between the clusters, corosync and drbd uses this link.<br>
<br>I was able to see that the fence script added a constraint :<br><br> location drbd-fence-by-handler-ms-drbd-supervision ms-drbd-supervision \<br> rule $id="drbd-fence-by-handler-rule-ms-drbd-supervision" $role="Master" -inf: #uname ne host<br>
<br>But this made :<br><br>:StandAlone ro:Secondary/Unknown ds:UpToDate/Outdated on drbd.<br><br>I don't really understand what I should be expected from those handlers ?<br>When cleaning up the errors, I shoudl delete the constraint right ?<br>
<br>Regards,<br><br>Hugo<br><br><div class="gmail_quote">On 26 July 2011 19:27, Digimer <span dir="ltr"><<a href="mailto:linux@alteeve.com">linux@alteeve.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">On 07/26/2011 11:43 AM, Lars Ellenberg wrote:<br>
> On Wed, Jul 20, 2011 at 11:36:25AM -0400, Digimer wrote:<br>
>> On 07/20/2011 11:24 AM, Hugo Deprez wrote:<br>
>>> Hello Andrew,<br>
>>><br>
>>> in fact DRBD was in standalone mode but the cluster was working :<br>
>>><br>
>>> Here is the syslog of the drbd's split brain :<br>
>>><br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.052245] block drbd0: Handshake<br>
>>> successful: Agreed network protocol version 91<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.052267] block drbd0: conn(<br>
>>> WFConnection -> WFReportParams )<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.066677] block drbd0: Starting<br>
>>> asender thread (from drbd0_receiver [23281])<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.066863] block drbd0:<br>
>>> data-integrity-alg: <not-used><br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.079182] block drbd0:<br>
>>> drbd_sync_handshake:<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.079190] block drbd0: self<br>
>>> BBA9B794EDB65CDF:9E8FB52F896EF383:C5FE44742558F9E1:1F9E06135B8E296F<br>
>>> bits:75338 flags:0<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.079196] block drbd0: peer<br>
>>> 8343B5F30B2BF674:9E8FB52F896EF382:C5FE44742558F9E0:1F9E06135B8E296F<br>
>>> bits:769 flags:0<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.079200] block drbd0:<br>
>>> uuid_compare()=100 by rule 90<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.079203] block drbd0: Split-Brain<br>
>>> detected, dropping connection!<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.079439] block drbd0: helper<br>
>>> command: /sbin/drbdadm split-brain minor-0<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.083955] block drbd0: meta<br>
>>> connection shut down by peer.<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.084163] block drbd0: conn(<br>
>>> WFReportParams -> NetworkFailure )<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.084173] block drbd0: asender<br>
>>> terminated<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.084176] block drbd0: Terminating<br>
>>> asender thread<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.084406] block drbd0: helper<br>
>>> command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.084420] block drbd0: conn(<br>
>>> NetworkFailure -> Disconnecting )<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.084430] block drbd0: error<br>
>>> receiving ReportState, l: 4!<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.084789] block drbd0: Connection<br>
>>> closed<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.084813] block drbd0: conn(<br>
>>> Disconnecting -> StandAlone )<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.086345] block drbd0: receiver<br>
>>> terminated<br>
>>> Jul 15 08:45:34 node1 kernel: [1536023.086349] block drbd0: Terminating<br>
>>> receiver thread<br>
>><br>
>> This was a DRBD split-brain, not a pacemaker split. I think that might<br>
>> have been the source of confusion.<br>
>><br>
>> The split brain occurs when both DRBD nodes lose contact with one<br>
>> another and then proceed as StandAlone/Primary/UpToDate. To avoid this,<br>
>> configure fencing (stonith) in Pacemaker, then use 'crm-fence-peer.sh'<br>
>> in drbd.conf;<br>
>><br>
>> ===<br>
>> disk {<br>
>> fencing resource-and-stonith;<br>
>> }<br>
>><br>
>> handlers {<br>
>> outdate-peer "/path/to/crm-fence-peer.sh";<br>
>> }<br>
>> ===<br>
><br>
> Thanks, that is basically right.<br>
> Let me fill in some details, though:<br>
><br>
>> This will tell DRBD to block (resource) and fence (stonith). DRBD will<br>
><br>
> drbd fencing options are "fencing resource-only",<br>
> and "fencing resource-and-stonith".<br>
><br>
> "resource-only" does *not* block IO while the fencing handler runs.<br>
><br>
> "resource-and-stonith" does block IO.<br>
<br>
</div></div>Ahhh, that's why I was confused. I thought the 'resource' meant the same<br>
thing in both cases, but had only read the 'resource-and-stonith' section.<br>
<div class="im"><br>
>> not resume IO until either the fence script exits with a success, or<br>
>> until an admit types 'drbdadm resume-io <res>'.<br>
><br>
><br>
>> The CRM script simply calls pacemaker and asks it to fence the other<br>
>> node.<br>
><br>
> No. It tries to place a constraint forcing the Master role off of any<br>
> node but the one with the good data.<br>
<br>
</div>Ok, I thought it was akin to the 'obliterate-peer.sh' script, which<br>
calls 'fence_node'... I made an assumption, which was not correct.<br>
<div class="im"><br>
>> When a node has actually failed, then the lost no is fenced. If<br>
>> both nodes are up but disconnected, as you had, then only the fastest<br>
>> node will succeed in calling the fence, and the slower node will be<br>
>> fenced before it can call a fence.<br>
><br>
> "fenced" may be "restricted from being/becoming Master" by that fencing<br>
> constraint. Or, if pacemaker decided to do so, actually "shot" by some<br>
> node level fencing agent (stonith).<br>
><br>
> All that resource-level fencing by placing some constraint stuff<br>
> obviously only works as long as the cluster communication is still up.<br>
> It not only the drbd replication link had issues, but the cluster<br>
> communication was down as well, it becomes a bit more complex.<br>
<br>
</div>Thanks for the clarity. Today I learned. :)<br>
<div class="im HOEnZb"><br>
--<br>
Digimer<br>
E-Mail: <a href="mailto:digimer@alteeve.com">digimer@alteeve.com</a><br>
Freenode handle: digimer<br>
Papers and Projects: <a href="http://alteeve.com" target="_blank">http://alteeve.com</a><br>
Node Assassin: <a href="http://nodeassassin.org" target="_blank">http://nodeassassin.org</a><br>
"At what point did we forget that the Space Shuttle was, essentially,<br>
a program that strapped human beings to an explosion and tried to stab<br>
through the sky with fire and math?"<br>
<br>
</div><div class="HOEnZb"><div class="h5">_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
</div></div></blockquote></div><br>