<div dir="ltr"><font color="#c27ba0"><span style="font-size:12.8px">Not true, the faster node will kill the slower node first. It is</span><br style="font-size:12.8px"><span style="font-size:12.8px">possible that through misconfiguration, both could die, but it's rare</span><br style="font-size:12.8px"><span style="font-size:12.8px">and easily avoided with a 'delay="15"' set on the fence config for the</span><br style="font-size:12.8px"><span style="font-size:12.8px">node you want to win.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Don't use a delay on the other node, just the node you want to live in</span><br style="font-size:12.8px"></font><span style="font-size:12.8px"><font color="#c27ba0">such a case.</font><br></span><br>**<div style="font-size:12.8px"> 1. Given Active/Passive setup, resources are active on Node1</div><div style="font-size:12.8px"> 2. fence1(prefers to Node1, delay=15) and fence2(prefers to Node2, delay=30)</div><div style="font-size:12.8px"> 3. Node2 goes down</div><div style="font-size:12.8px"> 4. Node1 thinks Node2 goes down / Node2 thinks Node1 goes down</div><div style="font-size:12.8px"> 5. fence1 counts 15 seconds before he fence Node1 while fence2 counts 30 seconds before he fence Node2</div><div style="font-size:12.8px"> 6. Since fence1 do have shorter time than fence2, fence1 executes and shutdown Node1. </div><div style="font-size:12.8px"> 7. fence1(action: shutdown Node1) will trigger first always because it has shorter delay than fence2.<br><br>** Okay what's important is that they should be different. But in the case above, even though Node2 goes down but Node1 has shorter delay, Node1 gets fenced/shutdown. This is a sample scenario. I don't get the point. Can you comment on this? <br><br>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 10, 2018 at 12:18 AM, Klaus Wenninger <span dir="ltr"><<a href="mailto:kwenning@redhat.com" target="_blank">kwenning@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 07/09/2018 05:53 PM, Digimer wrote:<br>
> On 2018-07-09 11:45 AM, Klaus Wenninger wrote:<br>
>> On 07/09/2018 05:33 PM, Digimer wrote:<br>
>>> On 2018-07-09 09:56 AM, Klaus Wenninger wrote:<br>
>>>> On 07/09/2018 03:49 PM, Digimer wrote:<br>
>>>>> On 2018-07-09 08:31 AM, Klaus Wenninger wrote:<br>
>>>>>> On 07/09/2018 02:04 PM, Confidential Company wrote:<br>
>>>>>>> Hi,<br>
>>>>>>><br>
>>>>>>> Any ideas what triggers fencing script or stonith?<br>
>>>>>>><br>
>>>>>>> Given the setup below:<br>
>>>>>>> 1. I have two nodes<br>
>>>>>>> 2. Configured fencing on both nodes<br>
>>>>>>> 3. Configured delay=15 and delay=30 on fence1(for Node1) and<br>
>>>>>>> fence2(for Node2) respectively<br>
>>>>>>><br>
>>>>>>> *What does it mean to configured delay in stonith? wait for 15 seconds<br>
>>>>>>> before it fence the node?<br>
>>>>>> Given that on a 2-node-cluster you don't have real quorum to make one<br>
>>>>>> partial cluster fence the rest of the nodes the different delays are meant<br>
>>>>>> to prevent a fencing-race.<br>
>>>>>> Without different delays that would lead to both nodes fencing each<br>
>>>>>> other at the same time - finally both being down.<br>
>>>>> Not true, the faster node will kill the slower node first. It is<br>
>>>>> possible that through misconfiguration, both could die, but it's rare<br>
>>>>> and easily avoided with a 'delay="15"' set on the fence config for the<br>
>>>>> node you want to win.<br>
>>>> What exactly is not true? Aren't we saying the same?<br>
>>>> Of course one of the delays can be 0 (most important is that<br>
>>>> they are different).<br>
>>> Perhaps I misunderstood your message. It seemed to me that the<br>
>>> implication was that fencing in 2-node without a delay always ends up<br>
>>> with both nodes being down, which isn't the case. It can happen if the<br>
>>> fence methods are not setup right (ie: the node isn't set to immediately<br>
>>> power off on ACPI power button event).<br>
>> Yes, a misunderstanding I guess.<br>
>><br>
>> Should have been more verbose in saying that due to the<br>
>> time between the fencing-command fired off to the fencing<br>
>> device and the actual fencing taking place (as you state<br>
>> dependent on how it is configured in detail - but a measurable<br>
>> time in all cases) there is a certain probability that when<br>
>> both nodes start fencing at roughly the same time we will<br>
>> end up with 2 nodes down.<br>
>><br>
>> Everybody has to find his own tradeoff between reliability<br>
>> fence-races are prevented and fencing delay I guess.<br>
> We've used this;<br>
><br>
> 1. IPMI (with the guest OS set to immediately power off) as primary,<br>
> with a 15 second delay on the active node.<br>
><br>
> 2. Two Switched PDUs (two power circuits, two PSUs) as backup fencing<br>
> for when IPMI fails, with no delay.<br>
><br>
> In ~8 years, across dozens and dozens of clusters and countless fence<br>
> actions, we've never had a dual-fence event (where both nodes go down).<br>
> So it can be done safely, but as always, test test test before prod.<br>
<br>
</div></div>No doubt about that this setup is working reliably.<br>
You just have to know your fencing-devices and<br>
which delays they involve.<br>
<br>
If we are talking about SBD (with disk as otherwise<br>
it doesn't work in a sensible way in 2-node-clusters)<br>
for instance I would strongly advise using a delay.<br>
<br>
So I guess it is important to understand the basic<br>
idea behind this different delay-based fence-race<br>
avoidance.<br>
Afterwards you can still decide why it is no issue<br>
in your own setup.<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
>>> If the delay is set on both nodes, and they are different, it will work<br>
>>> fine. The reason not to do this is that if you use 0, then don't use<br>
>>> anything at all (0 is default), and any other value causes avoidable<br>
>>> fence delays.<br>
>>><br>
>>>>> Don't use a delay on the other node, just the node you want to live in<br>
>>>>> such a case.<br>
>>>>><br>
>>>>>>> *Given Node1 is active and Node2 goes down, does it mean fence1 will<br>
>>>>>>> first execute and shutdowns Node1 even though Node2 goes down?<br>
>>>>>> If Node2 managed to sign off properly it will not.<br>
>>>>>> If network-connection is down so that Node2 can't inform Node1 that it<br>
>>>>>> is going<br>
>>>>>> down and finally has stopped all resources it will be fenced by Node1.<br>
>>>>>><br>
>>>>>> Regards,<br>
>>>>>> Klaus<br>
>>>>> Fencing occurs in two cases;<br>
>>>>><br>
>>>>> 1. The node stops responding (meaning it's in an unknown state, so it is<br>
>>>>> fenced to force it into a known state).<br>
>>>>> 2. A resource / service fails to stop stop. In this case, the service is<br>
>>>>> in an unknown state, so the node is fenced to force the service into a<br>
>>>>> known state so that it can be safely recovered on the peer.<br>
>>>>><br>
>>>>> Graceful withdrawal of the node from the cluster, and graceful stopping<br>
>>>>> of services will not lead to a fence (because in both cases, the node /<br>
>>>>> service are in a known state - off).<br>
>>>>><br>
>>> <br>
><br>
<br>
</div></div></blockquote></div><br></div>