<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<body>
<div dir="auto">
<div dir="auto">What if node (especially vm) freezes for several minutes and then continues to write to a shared disk where other nodes already put their data?</div><div dir="auto">In my opinion, fencing, preferably two-level, is mandatory for lustre, trust me, I'd developed whole HA stack for both Exascaler and PangeaFS. We've seen so many points where data loss may occur...</div><div dir='auto'><br></div>
<div id="aqm-original" style="color: black;">
<!-- body start -->
<div class="aqm-original-body">
<div style="color: black;">
<p style="color: black; font-size: 10pt; font-family: sans-serif; margin: 8pt 0;">On December 19, 2023 19:42:56 Artem <tyomikh@gmail.com> wrote:</p>
<blockquote type="cite" class="gmail_quote" style="margin: 0 0 0 0.75ex; border-left: 1px solid #808080; padding-left: 0.75ex;">
<div dir="ltr">Andrei and Klaus thanks for prompt reply and clarification!<div>As I understand, design and behavior of Pacemaker is tightly coupled with the stonith concept. But isn't it too rigid?</div><div><br></div><div>Is there a way to leverage self-monitoring or pingd rules to trigger isolated node to umount its FS? Like vSphere High Availability host isolation response.<br>Can resource-stickiness=off (auto-failback) decrease risk of corruption by unresponsive node coming back online?<br>Is there a quorum feature not for cluster but for resource start/stop? Got lock - is welcome to mount, unable to refresh lease - force unmount. <br>Can on-fail=ignore break manual failover logic (stopped will be considered as failed and thus ignored)?<br></div><div><br></div><div>best regards,</div><div>Artem</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 19 Dec 2023 at 17:03, Klaus Wenninger <<a href="mailto:kwenning@redhat.com">kwenning@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 19, 2023 at 10:00 AM Andrei Borzenkov <<a href="mailto:arvidjaar@gmail.com" target="_blank">arvidjaar@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, Dec 19, 2023 at 10:41 AM Artem <<a href="mailto:tyomikh@gmail.com" target="_blank">tyomikh@gmail.com</a>> wrote:<br>
...<br>
> Dec 19 09:48:13 <a href="http://lustre-mds2.ntslab.ru" rel="noreferrer" target="_blank">lustre-mds2.ntslab.ru</a> pacemaker-schedulerd[785107] (update_resource_action_runnable) warning: OST4_stop_0 on lustre4 is unrunnable (node is offline)<br>
> Dec 19 09:48:13 <a href="http://lustre-mds2.ntslab.ru" rel="noreferrer" target="_blank">lustre-mds2.ntslab.ru</a> pacemaker-schedulerd[785107] (recurring_op_for_active) info: Start 20s-interval monitor for OST4 on lustre3<br>
> Dec 19 09:48:13 <a href="http://lustre-mds2.ntslab.ru" rel="noreferrer" target="_blank">lustre-mds2.ntslab.ru</a> pacemaker-schedulerd[785107] (log_list_item) notice: Actions: Stop OST4 ( lustre4 ) blocked<br>
<br>
This is the default for the failed stop operation. The only way<br>
pacemaker can resolve failure to stop a resource is to fence the node<br>
where this resource was active. If it is not possible (and IIRC you<br>
refuse to use stonith), pacemaker has no other choice as to block it.<br>
If you insist, you can of course sert on-fail=ignore, but this means<br>
unreachable node will continue to run resources. Whether it can lead<br>
to some corruption in your case I cannot guess.<br></blockquote><div><br></div><div>Don't know if I'm reading that correctly but I understand what you had written</div><div>above that you try to trigger the failover by stopping the VM (lustre4) without</div><div>ordered shutdown.</div><div>With fencing disabled what we are seeing is exactly what we would expect:</div><div>The state of the resource is unknown - pacemaker tries to stop it - doesn't work</div><div>as the node is offline - no fencing configured - so everything it can do is wait</div><div>till there is info if the resource is up or not.</div><div>I guess the strange output below is because of fencing disabled - quite an</div><div>unusual - also not recommended - configuration and so this might not have </div><div>shown up too often in that way.</div><div><br></div><div>Klaus</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> Dec 19 09:48:13 <a href="http://lustre-mds2.ntslab.ru" rel="noreferrer" target="_blank">lustre-mds2.ntslab.ru</a> pacemaker-schedulerd[785107] (pcmk__create_graph) crit: Cannot fence lustre4 because of OST4: blocked (OST4_stop_0)<br>
<br>
That is a rather strange phrase. The resource is blocked because the<br>
pacemaker could not fence the node, not the other way round.<br>
_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
</blockquote></div></div>
_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
</blockquote></div>
<div>_______________________________________________</div>
<div>Manage your subscription:</div>
<div><a class="aqm-autolink aqm-autowrap" href="https://lists.clusterlabs.org/mailman/listinfo/users">https://lists.clusterlabs.org/mailman/listinfo/users</a></div>
<div><br></div>
<div>ClusterLabs home: <a class="aqm-autolink aqm-autowrap" href="https://www.clusterlabs.org/">https://www.clusterlabs.org/</a></div>
<div><br></div>
</blockquote>
</div>
</div>
<!-- body end -->
</div><div dir="auto"><br></div>
</div></body>
</html>