<div dir="ltr">We are only using one mount, and that mount has nothing on it currently.<div><br></div><div><br></div><div>I have fixed the problem. Our OS is Ubuntu 16.04 LTS (Xenial). I added the 17.04 (Zesty) repo to get newer a newer version of Corosync. I upgraded Corosync, which upgraded a long list of other related packages (Pacemaker and gfs2 among them). My fencing now works properly. If a node loses network connection to the cluster, only that node is fenced. Presumably, it is a bug in one of the packages in the Xenial repo that has been fixed by in versions in Zesty.</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-------</div>Seth Reid<div><br></div><div></div><div><br></div></div></div></div>
<br><div class="gmail_quote">On Fri, Mar 31, 2017 at 10:31 AM, Bob Peterson <span dir="ltr"><<a href="mailto:rpeterso@redhat.com" target="_blank">rpeterso@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">----- Original Message -----<br>
| I can confirm that doing an ifdown is not the source of my corosync issues.<br>
| My cluster is in another state, so I can't pull a cable, but I can down a<br>
| port on a switch. That had the exact same affects as doing an ifdown. Two<br>
| machines got fenced when it should have only been one.<br>
|<br>
| -------<br>
| Seth Reid<br><br>
<br>
</span>Hi Seth,<br>
<br>
I don't know if your problem is the same thing I'm looking at BUT:<br>
I'm currently working on a fix to the GFS2 file system for a<br>
similar problem. The scenario is something like this:<br>
<br>
1. Node X goes down for some reason.<br>
2. Node X gets fenced by one of the other nodes.<br>
3. As part of the recovery, GFS2 on all the other nodes have to<br>
replay the journals for all the file systems mounted on X.<br>
4. GFS2 journal replay hogs the CPU, which causes corosync to be<br>
starved for CPU on some node (say node Y).<br>
5. Since corosync on node Y was starved for CPU, it doesn't respond<br>
in time to the other nodes (say node Z).<br>
6. Thus, node Z fences node Y.<br>
<br>
In my case, the solution is to fix GFS2 so that it does some<br>
"cond_resched()" (conditional schedule) statements to allow corosync<br>
(and dlm) to get some work done. Thus, corosync isn't starved for<br>
CPU and does its work, and therefore, it doesn't get fenced.<br>
<br>
I don't know if that's what is happening in your case.<br>
Do you have a lot of GFS2 mount points that would need recovery<br>
when the first fence event occurs?<br>
In my case, I can recreate the problem by having 60 GFS2 mount points.<br>
<br>
Hopefully I'll be sending a GFS2 patch to the cluster-devel<br>
mailing list for this problem soon.<br>
<br>
In testing my fix, I've periodically experienced some weirdness<br>
and other unexplained fencing, so maybe there's a second problem<br>
lurking (or maybe there's just something weird in the experimental<br>
kernel I'm using as a base). Hopefully testing will prove whether<br>
my fix to GFS2 recovery is enough or if there's another problem.<br>
<span class="im HOEnZb"><br>
Regards,<br>
<br>
Bob Peterson<br>
Red Hat File Systems<br>
<br>
</span><div class="HOEnZb"><div class="h5">______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div></div>