<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >Hey Andrei, Ulrich<br><br>I am working with Janghyuk on his testing effort. Thank you for your responses, you have clarified some of the terminology we have been misusing.<br><br>As Janghyuk mentions previously, we have two "full cluster" nodes using two-node quorum and multiple heart beat rings + two more servers as pacemaker-remotes. The pacemaker-remote connection resources each prefer a specific full cluster node to run on, however they are configured such that they can fail over to the other cluster node if needed. Here is the configuration again...<br><br><strong>corosync.conf - nodelist & quorum</strong></div>
<div dir="ltr" ><div><span style="font-family:Courier New,Courier,monospace;" >nodelist {<br> node {<br> ring0_addr: node-1-subnet-1<br> ring1_addr: node-1-subnet-2<br> name: jangcluster-srv-1<br> nodeid: 1<br> }<br> node {<br> ring0_addr: node-2-subnet-1<br> ring1_addr: node-2-subnet-2<br> name: jangcluster-srv-2<br> nodeid: 2<br> }<br>}<br>quorum {<br> provider: corosync_votequorum<br> two_node: 1<br>}</span></div>
<div> </div>
<div><strong>crm config show</strong><br> </div>
<div><span style="font-family:Courier New,Courier,monospace;" >node 1: jangcluster-srv-1<br>node 2: jangcluster-srv-2<br>node jangcluster-srv-3:remote<br>node jangcluster-srv-4:remote<br>primitive GPFS-Fence stonith:fence_gpfs \<br> params instance=regress1 shared_filesystem="<shared filesystem path>" pcmk_host_list=" jangcluster-srv-1 jangcluster-srv-2 jangcluster-srv-3 jangcluster-srv-4" secure=true \<br> op monitor interval=30s timeout=500s \<br> op off interval=0 \<br> meta is-managed=true<br>primitive jangcluster-srv-3 ocf:pacemaker:remote \<br> params server=jangcluster-srv-3 reconnect_interval=1m \<br> op monitor interval=30s \<br> op_params migration-threshold=1 \<br> op stop interval=0 \<br> meta is-managed=true<br>primitive jangcluster-srv-4 ocf:pacemaker:remote \<br> params server=jangcluster-srv-4 reconnect_interval=1m \<br> op monitor interval=30s \<br> op_params migration-threshold=1 \<br> meta is-managed=true<br>location prefer-CF-Hosts GPFS-Fence \<br> rule 100: #uname eq jangcluster-srv-1 or #uname eq jangcluster-srv-2<br>location prefer-node-jangcluster-srv-3 jangcluster-srv-3 100: jangcluster-srv-1<br>location prefer-node-jangcluster-srv-3-2 jangcluster-srv-3 50: jangcluster-srv-2<br>location prefer-node-jangcluster-srv-4 jangcluster-srv-4 100: jangcluster-srv-2<br>location prefer-node-jangcluster-srv-4-2 jangcluster-srv-4 50: jangcluster-srv-1</span></div>
<div> </div>We are testing several failure scenarios... In most cases the pacemaker-remote connection resource will successfully fail over to the other cluster node. For example, if we reboot, shutdown, halt, or crash srv-2, the pacemaker-remote connection resource for srv-4 will fail over and start running on srv-1 without srv-3's physical host getting fenced. Manually fencing srv-2 via stonith_admin also works.<br><br>However when we attempt to simulate a communication failure on srv-2's Ethernet adapter via iptables, we observe srv-3's host getting fenced before the connection resource fails over to srv-1.<br>The concern here is that in the future we may have many remotes connecting to a single cluster host, and so far it seems like a Ethernet adapter issue on the cluster host could lead to many remote hosts getting unnecessarily fenced.<br><br>Here are the updated iptables commands that we run on srv-2 to simulate srv-2 losing the ability to communicate to srv-4.<br><span style="font-family:Courier New,Courier,monospace;" >iptables -A INPUT -s [IP of srv-1] -j DROP ; iptables -A OUTPUT -d [IP of srv-1] -j DROP<br>iptables -A INPUT -s [IP of srv-3] -j DROP ; iptables -A OUTPUT -d [IP of srv-3] -j DROP<br>iptables -A INPUT -s [IP of srv-4] -j DROP ; iptables -A OUTPUT -d [IP of srv-4] -j DROP</span><br><br>As Janghyuk has shown previously, it seems that the pacemaker-remote connection monitor timesout and causes the remote host to get fenced. Here are the logs that I think are most relevant.<br><br><span style="font-family:Courier New,Courier,monospace;" >Oct 22 12:21:09.389 jangcluster-srv-2 pacemaker-schedulerd[776553] (pe_get_failcount) info: jangcluster-srv-4 has failed 1 times on jangcluster-srv-2<br>Oct 22 12:21:09.389 jangcluster-srv-2 pacemaker-schedulerd[776553] (pe_get_failcount) info: jangcluster-srv-4 has failed 1 times on jangcluster-srv-2<br>Oct 22 12:21:09.389 jangcluster-srv-2 pacemaker-schedulerd[776553] (pe_get_failcount) info: jangcluster-srv-4 has failed 1 times on jangcluster-srv-2<br>Oct 22 12:21:09.389 jangcluster-srv-2 pacemaker-schedulerd[776553] (pe_get_failcount) info: jangcluster-srv-4 has failed 1 times on jangcluster-srv-2<br>Oct 22 12:21:09.389 jangcluster-srv-2 pacemaker-schedulerd[776553] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for monitor of jangcluster-srv-4 on jangcluster-srv-2 at Oct 22 12:21:09 2021 | rc=1 id=jangcluster-srv-4_last_failure_0<br>Oct 22 12:21:09.389 jangcluster-srv-2 pacemaker-schedulerd[776553] (unpack_rsc_op_failure) notice: jangcluster-srv-4 will not be started under current conditions<br>Oct 22 12:21:09.389 jangcluster-srv-2 pacemaker-schedulerd[776553] (pe_fence_node) warning: Remote node jangcluster-srv-4 will be fenced: remote connection is unrecoverable</span><br><br>What we also found to be interesting is that if the cluster is only using a single heartbeat ring, then srv-2 will get fenced instead, and the pacemaker-remote connection resources will successfully fail over without any additional fencing to the remote nodes themselves. It seems a little backwards to us since our reasoning for configuring multiple heartbeat rings was to increase the clusters reliability/robustness of the cluster, but it seems to do the opposite when using pacemaker-remote. :(<br><br>Any suggestions/comments on our configuration / test scenario's are appreciated!<br> </div>
<div dir="ltr" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" ><span style="font-size:10pt;" ><span style="font-family:Arial,Helvetica,sans-serif;" ><span style="color:#000000;" >Gerry Sommerville</span></span></span></div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div><span style="font-weight: bold; color: #336699;" >E-mail: </span><a href="mailto:gerry@ca.ibm.com" style="color: #555" target="_blank">gerry@ca.ibm.com</a></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Andrei Borzenkov" <arvidjaar@gmail.com><br>Sent by: "Users" <users-bounces@clusterlabs.org><br>To: "Cluster Labs - All topics related to open-source clustering welcomed" <users@clusterlabs.org><br>Cc:<br>Subject: [EXTERNAL] Re: [ClusterLabs] Antw: [EXT] Inquiry - remote node fencing issue<br>Date: Thu, Oct 28, 2021 3:58 AM<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >On Thu, Oct 28, 2021 at 10:30 AM Ulrich Windl<br><Ulrich.Windl@rz.uni-regensburg.de> wrote:<br>><br>> Fencing _is_ a part of failover!<br>><br><br>As any blanket answer this is mostly incorrect in this context.<br><br>There are two separate objects here - remote host itself and pacemaker<br>resource used to connect to and monitor state of remote host.<br><br>Remote host itself does not failover. Resources on this host do, but<br>OP does not ask about it.<br><br>Pacemaker resource used to monitor remote host may failover as any<br>other cluster resource. This failover does not require any fencing *of<br>remote host itself*, and in this particular case connection between<br>two cluster nodes was present all the time (at least, as long as we<br>can believe logs) so there was no reason for fencing as well. Whether<br>pacemaker should attempt to failover this resource to another node if<br>connection to remote host fails, is subject to discussion.<br><br>So fencing of the remote host itself is most certainly *not* part of<br>the failover of the resource that monitors this remote host.<br><br>> >>> "Janghyuk Boo" <Janghyuk.Boo@ibm.com> schrieb am 26.10.2021 um 22:09 in<br>> Nachricht<br>> <OF6751AF09.DD2C657C-ON0025877A.006EA8CB-0025877A.006EB632@ibm.com>:<br>> Dear Community ,<br>> Thank you Ken for your reply last time.<br>> I attached the log messages as requested from the last thread.<br>> I have a Pacemaker cluster with two cluster nodes with two network interfaces<br>> each, and two remote nodes and a prototyped fencing agent(GPFS-Fence) to cut a<br>> hosts access from the clustered filesystem.<br>> I noticed that remote node gets fenced when the quorum node its connected to<br>> gets fenced or experiences network failure.<br>> For example, when I disconnected srv-2 from the rest of the cluster by using<br>> iptables on srv-2<br>> iptables -A INPUT -s [IP of srv-1] -j DROP ; iptables -A OUTPUT -s [IP of<br>> srv-1] -j DROP<br>> iptables -A INPUT -s [IP of srv-3] -j DROP ; iptables -A OUTPUT -s [IP of<br>> srv-3] -j DROP<br>> iptables -A INPUT -s [IP of srv-4] -j DROP ; iptables -A OUTPUT -s [IP of<br>> srv-4] -j DROP<br>> I expected that remote node jangcluster-srv-4 would failover to srv-1 given my<br>> location constraints,<br>> but remote node’s monitor ‘jangcluster-srv-4_monitor’ failed and srv-4 was<br>> getting fenced before attempting to failover.<br>> What would be the proper way to simulate the network failover?<br>> How can I configure the cluster so that remote node srv-4 fails over instead<br>> of getting fenced?<br>><br>><br>><br>> _______________________________________________<br>> Manage your subscription:<br>> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a> <br>><br>> ClusterLabs home: <a href="https://www.clusterlabs.org/" target="_blank">https://www.clusterlabs.org/</a> <br>_______________________________________________<br>Manage your subscription:<br><a href="https://lists.clusterlabs.org/mailman/listinfo/users" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a> <br><br>ClusterLabs home: <a href="https://www.clusterlabs.org/" target="_blank">https://www.clusterlabs.org/</a> </font></div></blockquote>
<div dir="ltr" > </div></div></div><BR>
<BR>