<html><head><meta http-equiv="content-type" content="text/html; charset=us-ascii"><style>body { line-height: 1.5; }body { font-size: 14px; font-family: "Microsoft YaHei UI"; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>
<div><span></span><br></div><div><div>Background:</div><div>There are 4 physical machines, with two virtual machines running on each physical machine. lustre-mds-nodexx runs the Lustre MDS server, and lustre-oss-nodexx runs the Lustre OSS service. Each virtual machine is directly connected to two network interfaces, service1(<span style="background-color: transparent;">ens6f0np0</span><span style="background-color: transparent;">) and service2(</span><span style="background-color: transparent;">ens6f1np1</span><span style="background-color: transparent;">). Pacemaker is used to ensure high availability of the Lustre services.</span></div><div>Software versions:</div><div>Lustre: 2.15.5</div><div>Corosync: 3.1.5</div><div>Pacemaker: 2.1.0-8.el8</div><div>PCS: 0.10.8</div><div><br></div><div><br></div><div>Operation:</div><div>During testing, the network interfaces service1 and service2 on lustre-oss-node40 and lustre-mds-node40 were repeatedly brought up and down every 20 seconds (to simulate a network failure).</div><div><br></div><div>for i in {1..10}; do date; ifconfig ens6f0np0 down && <span style="background-color: transparent;">ifconfig ens6f1np1 down; sleep 20; date; </span><span style="background-color: transparent;">ifconfig ens6f0np0 up && </span><span style="background-color: transparent;">ifconfig ens6f1np1 up; date;sleep 30</span></div><div><br></div><div><br></div><div>Issue:</div><div>Theoretically, lustre-oss-node40 and lustre-mds-node40 should have been fenced, but lustre-mds-node32 was fenced instead.</div><div><br></div><div><br></div><div>Related Logs:</div><div>Jun 09 17:54:51 node32 fence_virtd[2502]: Destroying domain 60e80c07-107e-4e8a-ba42-39e48b3e6bb7 // This log indicates that lustre-mds-node32 was fenced.</div><div><br></div><div><br></div><div>* turning off of lustre-mds-node32 successful: delegate=lustre-mds-node42, client=pacemaker-controld.8918, origin=lustre-mds-node42, completed='2025-06-09 17:54:54.527116 +08:00'</div><div><br></div><div><br></div><div><br></div><div>Jun 09 17:54:10 [1429] lustre-mds-node32 corosync info [KNET ] link: Resetting MTU for link 0 because host 1 joined</div><div>Jun 09 17:54:10 [1429] lustre-mds-node32 corosync info [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)</div><div>Jun 09 17:54:10 [1429] lustre-mds-node32 corosync info [KNET ] pmtud: Global data MTU changed to: 1397</div><div>Jun 09 17:54:31 [1429] lustre-mds-node32 corosync info [KNET ] link: host: 1 link: 0 is down</div><div>Jun 09 17:54:31 [1429] lustre-mds-node32 corosync info [KNET ] host: host: 1 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:34 [1429] lustre-mds-node32 corosync info [KNET ] link: host: 1 link: 1 is down</div><div>Jun 09 17:54:34 [1429] lustre-mds-node32 corosync info [KNET ] host: host: 1 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:34 [1429] lustre-mds-node32 corosync warning [KNET ] host: host: 1 has no active links</div><div>Jun 09 17:54:36 [1429] lustre-mds-node32 corosync notice [TOTEM ] Token has not been received in 8475 ms</div><div>Jun 09 17:57:44 [1419] lustre-mds-node32 corosync notice [MAIN ] Corosync Cluster Engine 3.1.8 starting up</div><div><br></div><div><br></div><div>Jun 09 17:54:31 [1412] lustre-mds-node40 corosync info [KNET ] link: host: 4 link: 0 is down</div><div>Jun 09 17:54:31 [1412] lustre-mds-node40 corosync info [KNET ] link: host: 3 link: 0 is down</div><div>Jun 09 17:54:31 [1412] lustre-mds-node40 corosync info [KNET ] link: host: 2 link: 0 is down</div><div>Jun 09 17:54:31 [1412] lustre-mds-node40 corosync info [KNET ] host: host: 4 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:31 [1412] lustre-mds-node40 corosync info [KNET ] host: host: 3 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:31 [1412] lustre-mds-node40 corosync info [KNET ] host: host: 2 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:34 [1412] lustre-mds-node40 corosync notice [TOTEM ] Token has not been received in 8475 ms</div><div>Jun 09 17:54:34 [1412] lustre-mds-node40 corosync info [KNET ] link: host: 4 link: 1 is down</div><div>Jun 09 17:54:34 [1412] lustre-mds-node40 corosync info [KNET ] link: host: 3 link: 1 is down</div><div>Jun 09 17:54:34 [1412] lustre-mds-node40 corosync info [KNET ] link: host: 2 link: 1 is down</div><div>Jun 09 17:54:34 [1412] lustre-mds-node40 corosync info [KNET ] host: host: 4 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:34 [1412] lustre-mds-node40 corosync warning [KNET ] host: host: 4 has no active links</div><div>Jun 09 17:54:34 [1412] lustre-mds-node40 corosync info [KNET ] host: host: 3 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:34 [1412] lustre-mds-node40 corosync warning [KNET ] host: host: 3 has no active links</div><div>Jun 09 17:54:34 [1412] lustre-mds-node40 corosync info [KNET ] host: host: 2 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:34 [1412] lustre-mds-node40 corosync warning [KNET ] host: host: 2 has no active links</div><div>Jun 09 17:54:37 [1412] lustre-mds-node40 corosync notice [TOTEM ] A processor failed, forming new configuration: token timed out (11300ms), waiting 13560ms for consensus.</div><div>Jun 09 17:54:46 [1412] lustre-mds-node40 corosync info [KNET ] link: Resetting MTU for link 1 because host 3 joined</div><div>Jun 09 17:54:46 [1412] lustre-mds-node40 corosync info [KNET ] host: host: 3 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:46 [1412] lustre-mds-node40 corosync info [KNET ] pmtud: Global data MTU changed to: 1397</div><div>Jun 09 17:54:47 [1412] lustre-mds-node40 corosync info [KNET ] link: Resetting MTU for link 1 because host 2 joined</div><div>Jun 09 17:54:47 [1412] lustre-mds-node40 corosync info [KNET ] host: host: 2 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:47 [1412] lustre-mds-node40 corosync info [KNET ] pmtud: Global data MTU changed to: 1397</div><div>Jun 09 17:54:50 [1412] lustre-mds-node40 corosync notice [QUORUM] Sync members[3]: 1 2 3</div><div>Jun 09 17:54:50 [1412] lustre-mds-node40 corosync notice [QUORUM] Sync left[1]: 4</div><div>Jun 09 17:54:50 [1412] lustre-mds-node40 corosync notice [TOTEM ] A new membership (1.45) was formed. Members left: 4</div><div>Jun 09 17:54:50 [1412] lustre-mds-node40 corosync notice [TOTEM ] Failed to receive the leave message. failed: 4</div><div><br></div><div><br></div><div>Jun 09 17:54:29 [8913] lustre-mds-node41 corosync info [KNET ] link: host: 1 link: 0 is down</div><div>Jun 09 17:54:29 [8913] lustre-mds-node41 corosync info [KNET ] host: host: 1 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:30 [8913] lustre-mds-node41 corosync info [KNET ] link: host: 1 link: 1 is down</div><div>Jun 09 17:54:30 [8913] lustre-mds-node41 corosync info [KNET ] host: host: 1 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:30 [8913] lustre-mds-node41 corosync warning [KNET ] host: host: 1 has no active links</div><div>Jun 09 17:54:36 [8913] lustre-mds-node41 corosync notice [TOTEM ] Token has not been received in 8475 ms</div><div>Jun 09 17:54:39 [8913] lustre-mds-node41 corosync notice [TOTEM ] A processor failed, forming new configuration: token timed out (11300ms), waiting 13560ms for consensus.</div><div>Jun 09 17:54:47 [8913] lustre-mds-node41 corosync info [KNET ] rx: host: 1 link: 1 is up</div><div>Jun 09 17:54:47 [8913] lustre-mds-node41 corosync info [KNET ] link: Resetting MTU for link 1 because host 1 joined</div><div>Jun 09 17:54:47 [8913] lustre-mds-node41 corosync info [KNET ] host: host: 1 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:47 [8913] lustre-mds-node41 corosync info [KNET ] pmtud: Global data MTU changed to: 1397</div><div>Jun 09 17:54:50 [8913] lustre-mds-node41 corosync notice [QUORUM] Sync members[3]: 1 2 3</div><div>Jun 09 17:54:50 [8913] lustre-mds-node41 corosync notice [QUORUM] Sync left[1]: 4</div><div>Jun 09 17:54:50 [8913] lustre-mds-node41 corosync notice [TOTEM ] A new membership (1.45) was formed. Members left: 4</div><div>Jun 09 17:54:50 [8913] lustre-mds-node41 corosync notice [TOTEM ] Failed to receive the leave message. failed: 4</div><div><br></div><div><br></div><div>Jun 09 17:54:28 [8900] lustre-mds-node42 corosync info [KNET ] link: host: 1 link: 0 is down</div><div>Jun 09 17:54:28 [8900] lustre-mds-node42 corosync info [KNET ] host: host: 1 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:30 [8900] lustre-mds-node42 corosync info [KNET ] link: host: 1 link: 1 is down</div><div>Jun 09 17:54:30 [8900] lustre-mds-node42 corosync info [KNET ] host: host: 1 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:30 [8900] lustre-mds-node42 corosync warning [KNET ] host: host: 1 has no active links</div><div>Jun 09 17:54:36 [8900] lustre-mds-node42 corosync notice [TOTEM ] Token has not been received in 8475 ms</div><div>Jun 09 17:54:45 [8900] lustre-mds-node42 corosync info [KNET ] rx: host: 1 link: 1 is up</div><div>Jun 09 17:54:45 [8900] lustre-mds-node42 corosync info [KNET ] link: Resetting MTU for link 1 because host 1 joined</div><div>Jun 09 17:54:45 [8900] lustre-mds-node42 corosync info [KNET ] host: host: 1 (passive) best link: 1 (pri: 1)</div><div>Jun 09 17:54:45 [8900] lustre-mds-node42 corosync info [KNET ] pmtud: Global data MTU changed to: 1397</div><div>Jun 09 17:54:50 [8900] lustre-mds-node42 corosync notice [QUORUM] Sync members[3]: 1 2 3</div><div>Jun 09 17:54:50 [8900] lustre-mds-node42 corosync notice [QUORUM] Sync left[1]: 4</div><div>Jun 09 17:54:50 [8900] lustre-mds-node42 corosync notice [TOTEM ] A new membership (1.45) was formed. Members left: 4</div><div>Jun 09 17:54:50 [8900] lustre-mds-node42 corosync notice [TOTEM ] Failed to receive the leave message. failed: 4</div><div><br></div><div><br></div><div><br></div><div><br></div><div>/etc/corosync/corosync.conf</div><div>totem {</div><div> version: 2</div><div> cluster_name: mds_cluster</div><div> transport: knet</div><div> crypto_cipher: aes256</div><div> crypto_hash: sha256</div><div> cluster_uuid: 11f2c4097ac44d5981769a9ed579c99e</div><div> token: 10000</div><div>}</div><div><br></div><div>nodelist {</div><div> node {</div><div> ring0_addr: 10.255.153.240</div><div> ring1_addr: 10.255.153.241</div><div> name: lustre-mds-node40</div><div> nodeid: 1</div><div> }</div><div><br></div><div> node {</div><div> ring0_addr: 10.255.153.244</div><div> ring1_addr: 10.255.153.245</div><div> name: lustre-mds-node41</div><div> nodeid: 2</div><div> }</div><div><br></div><div> node {</div><div> ring0_addr: 10.255.153.248</div><div> ring1_addr: 10.255.153.249</div><div> name: lustre-mds-node42</div><div> nodeid: 3</div><div> }</div><div><br></div><div> node {</div><div> ring0_addr: 10.255.153.236</div><div> ring1_addr: 10.255.153.237</div><div> name: lustre-mds-node32</div><div> nodeid: 4</div><div> }</div><div>}</div><div><br></div><div>quorum {</div><div> provider: corosync_votequorum</div><div>}</div><div><br></div><div>logging {</div><div> to_logfile: yes</div><div> logfile: /var/log/cluster/corosync.log</div><div> to_syslog: yes</div><div> timestamp: on</div><div>}</div></div><div><br></div><div><br></div><div><br></div>
<div><br></div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><div>chenzufei@gmail.com</div></div></span></div>
</body></html>