[ClusterLabs] Cluster unable to find back together

[Leditzky, Fabian]

Hello

We have been dealing with our pacemaker/corosync clusters becoming unstable.
The OS is Debian 10 and we use Debian packages for pacemaker and corosync,
version 3.0.1-5+deb10u1 and 3.0.1-2+deb10u1 respectively.
We use knet over UDP transport.

We run multiple 2-node and 4-8 node clusters, primarily managing VIP resources.
The issue we experience presents itself as a spontaneous disagreement of
the status of cluster members. In two node clusters, each node spontaneously
sees the other node as offline, despite network connectivity being OK.
In larger clusters, the status can be inconsistent across the nodes.
E.g.: node1 sees 2,4 as offline, node 2 sees 1,4 as offline while node 3 and 4 see every node as online.
The cluster becomes generally unresponsive to resource actions in this state.
Thus far we have been unable to restore cluster health without restarting corosync.

We are running packet captures 24/7 on the clusters and have custom tooling
to detect lost UDP packets on knet ports. So far we could not see significant
packet loss trigger an event, at most we have seen a single UDP packet dropped
some seconds before the cluster fails.

However, even if the root cause is indeed a flaky network, we do not understand
why the cluster cannot recover on its own in any way. The issues definitely persist
beyond the presence of any intermittent network problem.

We were able to artificially break clusters by inducing packet loss with an iptables rule.
Dropping packets on a single node of an 8-node cluster can cause malfunctions on
multiple other cluster nodes. The expected behavior would be detecting that the
artificially broken node failed but keeping the rest of the cluster stable.
We were able to reproduce this also on Debian 11 with more recent corosync/pacemaker
versions.

Our configuration basic, we do not significantly deviate from the defaults.

We will be very grateful for any insights into this problem.

Thanks,
Fabian

// corosync.conf
totem {
    version: 2
    cluster_name: cluster01
    crypto_cipher: aes256
    crypto_hash: sha512
    transport: knet
}
logging {
    fileline: off
    to_stderr: no
    to_logfile: no
    to_syslog: yes
    debug: off
    timestamp: on
    logger_subsys {
        subsys: QUORUM
        debug: off
    }
}
quorum {
    provider: corosync_votequorum
    two_node: 1
    expected_votes: 2
}
nodelist {
    node {
        name: node01
        nodeid: 01
        ring0_addr: 10.0.0.10
    }
    node {
        name: node02
        nodeid: 02
        ring0_addr: 10.0.0.11
    }
}

// crm config show
node 1: node01 \
    attributes standby=off
node 2: node02 \
    attributes standby=off maintenance=off
primitive IP-clusterC1 IPaddr2 \
    params ip=10.0.0.20 nic=eth0 cidr_netmask=24 \
    meta migration-threshold=2 target-role=Started is-managed=true \
    op monitor interval=20 timeout=60 on-fail=restart
primitive IP-clusterC2 IPaddr2 \
    params ip=10.0.0.21 nic=eth0 cidr_netmask=24 \
    meta migration-threshold=2 target-role=Started is-managed=true \
    op monitor interval=20 timeout=60 on-fail=restart
location STICKY-IP-clusterC1 IP-clusterC1 100: node01
location STICKY-IP-clusterC2 IP-clusterC2 100: node02
property cib-bootstrap-options: \
    have-watchdog=false \
    dc-version=2.0.1-9e909a5bdd \
    cluster-infrastructure=corosync \
    cluster-name=cluster01 \
    stonith-enabled=no \
    no-quorum-policy=ignore \
    last-lrm-refresh=1632230917


________________________________
 [https://go.aciworldwide.com/rs/030-ROK-804/images/aci-footer.jpg] <http://www.aciworldwide.com>
This email message and any attachments may contain confidential, proprietary or non-public information. The information is intended solely for the designated recipient(s). If an addressing or transmission error has misdirected this email, please notify the sender immediately and destroy this email. Any review, dissemination, use or reliance upon this information by unintended recipients is prohibited. Any opinions expressed in this email are those of the author personally.