[ClusterLabs] Cluster unable to find back together

[Jan Friesse]

Hi,

On 19/05/2022 10:16, Leditzky, Fabian via Users wrote:
> Hello
> 
> We have been dealing with our pacemaker/corosync clusters becoming unstable.
> The OS is Debian 10 and we use Debian packages for pacemaker and corosync,
> version 3.0.1-5+deb10u1 and 3.0.1-2+deb10u1 respectively.

Seems like pcmk version is not so important for behavior you've 
described. Corosync 3.0.1 is super old, are you able to reproduce the 
behavior with 3.1.6? What is the version of knet? There were quite a few 
fixes so last one (1.23) is really recommended.

You can try to compile yourself, or use proxmox repo 
(http://download.proxmox.com/debian/pve/) which contains newer version 
of packages.

> We use knet over UDP transport.
> 
> We run multiple 2-node and 4-8 node clusters, primarily managing VIP resources.
> The issue we experience presents itself as a spontaneous disagreement of
> the status of cluster members. In two node clusters, each node spontaneously
> sees the other node as offline, despite network connectivity being OK.
> In larger clusters, the status can be inconsistent across the nodes.
> E.g.: node1 sees 2,4 as offline, node 2 sees 1,4 as offline while node 3 and 4 see every node as online.

This really shouldn't happen.

> The cluster becomes generally unresponsive to resource actions in this state.

Expected

> Thus far we have been unable to restore cluster health without restarting corosync.
> 
> We are running packet captures 24/7 on the clusters and have custom tooling
> to detect lost UDP packets on knet ports. So far we could not see significant
> packet loss trigger an event, at most we have seen a single UDP packet dropped
> some seconds before the cluster fails.
> 
> However, even if the root cause is indeed a flaky network, we do not understand
> why the cluster cannot recover on its own in any way. The issues definitely persist
> beyond the presence of any intermittent network problem.

Try newer version. If problem persist, it's good idea to monitor if 
packets are really passed thru. Corosync always (at least) creates 
single node membership.

Regards,
   Honza

> 
> We were able to artificially break clusters by inducing packet loss with an iptables rule.
> Dropping packets on a single node of an 8-node cluster can cause malfunctions on
> multiple other cluster nodes. The expected behavior would be detecting that the
> artificially broken node failed but keeping the rest of the cluster stable.
> We were able to reproduce this also on Debian 11 with more recent corosync/pacemaker
> versions.
> 
> Our configuration basic, we do not significantly deviate from the defaults.
> 
> We will be very grateful for any insights into this problem.
> 
> Thanks,
> Fabian
> 
> // corosync.conf
> totem {
>      version: 2
>      cluster_name: cluster01
>      crypto_cipher: aes256
>      crypto_hash: sha512
>      transport: knet
> }
> logging {
>      fileline: off
>      to_stderr: no
>      to_logfile: no
>      to_syslog: yes
>      debug: off
>      timestamp: on
>      logger_subsys {
>          subsys: QUORUM
>          debug: off
>      }
> }
> quorum {
>      provider: corosync_votequorum
>      two_node: 1
>      expected_votes: 2
> }
> nodelist {
>      node {
>          name: node01
>          nodeid: 01
>          ring0_addr: 10.0.0.10
>      }
>      node {
>          name: node02
>          nodeid: 02
>          ring0_addr: 10.0.0.11
>      }
> }
> 
> // crm config show
> node 1: node01 \
>      attributes standby=off
> node 2: node02 \
>      attributes standby=off maintenance=off
> primitive IP-clusterC1 IPaddr2 \
>      params ip=10.0.0.20 nic=eth0 cidr_netmask=24 \
>      meta migration-threshold=2 target-role=Started is-managed=true \
>      op monitor interval=20 timeout=60 on-fail=restart
> primitive IP-clusterC2 IPaddr2 \
>      params ip=10.0.0.21 nic=eth0 cidr_netmask=24 \
>      meta migration-threshold=2 target-role=Started is-managed=true \
>      op monitor interval=20 timeout=60 on-fail=restart
> location STICKY-IP-clusterC1 IP-clusterC1 100: node01
> location STICKY-IP-clusterC2 IP-clusterC2 100: node02
> property cib-bootstrap-options: \
>      have-watchdog=false \
>      dc-version=2.0.1-9e909a5bdd \
>      cluster-infrastructure=corosync \
>      cluster-name=cluster01 \
>      stonith-enabled=no \
>      no-quorum-policy=ignore \
>      last-lrm-refresh=1632230917
> 
> 
> ________________________________
>   [https://go.aciworldwide.com/rs/030-ROK-804/images/aci-footer.jpg] <http://www.aciworldwide.com>
> This email message and any attachments may contain confidential, proprietary or non-public information. The information is intended solely for the designated recipient(s). If an addressing or transmission error has misdirected this email, please notify the sender immediately and destroy this email. Any review, dissemination, use or reliance upon this information by unintended recipients is prohibited. Any opinions expressed in this email are those of the author personally.
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
>