[ClusterLabs] Two node cluster goes into split brain scenario during CPU intensive tasks

[Somanath Jeeva]

Hi All,

I have a two node cluster with multicast (udp) transport . The multicast IP used in 224.1.1.1 .

Whenever there is a CPU intensive task the pcs cluster goes into split brain scenario and doesn't recover automatically . We have to do a manual restart of services to bring both nodes online again. Before the nodes goes into split brain , the corosync log shows ,

May 24 15:10:02 server1 corosync[4745]:  [TOTEM ] Retransmit List: 7c 7e
May 24 15:10:02 server1 corosync[4745]:  [TOTEM ] Retransmit List: 7c 7e
May 24 15:10:02 server1 corosync[4745]:  [TOTEM ] Retransmit List: 7c 7e
May 24 15:10:02 server1 corosync[4745]:  [TOTEM ] Retransmit List: 7c 7e
May 24 15:10:02 server1 corosync[4745]:  [TOTEM ] Retransmit List: 7c 7e
May 24 15:51:42 server1 corosync[4745]:  [TOTEM ] A processor failed, forming new configuration.
May 24 16:41:42 server1 corosync[4745]:  [TOTEM ] A new membership (10.241.31.12:29276) was formed. Members left: 1
May 24 16:41:42 server1 corosync[4745]:  [TOTEM ] Failed to receive the leave message. failed: 1

Is there any way we can overcome this or this may be due to any multicast issues in the network side.

With Regards
Somanath Thilak J




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20190623/1808eb2c/attachment.html>