[ClusterLabs] [PaceMaker] Help troubleshooting frequent disjoin issue

Jérôme BECOT jerome.becot at deveryware.com
Wed Mar 22 13:17:41 EDT 2023


Thanks for your advices. As I suspected, the zabbix-proxy service has 
been started on the backup node today at 16h46

Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd     [946] 
(pcmk_cpg_membership)     info: Group attrd event 95: 
zabbix-proxy-dc2-01 (node 2 pid 965) left via cluster exit
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd     [946] 
(crm_update_peer_proc)    info: pcmk_cpg_membership: Node 
zabbix-proxy-dc2-01[2] - corosync-cpg is now offline
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(pcmk_cpg_membership)     info: Group cib event 96: zabbix-proxy-dc2-01 
(node 2 pid 962) left via cluster exit
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd     [946] 
(attrd_remove_voter)      notice: Lost attribute writer zabbix-proxy-dc2-01
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(crm_update_peer_proc)    info: pcmk_cpg_membership: Node 
zabbix-proxy-dc2-01[2] - corosync-cpg is now offline
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd     [946] 
(attrd_start_election_if_needed)  info: Starting an election to 
determine the writer
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd     [946] 
(crm_cs_flush)    info: Sent 0 CPG messages  (1 remaining, last=262): 
Try again (6)
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(crm_update_peer_state_iter)      notice: Node zabbix-proxy-dc2-01 state 
is now lost | nodeid=2 previous=member source=crm_update_peer_proc
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(reap_crm_member)         notice: Purged 1 peer with id=2 and/or 
uname=zabbix-proxy-dc2-01 from the membership cache
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(pcmk_cpg_membership)     info: Group cib event 96: zabbix-proxy-dc1-01 
(node 1 pid 942) is member
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd     [946] 
(attrd_peer_remove)       notice: Removing all zabbix-proxy-dc2-01 
attributes for peer loss
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd     [946] 
(crm_reap_dead_member)    info: Removing node with name 
zabbix-proxy-dc2-01 and id 2 from membership cache
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd     [946] 
(reap_crm_member)         notice: Purged 1 peer with id=2 and/or 
uname=zabbix-proxy-dc2-01 from the membership cache
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(cib_process_readwrite)   info: We are now in R/W mode
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(cib_process_request)     info: Completed cib_master operation for 
section 'all': OK (rc=0, origin=local/crmd/971, version=0.70.0)
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(cib_process_request)     info: Forwarding cib_modify operation for 
section cib to all (origin=local/crmd/972)
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(cib_process_request)     info: Completed cib_modify operation for 
section cib: OK (rc=0, origin=zabbix-proxy-dc1-01/crmd/972, version=0.70.0)
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(cib_process_request)     info: Forwarding cib_modify operation for 
section crm_config to all (origin=local/crmd/974)
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(cib_process_request)     info: Completed cib_modify operation for 
section crm_config: OK (rc=0, origin=zabbix-proxy-dc1-01/crmd/974, 
version=0.70.0)
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(cib_process_request)     info: Forwarding cib_modify operation for 
section crm_config to all (origin=local/crmd/976)
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based     [942] 
(cib_process_request)     info: Completed cib_modify operation for 
section crm_config: OK (rc=0, origin=zabbix-proxy-dc1-01/crmd/976, 
version=0.70.0)
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-controld  [948] 
(crm_update_peer_join)    info: initialize_join: Node 
zabbix-proxy-dc1-01[1] - join-31 phase confirmed -> none
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-controld  [948] 
(join_make_offer)         info: Not making an offer to 
zabbix-proxy-dc2-01: not active (lost)

I think that quorum setup has something wrong, I need some help to 
figure out:

pcs quorum status
Quorum information
------------------
Date:             Wed Mar 22 18:10:57 2023
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          1
Ring ID:          1/22376
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
     Nodeid      Votes    Qdevice Name
          1          1    A,V,NMW zabbix-proxy-dc1-01 (local)
          2          1    A,V,NMW zabbix-proxy-dc2-01
          0          1            Qdevice

It looks okay to me, but you said if the node is isolated it should stop 
everything, and this is not happening.

Thank you

Le 21/03/2023 à 12:01, Jehan-Guillaume de Rorthais a écrit :
> On Tue, 21 Mar 2023 11:47:23 +0100
> Jérôme BECOT<jerome.becot at deveryware.com>  wrote:
>
>> Le 21/03/2023 à 11:00, Jehan-Guillaume de Rorthais a écrit :
>>> Hi,
>>>
>>> On Tue, 21 Mar 2023 09:33:04 +0100
>>> Jérôme BECOT<jerome.becot at deveryware.com>   wrote:
>>>   
>>>> We have several clusters running for different zabbix components. Some
>>>> of these clusters consist of 2 zabbix proxies,where nodes run Mysql,
>>>> Zabbix-proxy server and a VIP, and a corosync-qdevice.
>>> I'm not sure to understand your topology. The corosync-device is not
>>> supposed to be on a cluster node. It is supposed to be on a remote node and
>>> provide some quorum features to one or more cluster without setting up the
>>> whole pacemaker/corosync stack.
>> I was not clear, the qdevice is deployed on a remote node, as intended.
> ok
>
>>>> The MySQL servers are always up to replicate, and are configured in
>>>> Master/Master (they both replicate from the other but only one is supposed
>>>> to be updated by the proxy running on the master node).
>>> Why do you bother with Master/Master when a simple (I suppose, I'm not a
>>> MySQL cluster guy) Primary-Secondary topology or even a shared storage
>>> would be enough and would keep your logic (writes on one node only) safe
>>> from incidents, failures, errors, etc?
>>>
>>> HA must be a simple as possible. Remove useless parts when you can.
>> A shared storage moves the complexity somewhere else.
> Yes, on storage/SAN side.
>
>> A classic Primary / secondary can be an option if PaceMaker manages to start
>> the client on the slave node,
> I suppose this can be done using a location constraint.
>
>> but it would become Master/Master during the split brain.
> No, and if you do have real split brain, then you might have something wrong in
> your setup. See below.
>
>
>>>> One cluster is prompt to frequent sync errors, with duplicate entries
>>>> errors in SQL. When I look at the logs, I can see "Mar 21 09:11:41
>>>> zabbix-proxy-01 pacemaker-controld  [948] (pcmk_cpg_membership)
>>>> info: Group crmd event 89: zabbix-proxy-02 (node 2 pid 967) left via
>>>> cluster exit", and within the next second, a rejoin. The same messages
>>>> are in the other node logs, suggesting a split brain, which should not
>>>> happen, because there is a quorum device.
>>> Would it be possible your SQL sync errors and the left/join issues are
>>> correlated and are both symptoms of another failure? Look at your log for
>>> some explanation about why the node decided to leave the cluster.
>> My guess is that maybe a high latency in network cause the disjoin,
>> hence starting Zabbix-proxy on both nodes causes the replication error.
>> It is configured to use the vip which is up locally because there is a
>> split brain.
> If you have a split brain, that means your quorum setup is failing.
>
> No node could start/promote a resource without having the quorum. If a node is
> isolated from the cluster and quorum-device, it should stop its resources, not
> recover/promote them.
>
> If both nodes lost connection with each others, but are still connected to the
> quorum-device, the later should be able to grant the quorum on one side only.
>
> Lastly, quorum is a split brain protection when "things are going fine".
> Fencing is a split brain protection for all other situations. Fencing is hard
> and painful, but it saves from many split brain situation.
>
>> This is why I'm requesting guidance to check/monitor these nodes to find
>> out if it is temporary network latency that is causing the disjoin.
> A cluster is always very sensitive to network latency/failures. You need to
> build on stronger fondations.
>
> Regards,
-- 
*Jérôme BECOT*
Ingénieur DevOps Infrastructure

Téléphone fixe: 01 82 28 37 06
Mobile : +33 757 173 193
Deveryware - 43 rue Taitbout - 75009 PARIS
https://www.deveryware.com <https://www.deveryware.com>

Deveryware_Logo
<https://www.deveryware.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20230322/7b2e3635/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: baniere_signature_dw_2022.png
Type: image/png
Size: 471995 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20230322/7b2e3635/attachment-0001.png>


More information about the Users mailing list