<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Thanks for your advices. As I suspected, the zabbix-proxy service
has been started on the backup node today at 16h46<br>
</p>
<div class="moz-cite-prefix">Mar 22 16:46:52 zabbix-proxy-dc1-01
pacemaker-attrd [946] (pcmk_cpg_membership) info: Group
attrd event 95: zabbix-proxy-dc2-01 (node 2 pid 965) left via
cluster exit<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd [946]
(crm_update_peer_proc) info: pcmk_cpg_membership: Node
zabbix-proxy-dc2-01[2] - corosync-cpg is now offline<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-based [942]
(pcmk_cpg_membership) info: Group cib event 96:
zabbix-proxy-dc2-01 (node 2 pid 962) left via cluster exit<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd [946]
(attrd_remove_voter) notice: Lost attribute writer
zabbix-proxy-dc2-01<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-based [942]
(crm_update_peer_proc) info: pcmk_cpg_membership: Node
zabbix-proxy-dc2-01[2] - corosync-cpg is now offline<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd [946]
(attrd_start_election_if_needed) info: Starting an election to
determine the writer<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd [946]
(crm_cs_flush) info: Sent 0 CPG messages (1 remaining,
last=262): Try again (6)<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-based [942]
(crm_update_peer_state_iter) notice: Node zabbix-proxy-dc2-01
state is now lost | nodeid=2 previous=member
source=crm_update_peer_proc<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-based [942]
(reap_crm_member) notice: Purged 1 peer with id=2 and/or
uname=zabbix-proxy-dc2-01 from the membership cache<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-based [942]
(pcmk_cpg_membership) info: Group cib event 96:
zabbix-proxy-dc1-01 (node 1 pid 942) is member<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd [946]
(attrd_peer_remove) notice: Removing all zabbix-proxy-dc2-01
attributes for peer loss<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd [946]
(crm_reap_dead_member) info: Removing node with name
zabbix-proxy-dc2-01 and id 2 from membership cache<br>
Mar 22 16:46:52 zabbix-proxy-dc1-01 pacemaker-attrd [946]
(reap_crm_member) notice: Purged 1 peer with id=2 and/or
uname=zabbix-proxy-dc2-01 from the membership cache</div>
<div class="moz-cite-prefix">Mar 22 16:46:53 zabbix-proxy-dc1-01
pacemaker-based [942] (cib_process_readwrite) info: We are
now in R/W mode<br>
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based [942]
(cib_process_request) info: Completed cib_master operation for
section 'all': OK (rc=0, origin=local/crmd/971, version=0.70.0)<br>
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based [942]
(cib_process_request) info: Forwarding cib_modify operation
for section cib to all (origin=local/crmd/972)<br>
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based [942]
(cib_process_request) info: Completed cib_modify operation for
section cib: OK (rc=0, origin=zabbix-proxy-dc1-01/crmd/972,
version=0.70.0)<br>
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based [942]
(cib_process_request) info: Forwarding cib_modify operation
for section crm_config to all (origin=local/crmd/974)<br>
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based [942]
(cib_process_request) info: Completed cib_modify operation for
section crm_config: OK (rc=0, origin=zabbix-proxy-dc1-01/crmd/974,
version=0.70.0)<br>
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based [942]
(cib_process_request) info: Forwarding cib_modify operation
for section crm_config to all (origin=local/crmd/976)<br>
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-based [942]
(cib_process_request) info: Completed cib_modify operation for
section crm_config: OK (rc=0, origin=zabbix-proxy-dc1-01/crmd/976,
version=0.70.0)<br>
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-controld [948]
(crm_update_peer_join) info: initialize_join: Node
zabbix-proxy-dc1-01[1] - join-31 phase confirmed -> none<br>
Mar 22 16:46:53 zabbix-proxy-dc1-01 pacemaker-controld [948]
(join_make_offer) info: Not making an offer to
zabbix-proxy-dc2-01: not active (lost)<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">I think that quorum setup has something
wrong, I need some help to figure out:</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">pcs quorum status<br>
Quorum information<br>
------------------<br>
Date: Wed Mar 22 18:10:57 2023<br>
Quorum provider: corosync_votequorum<br>
Nodes: 2<br>
Node ID: 1<br>
Ring ID: 1/22376<br>
Quorate: Yes<br>
<br>
Votequorum information<br>
----------------------<br>
Expected votes: 3<br>
Highest expected: 3<br>
Total votes: 3<br>
Quorum: 2 <br>
Flags: Quorate Qdevice <br>
<br>
Membership information<br>
----------------------<br>
Nodeid Votes Qdevice Name<br>
1 1 A,V,NMW zabbix-proxy-dc1-01 (local)<br>
2 1 A,V,NMW zabbix-proxy-dc2-01<br>
0 1 Qdevice<br>
<br>
</div>
<div class="moz-cite-prefix">It looks okay to me, but you said if
the node is isolated it should stop everything, and this is not
happening.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Thank you<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Le 21/03/2023 à 12:01, Jehan-Guillaume
de Rorthais a écrit :<br>
</div>
<blockquote type="cite" cite="mid:20230321120148.6301f9e4@karst">
<pre class="moz-quote-pre" wrap="">On Tue, 21 Mar 2023 11:47:23 +0100
Jérôme BECOT <a class="moz-txt-link-rfc2396E" href="mailto:jerome.becot@deveryware.com"><jerome.becot@deveryware.com></a> wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Le 21/03/2023 à 11:00, Jehan-Guillaume de Rorthais a écrit :
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hi,
On Tue, 21 Mar 2023 09:33:04 +0100
Jérôme BECOT<a class="moz-txt-link-rfc2396E" href="mailto:jerome.becot@deveryware.com"><jerome.becot@deveryware.com></a> wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">We have several clusters running for different zabbix components. Some
of these clusters consist of 2 zabbix proxies,where nodes run Mysql,
Zabbix-proxy server and a VIP, and a corosync-qdevice.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">I'm not sure to understand your topology. The corosync-device is not
supposed to be on a cluster node. It is supposed to be on a remote node and
provide some quorum features to one or more cluster without setting up the
whole pacemaker/corosync stack.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">I was not clear, the qdevice is deployed on a remote node, as intended.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
ok
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">The MySQL servers are always up to replicate, and are configured in
Master/Master (they both replicate from the other but only one is supposed
to be updated by the proxy running on the master node).
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">Why do you bother with Master/Master when a simple (I suppose, I'm not a
MySQL cluster guy) Primary-Secondary topology or even a shared storage
would be enough and would keep your logic (writes on one node only) safe
from incidents, failures, errors, etc?
HA must be a simple as possible. Remove useless parts when you can.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">A shared storage moves the complexity somewhere else.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
Yes, on storage/SAN side.
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">A classic Primary / secondary can be an option if PaceMaker manages to start
the client on the slave node,
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
I suppose this can be done using a location constraint.
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">but it would become Master/Master during the split brain.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
No, and if you do have real split brain, then you might have something wrong in
your setup. See below.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">One cluster is prompt to frequent sync errors, with duplicate entries
errors in SQL. When I look at the logs, I can see "Mar 21 09:11:41
zabbix-proxy-01 pacemaker-controld [948] (pcmk_cpg_membership)
info: Group crmd event 89: zabbix-proxy-02 (node 2 pid 967) left via
cluster exit", and within the next second, a rejoin. The same messages
are in the other node logs, suggesting a split brain, which should not
happen, because there is a quorum device.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">Would it be possible your SQL sync errors and the left/join issues are
correlated and are both symptoms of another failure? Look at your log for
some explanation about why the node decided to leave the cluster.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
My guess is that maybe a high latency in network cause the disjoin,
hence starting Zabbix-proxy on both nodes causes the replication error.
It is configured to use the vip which is up locally because there is a
split brain.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
If you have a split brain, that means your quorum setup is failing.
No node could start/promote a resource without having the quorum. If a node is
isolated from the cluster and quorum-device, it should stop its resources, not
recover/promote them.
If both nodes lost connection with each others, but are still connected to the
quorum-device, the later should be able to grant the quorum on one side only.
Lastly, quorum is a split brain protection when "things are going fine".
Fencing is a split brain protection for all other situations. Fencing is hard
and painful, but it saves from many split brain situation.
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">This is why I'm requesting guidance to check/monitor these nodes to find
out if it is temporary network latency that is causing the disjoin.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
A cluster is always very sensitive to network latency/failures. You need to
build on stronger fondations.
Regards,
</pre>
</blockquote>
<div class="moz-signature">-- <br>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title></title>
<div class="moz-signature"><b><span style="color:#002060">Jérôme
BECOT</span></b> <span style="color:#002060"></span><br>
<span style="color:#002060">Ingénieur DevOps Infrastructure </span><br>
<br>
<span style="color:#002060">Téléphone fixe: </span> <span
style="color:#002060;mso-fareast-language:FR">01 82 28 37 06</span><br>
<span style="color:#002060">Mobile : +33 757 173 193</span><br>
<span style="color:#002060">Deveryware - 43 rue Taitbout - 75009
PARIS</span><br>
<a moz-do-not-send="true" href="https://www.deveryware.com"> <span
style="color:#002060"><span tyle="color:#002060">
https://www.deveryware.com</span></span></a></div>
<div class="moz-signature"> <span
style="color:#002060;mso-fareast-language:FR"></span><br>
<img moz-do-not-send="false"
src="cid:part1.elLrj70f.4UOn1hKn@deveryware.com"
alt="Deveryware_Logo" width="402" height="107"><br>
<a href="https://www.deveryware.com"> <span
style="font-size:10.0pt;color:#08638F;mso-fareast-language:FR;text-decoration:none"></span></a></div>
</div>
</body>
</html>