<div dir="ltr">Hello Jan.<div><br></div><div>Thank you very much for your help, It was definitely related to the way I was blocking the packets.</div><div><br>Since I'm running these tests with VMs, I just paused node1 and then node2 got the IP after a few seconds.</div><div><br></div><div>About the issues in the documentation: I'll recreate the scenario from scratch to replicate the steps (since I want to automate them with Ansible), and I'll use this opportunity to check exactly what is missing and report it.</div><div><br></div><div>Have a great weekend.</div><div><br></div><div>Regards,</div><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Marcelo H. Terres <<a href="mailto:mhterres@gmail.com" target="_blank">mhterres@gmail.com</a>><br><a href="https://www.mundoopensource.com.br" target="_blank">https://www.mundoopensource.com.br</a><br><a href="https://twitter.com/mhterres" target="_blank">https://twitter.com/mhterres</a><br><a href="https://linkedin.com/in/marceloterres" target="_blank">https://linkedin.com/in/marceloterres</a></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 19 Mar 2021 at 08:25, Jan Friesse <<a href="mailto:jfriesse@redhat.com">jfriesse@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Marcelo,<br>
<br>
> Hello.<br>
> <br>
> I have configured corosync with 2 nodes and added a qdevice to help with<br>
> the quorum.<br>
> <br>
> On node1 I added firewall rules to block connections from node2 and the<br>
> qdevice, trying to simulate a network issue.<br>
<br>
Just please make sure to block both incoming and also outgoing packets. <br>
Qdevice will handle blocking of just one direction well (because of tcp) <br>
and corosync 3.x with knet too. But corosync 2.x has a big problem with <br>
"asymmetric" blocking. Also config suggest that multicast is used - <br>
please make sure to block also multicast in that case.<br>
<br>
> <br>
> The problem I'm having is that one node1 I can see it dropping the<br>
> service (the IP), but on node2 it never gets the IP, it is like the qdevice<br>
> is not voting.<br>
> <br>
> This is my corosync.conf:<br>
> <br>
> totem {<br>
> version: 2<br>
> cluster_name: cluster1<br>
> token: 3000<br>
> token_retransmits_before_loss_const: 10<br>
> clear_node_high_bit: yes<br>
> crypto_cipher: none<br>
> crypto_hash: none<br>
> }<br>
> interface {<br>
> ringnumber: 0<br>
> bindnetaddr: X.X.X.X<br>
> mcastaddr: 239.255.43.2<br>
> mcastport: 5405<br>
> ttl: 1<br>
> }<br>
> nodelist{<br>
> node {<br>
> ring0_addr: X.X.X.2<br>
> name: <a href="http://node1.domain.com" rel="noreferrer" target="_blank">node1.domain.com</a><br>
> nodeid: 2<br>
> }<br>
> node {<br>
> ring0_addr: X.X.X.3<br>
> name: <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a><br>
> nodeid: 3<br>
> }<br>
> }<br>
> <br>
> logging {<br>
> to_logfile: yes<br>
> logfile: /var/log/cluster/corosync.log<br>
> to_syslog: yes<br>
> }<br>
> <br>
> #}<br>
> <br>
> quorum {<br>
> provider: corosync_votequorum<br>
> device {<br>
> votes: 1<br>
> model: net<br>
> net {<br>
> tls: off<br>
> host: <a href="http://qdevice.domain.com" rel="noreferrer" target="_blank">qdevice.domain.com</a><br>
> algorithm: lms<br>
> }<br>
> heuristics {<br>
> mode: on<br>
> exec_ping: /usr/bin/ping -q -c 1 "<a href="http://qdevice.domain.com" rel="noreferrer" target="_blank">qdevice.domain.com</a>"<br>
> }<br>
> }<br>
> }<br>
> <br>
> <br>
> I'm getting this on the qdevice host (before adding the firewall rules), so<br>
> looks like the cluster is properly configured:<br>
> <br>
> pcs qdevice status net --full<br>
<br>
<br>
Correct. What is the status after blocking is enabled?<br>
<br>
> QNetd address: *:5403<br>
> TLS: Supported (client certificate required)<br>
> Connected clients: 2<br>
> Connected clusters: 1<br>
> Maximum send/receive size: 32768/32768 bytes<br>
> Cluster "cluster1":<br>
> Algorithm: LMS<br>
> Tie-breaker: Node with lowest node ID<br>
> Node ID 3:<br>
> Client address: ::ffff:X.X.X.3:59746<br>
> HB interval: 8000ms<br>
> Configured node list: 2, 3<br>
> Ring ID: 2.95d<br>
> Membership node list: 2, 3<br>
> Heuristics: Pass (membership: Pass, regular: Undefined)<br>
> TLS active: No<br>
> Vote: ACK (ACK)<br>
> Node ID 2:<br>
> Client address: ::ffff:X.X.X.2:33944<br>
> HB interval: 8000ms<br>
> Configured node list: 2, 3<br>
> Ring ID: 2.95d<br>
> Membership node list: 2, 3<br>
> Heuristics: Pass (membership: Pass, regular: Undefined)<br>
> TLS active: No<br>
> Vote: ACK (ACK)<br>
> <br>
> These are partial logs on node2 after activating the firewall rules on<br>
> node1. These logs repeats all the time until I remove the firewall rules:<br>
> <br>
> Mar 18 12:48:56 [7202] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> stonith-ng: info: crm_cs_flush:<br>
> Sent 0 CPG messages (1 remaining, last=16): Try again (6)<br>
> Mar 18 12:48:56 [7201] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> cib: info: crm_cs_flush:<br>
> Sent 0 CPG messages (2 remaining, last=87): Try again (6)<br>
> Mar 18 12:48:56 [7202] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> stonith-ng: info: crm_cs_flush:<br>
> Sent 0 CPG messages (1 remaining, last=16): Try again (6)<br>
> Mar 18 12:48:56 [7185] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> pacemakerd: info: crm_cs_flush:<br>
> Sent 0 CPG messages (1 remaining, last=13): Try again (6)<br>
> [7177] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> corosyncinfo [VOTEQ ] waiting for quorum device<br>
> Qdevice poll (but maximum for 30000 ms)<br>
> [7177] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> corosyncnotice [TOTEM ] A new membership<br>
> (X.X.X.3:2469) was formed. Members<br>
<br>
^^ This is weird. I'm pretty sure something is broken with the way how <br>
packets are blocked (or log is incomplete)<br>
<br>
> [7177] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> corosyncwarning [CPG ] downlist left_list: 0<br>
> received<br>
> [7177] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> corosyncwarning [TOTEM ] Discarding JOIN message<br>
> during flush, nodeid=3<br>
> Mar 18 12:48:56 [7201] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> cib: info: crm_cs_flush:<br>
> Sent 0 CPG messages (2 remaining, last=87): Try again (6)<br>
> Mar 18 12:48:56 [7202] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> stonith-ng: info: crm_cs_flush:<br>
> Sent 0 CPG messages (1 remaining, last=16): Try again (6)<br>
> Mar 18 12:48:56 [7185] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> pacemakerd: info: crm_cs_flush:<br>
> Sent 0 CPG messages (1 remaining, last=13): Try again (6)<br>
> Mar 18 12:48:56 [7201] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> cib: info: crm_cs_flush:<br>
> Sent 0 CPG messages (2 remaining, last=87): Try again (6)<br>
> Mar 18 12:48:56 [7185] <a href="http://node2.domain.com" rel="noreferrer" target="_blank">node2.domain.com</a> pacemakerd: info: crm_cs_flush:<br>
> Sent 0 CPG messages (1 remaining, last=13): Try again (6)<br>
<br>
If it repeats over and over again then it's 99.9% because of way packets <br>
are blocked.<br>
<br>
> <br>
> Also on node2:<br>
> <br>
> pcs quorum status<br>
> Error: Unable to get quorum status: Unable to get node address for nodeid<br>
> 2: CS_ERR_NOT_EXIST<br>
> <br>
> And these are the logs on the qdevice host:<br>
> <br>
> Mar 18 12:48:50 debug algo-lms: membership list from node 3 partition<br>
> (3.99d)<br>
> Mar 18 12:48:50 debug algo-util: all_ring_ids_match: seen nodeid 2<br>
> (client 0x55a99ce070d0) ring_id (2.995)<br>
> Mar 18 12:48:50 debug algo-util: nodeid 2 in our partition has different<br>
> ring_id (2.995) to us (3.99d)<br>
> Mar 18 12:48:50 debug algo-lms: nodeid 3: ring ID (3.99d) not unique in<br>
> this membership, waiting<br>
> Mar 18 12:48:50 debug Algorithm result vote is Wait for reply<br>
> Mar 18 12:48:52 debug algo-lms: Client 0x55a99cdfe590 (cluster cluster1,<br>
> node_id 3) Timer callback<br>
> Mar 18 12:48:52 debug algo-util: all_ring_ids_match: seen nodeid 2<br>
> (client 0x55a99ce070d0) ring_id (2.995)<br>
> Mar 18 12:48:52 debug algo-util: nodeid 2 in our partition has different<br>
> ring_id (2.995) to us (3.99d)<br>
> Mar 18 12:48:52 debug algo-lms: nodeid 3: ring ID (3.99d) not unique in<br>
> this membership, waiting<br>
> Mar 18 12:48:52 debug Algorithm for client ::ffff:X.X.X.3:59762 decided<br>
> to reschedule timer and not send vote with value Wait for reply<br>
> Mar 18 12:48:53 debug Client closed connection<br>
> Mar 18 12:48:53 debug Client ::ffff:X.X.X.2:33960 (init_received 1,<br>
> cluster cluster1, node_id 2) disconnect<br>
> Mar 18 12:48:53 debug algo-lms: Client 0x55a99ce070d0 (cluster cluster1,<br>
> node_id 2) disconnect<br>
> Mar 18 12:48:53 info algo-lms: server going down 0<br>
> Mar 18 12:48:54 debug algo-lms: Client 0x55a99cdfe590 (cluster cluster1,<br>
> node_id 3) Timer callback<br>
> Mar 18 12:48:54 debug algo-util: partition (3.99d) (0x55a99ce07780) has 1<br>
> nodes<br>
> Mar 18 12:48:54 debug algo-lms: Only 1 partition. This is votequorum's<br>
> problem, not ours<br>
> Mar 18 12:48:54 debug Algorithm for client ::ffff:X.X.X.3:59762 decided<br>
> to not reschedule timer and send vote with value ACK<br>
> Mar 18 12:48:54 debug Sending vote info to client ::ffff:X.X.X.3:59762<br>
> (cluster cluster1, node_id 3)<br>
> Mar 18 12:48:54 debug msg seq num = 1<br>
> Mar 18 12:48:54 debug vote = ACK<br>
> Mar 18 12:48:54 debug Client ::ffff:X.X.X.3:59762 (cluster cluster1,<br>
> node_id 3) replied back to vote info message<br>
> Mar 18 12:48:54 debug msg seq num = 1<br>
> Mar 18 12:48:54 debug algo-lms: Client 0x55a99cdfe590 (cluster cluster1,<br>
> node_id 3) replied back to vote info message<br>
> Mar 18 12:48:54 debug Client ::ffff:X.X.X.3:59762 (cluster cluster1,<br>
> node_id 3) sent membership node list.<br>
> Mar 18 12:48:54 debug msg seq num = 8<br>
> Mar 18 12:48:54 debug ring id = (3.9a1)<br>
> Mar 18 12:48:54 debug heuristics = Pass<br>
> Mar 18 12:48:54 debug node list:<br>
> Mar 18 12:48:54 debug node_id = 3, data_center_id = 0, node_state =<br>
> not set<br>
> Mar 18 12:48:54 debug<br>
> Mar 18 12:48:54 debug algo-lms: membership list from node 3 partition<br>
> (3.9a1)<br>
> Mar 18 12:48:54 debug algo-util: partition (3.99d) (0x55a99ce073f0) has 1<br>
> nodes<br>
> Mar 18 12:48:54 debug algo-lms: Only 1 partition. This is votequorum's<br>
> problem, not ours<br>
> Mar 18 12:48:54 debug Algorithm result vote is ACK<br>
> Mar 18 12:48:58 debug Client ::ffff:X.X.X.3:59762 (cluster cluster1,<br>
> node_id 3) sent membership node list.<br>
> Mar 18 12:48:58 debug msg seq num = 9<br>
> Mar 18 12:48:58 debug ring id = (3.9a5)<br>
> Mar 18 12:48:58 debug heuristics = Pass<br>
> Mar 18 12:48:58 debug node list:<br>
> Mar 18 12:48:58 debug node_id = 3, data_center_id = 0, node_state =<br>
> not set<br>
> <br>
> <br>
> I'm running it on CentOS7 servers and tried to follow the RH7 official<br>
> docs, but I found a few issues there, and a bug that they won't correct,<br>
<br>
What issues you've found? Could you please report them so doc team can <br>
fix them?<br>
<br>
> since there is a workaround. In the end, looks like it is working fine,<br>
> except for this voting issue.<br>
> <br>
> After lots of time looking for answers on Google, I decided to send a<br>
> message here, and hopefully you can help me to fix it (it is probably a<br>
> silly mistake).<br>
<br>
I would bet it's really way how traffic is blocked.<br>
<br>
Regards,<br>
Honza<br>
<br>
> <br>
> Any help will be appreciated.<br>
> <br>
> Thank you.<br>
> <br>
> Marcelo H. Terres <<a href="mailto:mhterres@gmail.com" target="_blank">mhterres@gmail.com</a>><br>
> <a href="https://www.mundoopensource.com.br" rel="noreferrer" target="_blank">https://www.mundoopensource.com.br</a><br>
> <a href="https://twitter.com/mhterres" rel="noreferrer" target="_blank">https://twitter.com/mhterres</a><br>
> <a href="https://linkedin.com/in/marceloterres" rel="noreferrer" target="_blank">https://linkedin.com/in/marceloterres</a><br>
> <br>
> <br>
> <br>
> _______________________________________________<br>
> Manage your subscription:<br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
> <br>
> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
> <br>
<br>
</blockquote></div>