<div dir="ltr">BTW, where I can download Corosync 3.x?<div>I've only seen Corosync 2.99.3 Alpha4 at <a href="http://corosync.github.io/corosync/">http://corosync.github.io/corosync/</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">2018-08-23 9:11 GMT+02:00 David Tolosa <span dir="ltr"><<a href="mailto:david.tolosa@upcnet.es" target="_blank">david.tolosa@upcnet.es</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm currently using an Ubuntu 18.04 server configuration with netplan.<div><br><div>Here you have my current YAML configuration:</div></div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div># This file describes the network interfaces available on your system</div></div><div><div># For more information, see netplan(5).</div></div><div><div>network:</div></div><div><div> version: 2</div></div><div><div> renderer: networkd</div></div><div><div> ethernets:</div></div><div><div> eno1:</div></div><div><div> addresses: [<a href="http://192.168.0.1/24" target="_blank">192.168.0.1/24</a>]</div></div><div><div> enp4s0f0:</div></div><div><div> addresses: [<a href="http://192.168.1.1/24" target="_blank">192.168.1.1/24</a>]</div></div><div><div> enp5s0f0:</div></div><div><div> {}</div></div><div><div> vlans:</div></div><div><div> vlan.XXX:</div></div><div><div> id: XXX</div></div><div><div> link: enp5s0f0</div></div><div><div> addresses: [ <a href="http://10.1.128.5/29" target="_blank">10.1.128.5/29</a> ]</div></div><div><div> gateway4: 10.1.128.1</div></div><div><div> nameservers:</div></div><div><div> addresses: [ 8.8.8.8, 8.8.4.4 ]</div></div><div><div> search: [ <a href="http://foo.com" target="_blank">foo.com</a>, <a href="http://bar.com" target="_blank">bar.com</a> ]</div></div><div><div> vlan.YYY:</div></div><div><div> id: YYY</div></div><div><div> link: enp5s0f0</div></div><div><div> addresses: [ <a href="http://10.1.128.5/29" target="_blank">10.1.128.5/29</a> ]</div></div></blockquote><div><br></div><div>So, eno1 and <span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">enp4s0f0 are the two ethernet ports connected each other with crossover cables to node2. </span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">enp5s0f0 port is used to connect outside/services using vlans defined in the same file.</span></span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">In short, I'm using </span></span>systemd-networkd default Ubuntu 18 server service for manage networks. Im not detecting any <span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">NetworkManager-config-<wbr>server package in my repository neither.</span><br></div><div><span style="font-size:12.8px">So the only solution that I have left, I suppose, is to test corosync 3.x and see if it works better handling RRP.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thank you for your quick response!</span></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">2018-08-23 8:40 GMT+02:00 Jan Friesse <span dir="ltr"><<a href="mailto:jfriesse@redhat.com" target="_blank">jfriesse@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">David,<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
Im getting crazy about this problem, that I expect to resolve here, with<br>
your help guys:<br>
<br>
I have 2 nodes with Corosync redundant ring feature.<br>
<br>
Each node has 2 similarly connected/configured NIC's. Both nodes are<br>
connected each other by two crossover cables.<br>
</blockquote>
<br></span>
I believe this is root of the problem. Are you using NetworkManager? If so, have you installed NetworkManager-config-server? If not, please install it and test again.<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I configured both nodes with rrp mode passive. Everything is working well<br>
at this point, but when I shutdown 1 node to test failover, and this node > returns to be online, corosync is marking the interface as FAULTY and rrp<br>
</blockquote>
<br></span>
I believe it's because with crossover cables configuration when other side is shutdown, NetworkManager detects it and does ifdown of the interface. And corosync is unable to handle ifdown properly. Ifdown is bad with single ring, but it's just killer with RRP (127.0.0.1 poisons every node in the cluster).<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
fails to recover the initial state:<br>
<br>
1. Initial scenario:<br>
<br>
# corosync-cfgtool -s<br>
Printing ring status.<br>
Local node ID 1<br>
RING ID 0<br>
id = 192.168.0.1<br>
status = ring 0 active with no faults<br>
RING ID 1<br>
id = 192.168.1.1<br>
status = ring 1 active with no faults<br>
<br>
<br>
2. When I shutdown the node 2, all continues with no faults. Sometimes the<br>
ring ID's are bonding with 127.0.0.1 and then bond back to their respective<br>
heartbeat IP.<br>
</blockquote>
<br></span>
Again, result of ifdown.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_1807907226360386776h5">
<br>
3. When node 2 is back online:<br>
<br>
# corosync-cfgtool -s<br>
Printing ring status.<br>
Local node ID 1<br>
RING ID 0<br>
id = 192.168.0.1<br>
status = ring 0 active with no faults<br>
RING ID 1<br>
id = 192.168.1.1<br>
status = Marking ringid 1 interface 192.168.1.1 FAULTY<br>
<br>
<br>
# service corosync status<br>
● corosync.service - Corosync Cluster Engine<br>
Loaded: loaded (/lib/systemd/system/corosync.<wbr>service; enabled; vendor<br>
preset: enabled)<br>
Active: active (running) since Wed 2018-08-22 14:44:09 CEST; 1min 38s ago<br>
Docs: man:corosync<br>
man:corosync.conf<br>
man:corosync_overview<br>
Main PID: 1439 (corosync)<br>
Tasks: 2 (limit: 4915)<br>
CGroup: /system.slice/corosync.service<br>
└─1439 /usr/sbin/corosync -f<br>
<br>
<br>
Aug 22 14:44:11 node1 corosync[1439]: Aug 22 14:44:11 notice [TOTEM ] The<br>
network interface [192.168.0.1] is now up.<br>
Aug 22 14:44:11 node1 corosync[1439]: [TOTEM ] The network interface<br>
[192.168.0.1] is now up.<br>
Aug 22 14:44:11 node1 corosync[1439]: Aug 22 14:44:11 notice [TOTEM ] The<br>
network interface [192.168.1.1] is now up.<br>
Aug 22 14:44:11 node1 corosync[1439]: [TOTEM ] The network interface<br>
[192.168.1.1] is now up.<br>
Aug 22 14:44:26 node1 corosync[1439]: Aug 22 14:44:26 notice [TOTEM ] A<br>
new membership (<a href="http://192.168.0.1:601760" rel="noreferrer" target="_blank">192.168.0.1:601760</a>) was formed. Members<br>
Aug 22 14:44:26 node1 corosync[1439]: [TOTEM ] A new membership (<br>
<a href="http://192.168.0.1:601760" rel="noreferrer" target="_blank">192.168.0.1:601760</a>) was formed. Members<br>
Aug 22 14:44:32 node1 corosync[1439]: Aug 22 14:44:32 notice [TOTEM ] A<br>
new membership (<a href="http://192.168.0.1:601764" rel="noreferrer" target="_blank">192.168.0.1:601764</a>) was formed. Members joined: 2<br>
Aug 22 14:44:32 node1 corosync[1439]: [TOTEM ] A new membership (<br>
<a href="http://192.168.0.1:601764" rel="noreferrer" target="_blank">192.168.0.1:601764</a>) was formed. Members joined: 2<br>
Aug 22 14:44:34 node1 corosync[1439]: Aug 22 14:44:34 error [TOTEM ]<br>
Marking ringid 1 interface 192.168.1.1 FAULTY<br>
Aug 22 14:44:34 node1 corosync[1439]: [TOTEM ] Marking ringid 1 interface<br>
192.168.1.1 FAULTY<br>
<br>
<br>
If I execute corosync-cfgtool, clears the faulty error but after some<br>
seconds return to be FAULTY.<br>
The only thing that it resolves the problem is to restart de service with<br>
service corosync restart.<br>
<br>
Here you have some of my configuration settings on node 1 (I probed already<br>
to change rrp_mode):<br>
<br></div></div>
*- corosync.conf*<div><div class="m_1807907226360386776h5"><br>
<br>
totem {<br>
version: 2<br>
cluster_name: node<br>
token: 5000<br>
token_retransmits_before_loss<wbr>_const: 10<br>
secauth: off<br>
threads: 0<br>
rrp_mode: passive<br>
nodeid: 1<br>
interface {<br>
ringnumber: 0<br>
bindnetaddr: 192.168.0.0<br>
#mcastaddr: 226.94.1.1<br>
mcastport: 5405<br>
broadcast: yes<br>
}<br>
interface {<br>
ringnumber: 1<br>
bindnetaddr: 192.168.1.0<br>
#mcastaddr: 226.94.1.2<br>
mcastport: 5407<br>
broadcast: yes<br>
}<br>
}<br>
<br>
logging {<br>
fileline: off<br>
to_stderr: yes<br>
to_syslog: yes<br>
to_logfile: yes<br>
logfile: /var/log/corosync/corosync.log<br>
debug: off<br>
timestamp: on<br>
logger_subsys {<br>
subsys: AMF<br>
debug: off<br>
}<br>
}<br>
<br>
amf {<br>
mode: disabled<br>
}<br>
<br>
quorum {<br>
provider: corosync_votequorum<br>
expected_votes: 2<br>
}<br>
<br>
nodelist {<br>
node {<br>
nodeid: 1<br>
ring0_addr: 192.168.0.1<br>
ring1_addr: 192.168.1.1<br>
}<br>
<br>
node {<br>
nodeid: 2<br>
ring0_addr: 192.168.0.2<br>
ring1_addr: 192.168.1.2<br>
}<br>
}<br>
<br>
aisexec {<br>
user: root<br>
group: root<br>
}<br>
<br>
service {<br>
name: pacemaker<br>
ver: 1<br>
}<br>
<br>
<br>
<br></div></div>
*- /etc/hosts*<span><br>
<br>
<br>
127.0.0.1 localhost<br>
10.4.172.5 <a href="http://node1.upc.edu" rel="noreferrer" target="_blank">node1.upc.edu</a> node1<br>
10.4.172.6 <a href="http://node2.upc.edu" rel="noreferrer" target="_blank">node2.upc.edu</a> node2<br>
<br>
</span></blockquote>
<br>
So machines have 3 NICs? 2 for corosync/cluster traffic and one for regular traffic/services/outside world?<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thank you for you help in advance!<br>
</blockquote>
<br></span>
To conclude:<br>
- If you are using NetworkManager, try to install NetworkManager-config-server, it will probably help<br>
- If you are brave enough, try corosync 3.x (current Alpha4 is pretty stable - actually some other projects gain this stability with SP1 :) ) that has no RRP but uses knet for support redundant links (up-to 8 links can be configured) and doesn't have problems with ifdown.<br>
<br>
Honza<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</blockquote>
<br>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="">-- <br><div class="m_1807907226360386776gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr" style="font-size:12.8px"><div><div><b><font color="#666666">David Tolosa Martínez</font></b><br></div><span style="color:rgb(136,136,136);font-size:12.8px">Customer Support & Infrastructure</span><br></div><font color="#888888">UPCnet - Edifici Vèrtex</font></div><div dir="ltr" style="font-size:12.8px"><span style="color:rgb(136,136,136);font-size:12.8px">Plaça d'Eusebi Güell, 6, 08034 Barcelona</span><font color="#888888"><br>Tel: 934054555</font><br><br></div><div dir="ltr"><a href="https://www.upcnet.es" target="_blank"><img src="https://www.upcnet.es/ca/img/upcnet.png"></a><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr" style="font-size:12.8px"><div><div><b><font color="#666666">David Tolosa Martínez</font></b><br></div><span style="color:rgb(136,136,136);font-size:12.8px">Customer Support & Infrastructure</span><br></div><font color="#888888">UPCnet - Edifici Vèrtex</font></div><div dir="ltr" style="font-size:12.8px"><span style="color:rgb(136,136,136);font-size:12.8px">Plaça d'Eusebi Güell, 6, 08034 Barcelona</span><font color="#888888"><br>Tel: 934054555</font><br><br></div><div dir="ltr"><a href="https://www.upcnet.es" target="_blank"><img src="https://www.upcnet.es/ca/img/upcnet.png"></a><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>
<br>
<br><br><p dir="ltr" style="line-height:1.3090909090909089;margin-top:0pt;margin-bottom:0pt;border-top:solid #eeeeee 0.75pt;background-color:#ffffff;padding:11.25pt 0pt 0pt 0pt"><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">INFORMACIÓ BÀSICA SOBRE PROTECCIÓ DE DADES:</span></p><p dir="ltr" style="line-height:1.3090909090909089;margin-top:0pt;margin-bottom:0pt;background-color:#ffffff"> </p><p dir="ltr" style="line-height:1.3090909090909089;margin-top:0pt;margin-bottom:0pt;background-color:#ffffff"><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">Responsable:</span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> UPCNET, Serveis d'Accés a Internet de la Universitat Politècnica de Catalunya, SLU | </span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">Finalitat:</span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> gestionar els contactes i les relacions professionals i comercials amb els nostres clients i proveïdors | </span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">Base legal:</span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> consentiment, interès legítim i/o relació contractual | </span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">Destinataris:</span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> no seran comunicades a tercers excepte per obligació legal | </span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">Drets:</span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> pots exercir els teus drets d’accés, rectificació i supressió, així com els altres drets reconeguts a la normativa vigent, enviant-nos un missatge a </span><span style="font-size:8.5pt;font-family:Arial;color:#207ebe;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"><a href="mailto:privacy@upcnet.es" target="_blank">privacy@upcnet.es</a></span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> | </span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">Més informació</span><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">: consulta la nostra </span><a href="https://www.upcnet.es/politica-de-privacitat" style="text-decoration:none" target="_blank"><span style="font-size:8.5pt;font-family:Arial;color:#207ebe;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre;white-space:pre-wrap">política completa de protecció de dades</span></a><span style="font-size:8.5pt;font-family:Arial;color:#999999;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">.</span></p><p dir="ltr" style="line-height:1.3090909090909089;margin-top:0pt;margin-bottom:0pt;background-color:#ffffff"> </p><p dir="ltr" style="line-height:1.3090909090909089;margin-top:0pt;margin-bottom:0pt;background-color:#ffffff"><a href="https://www.upcnet.es/ca/avis-de-confidencialitat" style="text-decoration:none" target="_blank"><span style="font-size:8.5pt;font-family:Arial;color:#207ebe;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre;white-space:pre-wrap">AVÍS DE CONFIDENCIALITAT</span></a></p>