[ClusterLabs] Redundant ring not recovering after node is back
David Tolosa
david.tolosa at upcnet.es
Thu Aug 23 03:15:52 EDT 2018
BTW, where I can download Corosync 3.x?
I've only seen Corosync 2.99.3 Alpha4 at http://corosync.github.io/corosync/
2018-08-23 9:11 GMT+02:00 David Tolosa <david.tolosa at upcnet.es>:
> I'm currently using an Ubuntu 18.04 server configuration with netplan.
>
> Here you have my current YAML configuration:
>
> # This file describes the network interfaces available on your system
> # For more information, see netplan(5).
> network:
> version: 2
> renderer: networkd
> ethernets:
> eno1:
> addresses: [192.168.0.1/24]
> enp4s0f0:
> addresses: [192.168.1.1/24]
> enp5s0f0:
> {}
> vlans:
> vlan.XXX:
> id: XXX
> link: enp5s0f0
> addresses: [ 10.1.128.5/29 ]
> gateway4: 10.1.128.1
> nameservers:
> addresses: [ 8.8.8.8, 8.8.4.4 ]
> search: [ foo.com, bar.com ]
> vlan.YYY:
> id: YYY
> link: enp5s0f0
> addresses: [ 10.1.128.5/29 ]
>
>
> So, eno1 and enp4s0f0 are the two ethernet ports connected each other
> with crossover cables to node2.
> enp5s0f0 port is used to connect outside/services using vlans defined in
> the same file.
>
> In short, I'm using systemd-networkd default Ubuntu 18 server service for
> manage networks. Im not detecting any NetworkManager-config-server
> package in my repository neither.
> So the only solution that I have left, I suppose, is to test corosync 3.x
> and see if it works better handling RRP.
>
> Thank you for your quick response!
>
> 2018-08-23 8:40 GMT+02:00 Jan Friesse <jfriesse at redhat.com>:
>
>> David,
>>
>> Hello,
>>> Im getting crazy about this problem, that I expect to resolve here, with
>>> your help guys:
>>>
>>> I have 2 nodes with Corosync redundant ring feature.
>>>
>>> Each node has 2 similarly connected/configured NIC's. Both nodes are
>>> connected each other by two crossover cables.
>>>
>>
>> I believe this is root of the problem. Are you using NetworkManager? If
>> so, have you installed NetworkManager-config-server? If not, please install
>> it and test again.
>>
>>
>>> I configured both nodes with rrp mode passive. Everything is working well
>>> at this point, but when I shutdown 1 node to test failover, and this
>>> node > returns to be online, corosync is marking the interface as FAULTY
>>> and rrp
>>>
>>
>> I believe it's because with crossover cables configuration when other
>> side is shutdown, NetworkManager detects it and does ifdown of the
>> interface. And corosync is unable to handle ifdown properly. Ifdown is bad
>> with single ring, but it's just killer with RRP (127.0.0.1 poisons every
>> node in the cluster).
>>
>> fails to recover the initial state:
>>>
>>> 1. Initial scenario:
>>>
>>> # corosync-cfgtool -s
>>> Printing ring status.
>>> Local node ID 1
>>> RING ID 0
>>> id = 192.168.0.1
>>> status = ring 0 active with no faults
>>> RING ID 1
>>> id = 192.168.1.1
>>> status = ring 1 active with no faults
>>>
>>>
>>> 2. When I shutdown the node 2, all continues with no faults. Sometimes
>>> the
>>> ring ID's are bonding with 127.0.0.1 and then bond back to their
>>> respective
>>> heartbeat IP.
>>>
>>
>> Again, result of ifdown.
>>
>>
>>> 3. When node 2 is back online:
>>>
>>> # corosync-cfgtool -s
>>> Printing ring status.
>>> Local node ID 1
>>> RING ID 0
>>> id = 192.168.0.1
>>> status = ring 0 active with no faults
>>> RING ID 1
>>> id = 192.168.1.1
>>> status = Marking ringid 1 interface 192.168.1.1 FAULTY
>>>
>>>
>>> # service corosync status
>>> ● corosync.service - Corosync Cluster Engine
>>> Loaded: loaded (/lib/systemd/system/corosync.service; enabled;
>>> vendor
>>> preset: enabled)
>>> Active: active (running) since Wed 2018-08-22 14:44:09 CEST; 1min
>>> 38s ago
>>> Docs: man:corosync
>>> man:corosync.conf
>>> man:corosync_overview
>>> Main PID: 1439 (corosync)
>>> Tasks: 2 (limit: 4915)
>>> CGroup: /system.slice/corosync.service
>>> └─1439 /usr/sbin/corosync -f
>>>
>>>
>>> Aug 22 14:44:11 node1 corosync[1439]: Aug 22 14:44:11 notice [TOTEM ]
>>> The
>>> network interface [192.168.0.1] is now up.
>>> Aug 22 14:44:11 node1 corosync[1439]: [TOTEM ] The network interface
>>> [192.168.0.1] is now up.
>>> Aug 22 14:44:11 node1 corosync[1439]: Aug 22 14:44:11 notice [TOTEM ]
>>> The
>>> network interface [192.168.1.1] is now up.
>>> Aug 22 14:44:11 node1 corosync[1439]: [TOTEM ] The network interface
>>> [192.168.1.1] is now up.
>>> Aug 22 14:44:26 node1 corosync[1439]: Aug 22 14:44:26 notice [TOTEM ] A
>>> new membership (192.168.0.1:601760) was formed. Members
>>> Aug 22 14:44:26 node1 corosync[1439]: [TOTEM ] A new membership (
>>> 192.168.0.1:601760) was formed. Members
>>> Aug 22 14:44:32 node1 corosync[1439]: Aug 22 14:44:32 notice [TOTEM ] A
>>> new membership (192.168.0.1:601764) was formed. Members joined: 2
>>> Aug 22 14:44:32 node1 corosync[1439]: [TOTEM ] A new membership (
>>> 192.168.0.1:601764) was formed. Members joined: 2
>>> Aug 22 14:44:34 node1 corosync[1439]: Aug 22 14:44:34 error [TOTEM ]
>>> Marking ringid 1 interface 192.168.1.1 FAULTY
>>> Aug 22 14:44:34 node1 corosync[1439]: [TOTEM ] Marking ringid 1
>>> interface
>>> 192.168.1.1 FAULTY
>>>
>>>
>>> If I execute corosync-cfgtool, clears the faulty error but after some
>>> seconds return to be FAULTY.
>>> The only thing that it resolves the problem is to restart de service with
>>> service corosync restart.
>>>
>>> Here you have some of my configuration settings on node 1 (I probed
>>> already
>>> to change rrp_mode):
>>>
>>> *- corosync.conf*
>>>
>>>
>>> totem {
>>> version: 2
>>> cluster_name: node
>>> token: 5000
>>> token_retransmits_before_loss_const: 10
>>> secauth: off
>>> threads: 0
>>> rrp_mode: passive
>>> nodeid: 1
>>> interface {
>>> ringnumber: 0
>>> bindnetaddr: 192.168.0.0
>>> #mcastaddr: 226.94.1.1
>>> mcastport: 5405
>>> broadcast: yes
>>> }
>>> interface {
>>> ringnumber: 1
>>> bindnetaddr: 192.168.1.0
>>> #mcastaddr: 226.94.1.2
>>> mcastport: 5407
>>> broadcast: yes
>>> }
>>> }
>>>
>>> logging {
>>> fileline: off
>>> to_stderr: yes
>>> to_syslog: yes
>>> to_logfile: yes
>>> logfile: /var/log/corosync/corosync.log
>>> debug: off
>>> timestamp: on
>>> logger_subsys {
>>> subsys: AMF
>>> debug: off
>>> }
>>> }
>>>
>>> amf {
>>> mode: disabled
>>> }
>>>
>>> quorum {
>>> provider: corosync_votequorum
>>> expected_votes: 2
>>> }
>>>
>>> nodelist {
>>> node {
>>> nodeid: 1
>>> ring0_addr: 192.168.0.1
>>> ring1_addr: 192.168.1.1
>>> }
>>>
>>> node {
>>> nodeid: 2
>>> ring0_addr: 192.168.0.2
>>> ring1_addr: 192.168.1.2
>>> }
>>> }
>>>
>>> aisexec {
>>> user: root
>>> group: root
>>> }
>>>
>>> service {
>>> name: pacemaker
>>> ver: 1
>>> }
>>>
>>>
>>>
>>> *- /etc/hosts*
>>>
>>>
>>> 127.0.0.1 localhost
>>> 10.4.172.5 node1.upc.edu node1
>>> 10.4.172.6 node2.upc.edu node2
>>>
>>>
>> So machines have 3 NICs? 2 for corosync/cluster traffic and one for
>> regular traffic/services/outside world?
>>
>>
>>> Thank you for you help in advance!
>>>
>>
>> To conclude:
>> - If you are using NetworkManager, try to install
>> NetworkManager-config-server, it will probably help
>> - If you are brave enough, try corosync 3.x (current Alpha4 is pretty
>> stable - actually some other projects gain this stability with SP1 :) )
>> that has no RRP but uses knet for support redundant links (up-to 8 links
>> can be configured) and doesn't have problems with ifdown.
>>
>> Honza
>>
>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list: Users at clusterlabs.org
>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>>
>>
>
>
> --
> *David Tolosa Martínez*
> Customer Support & Infrastructure
> UPCnet - Edifici Vèrtex
> Plaça d'Eusebi Güell, 6, 08034 Barcelona
> Tel: 934054555
>
> <https://www.upcnet.es>
>
--
*David Tolosa Martínez*
Customer Support & Infrastructure
UPCnet - Edifici Vèrtex
Plaça d'Eusebi Güell, 6, 08034 Barcelona
Tel: 934054555
<https://www.upcnet.es>
--
INFORMACIÓ BÀSICA SOBRE PROTECCIÓ DE DADES:
Responsable: UPCNET,
Serveis d'Accés a Internet de la Universitat Politècnica de Catalunya, SLU
| Finalitat: gestionar els contactes i les relacions professionals i
comercials amb els nostres clients i proveïdors | Base legal:
consentiment, interès legítim i/o relació contractual | Destinataris:
no seran comunicades a tercers excepte per obligació legal | Drets:
pots exercir els teus drets d’accés, rectificació i supressió, així com els
altres drets reconeguts a la normativa vigent, enviant-nos un missatge a
privacy at upcnet.es <mailto:privacy at upcnet.es> | Més informació: consulta
la nostra política completa de protecció de dades
<https://www.upcnet.es/politica-de-privacitat>.
AVÍS DE
CONFIDENCIALITAT <https://www.upcnet.es/ca/avis-de-confidencialitat>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20180823/a601b0ea/attachment-0001.html>
More information about the Users
mailing list