[ClusterLabs] corosync not able to form cluster

Christine Caulfield ccaulfie at redhat.com
Thu Jun 7 09:46:03 EDT 2018


On 07/06/18 12:00, Prasad Nagaraj wrote:
> And here is how startup log looks on one of the nodes.
> Its pretty much same on the other nodes as well
> 

That all looks fine to me (from a corosync POV anyway). Have you checked
for any iptables rules getting in the way?

Chrissie

> Jun 07 10:41:37 corosync [MAIN  ] Corosync Cluster Engine ('UNKNOWN'):
> started and ready to provide service.
> Jun 07 10:41:37 corosync [MAIN  ] Corosync built-in features: nss
> Jun 07 10:41:37 corosync [MAIN  ] Successfully read main configuration
> file '/etc/corosync/corosync.conf'.
> Jun 07 10:41:38 corosync [TOTEM ] Initializing transport (UDP/IP Unicast).
> Jun 07 10:41:38 corosync [TOTEM ] Initializing transmit/receive
> security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Jun 07 10:41:38 corosync [TOTEM ] the ip i gave is  [0.0.0.0]
> Set r/w permissions for uid=0, gid=0 on /var/log/cluster/corosync.log
> Jun 07 10:41:38 corosync [TOTEM ] The network interface [172.22.0.4] is
> now up.
> Jun 07 10:41:38 corosync [pcmk  ] info: process_ais_conf: Reading configure
> Jun 07 10:41:38 corosync [pcmk  ] ERROR: process_ais_conf: You have
> configured a cluster using the Pacemaker plugin for Corosync. The plugin
> is not supported in this environment and will be removed very soon.
> Jun 07 10:41:38 corosync [pcmk  ] ERROR: process_ais_conf:  Please see
> Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc)
> for details on using Pacemaker with CMAN
> Jun 07 10:41:38 corosync [pcmk  ] info: config_find_init: Local handle:
> 2730409743423111169 for logging
> Jun 07 10:41:38 corosync [pcmk  ] info: config_find_next: Processing
> additional logging options...
> Jun 07 10:41:38 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'on' for option: debug
> Jun 07 10:41:38 corosync [pcmk  ] info: get_config_opt: Found 'yes' for
> option: to_logfile
> Jun 07 10:41:38 corosync [pcmk  ] info: get_config_opt: Found
> '/var/log/cluster/corosync.log' for option: logfile
> Jun 07 10:41:38 corosync [pcmk  ] info: get_config_opt: Found 'no' for
> option: to_syslog
> Jun 07 10:41:38 corosync [pcmk  ] info: process_ais_conf: User
> configured file based logging and explicitly disabled syslog.
> Jun 07 10:41:38 corosync [pcmk  ] info: config_find_init: Local handle:
> 5880381755227111426 for quorum
> Jun 07 10:41:38 corosync [pcmk  ] info: config_find_next: No additional
> configuration supplied for: quorum
> Jun 07 10:41:38 corosync [pcmk  ] info: get_config_opt: No default for
> option: provider
> Jun 07 10:41:38 corosync [pcmk  ] info: config_find_init: Local handle:
> 4835695805891346435 for service
> Jun 07 10:41:38 corosync [pcmk  ] info: config_find_next: Processing
> additional service options...
> Jun 07 10:41:38 corosync [pcmk  ] info: get_config_opt: Found '1' for
> option: ver
> Jun 07 10:41:38 corosync [pcmk  ] info: process_ais_conf: Enabling MCP
> mode: Use the Pacemaker init script to complete Pacemaker startup
> Jun 07 10:41:38 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'pcmk' for option: clustername
> Jun 07 10:41:38 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'no' for option: use_logd
> Jun 07 10:41:38 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'no' for option: use_mgmtd
> Jun 07 10:41:38 corosync [pcmk  ] info: pcmk_startup: CRM: Initialized
> Jun 07 10:41:38 corosync [pcmk  ] Logging: Initialized pcmk_startup
> Jun 07 10:41:38 corosync [pcmk  ] info: pcmk_startup: Maximum core file
> size is: 18446744073709551615
> Jun 07 10:41:38 corosync [pcmk  ] info: pcmk_startup: Service: 9
> Jun 07 10:41:38 corosync [pcmk  ] info: pcmk_startup: Local hostname:
> vme6c95166f0
> Jun 07 10:41:38 corosync [pcmk  ] info: pcmk_update_nodeid: Local node
> id: 67114668
> Jun 07 10:41:38 corosync [pcmk  ] info: update_member: Creating entry
> for node 67114668 born on 0
> Jun 07 10:41:38 corosync [pcmk  ] info: update_member: 0x1ffdf20 Node
> 67114668 now known as vme6c95166f0 (was: (null))
> Jun 07 10:41:38 corosync [pcmk  ] info: update_member: Node vme6c95166f0
> now has 1 quorum votes (was 0)
> Jun 07 10:41:38 corosync [pcmk  ] info: update_member: Node
> 67114668/vme6c95166f0 is now: member
> Jun 07 10:41:38 corosync [SERV  ] Service engine loaded: Pacemaker
> Cluster Manager 1.1.14-8.el6_8.1
> Jun 07 10:41:38 corosync [SERV  ] Service engine loaded: corosync
> extended virtual synchrony service
> Jun 07 10:41:38 corosync [SERV  ] Service engine loaded: corosync
> configuration service
> Jun 07 10:41:38 corosync [SERV  ] Service engine loaded: corosync
> cluster closed process group service v1.01
> Jun 07 10:41:38 corosync [SERV  ] Service engine loaded: corosync
> cluster config database access v1.01
> Jun 07 10:41:38 corosync [SERV  ] Service engine loaded: corosync
> profile loading service
> Jun 07 10:41:38 corosync [SERV  ] Service engine loaded: corosync
> cluster quorum service v0.1
> Jun 07 10:41:38 corosync [MAIN  ] Compatibility mode set to whitetank. 
> Using V1 and V2 of the synchronization engine.
> Jun 07 10:41:38 corosync [TOTEM ] adding new UDPU member {172.22.0.4}
> Jun 07 10:41:38 corosync [TOTEM ] adding new UDPU member {172.22.0.11}
> Jun 07 10:41:38 corosync [TOTEM ] adding new UDPU member {172.22.0.13}
> Jun 07 10:41:38 corosync [pcmk  ] notice: pcmk_peer_update: Transitional
> membership event on ring 77856: memb=0, new=0, lost=0
> Jun 07 10:41:38 corosync [pcmk  ] notice: pcmk_peer_update: Stable
> membership event on ring 77856: memb=1, new=1, lost=0
> Jun 07 10:41:38 corosync [pcmk  ] info: pcmk_peer_update: NEW: 
> vme6c95166f0 67114668
> Jun 07 10:41:38 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> vme6c95166f0 67114668
> Jun 07 10:41:38 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> Jun 07 10:41:38 corosync [CPG   ] chosen downlist: sender r(0)
> ip(172.22.0.4) ; members(old:0 left:0)
> Jun 07 10:41:38 corosync [MAIN  ] Completed service synchronization,
> ready to provide service.
> Jun 07 10:42:21 corosync [pcmk  ] notice: pcmk_peer_update: Transitional
> membership event on ring 77868: memb=1, new=0, lost=0
> Jun 07 10:42:21 corosync [pcmk  ] info: pcmk_peer_update: memb:
> vme6c95166f0 67114668
> Jun 07 10:42:21 corosync [pcmk  ] notice: pcmk_peer_update: Stable
> membership event on ring 77868: memb=1, new=0, lost=0
> Jun 07 10:42:21 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> vme6c95166f0 67114668
> Jun 07 10:42:21 corosync [pcmk  ] info: update_member: 0x1ffdf20 Node
> 67114668 ((null)) born on: 77856
> Jun 07 10:42:21 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> Jun 07 10:42:21 corosync [CPG   ] chosen downlist: sender r(0)
> ip(172.22.0.4) ; members(old:1 left:0)
> Jun 07 10:42:21 corosync [MAIN  ] Completed service synchronization,
> ready to provide service.
> Jun 07 10:42:34 corosync [pcmk  ] notice: pcmk_peer_update: Transitional
> membership event on ring 77876: memb=1, new=0, lost=0
> Jun 07 10:42:34 corosync [pcmk  ] info: pcmk_peer_update: memb:
> vme6c95166f0 67114668
> Jun 07 10:42:34 corosync [pcmk  ] notice: pcmk_peer_update: Stable
> membership event on ring 77876: memb=1, new=0, lost=0
> Jun 07 10:42:34 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> vme6c95166f0 67114668
> Jun 07 10:42:34 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> Jun 07 10:49:47 corosync [MAIN  ] Corosync Cluster Engine ('UNKNOWN'):
> started and ready to provide service.
> Jun 07 10:49:47 corosync [MAIN  ] Corosync built-in features: nss
> Jun 07 10:49:47 corosync [MAIN  ] Successfully read main configuration
> file '/etc/corosync/corosync.conf'.
> Jun 07 10:49:47 corosync [TOTEM ] Initializing transport (UDP/IP Unicast).
> Jun 07 10:49:47 corosync [TOTEM ] Initializing transmit/receive
> security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Jun 07 10:49:47 corosync [TOTEM ] the ip i gave is  [0.0.0.0]
> Jun 07 10:49:47 corosync [TOTEM ] The network interface [172.22.0.13] is
> now up.
> Set r/w permissions for uid=0, gid=0 on /var/log/cluster/corosync.log
> Jun 07 10:49:47 corosync [pcmk  ] info: process_ais_conf: Reading configure
> Jun 07 10:49:47 corosync [pcmk  ] ERROR: process_ais_conf: You have
> configured a cluster using the Pacemaker plugin for Corosync. The plugin
> is not supported in this environment and will be removed very soon.
> Jun 07 10:49:47 corosync [pcmk  ] ERROR: process_ais_conf:  Please see
> Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc)
> for details on using Pacemaker with CMAN
> Jun 07 10:49:47 corosync [pcmk  ] info: config_find_init: Local handle:
> 2730409743423111169 for logging
> Jun 07 10:49:47 corosync [pcmk  ] info: config_find_next: Processing
> additional logging options...
> Jun 07 10:49:47 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'on' for option: debug
> Jun 07 10:49:47 corosync [pcmk  ] info: get_config_opt: Found 'yes' for
> option: to_logfile
> Jun 07 10:49:47 corosync [pcmk  ] info: get_config_opt: Found
> '/var/log/cluster/corosync.log' for option: logfile
> Jun 07 10:49:47 corosync [pcmk  ] info: get_config_opt: Found 'no' for
> option: to_syslog
> Jun 07 10:49:47 corosync [pcmk  ] info: process_ais_conf: User
> configured file based logging and explicitly disabled syslog.
> Jun 07 10:49:47 corosync [pcmk  ] info: config_find_init: Local handle:
> 5880381755227111426 for quorum
> Jun 07 10:49:47 corosync [pcmk  ] info: config_find_next: No additional
> configuration supplied for: quorum
> Jun 07 10:49:47 corosync [pcmk  ] info: get_config_opt: No default for
> option: provider
> Jun 07 10:49:47 corosync [pcmk  ] info: config_find_init: Local handle:
> 4835695805891346435 for service
> Jun 07 10:49:47 corosync [pcmk  ] info: config_find_next: Processing
> additional service options...
> Jun 07 10:49:47 corosync [pcmk  ] info: get_config_opt: Found '1' for
> option: ver
> Jun 07 10:49:47 corosync [pcmk  ] info: process_ais_conf: Enabling MCP
> mode: Use the Pacemaker init script to complete Pacemaker startup
> Jun 07 10:49:47 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'pcmk' for option: clustername
> Jun 07 10:49:47 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'no' for option: use_logd
> Jun 07 10:49:47 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'no' for option: use_mgmtd
> Jun 07 10:49:47 corosync [pcmk  ] info: pcmk_startup: CRM: Initialized
> Jun 07 10:49:47 corosync [pcmk  ] Logging: Initialized pcmk_startup
> Jun 07 10:49:47 corosync [pcmk  ] info: pcmk_startup: Maximum core file
> size is: 18446744073709551615
> Jun 07 10:49:47 corosync [pcmk  ] info: pcmk_startup: Service: 9
> Jun 07 10:49:47 corosync [pcmk  ] info: pcmk_startup: Local hostname:
> vme0f7614459
> Jun 07 10:49:47 corosync [pcmk  ] info: pcmk_update_nodeid: Local node
> id: 218109612
> Jun 07 10:49:47 corosync [pcmk  ] info: update_member: Creating entry
> for node 218109612 born on 0
> Jun 07 10:49:47 corosync [pcmk  ] info: update_member: 0x228ef20 Node
> 218109612 now known as vme0f7614459 (was: (null))
> Jun 07 10:49:47 corosync [pcmk  ] info: update_member: Node vme0f7614459
> now has 1 quorum votes (was 0)
> Jun 07 10:49:47 corosync [pcmk  ] info: update_member: Node
> 218109612/vme0f7614459 is now: member
> Jun 07 10:49:47 corosync [SERV  ] Service engine loaded: Pacemaker
> Cluster Manager 1.1.14-8.el6_8.1
> Jun 07 10:49:47 corosync [SERV  ] Service engine loaded: corosync
> extended virtual synchrony service
> Jun 07 10:49:47 corosync [SERV  ] Service engine loaded: corosync
> configuration service
> Jun 07 10:49:47 corosync [SERV  ] Service engine loaded: corosync
> cluster closed process group service v1.01
> Jun 07 10:49:47 corosync [SERV  ] Service engine loaded: corosync
> cluster config database access v1.01
> Jun 07 10:49:47 corosync [SERV  ] Service engine loaded: corosync
> profile loading service
> Jun 07 10:49:47 corosync [SERV  ] Service engine loaded: corosync
> cluster quorum service v0.1
> Jun 07 10:49:47 corosync [MAIN  ] Compatibility mode set to whitetank. 
> Using V1 and V2 of the synchronization engine.
> Jun 07 10:49:47 corosync [TOTEM ] adding new UDPU member {172.22.0.4}
> Jun 07 10:49:47 corosync [TOTEM ] adding new UDPU member {172.22.0.11}
> Jun 07 10:49:47 corosync [TOTEM ] adding new UDPU member {172.22.0.13}
> Jun 07 10:50:17 corosync [pcmk  ] notice: pcmk_peer_update: Transitional
> membership event on ring 78176: memb=0, new=0, lost=0
> Jun 07 10:50:17 corosync [pcmk  ] notice: pcmk_peer_update: Stable
> membership event on ring 78176: memb=1, new=1, lost=0
> Jun 07 10:50:17 corosync [pcmk  ] info: pcmk_peer_update: NEW: 
> vme0f7614459 218109612
> Jun 07 10:50:17 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> vme0f7614459 218109612
> Jun 07 10:50:17 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> Jun 07 10:50:17 corosync [CPG   ] chosen downlist: sender r(0)
> ip(172.22.0.13) ; members(old:0 left:0)
> Jun 07 10:50:17 corosync [MAIN  ] Completed service synchronization,
> ready to provide service.
> Jun 07 10:50:29 corosync [pcmk  ] notice: pcmk_peer_update: Transitional
> membership event on ring 78188: memb=1, new=0, lost=0
> Jun 07 10:50:29 corosync [pcmk  ] info: pcmk_peer_update: memb:
> vme0f7614459 218109612
> Jun 07 10:50:29 corosync [pcmk  ] notice: pcmk_peer_update: Stable
> membership event on ring 78188: memb=1, new=0, lost=0
> Jun 07 10:50:29 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> vme0f7614459 218109612
> Jun 07 10:50:29 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> Jun 07 10:50:29 corosync [CPG   ] chosen downlist: sender r(0)
> ip(172.22.0.13) ; members(old:1 left:0)
> Jun 07 10:50:29 corosync [MAIN  ] Completed service synchronization,
> ready to provide service.
> Jun 07 10:50:41 corosync [pcmk  ] notice: pcmk_peer_update: Transitional
> membership event on ring 78200: memb=1, new=0, lost=0
> Jun 07 10:50:41 corosync [pcmk  ] info: pcmk_peer_update: memb:
> vme0f7614459 218109612
> Jun 07 10:50:41 corosync [pcmk  ] notice: pcmk_peer_update: Stable
> membership event on ring 78200: memb=1, new=0, lost=0
> Jun 07 10:50:41 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> vme0f7614459 218109612
> Jun 07 10:50:41 corosync [pcmk  ] info: update_member: 0x228ef20 Node
> 218109612 ((null)) born on: 78176
> Jun 07 10:50:41 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> Jun 07 10:48:49 corosync [MAIN  ] Corosync Cluster Engine ('UNKNOWN'):
> started and ready to provide service.
> Jun 07 10:48:49 corosync [MAIN  ] Corosync built-in features: nss
> Jun 07 10:48:49 corosync [MAIN  ] Successfully read main configuration
> file '/etc/corosync/corosync.conf'.
> Jun 07 10:48:49 corosync [TOTEM ] Initializing transport (UDP/IP Unicast).
> Jun 07 10:48:49 corosync [TOTEM ] Initializing transmit/receive
> security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Jun 07 10:48:49 corosync [TOTEM ] the ip i gave is  [0.0.0.0]
> Jun 07 10:48:49 corosync [TOTEM ] The network interface [172.22.0.11] is
> now up.
> Set r/w permissions for uid=0, gid=0 on /var/log/cluster/corosync.log
> Jun 07 10:48:49 corosync [pcmk  ] info: process_ais_conf: Reading configure
> Jun 07 10:48:49 corosync [pcmk  ] ERROR: process_ais_conf: You have
> configured a cluster using the Pacemaker plugin for Corosync. The plugin
> is not supported in this environment and will be removed very soon.
> Jun 07 10:48:49 corosync [pcmk  ] ERROR: process_ais_conf:  Please see
> Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc)
> for details on using Pacemaker with CMAN
> Jun 07 10:48:49 corosync [pcmk  ] info: config_find_init: Local handle:
> 2730409743423111169 for logging
> Jun 07 10:48:49 corosync [pcmk  ] info: config_find_next: Processing
> additional logging options...
> Jun 07 10:48:49 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'on' for option: debug
> Jun 07 10:48:49 corosync [pcmk  ] info: get_config_opt: Found 'yes' for
> option: to_logfile
> Jun 07 10:48:49 corosync [pcmk  ] info: get_config_opt: Found
> '/var/log/cluster/corosync.log' for option: logfile
> Jun 07 10:48:49 corosync [pcmk  ] info: get_config_opt: Found 'no' for
> option: to_syslog
> Jun 07 10:48:49 corosync [pcmk  ] info: process_ais_conf: User
> configured file based logging and explicitly disabled syslog.
> Jun 07 10:48:49 corosync [pcmk  ] info: config_find_init: Local handle:
> 5880381755227111426 for quorum
> Jun 07 10:48:49 corosync [pcmk  ] info: config_find_next: No additional
> configuration supplied for: quorum
> Jun 07 10:48:49 corosync [pcmk  ] info: get_config_opt: No default for
> option: provider
> Jun 07 10:48:49 corosync [pcmk  ] info: config_find_init: Local handle:
> 4835695805891346435 for service
> Jun 07 10:48:49 corosync [pcmk  ] info: config_find_next: Processing
> additional service options...
> Jun 07 10:48:49 corosync [pcmk  ] info: get_config_opt: Found '1' for
> option: ver
> Jun 07 10:48:49 corosync [pcmk  ] info: process_ais_conf: Enabling MCP
> mode: Use the Pacemaker init script to complete Pacemaker startup
> Jun 07 10:48:49 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'pcmk' for option: clustername
> Jun 07 10:48:49 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'no' for option: use_logd
> Jun 07 10:48:49 corosync [pcmk  ] info: get_config_opt: Defaulting to
> 'no' for option: use_mgmtd
> Jun 07 10:48:49 corosync [pcmk  ] info: pcmk_startup: CRM: Initialized
> Jun 07 10:48:49 corosync [pcmk  ] Logging: Initialized pcmk_startup
> Jun 07 10:48:49 corosync [pcmk  ] info: pcmk_startup: Maximum core file
> size is: 18446744073709551615
> Jun 07 10:48:49 corosync [pcmk  ] info: pcmk_startup: Service: 9
> Jun 07 10:48:49 corosync [pcmk  ] info: pcmk_startup: Local hostname:
> vm2883711991
> Jun 07 10:48:49 corosync [pcmk  ] info: pcmk_update_nodeid: Local node
> id: 184555180
> Jun 07 10:48:49 corosync [pcmk  ] info: update_member: Creating entry
> for node 184555180 born on 0
> Jun 07 10:48:49 corosync [pcmk  ] info: update_member: 0x1576f20 Node
> 184555180 now known as vm2883711991 (was: (null))
> Jun 07 10:48:49 corosync [pcmk  ] info: update_member: Node vm2883711991
> now has 1 quorum votes (was 0)
> Jun 07 10:48:49 corosync [pcmk  ] info: update_member: Node
> 184555180/vm2883711991 is now: member
> Jun 07 10:48:49 corosync [SERV  ] Service engine loaded: Pacemaker
> Cluster Manager 1.1.14-8.el6_8.1
> Jun 07 10:48:49 corosync [SERV  ] Service engine loaded: corosync
> extended virtual synchrony service
> Jun 07 10:48:49 corosync [SERV  ] Service engine loaded: corosync
> configuration service
> Jun 07 10:48:49 corosync [SERV  ] Service engine loaded: corosync
> cluster closed process group service v1.01
> Jun 07 10:48:49 corosync [SERV  ] Service engine loaded: corosync
> cluster config database access v1.01
> Jun 07 10:48:49 corosync [SERV  ] Service engine loaded: corosync
> profile loading service
> Jun 07 10:48:49 corosync [SERV  ] Service engine loaded: corosync
> cluster quorum service v0.1
> Jun 07 10:48:49 corosync [MAIN  ] Compatibility mode set to whitetank. 
> Using V1 and V2 of the synchronization engine.
> Jun 07 10:48:49 corosync [TOTEM ] adding new UDPU member {172.22.0.4}
> Jun 07 10:48:49 corosync [TOTEM ] adding new UDPU member {172.22.0.11}
> Jun 07 10:48:49 corosync [TOTEM ] adding new UDPU member {172.22.0.13}
> Jun 07 10:49:16 corosync [pcmk  ] notice: pcmk_peer_update: Transitional
> membership event on ring 78140: memb=0, new=0, lost=0
> Jun 07 10:49:16 corosync [pcmk  ] notice: pcmk_peer_update: Stable
> membership event on ring 78140: memb=1, new=1, lost=0
> Jun 07 10:49:16 corosync [pcmk  ] info: pcmk_peer_update: NEW: 
> vm2883711991 184555180
> Jun 07 10:49:16 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> vm2883711991 184555180
> Jun 07 10:49:16 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> Jun 07 10:49:28 corosync [pcmk  ] notice: pcmk_peer_update: Transitional
> membership event on ring 78148: memb=1, new=0, lost=0
> Jun 07 10:49:28 corosync [pcmk  ] info: pcmk_peer_update: memb:
> vm2883711991 184555180
> Jun 07 10:49:28 corosync [pcmk  ] notice: pcmk_peer_update: Stable
> membership event on ring 78148: memb=1, new=0, lost=0
> Jun 07 10:49:28 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> vm2883711991 184555180
> Jun 07 10:49:28 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> Jun 07 10:49:40 corosync [pcmk  ] notice: pcmk_peer_update: Transitional
> membership event on ring 78152: memb=1, new=0, lost=0
> Jun 07 10:49:40 corosync [pcmk  ] info: pcmk_peer_update: memb:
> vm2883711991 184555180
> Jun 07 10:49:40 corosync [pcmk  ] notice: pcmk_peer_update: Stable
> membership event on ring 78152: memb=1, new=0, lost=0
> Jun 07 10:49:40 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> vm2883711991 184555180
> Jun 07 10:49:40 corosync [pcmk  ] info: update_member: 0x1576f20 Node
> 184555180 ((null)) born on: 78140
> Jun 07 10:49:40 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> Jun 07 10:49:40 corosync [CPG   ] chosen downlist: sender r(0)
> ip(172.22.0.11) ; members(old:0 left:0)
> Jun 07 10:49:40 corosync [MAIN  ] Completed service synchronization,
> ready to provide service.
> Jun 07 10:49:52 corosync [pcmk  ] notice: pcmk_peer_update: Transitional
> membership event on ring 78160: memb=1, new=0, lost=0
> Jun 07 10:49:52 corosync [pcmk  ] info: pcmk_peer_update: memb:
> vm2883711991 184555180
> Jun 07 10:49:52 corosync [pcmk  ] notice: pcmk_peer_update: Stable
> membership event on ring 78160: memb=1, new=0, lost=0
> Jun 07 10:49:52 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> vm2883711991 184555180
> Jun 07 10:49:52 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> 
> 
> 
> On Thu, Jun 7, 2018 at 4:01 PM, Prasad Nagaraj
> <prasad.nagaraj76 at gmail.com <mailto:prasad.nagaraj76 at gmail.com>> wrote:
> 
>     Hi Christine -
> 
>     Thanks for looking into this and here are the details.
>     All the nodes are pingable from each other and actively exchanging
>     corosync packets from each other as seen from tcpdump
> 
>     Here is the ifconfig out from each of the node
>     # ifconfig
>     eth0      Link encap:Ethernet  HWaddr 00:0D:3A:03:35:64
>               inet addr:172.22.0.4  Bcast:172.22.0.255  Mask:255.255.255.0
>               inet6 addr: fe80::20d:3aff:fe03:3564/64 Scope:Link
>               UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>               RX packets:3721169 errors:0 dropped:0 overruns:0 frame:0
>               TX packets:2780455 errors:0 dropped:0 overruns:0 carrier:0
>               collisions:0 txqueuelen:1000
>               RX bytes:1229505889 (1.1 GiB)  TX bytes:982021535 (936.5 MiB)
> 
>     lo        Link encap:Local Loopback
>               inet addr:127.0.0.1  Mask:255.0.0.0
>               inet6 addr: ::1/128 Scope:Host
>               UP LOOPBACK RUNNING  MTU:65536  Metric:1
>               RX packets:1367018 errors:0 dropped:0 overruns:0 frame:0
>               TX packets:1367018 errors:0 dropped:0 overruns:0 carrier:0
>               collisions:0 txqueuelen:0
>               RX bytes:459591075 (438.3 MiB)  TX bytes:459591075 (438.3 MiB)
>     -----------------------------------------------------------------------------------------------------
>     # ifconfig
>     eth0      Link encap:Ethernet  HWaddr 00:0D:3A:03:38:D7
>               inet addr:172.22.0.11  Bcast:172.22.0.255  Mask:255.255.255.0
>               inet6 addr: fe80::20d:3aff:fe03:38d7/64 Scope:Link
>               UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>               RX packets:4052226 errors:0 dropped:0 overruns:0 frame:0
>               TX packets:3744671 errors:0 dropped:0 overruns:0 carrier:0
>               collisions:0 txqueuelen:1000
>               RX bytes:1930027786 (1.7 GiB)  TX bytes:1180930029 (1.0 GiB)
> 
>     lo        Link encap:Local Loopback
>               inet addr:127.0.0.1  Mask:255.0.0.0
>               inet6 addr: ::1/128 Scope:Host
>               UP LOOPBACK RUNNING  MTU:65536  Metric:1
>               RX packets:1394930 errors:0 dropped:0 overruns:0 frame:0
>               TX packets:1394930 errors:0 dropped:0 overruns:0 carrier:0
>               collisions:0 txqueuelen:0
>               RX bytes:508170210 (484.6 MiB)  TX bytes:508170210 (484.6 MiB)
>     --------------------------------------------------------------------------------------------------
> 
>     # ifconfig
>     eth0      Link encap:Ethernet  HWaddr 00:0D:3A:04:06:F6
>               inet addr:172.22.0.13  Bcast:172.22.0.255  Mask:255.255.255.0
>               inet6 addr: fe80::20d:3aff:fe04:6f6/64 Scope:Link
>               UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>               RX packets:3974698 errors:0 dropped:0 overruns:0 frame:0
>               TX packets:3891546 errors:0 dropped:0 overruns:0 carrier:0
>               collisions:0 txqueuelen:1000
>               RX bytes:1903617077 (1.7 GiB)  TX bytes:1234961001 (1.1 GiB)
> 
>     lo        Link encap:Local Loopback
>               inet addr:127.0.0.1  Mask:255.0.0.0
>               inet6 addr: ::1/128 Scope:Host
>               UP LOOPBACK RUNNING  MTU:65536  Metric:1
>               RX packets:1503643 errors:0 dropped:0 overruns:0 frame:0
>               TX packets:1503643 errors:0 dropped:0 overruns:0 carrier:0
>               collisions:0 txqueuelen:0
>               RX bytes:541177718 (516.1 MiB)  TX bytes:541177718 (516.1 MiB)
>     ---------------------------------------------------------------------------------------------------------------
>     tcpdump snippet from each node is as below
>     -------------------------------------------------------------------------------------------------------------
>     ------------------------------------------------------------------------------------------------------------
> 
>     ]# tcpdump udp port 5405
>     tcpdump: verbose output suppressed, use -v or -vv for full protocol
>     decode
>     listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
>     10:21:23.077694 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:21:23.084548 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:21:23.095320 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:21:23.095334 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:21:23.144459 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:21:23.144477 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:21:23.155382 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:21:23.155398 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:21:23.204543 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:21:23.204563 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:21:23.205786 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:21:23.206175 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:21:23.264298 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:21:23.264322 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:21:23.264954 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:21:23.265077 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:21:23.323183 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:21:23.323202 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:21:23.324683 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:21:23.324696 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:21:23.382816 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:21:23.382834 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:21:23.383349 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:21:23.383802 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     ------------------------------------------------------------------------------------------------------------------
> 
>      tcpdump udp port 5405
>     tcpdump: verbose output suppressed, use -v or -vv for full protocol
>     decode
>     listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
>     10:23:16.891854 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:23:16.891871 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:23:16.891927 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:23:16.891939 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:23:16.913586 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP,
>     length 156
>     10:23:16.945218 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:23:16.945236 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:23:16.945305 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:23:16.945326 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:23:16.972761 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP,
>     length 156
>     10:23:17.004903 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:23:17.004919 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:23:17.004928 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:23:17.004938 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:23:17.031846 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP,
>     length 156
>     10:23:17.059219 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:23:17.059306 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:23:17.059317 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:23:17.059941 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:23:17.091094 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP,
>     length 156
>     10:23:17.116600 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:23:17.116662 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:23:17.116672 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP,
>     length 376
>     10:23:17.117347 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:23:17.140960 IP 172.22.0.8.50438 > 172.22.0.4.netsupport: UDP,
>     length 82
>     10:23:17.141319 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP,
>     length 156
>     10:23:17.176250 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     ---------------------------------------------------------------------------------------------------------------------
>     # tcpdump udp port 5405
>     tcpdump: verbose output suppressed, use -v or -vv for full protocol
>     decode
>     listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
>     10:25:30.562549 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:25:30.562576 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:25:30.588697 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:25:30.588728 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:25:30.619383 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:25:30.624817 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:25:30.647937 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:25:30.647968 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:25:30.672207 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:25:30.684604 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:25:30.707733 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:25:30.707760 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:25:30.731354 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:25:30.744345 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:25:30.767456 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:25:30.767483 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:25:30.791532 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:25:30.804432 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:25:30.827539 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:25:30.827563 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:25:30.850832 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:25:30.863531 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:25:30.886664 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:25:30.886691 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:25:30.910820 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:25:30.923403 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:25:30.946507 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:25:30.946531 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:25:30.970931 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
>     10:25:30.983055 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP,
>     length 332
>     10:25:31.006306 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP,
>     length 332
>     10:25:31.006339 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP,
>     length 332
>     10:25:31.030207 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP,
>     length 376
> 
>     ----------------------------------------------------------------------------------------------------------------
>     And here is the lsof output for each node.
>     lsof -i | grep corosync
>     corosync  47873      root   10u  IPv4 1193147      0t0  UDP
>     172.22.0.4:netsupport
>     corosync  47873      root   13u  IPv4 1193151      0t0  UDP
>     172.22.0.4:45846 <http://172.22.0.4:45846>
>     corosync  47873      root   14u  IPv4 1193152      0t0  UDP
>     172.22.0.4:34060 <http://172.22.0.4:34060>
>     corosync  47873      root   15u  IPv4 1193153      0t0  UDP
>     172.22.0.4:40755 <http://172.22.0.4:40755>
> 
>     lsof -i | grep corosync
>     corosync  11039      root   10u  IPv4   54862      0t0  UDP
>     172.22.0.13:netsupport
>     corosync  11039      root   13u  IPv4   54869      0t0  UDP
>     172.22.0.13:50468 <http://172.22.0.13:50468>
>     corosync  11039      root   14u  IPv4   54870      0t0  UDP
>     172.22.0.13:57332 <http://172.22.0.13:57332>
>     corosync  11039      root   15u  IPv4   54871      0t0  UDP
>     172.22.0.13:46460 <http://172.22.0.13:46460>
> 
>      lsof -i | grep corosync
>     corosync  75188      root   10u  IPv4 1582737      0t0  UDP
>     172.22.0.11:netsupport
>     corosync  75188      root   13u  IPv4 1582741      0t0  UDP
>     172.22.0.11:54545 <http://172.22.0.11:54545>
>     corosync  75188      root   14u  IPv4 1582742      0t0  UDP
>     172.22.0.11:53213 <http://172.22.0.11:53213>
>     corosync  75188      root   15u  IPv4 1582743      0t0  UDP
>     172.22.0.11:44864 <http://172.22.0.11:44864>
> 
> 
>     Thanks!
> 
> 
> 
>     On Thu, Jun 7, 2018 at 3:33 PM, Christine Caulfield
>     <ccaulfie at redhat.com <mailto:ccaulfie at redhat.com>> wrote:
> 
>         On 07/06/18 09:21, Prasad Nagaraj wrote:
>         > Hi - I am running corosync on  3 nodes of CentOS release 6.9 (Final).
>         > Corosync version is  corosync-1.4.7.
>         > The nodes are not seeing each other and not able to form memberships.
>         > What I see is continuous message about " A processor joined or left the
>         > membership and a new membership was formed."
>         > For example:on node:  vm2883711991 
>         > 
> 
>         I can't draw any conclusions from the logs, we'd need to see what
>         corosync though it was binding to and the IP addresses of the hosts.
> 
>         Have a look at the start of the logs and see if they match what
>         you'd
>         expect (ie are similar to the ones on the working clusters),
>         Also check
>         using lsof, to see what addresses corosync is bound to. tcpdump
>         on port
>         5405 will show you if traffic is leaving the nodes and being
>         received.
> 
>         Also check firewall settings and make sure the nodes can ping
>         each other.
> 
>         If you're still stumped them feel free to post more info here
>         for us to
>         look at, though if you have that configuration working on other
>         nodes it
>         might be something in your environment
> 
>         Chrissie
> 
> 
>         >
>         > Jun 07 07:54:52 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vm2883711991 184555180
>         > Jun 07 07:54:52 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:54:52 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.11) ; members(old:1 left:0)
>         > Jun 07 07:54:52 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         > Jun 07 07:55:04 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71084: memb=1, new=0, lost=0
>         > Jun 07 07:55:04 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vm2883711991 184555180
>         > Jun 07 07:55:04 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71084: memb=1, new=0, lost=0
>         > Jun 07 07:55:04 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vm2883711991 184555180
>         > Jun 07 07:55:04 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:55:04 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.11) ; members(old:1 left:0)
>         > Jun 07 07:55:04 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         > Jun 07 07:55:16 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71096: memb=1, new=0, lost=0
>         > Jun 07 07:55:16 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vm2883711991 184555180
>         > Jun 07 07:55:16 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71096: memb=1, new=0, lost=0
>         > Jun 07 07:55:16 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vm2883711991 184555180
>         > Jun 07 07:55:16 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:55:16 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.11) ; members(old:1 left:0)
>         > Jun 07 07:55:16 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         > Jun 07 07:55:28 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71108: memb=1, new=0, lost=0
>         > Jun 07 07:55:28 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vm2883711991 184555180
>         > Jun 07 07:55:28 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71108: memb=1, new=0, lost=0
>         > Jun 07 07:55:28 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vm2883711991 184555180
>         > Jun 07 07:55:28 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:55:28 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.11) ; members(old:1 left:0)
>         > Jun 07 07:55:28 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         > Jun 07 07:55:40 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71120: memb=1, new=0, lost=0
>         > Jun 07 07:55:40 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vm2883711991 184555180
>         > Jun 07 07:55:40 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71120: memb=1, new=0, lost=0
>         > Jun 07 07:55:40 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vm2883711991 184555180
>         > Jun 07 07:55:40 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:55:40 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.11) ; members(old:1 left:0)
>         > Jun 07 07:55:40 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         > Jun 07 07:55:52 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71132: memb=1, new=0, lost=0
>         > Jun 07 07:55:52 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vm2883711991 184555180
>         > Jun 07 07:55:52 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71132: memb=1, new=0, lost=0
>         > Jun 07 07:55:52 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vm2883711991 184555180
>         > Jun 07 07:55:52 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:55:52 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.11) ; members(old:1 left:0)
>         > Jun 07 07:55:52 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         > Jun 07 07:56:04 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71144: memb=1, new=0, lost=0
>         > Jun 07 07:56:04 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vm2883711991 184555180
>         > Jun 07 07:56:04 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71144: memb=1, new=0, lost=0
>         > Jun 07 07:56:04 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vm2883711991 184555180
>         > Jun 07 07:56:04 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:56:17 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71156: memb=1, new=0, lost=0
>         > Jun 07 07:56:17 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vm2883711991 184555180
>         > Jun 07 07:56:17 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71156: memb=1, new=0, lost=0
>         > Jun 07 07:56:17 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vm2883711991 184555180
>         > Jun 07 07:56:17 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:56:17 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.11) ; members(old:1 left:0)
>         > Jun 07 07:56:17 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         >
>         > Similarly on node:  vme6c95166f0 
>         >
>         > Jun 07 07:55:12 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71088: memb=1, new=0, lost=0
>         > Jun 07 07:55:12 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vme6c95166f0 67114668
>         > Jun 07 07:55:12 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71088: memb=1, new=0, lost=0
>         > Jun 07 07:55:12 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vme6c95166f0 67114668
>         > Jun 07 07:55:12 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:55:12 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.4) ; members(old:1 left:0)
>         > Jun 07 07:55:12 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         > Jun 07 07:55:24 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71100: memb=1, new=0, lost=0
>         > Jun 07 07:55:24 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vme6c95166f0 67114668
>         > Jun 07 07:55:24 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71100: memb=1, new=0, lost=0
>         > Jun 07 07:55:24 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vme6c95166f0 67114668
>         > Jun 07 07:55:24 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:55:24 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.4) ; members(old:1 left:0)
>         > Jun 07 07:55:24 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         > Jun 07 07:55:37 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71112: memb=1, new=0, lost=0
>         > Jun 07 07:55:37 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vme6c95166f0 67114668
>         > Jun 07 07:55:37 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71112: memb=1, new=0, lost=0
>         > Jun 07 07:55:37 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vme6c95166f0 67114668
>         > Jun 07 07:55:37 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:55:49 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71124: memb=1, new=0, lost=0
>         > Jun 07 07:55:49 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vme6c95166f0 67114668
>         > Jun 07 07:55:49 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71124: memb=1, new=0, lost=0
>         > Jun 07 07:55:49 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vme6c95166f0 67114668
>         > Jun 07 07:55:49 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:55:49 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.4) ; members(old:1 left:0)
>         > Jun 07 07:55:49 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         > Jun 07 07:56:02 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71136: memb=1, new=0, lost=0
>         > Jun 07 07:56:02 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vme6c95166f0 67114668
>         > Jun 07 07:56:02 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71136: memb=1, new=0, lost=0
>         > Jun 07 07:56:02 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vme6c95166f0 67114668
>         > Jun 07 07:56:02 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:56:02 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.4) ; members(old:1 left:0)
>         > Jun 07 07:56:02 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         > Jun 07 07:56:14 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71148: memb=1, new=0, lost=0
>         > Jun 07 07:56:14 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vme6c95166f0 67114668
>         > Jun 07 07:56:14 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71148: memb=1, new=0, lost=0
>         > Jun 07 07:56:14 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vme6c95166f0 67114668
>         > Jun 07 07:56:14 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:56:27 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71160: memb=1, new=0, lost=0
>         > Jun 07 07:56:27 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vme6c95166f0 67114668
>         > Jun 07 07:56:27 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71160: memb=1, new=0, lost=0
>         > Jun 07 07:56:27 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vme6c95166f0 67114668
>         > Jun 07 07:56:27 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:56:39 corosync [pcmk  ] notice: pcmk_peer_update:
>         Transitional
>         > membership event on ring 71172: memb=1, new=0, lost=0
>         > Jun 07 07:56:39 corosync [pcmk  ] info: pcmk_peer_update: memb:
>         > vme6c95166f0 67114668
>         > Jun 07 07:56:39 corosync [pcmk  ] notice: pcmk_peer_update: Stable
>         > membership event on ring 71172: memb=1, new=0, lost=0
>         > Jun 07 07:56:39 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
>         > vme6c95166f0 67114668
>         > Jun 07 07:56:39 corosync [TOTEM ] A processor joined or left the
>         > membership and a new membership was formed.
>         > Jun 07 07:56:39 corosync [CPG   ] chosen downlist: sender r(0)
>         > ip(172.22.0.4) ; members(old:1 left:0)
>         > Jun 07 07:56:39 corosync [MAIN  ] Completed service
>         synchronization,
>         > ready to provide service.
>         >
>         > and similar message on the third node as well.
>         >
>         > my corosync.conf is as follows:
>         > compatibility: whitetank
>         > totem {
>         >     version: 2
>         >     secauth: off
>         >     threads: 0
>         >     interface {
>         >     member {
>         >             memberaddr: 172.22.0.4
>         >         }
>         > member {
>         >             memberaddr: 172.22.0.11
>         >         }
>         > member {
>         >             memberaddr: 172.22.0.13
>         >         }
>         >
>         >     bindnetaddr: 172.22.0.4
>         >
>         >     ringnumber: 0
>         >     mcastport: 5405
>         >     ttl: 1
>         >     }
>         >     transport: udpu
>         >     token: 10000
>         >     token_retransmits_before_loss_const: 10
>         > }
>         >
>         > logging {
>         >     fileline: off
>         >     to_stderr: yes
>         >     to_logfile: yes
>         >     to_syslog: no
>         >     logfile: /var/log/cluster/corosync.log
>         >     timestamp: on
>         >     logger_subsys {
>         >     subsys: AMF
>         >     debug: on
>         >     }
>         > }
>         > service {
>         >     name: pacemaker
>         >     ver: 1
>         > }
>         > amf {
>         >     mode: disabled
>         > }
>         >
>         > In general , this configuration has worked for me in other
>         clusters I
>         > have, but this particular setup is running into this issue.
>         Request help
>         > on how to debug and resolve this condition.
>         >
>         > Thanks
>         >
>         >
>         >
>         > _______________________________________________
>         > Users mailing list: Users at clusterlabs.org
>         <mailto:Users at clusterlabs.org>
>         > https://lists.clusterlabs.org/mailman/listinfo/users
>         <https://lists.clusterlabs.org/mailman/listinfo/users>
>         >
>         > Project Home: http://www.clusterlabs.org
>         > Getting started:
>         http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>         <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>         > Bugs: http://bugs.clusterlabs.org
>         >
> 
>         _______________________________________________
>         Users mailing list: Users at clusterlabs.org
>         <mailto:Users at clusterlabs.org>
>         https://lists.clusterlabs.org/mailman/listinfo/users
>         <https://lists.clusterlabs.org/mailman/listinfo/users>
> 
>         Project Home: http://www.clusterlabs.org
>         Getting started:
>         http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>         <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>         Bugs: http://bugs.clusterlabs.org
> 
> 
> 
> 
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 




More information about the Users mailing list