<div dir="ltr">Hi Christine -<div><br></div><div>Thanks for looking into this and here are the details.</div><div>All the nodes are pingable from each other and actively exchanging corosync packets from each other as seen from tcpdump</div><div><br></div><div>Here is the ifconfig out from each of the node</div><div><div># ifconfig</div><div>eth0 Link encap:Ethernet HWaddr 00:0D:3A:03:35:64</div><div> inet addr:172.22.0.4 Bcast:172.22.0.255 Mask:255.255.255.0</div><div> inet6 addr: fe80::20d:3aff:fe03:3564/64 Scope:Link</div><div> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1</div><div> RX packets:3721169 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:2780455 errors:0 dropped:0 overruns:0 carrier:0</div><div> collisions:0 txqueuelen:1000</div><div> RX bytes:1229505889 (1.1 GiB) TX bytes:982021535 (936.5 MiB)</div><div><br></div><div>lo Link encap:Local Loopback</div><div> inet addr:127.0.0.1 Mask:255.0.0.0</div><div> inet6 addr: ::1/128 Scope:Host</div><div> UP LOOPBACK RUNNING MTU:65536 Metric:1</div><div> RX packets:1367018 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:1367018 errors:0 dropped:0 overruns:0 carrier:0</div><div> collisions:0 txqueuelen:0</div><div> RX bytes:459591075 (438.3 MiB) TX bytes:459591075 (438.3 MiB)</div></div><div>-----------------------------------------------------------------------------------------------------</div><div><div># ifconfig</div><div>eth0 Link encap:Ethernet HWaddr 00:0D:3A:03:38:D7</div><div> inet addr:172.22.0.11 Bcast:172.22.0.255 Mask:255.255.255.0</div><div> inet6 addr: fe80::20d:3aff:fe03:38d7/64 Scope:Link</div><div> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1</div><div> RX packets:4052226 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:3744671 errors:0 dropped:0 overruns:0 carrier:0</div><div> collisions:0 txqueuelen:1000</div><div> RX bytes:1930027786 (1.7 GiB) TX bytes:1180930029 (1.0 GiB)</div><div><br></div><div>lo Link encap:Local Loopback</div><div> inet addr:127.0.0.1 Mask:255.0.0.0</div><div> inet6 addr: ::1/128 Scope:Host</div><div> UP LOOPBACK RUNNING MTU:65536 Metric:1</div><div> RX packets:1394930 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:1394930 errors:0 dropped:0 overruns:0 carrier:0</div><div> collisions:0 txqueuelen:0</div><div> RX bytes:508170210 (484.6 MiB) TX bytes:508170210 (484.6 MiB)</div></div><div>--------------------------------------------------------------------------------------------------</div><div><br></div><div><div># ifconfig</div><div>eth0 Link encap:Ethernet HWaddr 00:0D:3A:04:06:F6</div><div> inet addr:172.22.0.13 Bcast:172.22.0.255 Mask:255.255.255.0</div><div> inet6 addr: fe80::20d:3aff:fe04:6f6/64 Scope:Link</div><div> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1</div><div> RX packets:3974698 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:3891546 errors:0 dropped:0 overruns:0 carrier:0</div><div> collisions:0 txqueuelen:1000</div><div> RX bytes:1903617077 (1.7 GiB) TX bytes:1234961001 (1.1 GiB)</div><div><br></div><div>lo Link encap:Local Loopback</div><div> inet addr:127.0.0.1 Mask:255.0.0.0</div><div> inet6 addr: ::1/128 Scope:Host</div><div> UP LOOPBACK RUNNING MTU:65536 Metric:1</div><div> RX packets:1503643 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:1503643 errors:0 dropped:0 overruns:0 carrier:0</div><div> collisions:0 txqueuelen:0</div><div> RX bytes:541177718 (516.1 MiB) TX bytes:541177718 (516.1 MiB)</div></div><div>---------------------------------------------------------------------------------------------------------------</div><div>tcpdump snippet from each node is as below</div><div>-------------------------------------------------------------------------------------------------------------</div><div>------------------------------------------------------------------------------------------------------------</div><div><br></div><div><div>]# tcpdump udp port 5405</div><div>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode</div><div>listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes</div><div>10:21:23.077694 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:21:23.084548 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:21:23.095320 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:21:23.095334 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:21:23.144459 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:21:23.144477 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:21:23.155382 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:21:23.155398 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:21:23.204543 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:21:23.204563 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:21:23.205786 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:21:23.206175 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:21:23.264298 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:21:23.264322 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:21:23.264954 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:21:23.265077 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:21:23.323183 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:21:23.323202 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:21:23.324683 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:21:23.324696 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:21:23.382816 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:21:23.382834 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:21:23.383349 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:21:23.383802 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div></div><div>------------------------------------------------------------------------------------------------------------------</div><div><br></div><div><div> tcpdump udp port 5405</div><div>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode</div><div>listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes</div><div>10:23:16.891854 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:23:16.891871 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:23:16.891927 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:23:16.891939 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:23:16.913586 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP, length 156</div><div>10:23:16.945218 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:23:16.945236 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:23:16.945305 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:23:16.945326 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:23:16.972761 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP, length 156</div><div>10:23:17.004903 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:23:17.004919 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:23:17.004928 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:23:17.004938 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:23:17.031846 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP, length 156</div><div>10:23:17.059219 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:23:17.059306 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:23:17.059317 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:23:17.059941 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:23:17.091094 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP, length 156</div><div>10:23:17.116600 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:23:17.116662 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:23:17.116672 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, length 376</div><div>10:23:17.117347 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:23:17.140960 IP 172.22.0.8.50438 > 172.22.0.4.netsupport: UDP, length 82</div><div>10:23:17.141319 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP, length 156</div><div>10:23:17.176250 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div></div><div>---------------------------------------------------------------------------------------------------------------------</div><div><div># tcpdump udp port 5405</div><div>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode</div><div>listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes</div><div>10:25:30.562549 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:25:30.562576 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:25:30.588697 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:25:30.588728 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:25:30.619383 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:25:30.624817 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:25:30.647937 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:25:30.647968 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:25:30.672207 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:25:30.684604 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:25:30.707733 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:25:30.707760 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:25:30.731354 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:25:30.744345 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:25:30.767456 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:25:30.767483 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:25:30.791532 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:25:30.804432 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:25:30.827539 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:25:30.827563 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:25:30.850832 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:25:30.863531 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:25:30.886664 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:25:30.886691 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:25:30.910820 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:25:30.923403 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:25:30.946507 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:25:30.946531 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:25:30.970931 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div><div>10:25:30.983055 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, length 332</div><div>10:25:31.006306 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, length 332</div><div>10:25:31.006339 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, length 332</div><div>10:25:31.030207 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, length 376</div></div><div><br></div><div>----------------------------------------------------------------------------------------------------------------</div><div>And here is the lsof output for each node.</div><div><div>lsof -i | grep corosync</div><div>corosync 47873 root 10u IPv4 1193147 0t0 UDP 172.22.0.4:netsupport</div><div>corosync 47873 root 13u IPv4 1193151 0t0 UDP <a href="http://172.22.0.4:45846">172.22.0.4:45846</a></div><div>corosync 47873 root 14u IPv4 1193152 0t0 UDP <a href="http://172.22.0.4:34060">172.22.0.4:34060</a></div><div>corosync 47873 root 15u IPv4 1193153 0t0 UDP <a href="http://172.22.0.4:40755">172.22.0.4:40755</a></div></div><div><br></div><div><div>lsof -i | grep corosync</div><div>corosync 11039 root 10u IPv4 54862 0t0 UDP 172.22.0.13:netsupport</div><div>corosync 11039 root 13u IPv4 54869 0t0 UDP <a href="http://172.22.0.13:50468">172.22.0.13:50468</a></div><div>corosync 11039 root 14u IPv4 54870 0t0 UDP <a href="http://172.22.0.13:57332">172.22.0.13:57332</a></div><div>corosync 11039 root 15u IPv4 54871 0t0 UDP <a href="http://172.22.0.13:46460">172.22.0.13:46460</a></div></div><div><br></div><div><div> lsof -i | grep corosync</div><div>corosync 75188 root 10u IPv4 1582737 0t0 UDP 172.22.0.11:netsupport</div><div>corosync 75188 root 13u IPv4 1582741 0t0 UDP <a href="http://172.22.0.11:54545">172.22.0.11:54545</a></div><div>corosync 75188 root 14u IPv4 1582742 0t0 UDP <a href="http://172.22.0.11:53213">172.22.0.11:53213</a></div><div>corosync 75188 root 15u IPv4 1582743 0t0 UDP <a href="http://172.22.0.11:44864">172.22.0.11:44864</a></div></div><div><br></div><div><br></div><div>Thanks!</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 7, 2018 at 3:33 PM, Christine Caulfield <span dir="ltr"><<a href="mailto:ccaulfie@redhat.com" target="_blank">ccaulfie@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 07/06/18 09:21, Prasad Nagaraj wrote:<br>
> Hi - I am running corosync on 3 nodes of CentOS release 6.9 (Final).<br>
> Corosync version is corosync-1.4.7.<br>
> The nodes are not seeing each other and not able to form memberships.<br>
> What I see is continuous message about " A processor joined or left the<br>
> membership and a new membership was formed."<br>
> For example:on node: vm2883711991 <br>
> <br>
<br>
</span>I can't draw any conclusions from the logs, we'd need to see what<br>
corosync though it was binding to and the IP addresses of the hosts.<br>
<br>
Have a look at the start of the logs and see if they match what you'd<br>
expect (ie are similar to the ones on the working clusters), Also check<br>
using lsof, to see what addresses corosync is bound to. tcpdump on port<br>
5405 will show you if traffic is leaving the nodes and being received.<br>
<br>
Also check firewall settings and make sure the nodes can ping each other.<br>
<br>
If you're still stumped them feel free to post more info here for us to<br>
look at, though if you have that configuration working on other nodes it<br>
might be something in your environment<br>
<br>
Chrissie<br>
<div><div class="h5"><br>
<br>
> <br>
> Jun 07 07:54:52 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vm2883711991 184555180<br>
> Jun 07 07:54:52 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:54:52 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.11) ; members(old:1 left:0)<br>
> Jun 07 07:54:52 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> Jun 07 07:55:04 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71084: memb=1, new=0, lost=0<br>
> Jun 07 07:55:04 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vm2883711991 184555180<br>
> Jun 07 07:55:04 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71084: memb=1, new=0, lost=0<br>
> Jun 07 07:55:04 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vm2883711991 184555180<br>
> Jun 07 07:55:04 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:55:04 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.11) ; members(old:1 left:0)<br>
> Jun 07 07:55:04 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> Jun 07 07:55:16 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71096: memb=1, new=0, lost=0<br>
> Jun 07 07:55:16 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vm2883711991 184555180<br>
> Jun 07 07:55:16 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71096: memb=1, new=0, lost=0<br>
> Jun 07 07:55:16 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vm2883711991 184555180<br>
> Jun 07 07:55:16 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:55:16 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.11) ; members(old:1 left:0)<br>
> Jun 07 07:55:16 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> Jun 07 07:55:28 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71108: memb=1, new=0, lost=0<br>
> Jun 07 07:55:28 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vm2883711991 184555180<br>
> Jun 07 07:55:28 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71108: memb=1, new=0, lost=0<br>
> Jun 07 07:55:28 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vm2883711991 184555180<br>
> Jun 07 07:55:28 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:55:28 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.11) ; members(old:1 left:0)<br>
> Jun 07 07:55:28 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> Jun 07 07:55:40 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71120: memb=1, new=0, lost=0<br>
> Jun 07 07:55:40 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vm2883711991 184555180<br>
> Jun 07 07:55:40 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71120: memb=1, new=0, lost=0<br>
> Jun 07 07:55:40 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vm2883711991 184555180<br>
> Jun 07 07:55:40 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:55:40 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.11) ; members(old:1 left:0)<br>
> Jun 07 07:55:40 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> Jun 07 07:55:52 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71132: memb=1, new=0, lost=0<br>
> Jun 07 07:55:52 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vm2883711991 184555180<br>
> Jun 07 07:55:52 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71132: memb=1, new=0, lost=0<br>
> Jun 07 07:55:52 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vm2883711991 184555180<br>
> Jun 07 07:55:52 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:55:52 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.11) ; members(old:1 left:0)<br>
> Jun 07 07:55:52 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> Jun 07 07:56:04 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71144: memb=1, new=0, lost=0<br>
> Jun 07 07:56:04 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vm2883711991 184555180<br>
> Jun 07 07:56:04 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71144: memb=1, new=0, lost=0<br>
> Jun 07 07:56:04 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vm2883711991 184555180<br>
> Jun 07 07:56:04 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:56:17 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71156: memb=1, new=0, lost=0<br>
> Jun 07 07:56:17 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vm2883711991 184555180<br>
> Jun 07 07:56:17 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71156: memb=1, new=0, lost=0<br>
> Jun 07 07:56:17 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vm2883711991 184555180<br>
> Jun 07 07:56:17 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:56:17 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.11) ; members(old:1 left:0)<br>
> Jun 07 07:56:17 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> <br>
> Similarly on node: vme6c95166f0 <br>
> <br>
> Jun 07 07:55:12 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71088: memb=1, new=0, lost=0<br>
> Jun 07 07:55:12 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:55:12 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71088: memb=1, new=0, lost=0<br>
> Jun 07 07:55:12 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:55:12 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:55:12 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.4) ; members(old:1 left:0)<br>
> Jun 07 07:55:12 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> Jun 07 07:55:24 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71100: memb=1, new=0, lost=0<br>
> Jun 07 07:55:24 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:55:24 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71100: memb=1, new=0, lost=0<br>
> Jun 07 07:55:24 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:55:24 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:55:24 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.4) ; members(old:1 left:0)<br>
> Jun 07 07:55:24 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> Jun 07 07:55:37 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71112: memb=1, new=0, lost=0<br>
> Jun 07 07:55:37 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:55:37 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71112: memb=1, new=0, lost=0<br>
> Jun 07 07:55:37 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:55:37 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:55:49 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71124: memb=1, new=0, lost=0<br>
> Jun 07 07:55:49 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:55:49 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71124: memb=1, new=0, lost=0<br>
> Jun 07 07:55:49 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:55:49 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:55:49 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.4) ; members(old:1 left:0)<br>
> Jun 07 07:55:49 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> Jun 07 07:56:02 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71136: memb=1, new=0, lost=0<br>
> Jun 07 07:56:02 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:56:02 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71136: memb=1, new=0, lost=0<br>
> Jun 07 07:56:02 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:56:02 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:56:02 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.4) ; members(old:1 left:0)<br>
> Jun 07 07:56:02 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> Jun 07 07:56:14 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71148: memb=1, new=0, lost=0<br>
> Jun 07 07:56:14 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:56:14 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71148: memb=1, new=0, lost=0<br>
> Jun 07 07:56:14 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:56:14 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:56:27 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71160: memb=1, new=0, lost=0<br>
> Jun 07 07:56:27 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:56:27 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71160: memb=1, new=0, lost=0<br>
> Jun 07 07:56:27 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:56:27 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:56:39 corosync [pcmk ] notice: pcmk_peer_update: Transitional<br>
> membership event on ring 71172: memb=1, new=0, lost=0<br>
> Jun 07 07:56:39 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:56:39 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 71172: memb=1, new=0, lost=0<br>
> Jun 07 07:56:39 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vme6c95166f0 67114668<br>
> Jun 07 07:56:39 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Jun 07 07:56:39 corosync [CPG ] chosen downlist: sender r(0)<br>
> ip(172.22.0.4) ; members(old:1 left:0)<br>
> Jun 07 07:56:39 corosync [MAIN ] Completed service synchronization,<br>
> ready to provide service.<br>
> <br>
> and similar message on the third node as well.<br>
> <br>
> my corosync.conf is as follows:<br>
> compatibility: whitetank<br>
> totem {<br>
> version: 2<br>
> secauth: off<br>
> threads: 0<br>
> interface {<br>
> member {<br>
> memberaddr: 172.22.0.4<br>
> }<br>
> member {<br>
> memberaddr: 172.22.0.11<br>
> }<br>
> member {<br>
> memberaddr: 172.22.0.13<br>
> }<br>
> <br>
> bindnetaddr: 172.22.0.4<br>
> <br>
> ringnumber: 0<br>
> mcastport: 5405<br>
> ttl: 1<br>
> }<br>
> transport: udpu<br>
> token: 10000<br>
> token_retransmits_before_loss_<wbr>const: 10<br>
> }<br>
> <br>
> logging {<br>
> fileline: off<br>
> to_stderr: yes<br>
> to_logfile: yes<br>
> to_syslog: no<br>
> logfile: /var/log/cluster/corosync.log<br>
> timestamp: on<br>
> logger_subsys {<br>
> subsys: AMF<br>
> debug: on<br>
> }<br>
> }<br>
> service {<br>
> name: pacemaker<br>
> ver: 1<br>
> }<br>
> amf {<br>
> mode: disabled<br>
> }<br>
> <br>
> In general , this configuration has worked for me in other clusters I<br>
> have, but this particular setup is running into this issue. Request help<br>
> on how to debug and resolve this condition.<br>
> <br>
> Thanks<br>
> <br>
> <br>
> <br>
</div></div>> ______________________________<wbr>_________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
> <br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
> <br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div><br></div>