<div dir="ltr">Hi Christine - Got it:)<div><br></div><div>I have collected few seconds of debug logs from all nodes after startup.</div><div>Please find them attached.</div><div>Please let me know if this will help us to identify rootcause.</div><div><br></div><div>Thanks!</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 7, 2018 at 8:43 PM, Christine Caulfield <span dir="ltr"><<a href="mailto:ccaulfie@redhat.com" target="_blank">ccaulfie@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 07/06/18 15:53, Prasad Nagaraj wrote:<br>
> Hi - As you can see in the corosync.conf details - i have already kept<br>
> debug: on<br>
> <br>
<br>
</span>But only in the (disabled) AMF subsystem, not for corosync as a whole :)<br>
<span class=""><br>
logger_subsys {<br>
subsys: AMF<br>
debug: on<br>
}<br>
<br>
<br>
</span>Chrissie<br>
<span class=""><br>
<br>
> <br>
> On Thu, 7 Jun 2018, 8:03 pm Christine Caulfield, <<a href="mailto:ccaulfie@redhat.com">ccaulfie@redhat.com</a><br>
</span><div><div class="h5">> <mailto:<a href="mailto:ccaulfie@redhat.com">ccaulfie@redhat.com</a>>> wrote:<br>
> <br>
> On 07/06/18 15:24, Prasad Nagaraj wrote:<br>
> ><br>
> > No iptables or otherwise firewalls are setup on these nodes.<br>
> ><br>
> > One observation is that each node sends messages on with its own ring<br>
> > sequence number which is not converging.. I have seen that in a good<br>
> > cluster, when nodes respond with same sequence number, the<br>
> membership is<br>
> > automatically formed. But in our case, that is not the case.<br>
> ><br>
> <br>
> That's just a side-effect of the cluster not forming. It's not causing<br>
> it. Can you enable full corosync debugging (just add debug:on to the end<br>
> of the logging {} stanza) and see if that has any more useful<br>
> information (I only need the corosync bits, not the pcmk ones)<br>
> <br>
> Chrissie<br>
> <br>
> > Example: we can see that one node sends<br>
> > Jun 07 07:55:04 corosync [pcmk ] notice: pcmk_peer_update:<br>
> Transitional<br>
> > membership event on ring 71084: memb=1, new=0, lost=0<br>
> > .....<br>
> > Jun 07 07:55:16 corosync [pcmk ] notice: pcmk_peer_update:<br>
> Transitional<br>
> > membership event on ring 71096: memb=1, new=0, lost=0<br>
> > Jun 07 07:55:16 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> > membership event on ring 71096: memb=1, new=0, lost=0<br>
> ><br>
> > other node sends messages with its own numbers<br>
> > Jun 07 07:55:12 corosync [pcmk ] notice: pcmk_peer_update:<br>
> Transitional<br>
> > membership event on ring 71088: memb=1, new=0, lost=0<br>
> > Jun 07 07:55:12 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> > membership event on ring 71088: memb=1, new=0, lost=0<br>
> > .......<br>
> > Jun 07 07:55:24 corosync [pcmk ] notice: pcmk_peer_update:<br>
> Transitional<br>
> > membership event on ring 71100: memb=1, new=0, lost=0<br>
> > Jun 07 07:55:24 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> > membership event on ring 71100: memb=1, new=0, lost=0<br>
> ><br>
> > Any idea why this happens, and why the seq. numbers from different<br>
> nodes<br>
> > are not converging ?<br>
> ><br>
> > Thanks!<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > ______________________________<wbr>_________________<br>
> > Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
</div></div>> <mailto:<a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a>><br>
<span class="">> > <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
> ><br>
> > Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> > Getting started:<br>
> <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
> > Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
> ><br>
> <br>
> ______________________________<wbr>_________________<br>
</span>> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a> <mailto:<a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a>><br>
<div class="HOEnZb"><div class="h5">> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
> <br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
> <br>
> <br>
> <br>
> ______________________________<wbr>_________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
> <br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
> <br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>