[ClusterLabs] [ClusterLab] : Corosync not initializing successfully
Nikhil Utane
nikhil.subscribed at gmail.com
Tue May 3 09:34:19 EDT 2016
Thanks for your response Dejan.
I do not know yet whether this has anything to do with endianness.
FWIW, there could be something quirky with the system so keeping all
options open. :)
I added some debug prints to understand what's happening under the hood.
*Success case: (on x86 machine): *
[TOTEM ] entering OPERATIONAL state.
[TOTEM ] A new membership (10.206.1.7:137220) was formed. Members joined:
181272839
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=0,
my_high_delivered=0
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
my_high_delivered=0
[TOTEM ] Delivering 0 to 1
[TOTEM ] Delivering MCAST message with seq 1 to pending delivery queue
[SYNC ] Nikhil: Inside sync_deliver_fn. header->id=1
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=2,
my_high_delivered=1
[TOTEM ] Delivering 1 to 2
[TOTEM ] Delivering MCAST message with seq 2 to pending delivery queue
[SYNC ] Nikhil: Inside sync_deliver_fn. header->id=0
[SYNC ] Nikhil: Entering sync_barrier_handler
[SYNC ] Committing synchronization for corosync configuration map access
.
[TOTEM ] Delivering 2 to 4
[TOTEM ] Delivering MCAST message with seq 3 to pending delivery queue
[TOTEM ] Delivering MCAST message with seq 4 to pending delivery queue
[CPG ] comparing: sender r(0) ip(10.206.1.7) ; members(old:0 left:0)
[CPG ] chosen downlist: sender r(0) ip(10.206.1.7) ; members(old:0 left:0)
[SYNC ] Committing synchronization for corosync cluster closed process
group service v1.01
*[MAIN ] Completed service synchronization, ready to provide service.*
*Failure case: (on ppc)*:
[TOTEM ] entering OPERATIONAL state.
[TOTEM ] A new membership (10.207.24.101:16) was formed. Members joined:
181344357
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=0,
my_high_delivered=0
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
my_high_delivered=0
[TOTEM ] Delivering 0 to 1
[TOTEM ] Delivering MCAST message with seq 1 to pending delivery queue
[SYNC ] Nikhil: Inside sync_deliver_fn header->id=1
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
my_high_delivered=1
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
my_high_delivered=1
Above message repeats continuously.
So it appears that in failure case I do not receive messages with sequence
number 2-4.
If somebody can throw some ideas that'll help a lot.
-Thanks
Nikhil
On Tue, May 3, 2016 at 5:26 PM, Dejan Muhamedagic <dejanmm at fastmail.fm>
wrote:
> Hi,
>
> On Mon, May 02, 2016 at 08:54:09AM +0200, Jan Friesse wrote:
> > >As your hardware is probably capable of running ppcle and if you have an
> > >environment
> > >at hand without too much effort it might pay off to try that.
> > >There are of course distributions out there support corosync on
> > >big-endian architectures
> > >but I don't know if there is an automatized regression for corosync on
> > >big-endian that
> > >would catch big-endian-issues right away with something as current as
> > >your 2.3.5.
> >
> > No we are not testing big-endian.
> >
> > So totally agree with Klaus. Give a try to ppcle. Also make sure all
> > nodes are little-endian. Corosync should work in mixed BE/LE
> > environment but because it's not tested, it may not work (and it's a
> > bug, so if ppcle works I will try to fix BE).
>
> I tested a cluster consisting of big endian/little endian nodes
> (s390 and x86-64), but that was a while ago. IIRC, all relevant
> bugs in corosync got fixed at that time. Don't know what is the
> situation with the latest version.
>
> Thanks,
>
> Dejan
>
> > Regards,
> > Honza
> >
> > >
> > >Regards,
> > >Klaus
> > >
> > >On 05/02/2016 06:44 AM, Nikhil Utane wrote:
> > >>Re-sending as I don't see my post on the thread.
> > >>
> > >>On Sun, May 1, 2016 at 4:21 PM, Nikhil Utane
> > >><nikhil.subscribed at gmail.com <mailto:nikhil.subscribed at gmail.com>>
> wrote:
> > >>
> > >> Hi,
> > >>
> > >> Looking for some guidance here as we are completely blocked
> > >> otherwise :(.
> > >>
> > >> -Regards
> > >> Nikhil
> > >>
> > >> On Fri, Apr 29, 2016 at 6:11 PM, Sriram <sriram.ec at gmail.com
> > >> <mailto:sriram.ec at gmail.com>> wrote:
> > >>
> > >> Corrected the subject.
> > >>
> > >> We went ahead and captured corosync debug logs for our ppc
> board.
> > >> After log analysis and comparison with the sucessful logs(
> > >> from x86 machine) ,
> > >> we didnt find *"[ MAIN ] Completed service synchronization,
> > >> ready to provide service.*" in ppc logs.
> > >> So, looks like corosync is not in a position to accept
> > >> connection from Pacemaker.
> > >> Even I tried with the new corosync.conf with no success.
> > >>
> > >> Any hints on this issue would be really helpful.
> > >>
> > >> Attaching ppc_notworking.log, x86_working.log, corosync.conf.
> > >>
> > >> Regards,
> > >> Sriram
> > >>
> > >>
> > >>
> > >> On Fri, Apr 29, 2016 at 2:44 PM, Sriram <sriram.ec at gmail.com
> > >> <mailto:sriram.ec at gmail.com>> wrote:
> > >>
> > >> Hi,
> > >>
> > >> I went ahead and made some changes in file system(Like I
> > >> brought in /etc/init.d/corosync and /etc/init.d/pacemaker,
> > >> /etc/sysconfig ), After that I was able to run "pcs
> > >> cluster start".
> > >> But it failed with the following error
> > >> # pcs cluster start
> > >> Starting Cluster...
> > >> Starting Pacemaker Cluster Manager[FAILED]
> > >> Error: unable to start pacemaker
> > >>
> > >> And in the /var/log/pacemaker.log, I saw these errors
> > >> pacemakerd: info: mcp_read_config: cmap connection
> > >> setup failed: CS_ERR_TRY_AGAIN. Retrying in 4s
> > >> Apr 29 08:53:47 [15863] node_cu pacemakerd: info:
> > >> mcp_read_config: cmap connection setup failed:
> > >> CS_ERR_TRY_AGAIN. Retrying in 5s
> > >> Apr 29 08:53:52 [15863] node_cu pacemakerd: warning:
> > >> mcp_read_config: Could not connect to Cluster
> > >> Configuration Database API, error 6
> > >> Apr 29 08:53:52 [15863] node_cu pacemakerd: notice:
> > >> main: Could not obtain corosync config data, exiting
> > >> Apr 29 08:53:52 [15863] node_cu pacemakerd: info:
> > >> crm_xml_cleanup: Cleaning up memory from libxml2
> > >>
> > >>
> > >> And in the /var/log/Debuglog, I saw these errors coming
> > >> from corosync
> > >> 20160429 085347.487050 <tel:085347.487050> airv_cu
> > >> daemon.warn corosync[12857]: [QB ] Denied connection,
> > >> is not ready (12857-15863-14)
> > >> 20160429 085347.487067 <tel:085347.487067> airv_cu
> > >> daemon.info <http://daemon.info> corosync[12857]: [QB
> > >> ] Denied connection, is not ready (12857-15863-14)
> > >>
> > >>
> > >> I browsed the code of libqb to find that it is failing in
> > >>
> > >>
> https://github.com/ClusterLabs/libqb/blob/master/lib/ipc_setup.c
> > >>
> > >> Line 600 :
> > >> handle_new_connection function
> > >>
> > >> Line 637:
> > >> if (auth_result == 0 &&
> > >> c->service->serv_fns.connection_accept) {
> > >> res = c->service->serv_fns.connection_accept(c,
> > >> c->euid, c->egid);
> > >> }
> > >> if (res != 0) {
> > >> goto send_response;
> > >> }
> > >>
> > >> Any hints on this issue would be really helpful for me to
> > >> go ahead.
> > >> Please let me know if any logs are required,
> > >>
> > >> Regards,
> > >> Sriram
> > >>
> > >> On Thu, Apr 28, 2016 at 2:42 PM, Sriram
> > >> <sriram.ec at gmail.com <mailto:sriram.ec at gmail.com>> wrote:
> > >>
> > >> Thanks Ken and Emmanuel.
> > >> Its a big endian machine. I will try with running "pcs
> > >> cluster setup" and "pcs cluster start"
> > >> Inside cluster.py, "service pacemaker start" and
> > >> "service corosync start" are executed to bring up
> > >> pacemaker and corosync.
> > >> Those service scripts and the infrastructure needed to
> > >> bring up the processes in the above said manner
> > >> doesn't exist in my board.
> > >> As it is a embedded board with the limited memory,
> > >> full fledged linux is not installed.
> > >> Just curious to know, what could be reason the
> > >> pacemaker throws that error.
> > >>
> > >> /"cmap connection setup failed: CS_ERR_TRY_AGAIN.
> > >> Retrying in 1s"
> > >>
> > >> /
> > >> Thanks for response.
> > >>
> > >> Regards,
> > >> Sriram.
> > >>
> > >> On Thu, Apr 28, 2016 at 8:55 AM, Ken Gaillot
> > >> <kgaillot at redhat.com <mailto:kgaillot at redhat.com>>
> wrote:
> > >>
> > >> On 04/27/2016 11:25 AM, emmanuel segura wrote:
> > >> > you need to use pcs to do everything, pcs
> > >> cluster setup and pcs
> > >> > cluster start, try to use the redhat docs for
> > >> more information.
> > >>
> > >> Agreed -- pcs cluster setup will create a proper
> > >> corosync.conf for you.
> > >> Your corosync.conf below uses corosync 1 syntax,
> > >> and there were
> > >> significant changes in corosync 2. In particular,
> > >> you don't need the
> > >> file created in step 4, because pacemaker is no
> > >> longer launched via a
> > >> corosync plugin.
> > >>
> > >> > 2016-04-27 17:28 GMT+02:00 Sriram
> > >> <sriram.ec at gmail.com <mailto:sriram.ec at gmail.com
> >>:
> > >> >> Dear All,
> > >> >>
> > >> >> I m trying to use pacemaker and corosync for
> > >> the clustering requirement that
> > >> >> came up recently.
> > >> >> We have cross compiled corosync, pacemaker and
> > >> pcs(python) for ppc
> > >> >> environment (Target board where pacemaker and
> > >> corosync are supposed to run)
> > >> >> I m having trouble bringing up pacemaker in
> > >> that environment, though I could
> > >> >> successfully bring up corosync.
> > >> >> Any help is welcome.
> > >> >>
> > >> >> I m using these versions of pacemaker and
> corosync
> > >> >> [root at node_cu pacemaker]# corosync -v
> > >> >> Corosync Cluster Engine, version '2.3.5'
> > >> >> Copyright (c) 2006-2009 Red Hat, Inc.
> > >> >> [root at node_cu pacemaker]# pacemakerd -$
> > >> >> Pacemaker 1.1.14
> > >> >> Written by Andrew Beekhof
> > >> >>
> > >> >> For running corosync, I did the following.
> > >> >> 1. Created the following directories,
> > >> >> /var/lib/pacemaker
> > >> >> /var/lib/corosync
> > >> >> /var/lib/pacemaker
> > >> >> /var/lib/pacemaker/cores
> > >> >> /var/lib/pacemaker/pengine
> > >> >> /var/lib/pacemaker/blackbox
> > >> >> /var/lib/pacemaker/cib
> > >> >>
> > >> >>
> > >> >> 2. Created a file called corosync.conf under
> > >> /etc/corosync folder with the
> > >> >> following contents
> > >> >>
> > >> >> totem {
> > >> >>
> > >> >> version: 2
> > >> >> token: 5000
> > >> >> token_retransmits_before_loss_const: 20
> > >> >> join: 1000
> > >> >> consensus: 7500
> > >> >> vsftype: none
> > >> >> max_messages: 20
> > >> >> secauth: off
> > >> >> cluster_name: mycluster
> > >> >> transport: udpu
> > >> >> threads: 0
> > >> >> clear_node_high_bit: yes
> > >> >>
> > >> >> interface {
> > >> >> ringnumber: 0
> > >> >> # The following three values
> > >> need to be set based on your
> > >> >> environment
> > >> >> bindnetaddr: 10.x.x.x
> > >> >> mcastaddr: 226.94.1.1
> > >> >> mcastport: 5405
> > >> >> }
> > >> >> }
> > >> >>
> > >> >> logging {
> > >> >> fileline: off
> > >> >> to_syslog: yes
> > >> >> to_stderr: no
> > >> >> to_syslog: yes
> > >> >> logfile: /var/log/corosync.log
> > >> >> syslog_facility: daemon
> > >> >> debug: on
> > >> >> timestamp: on
> > >> >> }
> > >> >>
> > >> >> amf {
> > >> >> mode: disabled
> > >> >> }
> > >> >>
> > >> >> quorum {
> > >> >> provider: corosync_votequorum
> > >> >> }
> > >> >>
> > >> >> nodelist {
> > >> >> node {
> > >> >> ring0_addr: node_cu
> > >> >> nodeid: 1
> > >> >> }
> > >> >> }
> > >> >>
> > >> >> 3. Created authkey under /etc/corosync
> > >> >>
> > >> >> 4. Created a file called pcmk under
> > >> /etc/corosync/service.d and contents as
> > >> >> below,
> > >> >> cat pcmk
> > >> >> service {
> > >> >> # Load the Pacemaker Cluster Resource
> > >> Manager
> > >> >> name: pacemaker
> > >> >> ver: 1
> > >> >> }
> > >> >>
> > >> >> 5. Added the node name "node_cu" in /etc/hosts
> > >> with 10.X.X.X ip
> > >> >>
> > >> >> 6. ./corosync -f -p & --> this step started
> > >> corosync
> > >> >>
> > >> >> [root at node_cu pacemaker]# netstat -alpn | grep
> > >> -i coros
> > >> >> udp 0 0 10.X.X.X:61841 0.0.0.0:
> *
> > >> >> 9133/corosync
> > >> >> udp 0 0 10.X.X.X:5405 0.0.0.0:
> *
> > >> >> 9133/corosync
> > >> >> unix 2 [ ACC ] STREAM LISTENING
> > >> 148888 9133/corosync
> > >> >> @quorum
> > >> >> unix 2 [ ACC ] STREAM LISTENING
> > >> 148884 9133/corosync
> > >> >> @cmap
> > >> >> unix 2 [ ACC ] STREAM LISTENING
> > >> 148887 9133/corosync
> > >> >> @votequorum
> > >> >> unix 2 [ ACC ] STREAM LISTENING
> > >> 148885 9133/corosync
> > >> >> @cfg
> > >> >> unix 2 [ ACC ] STREAM LISTENING
> > >> 148886 9133/corosync
> > >> >> @cpg
> > >> >> unix 2 [ ] DGRAM
> > >> 148840 9133/corosync
> > >> >>
> > >> >> 7. ./pacemakerd -f & gives the following error
> > >> and exits.
> > >> >> [root at node_cu pacemaker]# pacemakerd -f
> > >> >> cmap connection setup failed:
> > >> CS_ERR_TRY_AGAIN. Retrying in 1s
> > >> >> cmap connection setup failed:
> > >> CS_ERR_TRY_AGAIN. Retrying in 2s
> > >> >> cmap connection setup failed:
> > >> CS_ERR_TRY_AGAIN. Retrying in 3s
> > >> >> cmap connection setup failed:
> > >> CS_ERR_TRY_AGAIN. Retrying in 4s
> > >> >> cmap connection setup failed:
> > >> CS_ERR_TRY_AGAIN. Retrying in 5s
> > >> >> Could not connect to Cluster Configuration
> > >> Database API, error 6
> > >> >>
> > >> >> Can you please point me, what is missing in
> > >> these steps ?
> > >> >>
> > >> >> Before trying these steps, I tried running "pcs
> > >> cluster start", but that
> > >> >> command fails with "service" script not found.
> > >> As the root filesystem
> > >> >> doesn't contain either /etc/init.d/ or
> > >> /sbin/service
> > >> >>
> > >> >> So, the plan is to bring up corosync and
> > >> pacemaker manually, later do the
> > >> >> cluster configuration using "pcs" commands.
> > >> >>
> > >> >> Regards,
> > >> >> Sriram
> > >> >>
> > >> >> _______________________________________________
> > >> >> Users mailing list: Users at clusterlabs.org
> > >> <mailto:Users at clusterlabs.org>
> > >> >> http://clusterlabs.org/mailman/listinfo/users
> > >> >>
> > >> >> Project Home: http://www.clusterlabs.org
> > >> >> Getting started:
> > >>
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > >> >> Bugs: http://bugs.clusterlabs.org
> > >> >>
> > >> >
> > >> >
> > >> >
> > >>
> > >>
> > >> _______________________________________________
> > >> Users mailing list: Users at clusterlabs.org
> > >> <mailto:Users at clusterlabs.org>
> > >> http://clusterlabs.org/mailman/listinfo/users
> > >>
> > >> Project Home: http://www.clusterlabs.org
> > >> Getting started:
> > >>
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > >> Bugs: http://bugs.clusterlabs.org
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> _______________________________________________
> > >> Users mailing list: Users at clusterlabs.org
> > >> <mailto:Users at clusterlabs.org>
> > >> http://clusterlabs.org/mailman/listinfo/users
> > >>
> > >> Project Home: http://www.clusterlabs.org
> > >> Getting started:
> > >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > >> Bugs: http://bugs.clusterlabs.org
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>_______________________________________________
> > >>Users mailing list: Users at clusterlabs.org
> > >>http://clusterlabs.org/mailman/listinfo/users
> > >>
> > >>Project Home: http://www.clusterlabs.org
> > >>Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > >>Bugs: http://bugs.clusterlabs.org
> > >
> > >
> > >_______________________________________________
> > >Users mailing list: Users at clusterlabs.org
> > >http://clusterlabs.org/mailman/listinfo/users
> > >
> > >Project Home: http://www.clusterlabs.org
> > >Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > >Bugs: http://bugs.clusterlabs.org
> > >
> >
> >
> > _______________________________________________
> > Users mailing list: Users at clusterlabs.org
> > http://clusterlabs.org/mailman/listinfo/users
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160503/d6b9e5db/attachment-0003.html>
More information about the Users
mailing list