[ClusterLabs] [ClusterLab] : Corosync not initializing successfully
Jan Friesse
jfriesse at redhat.com
Thu May 5 05:28:49 EDT 2016
Nikhil
> Found the root-cause.
> In file schedwrk.c, the function handle2void() uses a union which was not
> initialized.
> Because of that the handle value was computed incorrectly (lower half was
> garbage).
>
> 56 static hdb_handle_t
> 57 void2handle (const void *v) { union u u={}; u.v = v; return u.h; }
> 58 static const void *
> 59 handle2void (hdb_handle_t h) { union u u={}; u.h = h; return u.v; }
>
> After initializing (as highlighted), the corosync initialization seems to
> be going through fine. Will check other things.
Your patch is incorrect and actually doesn't work. As I said (when
pointing you to schedwrk.c), I will send you proper patch, but fix that
issue correctly is not easy.
Regards,
Honza
>
> -Regards
> Nikhil
>
> On Tue, May 3, 2016 at 7:04 PM, Nikhil Utane <nikhil.subscribed at gmail.com>
> wrote:
>
>> Thanks for your response Dejan.
>>
>> I do not know yet whether this has anything to do with endianness.
>> FWIW, there could be something quirky with the system so keeping all
>> options open. :)
>>
>> I added some debug prints to understand what's happening under the hood.
>>
>> *Success case: (on x86 machine): *
>> [TOTEM ] entering OPERATIONAL state.
>> [TOTEM ] A new membership (10.206.1.7:137220) was formed. Members joined:
>> 181272839
>> [TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=0,
>> my_high_delivered=0
>> [TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
>> my_high_delivered=0
>> [TOTEM ] Delivering 0 to 1
>> [TOTEM ] Delivering MCAST message with seq 1 to pending delivery queue
>> [SYNC ] Nikhil: Inside sync_deliver_fn. header->id=1
>> [TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=2,
>> my_high_delivered=1
>> [TOTEM ] Delivering 1 to 2
>> [TOTEM ] Delivering MCAST message with seq 2 to pending delivery queue
>> [SYNC ] Nikhil: Inside sync_deliver_fn. header->id=0
>> [SYNC ] Nikhil: Entering sync_barrier_handler
>> [SYNC ] Committing synchronization for corosync configuration map access
>> .
>> [TOTEM ] Delivering 2 to 4
>> [TOTEM ] Delivering MCAST message with seq 3 to pending delivery queue
>> [TOTEM ] Delivering MCAST message with seq 4 to pending delivery queue
>> [CPG ] comparing: sender r(0) ip(10.206.1.7) ; members(old:0 left:0)
>> [CPG ] chosen downlist: sender r(0) ip(10.206.1.7) ; members(old:0
>> left:0)
>> [SYNC ] Committing synchronization for corosync cluster closed process
>> group service v1.01
>> *[MAIN ] Completed service synchronization, ready to provide service.*
>>
>>
>> *Failure case: (on ppc)*:
>> [TOTEM ] entering OPERATIONAL state.
>> [TOTEM ] A new membership (10.207.24.101:16) was formed. Members joined:
>> 181344357
>> [TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=0,
>> my_high_delivered=0
>> [TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
>> my_high_delivered=0
>> [TOTEM ] Delivering 0 to 1
>> [TOTEM ] Delivering MCAST message with seq 1 to pending delivery queue
>> [SYNC ] Nikhil: Inside sync_deliver_fn header->id=1
>> [TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
>> my_high_delivered=1
>> [TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
>> my_high_delivered=1
>> Above message repeats continuously.
>>
>> So it appears that in failure case I do not receive messages with sequence
>> number 2-4.
>> If somebody can throw some ideas that'll help a lot.
>>
>> -Thanks
>> Nikhil
>>
>> On Tue, May 3, 2016 at 5:26 PM, Dejan Muhamedagic <dejanmm at fastmail.fm>
>> wrote:
>>
>>> Hi,
>>>
>>> On Mon, May 02, 2016 at 08:54:09AM +0200, Jan Friesse wrote:
>>>>> As your hardware is probably capable of running ppcle and if you have
>>> an
>>>>> environment
>>>>> at hand without too much effort it might pay off to try that.
>>>>> There are of course distributions out there support corosync on
>>>>> big-endian architectures
>>>>> but I don't know if there is an automatized regression for corosync on
>>>>> big-endian that
>>>>> would catch big-endian-issues right away with something as current as
>>>>> your 2.3.5.
>>>>
>>>> No we are not testing big-endian.
>>>>
>>>> So totally agree with Klaus. Give a try to ppcle. Also make sure all
>>>> nodes are little-endian. Corosync should work in mixed BE/LE
>>>> environment but because it's not tested, it may not work (and it's a
>>>> bug, so if ppcle works I will try to fix BE).
>>>
>>> I tested a cluster consisting of big endian/little endian nodes
>>> (s390 and x86-64), but that was a while ago. IIRC, all relevant
>>> bugs in corosync got fixed at that time. Don't know what is the
>>> situation with the latest version.
>>>
>>> Thanks,
>>>
>>> Dejan
>>>
>>>> Regards,
>>>> Honza
>>>>
>>>>>
>>>>> Regards,
>>>>> Klaus
>>>>>
>>>>> On 05/02/2016 06:44 AM, Nikhil Utane wrote:
>>>>>> Re-sending as I don't see my post on the thread.
>>>>>>
>>>>>> On Sun, May 1, 2016 at 4:21 PM, Nikhil Utane
>>>>>> <nikhil.subscribed at gmail.com <mailto:nikhil.subscribed at gmail.com>>
>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Looking for some guidance here as we are completely blocked
>>>>>> otherwise :(.
>>>>>>
>>>>>> -Regards
>>>>>> Nikhil
>>>>>>
>>>>>> On Fri, Apr 29, 2016 at 6:11 PM, Sriram <sriram.ec at gmail.com
>>>>>> <mailto:sriram.ec at gmail.com>> wrote:
>>>>>>
>>>>>> Corrected the subject.
>>>>>>
>>>>>> We went ahead and captured corosync debug logs for our ppc
>>> board.
>>>>>> After log analysis and comparison with the sucessful logs(
>>>>>> from x86 machine) ,
>>>>>> we didnt find *"[ MAIN ] Completed service synchronization,
>>>>>> ready to provide service.*" in ppc logs.
>>>>>> So, looks like corosync is not in a position to accept
>>>>>> connection from Pacemaker.
>>>>>> Even I tried with the new corosync.conf with no success.
>>>>>>
>>>>>> Any hints on this issue would be really helpful.
>>>>>>
>>>>>> Attaching ppc_notworking.log, x86_working.log, corosync.conf.
>>>>>>
>>>>>> Regards,
>>>>>> Sriram
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 29, 2016 at 2:44 PM, Sriram <sriram.ec at gmail.com
>>>>>> <mailto:sriram.ec at gmail.com>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I went ahead and made some changes in file system(Like I
>>>>>> brought in /etc/init.d/corosync and
>>> /etc/init.d/pacemaker,
>>>>>> /etc/sysconfig ), After that I was able to run "pcs
>>>>>> cluster start".
>>>>>> But it failed with the following error
>>>>>> # pcs cluster start
>>>>>> Starting Cluster...
>>>>>> Starting Pacemaker Cluster Manager[FAILED]
>>>>>> Error: unable to start pacemaker
>>>>>>
>>>>>> And in the /var/log/pacemaker.log, I saw these errors
>>>>>> pacemakerd: info: mcp_read_config: cmap connection
>>>>>> setup failed: CS_ERR_TRY_AGAIN. Retrying in 4s
>>>>>> Apr 29 08:53:47 [15863] node_cu pacemakerd: info:
>>>>>> mcp_read_config: cmap connection setup failed:
>>>>>> CS_ERR_TRY_AGAIN. Retrying in 5s
>>>>>> Apr 29 08:53:52 [15863] node_cu pacemakerd: warning:
>>>>>> mcp_read_config: Could not connect to Cluster
>>>>>> Configuration Database API, error 6
>>>>>> Apr 29 08:53:52 [15863] node_cu pacemakerd: notice:
>>>>>> main: Could not obtain corosync config data, exiting
>>>>>> Apr 29 08:53:52 [15863] node_cu pacemakerd: info:
>>>>>> crm_xml_cleanup: Cleaning up memory from libxml2
>>>>>>
>>>>>>
>>>>>> And in the /var/log/Debuglog, I saw these errors coming
>>>>>> from corosync
>>>>>> 20160429 085347.487050 <tel:085347.487050> airv_cu
>>>>>> daemon.warn corosync[12857]: [QB ] Denied
>>> connection,
>>>>>> is not ready (12857-15863-14)
>>>>>> 20160429 085347.487067 <tel:085347.487067> airv_cu
>>>>>> daemon.info <http://daemon.info> corosync[12857]: [QB
>>>>>> ] Denied connection, is not ready (12857-15863-14)
>>>>>>
>>>>>>
>>>>>> I browsed the code of libqb to find that it is failing in
>>>>>>
>>>>>>
>>> https://github.com/ClusterLabs/libqb/blob/master/lib/ipc_setup.c
>>>>>>
>>>>>> Line 600 :
>>>>>> handle_new_connection function
>>>>>>
>>>>>> Line 637:
>>>>>> if (auth_result == 0 &&
>>>>>> c->service->serv_fns.connection_accept) {
>>>>>> res = c->service->serv_fns.connection_accept(c,
>>>>>> c->euid, c->egid);
>>>>>> }
>>>>>> if (res != 0) {
>>>>>> goto send_response;
>>>>>> }
>>>>>>
>>>>>> Any hints on this issue would be really helpful for me to
>>>>>> go ahead.
>>>>>> Please let me know if any logs are required,
>>>>>>
>>>>>> Regards,
>>>>>> Sriram
>>>>>>
>>>>>> On Thu, Apr 28, 2016 at 2:42 PM, Sriram
>>>>>> <sriram.ec at gmail.com <mailto:sriram.ec at gmail.com>>
>>> wrote:
>>>>>>
>>>>>> Thanks Ken and Emmanuel.
>>>>>> Its a big endian machine. I will try with running
>>> "pcs
>>>>>> cluster setup" and "pcs cluster start"
>>>>>> Inside cluster.py, "service pacemaker start" and
>>>>>> "service corosync start" are executed to bring up
>>>>>> pacemaker and corosync.
>>>>>> Those service scripts and the infrastructure needed
>>> to
>>>>>> bring up the processes in the above said manner
>>>>>> doesn't exist in my board.
>>>>>> As it is a embedded board with the limited memory,
>>>>>> full fledged linux is not installed.
>>>>>> Just curious to know, what could be reason the
>>>>>> pacemaker throws that error.
>>>>>>
>>>>>> /"cmap connection setup failed: CS_ERR_TRY_AGAIN.
>>>>>> Retrying in 1s"
>>>>>>
>>>>>> /
>>>>>> Thanks for response.
>>>>>>
>>>>>> Regards,
>>>>>> Sriram.
>>>>>>
>>>>>> On Thu, Apr 28, 2016 at 8:55 AM, Ken Gaillot
>>>>>> <kgaillot at redhat.com <mailto:kgaillot at redhat.com>>
>>> wrote:
>>>>>>
>>>>>> On 04/27/2016 11:25 AM, emmanuel segura wrote:
>>>>>> > you need to use pcs to do everything, pcs
>>>>>> cluster setup and pcs
>>>>>> > cluster start, try to use the redhat docs for
>>>>>> more information.
>>>>>>
>>>>>> Agreed -- pcs cluster setup will create a proper
>>>>>> corosync.conf for you.
>>>>>> Your corosync.conf below uses corosync 1 syntax,
>>>>>> and there were
>>>>>> significant changes in corosync 2. In particular,
>>>>>> you don't need the
>>>>>> file created in step 4, because pacemaker is no
>>>>>> longer launched via a
>>>>>> corosync plugin.
>>>>>>
>>>>>> > 2016-04-27 17:28 GMT+02:00 Sriram
>>>>>> <sriram.ec at gmail.com <mailto:sriram.ec at gmail.com
>>>>> :
>>>>>> >> Dear All,
>>>>>> >>
>>>>>> >> I m trying to use pacemaker and corosync for
>>>>>> the clustering requirement that
>>>>>> >> came up recently.
>>>>>> >> We have cross compiled corosync, pacemaker and
>>>>>> pcs(python) for ppc
>>>>>> >> environment (Target board where pacemaker and
>>>>>> corosync are supposed to run)
>>>>>> >> I m having trouble bringing up pacemaker in
>>>>>> that environment, though I could
>>>>>> >> successfully bring up corosync.
>>>>>> >> Any help is welcome.
>>>>>> >>
>>>>>> >> I m using these versions of pacemaker and
>>> corosync
>>>>>> >> [root at node_cu pacemaker]# corosync -v
>>>>>> >> Corosync Cluster Engine, version '2.3.5'
>>>>>> >> Copyright (c) 2006-2009 Red Hat, Inc.
>>>>>> >> [root at node_cu pacemaker]# pacemakerd -$
>>>>>> >> Pacemaker 1.1.14
>>>>>> >> Written by Andrew Beekhof
>>>>>> >>
>>>>>> >> For running corosync, I did the following.
>>>>>> >> 1. Created the following directories,
>>>>>> >> /var/lib/pacemaker
>>>>>> >> /var/lib/corosync
>>>>>> >> /var/lib/pacemaker
>>>>>> >> /var/lib/pacemaker/cores
>>>>>> >> /var/lib/pacemaker/pengine
>>>>>> >> /var/lib/pacemaker/blackbox
>>>>>> >> /var/lib/pacemaker/cib
>>>>>> >>
>>>>>> >>
>>>>>> >> 2. Created a file called corosync.conf under
>>>>>> /etc/corosync folder with the
>>>>>> >> following contents
>>>>>> >>
>>>>>> >> totem {
>>>>>> >>
>>>>>> >> version: 2
>>>>>> >> token: 5000
>>>>>> >> token_retransmits_before_loss_const:
>>> 20
>>>>>> >> join: 1000
>>>>>> >> consensus: 7500
>>>>>> >> vsftype: none
>>>>>> >> max_messages: 20
>>>>>> >> secauth: off
>>>>>> >> cluster_name: mycluster
>>>>>> >> transport: udpu
>>>>>> >> threads: 0
>>>>>> >> clear_node_high_bit: yes
>>>>>> >>
>>>>>> >> interface {
>>>>>> >> ringnumber: 0
>>>>>> >> # The following three values
>>>>>> need to be set based on your
>>>>>> >> environment
>>>>>> >> bindnetaddr: 10.x.x.x
>>>>>> >> mcastaddr: 226.94.1.1
>>>>>> >> mcastport: 5405
>>>>>> >> }
>>>>>> >> }
>>>>>> >>
>>>>>> >> logging {
>>>>>> >> fileline: off
>>>>>> >> to_syslog: yes
>>>>>> >> to_stderr: no
>>>>>> >> to_syslog: yes
>>>>>> >> logfile: /var/log/corosync.log
>>>>>> >> syslog_facility: daemon
>>>>>> >> debug: on
>>>>>> >> timestamp: on
>>>>>> >> }
>>>>>> >>
>>>>>> >> amf {
>>>>>> >> mode: disabled
>>>>>> >> }
>>>>>> >>
>>>>>> >> quorum {
>>>>>> >> provider: corosync_votequorum
>>>>>> >> }
>>>>>> >>
>>>>>> >> nodelist {
>>>>>> >> node {
>>>>>> >> ring0_addr: node_cu
>>>>>> >> nodeid: 1
>>>>>> >> }
>>>>>> >> }
>>>>>> >>
>>>>>> >> 3. Created authkey under /etc/corosync
>>>>>> >>
>>>>>> >> 4. Created a file called pcmk under
>>>>>> /etc/corosync/service.d and contents as
>>>>>> >> below,
>>>>>> >> cat pcmk
>>>>>> >> service {
>>>>>> >> # Load the Pacemaker Cluster Resource
>>>>>> Manager
>>>>>> >> name: pacemaker
>>>>>> >> ver: 1
>>>>>> >> }
>>>>>> >>
>>>>>> >> 5. Added the node name "node_cu" in /etc/hosts
>>>>>> with 10.X.X.X ip
>>>>>> >>
>>>>>> >> 6. ./corosync -f -p & --> this step started
>>>>>> corosync
>>>>>> >>
>>>>>> >> [root at node_cu pacemaker]# netstat -alpn |
>>> grep
>>>>>> -i coros
>>>>>> >> udp 0 0 10.X.X.X:61841
>>> 0.0.0.0:*
>>>>>> >> 9133/corosync
>>>>>> >> udp 0 0 10.X.X.X:5405
>>> 0.0.0.0:*
>>>>>> >> 9133/corosync
>>>>>> >> unix 2 [ ACC ] STREAM LISTENING
>>>>>> 148888 9133/corosync
>>>>>> >> @quorum
>>>>>> >> unix 2 [ ACC ] STREAM LISTENING
>>>>>> 148884 9133/corosync
>>>>>> >> @cmap
>>>>>> >> unix 2 [ ACC ] STREAM LISTENING
>>>>>> 148887 9133/corosync
>>>>>> >> @votequorum
>>>>>> >> unix 2 [ ACC ] STREAM LISTENING
>>>>>> 148885 9133/corosync
>>>>>> >> @cfg
>>>>>> >> unix 2 [ ACC ] STREAM LISTENING
>>>>>> 148886 9133/corosync
>>>>>> >> @cpg
>>>>>> >> unix 2 [ ] DGRAM
>>>>>> 148840 9133/corosync
>>>>>> >>
>>>>>> >> 7. ./pacemakerd -f & gives the following error
>>>>>> and exits.
>>>>>> >> [root at node_cu pacemaker]# pacemakerd -f
>>>>>> >> cmap connection setup failed:
>>>>>> CS_ERR_TRY_AGAIN. Retrying in 1s
>>>>>> >> cmap connection setup failed:
>>>>>> CS_ERR_TRY_AGAIN. Retrying in 2s
>>>>>> >> cmap connection setup failed:
>>>>>> CS_ERR_TRY_AGAIN. Retrying in 3s
>>>>>> >> cmap connection setup failed:
>>>>>> CS_ERR_TRY_AGAIN. Retrying in 4s
>>>>>> >> cmap connection setup failed:
>>>>>> CS_ERR_TRY_AGAIN. Retrying in 5s
>>>>>> >> Could not connect to Cluster Configuration
>>>>>> Database API, error 6
>>>>>> >>
>>>>>> >> Can you please point me, what is missing in
>>>>>> these steps ?
>>>>>> >>
>>>>>> >> Before trying these steps, I tried running
>>> "pcs
>>>>>> cluster start", but that
>>>>>> >> command fails with "service" script not found.
>>>>>> As the root filesystem
>>>>>> >> doesn't contain either /etc/init.d/ or
>>>>>> /sbin/service
>>>>>> >>
>>>>>> >> So, the plan is to bring up corosync and
>>>>>> pacemaker manually, later do the
>>>>>> >> cluster configuration using "pcs" commands.
>>>>>> >>
>>>>>> >> Regards,
>>>>>> >> Sriram
>>>>>> >>
>>>>>> >>
>>> _______________________________________________
>>>>>> >> Users mailing list: Users at clusterlabs.org
>>>>>> <mailto:Users at clusterlabs.org>
>>>>>> >> http://clusterlabs.org/mailman/listinfo/users
>>>>>> >>
>>>>>> >> Project Home: http://www.clusterlabs.org
>>>>>> >> Getting started:
>>>>>>
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>> >> Bugs: http://bugs.clusterlabs.org
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>> <mailto:Users at clusterlabs.org>
>>>>>> http://clusterlabs.org/mailman/listinfo/users
>>>>>>
>>>>>> Project Home: http://www.clusterlabs.org
>>>>>> Getting started:
>>>>>>
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>> <mailto:Users at clusterlabs.org>
>>>>>> http://clusterlabs.org/mailman/listinfo/users
>>>>>>
>>>>>> Project Home: http://www.clusterlabs.org
>>>>>> Getting started:
>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>> http://clusterlabs.org/mailman/listinfo/users
>>>>>>
>>>>>> Project Home: http://www.clusterlabs.org
>>>>>> Getting started:
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list: Users at clusterlabs.org
>>>>> http://clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started:
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list: Users at clusterlabs.org
>>>> http://clusterlabs.org/mailman/listinfo/users
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started:
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>
>>> _______________________________________________
>>> Users mailing list: Users at clusterlabs.org
>>> http://clusterlabs.org/mailman/listinfo/users
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>
>>
>
>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
More information about the Users
mailing list