[Pacemaker] Pacemaker core dumps

Andrew Beekhof andrew at beekhof.net
Wed May 1 19:06:42 EDT 2013 

On 01/05/2013, at 11:36 PM, Xavier Lashmar <xlashmar at uottawa.ca> wrote:

> I'm not sure if anyone has run into this issue but I can't seem to find a debuginfo package for one of the libraries for CentOS 6.3 with Kernel 2.6.32-279.9.1el6.x86_64 : libtool-ltdl
> 
> Here's what I get so far from the core dump, but I think it's incomplete:
> 
> ...
> ...
> ...
> Reading symbols from /lib64/libfreebl3.so...
> warning: the debug information found in "/usr/lib/debug//lib64/libfreebl3.so.debug" does not match "/lib64/libfreebl3.so" (CRC mismatch).
> 
> warning: the debug information found in "/usr/lib/debug/lib64/libfreebl3.so.debug" does not match "/lib64/libfreebl3.so" (CRC mismatch).
> 
> Missing separate debuginfo for /lib64/libfreebl3.so
> Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/68/195872ecfb188389d29aaf01031a976fd18168.debug
> (no debugging symbols found)...done.
> Loaded symbols for /lib64/libfreebl3.so
> Reading symbols from /lib64/libnss_files-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libnss_files-2.12.so
> Core was generated by `/usr/libexec/pacemaker/crmd'.
> Program terminated with signal 6, Aborted.
> #0  0x00007f81896ac8a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> 64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
> Missing separate debuginfos, use: debuginfo-install libtool-ltdl-2.2.6-15.5.el6.x86_64
> 
> Any info about either finding the right debuginfo files, or about the error itself would be greatly appreciated.

The libtool parts aren't so interesting.
Were there no other frames? (lines starting with # and a number)

> 
> Xavier Lashmar
> Analyste de Systèmes | Systems Analyst
> Service étudiants, service de l'informatique et des communications/Student services, computing and communications services.
> 1 Nicholas Street (810)
> Ottawa ON K1N 7B7
> Tél. | Tel. 613-562-5800 (2120)
>  
> 
> 
> 
> -----Original Message-----
> From: Andrew Beekhof [mailto:andrew at beekhof.net] 
> Sent: Monday, April 29, 2013 11:00 PM
> To: The Pacemaker cluster resource manager
> Subject: Re: [Pacemaker] Pacemaker core dumps
> 
> 
> On 30/04/2013, at 1:32 AM, Xavier Lashmar <xlashmar at uottawa.ca> wrote:
> 
>> Hello Andrew,
>> 
>> Thanks for your help.  We've upgrade to pacemaker 1.1.9 and still have the same issue.  
> 
> Thats a disappointing but useful data point.
> 
>> 
>> We are trying to get the core information but we are missing some debuginfo files which we are trying to get our hands on.  I'll try to forward this information soon.   
> 
> Great
> 
>> 
>> Is there something we need to do to the CIB when we upgrade?
> 
> No, anything that needs to happen will be done under the hood.
> 
>> 
>> 
>> Xavier Lashmar
>> Analyste de Systèmes | Systems Analyst Service étudiants, service de 
>> l'informatique et des communications/Student services, computing and communications services.
>> 1 Nicholas Street (810)
>> Ottawa ON K1N 7B7
>> Tél. | Tel. 613-562-5800 (2120)
>> 
>> 
>> 
>> -----Original Message-----
>> From: Andrew Beekhof [mailto:andrew at beekhof.net]
>> Sent: Thursday, April 25, 2013 8:15 PM
>> To: The Pacemaker cluster resource manager
>> Subject: Re: [Pacemaker] Pacemaker core dumps
>> 
>> 
>> On 26/04/2013, at 10:06 AM, Andrew Beekhof <andrew at beekhof.net> wrote:
>> 
>>> 
>>> On 25/04/2013, at 11:59 PM, Xavier Lashmar <xlashmar at uottawa.ca> wrote:
>>> 
>>>> Following further investigation, we were able to determine that upgrading both nodes (in a two node cluster) from Pacemaker 1.1.7-6 to Pacemaker 1.1.8-7 (CentOS 6.3 or Centos 6.4) caused these errors to begin happening:
>>> 
>>> Would you be able to try the 1.1.9 packages from http://www.clusterlabs.org/rpm-next to see if they are also affected?
>>> 
>>>> 
>>>> We were able to replicate the initiation of the errors by upgrading another cluster in the same manner.  This other cluster is now experiencing the same core-dumping and errors as the previous cluster:
>>>> 
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: Entity: line 1: parser error : invalid character in attribute value
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error:                                                                                ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: Entity: line 1: parser error : attributes construct error
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error:                                                                                ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: Entity: line 1: parser error : Couldn't find end of Start Tag lrmd_notify line 1
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error:                                                                                ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: Entity: line 1: parser error : Extra content at the end of the document
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error:                                                                                ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]:  warning: string2xml: Parsing 
>>>> failed (domain=1, level=3, code=5): Extra content at the end of the 
>>>> document Apr 25 09:46:22 xxxx crmd[1764]:  warning: string2xml: String start:
>>>> <lrmd_notify lrmd_origin="send_cmd_complete_notify
>>>> Apr 25 09:46:22 xxxx crmd[1764]:  warning: string2xml: String start+688: 0000" CRM_meta_start_delay="15000"/></lrmd_notify>
>>>> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_abort: string2xml: Forked child 4182 to record non-fatal assert at xml.c:605 : String parsing error
>> 
>> Also, it would be very useful if you could open up the core file for 
>> 4182 and print the contents of the input passed to string2xml() 
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org 
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> Project Home: http://www.clusterlabs.org Getting started: 
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>> 
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org 
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> Project Home: http://www.clusterlabs.org Getting started: 
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
> 
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

More information about the Pacemaker mailing list