<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Following further investigation, we were able to determine that upgrading both nodes (in a two node cluster) from Pacemaker 1.1.7-6 to Pacemaker 1.1.8-7 (CentOS 6.3 or Centos 6.4) caused these errors to begin
happening:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">We were able to replicate the initiation of the errors by upgrading another cluster in the same manner. This other cluster is now experiencing the same core-dumping and errors as the previous cluster:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity: line 1: parser error : invalid character in attribute value<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: ^<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity: line 1: parser error : attributes construct error<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: ^<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity: line 1: parser error : Couldn't find end of Start Tag lrmd_notify line 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: ^<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity: line 1: parser error : Extra content at the end of the document<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: ^<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: Parsing failed (domain=1, level=3, code=5): Extra content at the end of the document<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: String start: <lrmd_notify lrmd_origin="send_cmd_complete_notify<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: String start+688: 0000" CRM_meta_start_delay="15000"/></lrmd_notify><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Apr 25 09:46:22 xxxx crmd[1764]: error: crm_abort: string2xml: Forked child 4182 to record non-fatal assert at xml.c:605 : String parsing error<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Any assistance would be appreciated.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-CA" style="color:#1F497D">Xavier Lashmar<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA" style="color:#1F497D">X2120<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Xavier Lashmar [mailto:xlashmar@uottawa.ca]
<br>
<b>Sent:</b> Tuesday, April 23, 2013 3:41 PM<br>
<b>To:</b> Pacemaker@oss.clusterlabs.org<br>
<b>Subject:</b> [Pacemaker] Pacemaker core dumps<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-CA">Hello everyone,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Below is a message that my colleague tried to post to the mailing list, but somehow it didn’t make it yet, so I’m re-posting just in case it got lost somewhere:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Hi All,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">I currently have a pacemaker + drbd cluster. I am getting core dumps every 15 minutes. I can’t seem to figure out what has triggered this. How do you see what is being output in the core dump? Also, in my corosync.log
I found the following errors:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: Entity: line 1: parser error : invalid character in attribute value<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: e-9c6d-44bf-963b-87836613e694" lrmd_rsc_output="tomcat6 (pid 8899) is running...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: ^<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: Entity: line 1: parser error : attributes construct error<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: e-9c6d-44bf-963b-87836613e694" lrmd_rsc_output="tomcat6 (pid 8899) is running...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: ^<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: Entity: line 1: parser error : Couldn't find end of Start Tag lrmd_notify line 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: e-9c6d-44bf-963b-87836613e694" lrmd_rsc_output="tomcat6 (pid 8899) is running...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: ^<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: Entity: line 1: parser error : Extra content at the end of the document<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: e-9c6d-44bf-963b-87836613e694" lrmd_rsc_output="tomcat6 (pid 8899) is running...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_xml_err: XML Error: ^<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: warning: string2xml: Parsing failed (domain=1, level=3, code=5): Extra content at the end of the document<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: warning: string2xml: String start: <lrmd_notify lrmd_origin="send_cmd_complete_notify<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: warning: string2xml: String start+645: ="30000" CRM_meta_interval="15000"/></lrmd_notify><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Apr 22 10:37:15 [2058] crmd: error: crm_abort: string2xml: Forked child 22964 to record non-fatal assert at xml.c:605 : String parsing error<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">So I thought the errors might be in my xml config file so I extracted the file and found no errors in there. From the log file, this string “e-9c6d-44bf-963b-87836613e694” seems to be part of the transition-key. I am
not sure if that helps shed some light on the issue but any help would be appreciated.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" style="border-collapse:collapse">
<tbody>
<tr>
<td width="505" colspan="4" valign="top" style="width:378.9pt;padding:0in 5.4pt 0in 5.4pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto"><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:#BFBFBF"><img width="422" height="7" id="Picture_x0020_1" src="cid:image001.png@01CE419B.9DC5DF10" alt="Description: Description: cid:4608B9EF-A1C2-4029-A091-0E5BCC69E195"></span><span style="font-size:12.0pt;color:#1F497D;mso-fareast-language:EN-CA"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="116" style="width:86.8pt;padding:0in 5.4pt 0in 5.4pt">
<p class="MsoNormal" align="center" style="mso-margin-top-alt:auto;text-align:center">
<span style="color:#1F497D"><img width="77" height="77" id="Picture_x0020_2" src="cid:image002.png@01CE419B.9DC5DF10" alt="Description: Description: cid:D85E51EA-D618-4CBC-9F88-34F696123DED"></span><span style="font-size:12.0pt;color:#1F497D;mso-fareast-language:EN-CA"><o:p></o:p></span></p>
</td>
<td width="24" style="width:17.85pt;padding:0in 5.4pt 0in 5.4pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto"><span style="font-size:7.0pt;font-family:"Verdana","sans-serif";color:gray"><img width="3" height="87" id="Picture_x0020_3" src="cid:image003.png@01CE419B.9DC5DF10" alt="Description: Description: cid:3CA29A2F-1695-4743-8A3E-0DE3554B6C51"></span><span style="font-size:12.0pt;color:#1F497D;mso-fareast-language:EN-CA"><o:p></o:p></span></p>
</td>
<td width="246" style="width:184.25pt;padding:0in 5.4pt 0in 5.4pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto"><b><span lang="FR-CA" style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:gray;mso-fareast-language:EN-CA">Xavier Lashmar<br>
</span></b><span lang="FR-CA" style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:gray;mso-fareast-language:EN-CA">Analyste de Systèmes</span><span lang="FR-CA" style="color:black;mso-fareast-language:EN-CA">
</span><span lang="FR-CA" style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:gray;mso-fareast-language:EN-CA">| Systems Analyst<br>
Service étudiants, service de l'informatique et des communications/Student services, computing and communications services.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto"><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:gray;mso-fareast-language:EN-CA">1 Nicholas Street (810)</span><span style="color:#1F497D;mso-fareast-language:EN-CA"><br>
</span><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:gray;mso-fareast-language:EN-CA">Ottawa ON K1N 7B7</span><span style="color:#1F497D;mso-fareast-language:EN-CA"><br>
</span><span style="font-size:7.5pt;font-family:"Verdana","sans-serif";color:gray;mso-fareast-language:EN-CA">Tél. | Tel. 613-562-5800 (2120)</span><span style="font-size:12.0pt;color:#1F497D;mso-fareast-language:EN-CA"><o:p></o:p></span></p>
</td>
<td width="120" style="width:1.25in;padding:0in 0in 0in 0in">
<p class="MsoNormal"><span style="mso-fareast-language:EN-CA"> <o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>