<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="ProgId" content="Excel.Sheet">
<meta name="Generator" content="Microsoft Excel 14">
<link id="Main-File" rel="Main-File" href="file://localhost/Users/greg/Library/Caches/TemporaryItems/msoclip/0/clip.htm"><link rel="File-List" href="file://localhost/Users/greg/Library/Caches/TemporaryItems/msoclip/0/clip_filelist.xml"><style>
<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:1.0in .75in 1.0in .75in;
mso-header-margin:.5in;
mso-footer-margin:.5in;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:12.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:Calibri, sans-serif;
mso-font-charset:0;
mso-number-format:General;
text-align:general;
vertical-align:bottom;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
.xl63
{color:#1F1F1F;
font-size:10.0pt;
font-family:Verdana;
mso-generic-font-family:auto;
mso-font-charset:0;}
-->
</style>
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>I’ve also just started capturing the process’ mapped size every 5 minutes, and can see that on the active node this grows by exactly 132KB every 10 minutes. (Or at least I can see the growth every other time I capture the data):</div>
</div>
</div>
<div><br>
</div>
<div>
<table border="0" cellpadding="0" cellspacing="0" width="198" style="border-collapse: collapse; width: 198pt; empty-cells: show;">
<!--StartFragment--><colgroup><col width="121" style="mso-width-source:userset;mso-width-alt:5162;width:121pt"><col width="77" style="mso-width-source:userset;mso-width-alt:3285;width:77pt"></colgroup>
<tbody>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="even_row">
<td height="15" class="xl63" width="121" style="height:15.0pt;width:121pt;
color:inherit;border-top-left-radius: 2px;border-top-right-radius: 2px;
border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 12:35:00</td>
<td class="xl63" align="right" width="77" style="width:77pt;color:inherit;
border-top-left-radius: 2px;border-top-right-radius: 2px;border-bottom-right-radius: 2px;
border-bottom-left-radius: 2px">
1175072768</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="odd_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 12:29:59</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1175072768</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="even_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 12:25:00</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1174937600</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="odd_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 12:19:59</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1174937600</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="even_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 12:14:59</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1174802432</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="odd_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 12:09:59</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1174802432</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="even_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 12:04:59</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1174667264</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="odd_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 11:59:59</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1174667264</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="even_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 11:54:59</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1174532096</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="odd_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 11:49:59</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1174532096</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="even_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 11:44:59</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1174396928</td>
</tr>
<tr height="15" style="height:15.0pt;color:inherit;padding-bottom:0px;
padding-top:0px" origclass="odd_row">
<td height="15" class="xl63" style="height:15.0pt;color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
2014.Apr.30 11:40:00</td>
<td class="xl63" align="right" style="color:inherit;border-top-left-radius: 2px;
border-top-right-radius: 2px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px">
1174396928</td>
</tr>
<!--EndFragment-->
</tbody>
</table>
</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Greg Murphy <<a href="mailto:greg.murphy@gamesparks.com">greg.murphy@gamesparks.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>The Pacemaker cluster resource manager <<a href="mailto:pacemaker@oss.clusterlabs.org">pacemaker@oss.clusterlabs.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, 30 April 2014 12:01<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:pacemaker@oss.clusterlabs.org">pacemaker@oss.clusterlabs.org</a>" <<a href="mailto:pacemaker@oss.clusterlabs.org">pacemaker@oss.clusterlabs.org</a>><br>
<span style="font-weight:bold">Subject: </span>[Pacemaker] lrmd Memory Usage<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Hi</div>
<div><br>
</div>
<div>I’m running a two-node Pacemaker cluster on Ubuntu Saucy (13.10), kernel 3.11.0-17-generic and the Ubuntu Pacemaker package, version 1.1.10+git20130802-1ubuntu1. The cluster is configured with a DRBD master/slave set and then a failover resource group
containing MySQL (along with its DRBD filesystem) and a Zabbix Proxy and Agent.</div>
<div><br>
</div>
<div>Since I built the cluster around two months ago I’ve noticed that on the the active node the memory footprint of lrmd gradually grows to quite a significant size. The cluster was last restarted three weeks ago, and now lrmd has over 1GB of mapped memory
on the active node and only 151MB on the passive node. Current excerpts from /proc/PID/status are:</div>
<div><br>
</div>
<div><b><u>Active node</u></b></div>
<div>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmPeak:<span class="Apple-tab-span" style="white-space:pre"></span>1146740 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmSize:<span class="Apple-tab-span" style="white-space:pre"></span>1146740 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmLck:<span class="Apple-tab-span" style="white-space:pre"></span> 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmPin:<span class="Apple-tab-span" style="white-space:pre"></span> 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmHWM:<span class="Apple-tab-span" style="white-space:pre"></span> 267680 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmRSS:<span class="Apple-tab-span" style="white-space:pre"></span> 188764 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmData:<span class="Apple-tab-span" style="white-space:pre"></span>1065860 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmStk:<span class="Apple-tab-span" style="white-space:pre"></span> 136 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmExe:<span class="Apple-tab-span" style="white-space:pre"></span> 32 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmLib:<span class="Apple-tab-span" style="white-space:pre"></span> 10416 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmPTE:<span class="Apple-tab-span" style="white-space:pre"></span> 2164 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmSwap:<span class="Apple-tab-span" style="white-space:pre"></span> 822752 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;"><b style="font-family: Calibri, sans-serif; font-size: 14px;"><u>Passive node</u></b></p>
</div>
<div>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmPeak:<span class="Apple-tab-span" style="white-space:pre"></span> 220832 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmSize:<span class="Apple-tab-span" style="white-space:pre"></span> 155428 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmLck:<span class="Apple-tab-span" style="white-space:pre"></span> 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmPin:<span class="Apple-tab-span" style="white-space:pre"></span> 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmHWM:<span class="Apple-tab-span" style="white-space:pre"></span> 4568 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmRSS:<span class="Apple-tab-span" style="white-space:pre"></span> 3880 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmData:<span class="Apple-tab-span" style="white-space:pre"></span> 74548 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmStk:<span class="Apple-tab-span" style="white-space:pre"></span> 136 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmExe:<span class="Apple-tab-span" style="white-space:pre"></span> 32 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmLib:<span class="Apple-tab-span" style="white-space:pre"></span> 10416 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmPTE:<span class="Apple-tab-span" style="white-space:pre"></span> 172 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmSwap:<span class="Apple-tab-span" style="white-space:pre"></span> 0 kB</p>
</div>
<div><br>
</div>
<div>During the last week or so I’ve taken a couple of snapshots of /proc/PID/smaps on the active node, and the heap particularly stands out as growing: (I have the full outputs captured if they’ll help)</div>
<div><br>
</div>
<div>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;"><b><u>20140422</u></b></p>
</div>
<div>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">7f92e1578000-7f92f218b000 rw-p 00000000 00:00 0 [heap]</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Size: 274508 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Rss: 180152 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Pss: 180152 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Shared_Clean: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Shared_Dirty: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Private_Clean: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Private_Dirty: 180152 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Referenced: 120472 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Anonymous: 180152 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">AnonHugePages: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Swap: 91568 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">KernelPageSize: 4 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">MMUPageSize: 4 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Locked: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmFlags: rd wr mr mw me ac</p>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;"><b><u>20140423</u></b></p>
</div>
<div>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">7f92e1578000-7f92f305e000 rw-p 00000000 00:00 0 [heap]</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Size: 289688 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Rss: 184136 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Pss: 184136 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Shared_Clean: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Shared_Dirty: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Private_Clean: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Private_Dirty: 184136 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Referenced: 69748 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Anonymous: 184136 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">AnonHugePages: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Swap: 103112 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">KernelPageSize: 4 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">MMUPageSize: 4 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Locked: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmFlags: rd wr mr mw me ac</p>
</div>
<div><br>
</div>
<div>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;"><b><u>20140430</u></b></p>
</div>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">7f92e1578000-7f92fc01d000 rw-p 00000000 00:00 0 [heap]</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Size: 436884 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Rss: 140812 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Pss: 140812 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Shared_Clean: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Shared_Dirty: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Private_Clean: 744 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Private_Dirty: 140068 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Referenced: 43600 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Anonymous: 140812 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">AnonHugePages: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Swap: 287392 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">KernelPageSize: 4 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">MMUPageSize: 4 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">Locked: 0 kB</p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">VmFlags: rd wr mr mw me ac</p>
<div><br>
</div>
<div>I noticed in the release notes for 1.1.10-rc1 (<a href="https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.10-rc1">https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.10-rc1</a>) that there was work done to fix "crmd: lrmd:
stonithd: fixed memory leaks” but I’m not sure which particular bug this was related to. (And those fixes should be in the version I’m running anyway).</div>
<div><br>
</div>
<div>I’ve also spotted a few memory leak fixes in <a href="https://github.com/beekhof/pacemaker">https://github.com/beekhof/pacemaker</a>, but I’m not sure whether they relate to my issue (assuming I have a memory leak and this isn’t expected behaviour).</div>
<div><br>
</div>
<div>Is there additional debugging that I can perform to check whether I have a leak, or is there enough evidence to justify upgrading to 1.1.11?</div>
<div><br>
</div>
<div>Thanks in advance</div>
<div><br>
</div>
<div>Greg Murphy</div>
</div>
</div>
</span>
</body>
</html>