<div dir="ltr">OK. Will do that.<div><br></div><div>Actually I gave the /dev/shm usage when the cluster wasn't up.</div><div>When it is up, I see it occupies close to 300 MB (it's also the DC).</div><div><br></div><div><div>tmpfs 500.0M 329.4M 170.6M 66% /dev/shm</div></div><div><br></div><div>On another node the same is 115 MB.</div><div><br></div><div>Anyways, I'll monitor the usage to know what size is needed. </div><div><br></div><div>Thank you Ken and Ulrich. </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 17, 2016 at 8:23 PM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 05/17/2016 04:07 AM, Nikhil Utane wrote:<br>
> What I would like to understand is how much total shared memory<br>
> (approximately) would Pacemaker need so that accordingly I can define<br>
> the partition size. Currently it is 300 MB in our system. I recently ran<br>
> into insufficient shared memory issue because of improper clean-up. So<br>
> would like to understand how much Pacemaker would need for a 6-node<br>
> cluster so that accordingly I can increase it.<br>
<br>
</span>I have no idea :-)<br>
<br>
I don't think there's any way to pre-calculate it. The libqb library is<br>
the part of the software stack that actually manages the shared memory,<br>
but it's used by everything -- corosync (including its cpg and<br>
votequorum components) and each pacemaker daemon.<br>
<br>
The size depends directly on the amount of communication activity in the<br>
cluster, which is only indirectly related to the number of<br>
nodes/resources/etc., the size of the CIB, etc. A cluster with nodes<br>
joining/leaving frequently and resources moving around a lot will use<br>
more shared memory than a cluster of the same size that's quiet. Cluster<br>
options such as cluster-recheck-interval would also matter.<br>
<br>
Practically, I think all you can do is simulate expected cluster<br>
configurations and loads, and see what it comes out to be.<br>
<span class=""><br>
> # df -kh<br>
> tmpfs 300.0M 27.5M 272.5M 9% /dev/shm<br>
><br>
> Thanks<br>
> Nikhil<br>
><br>
> On Tue, May 17, 2016 at 12:09 PM, Ulrich Windl<br>
> <<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a><br>
</span><span class="">> <mailto:<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a>>> wrote:<br>
><br>
> Hi!<br>
><br>
> One of the main problems I identified with POSIX shared memory<br>
> (/dev/shm) in Linux is that changes to the shared memory don't<br>
> affect the i-node, so you cannot tell from a "ls -rtl" which<br>
> segments are still active and which are not. You can only see the<br>
> creation time.<br>
><br>
> Maybe there should be a tool that identifies and cleans up obsolete<br>
> shared memory.<br>
> I don't understand the part talking about the size of /dev/shm: It's<br>
> shared memory. See "kernel.shmmax" and "kernel.shmall" in you sysctl<br>
> settings (/etc/sysctl.conf).<br>
><br>
> Regards,<br>
> Ulrich<br>
><br>
> >>> Nikhil Utane <<a href="mailto:nikhil.subscribed@gmail.com">nikhil.subscribed@gmail.com</a><br>
</span>> <mailto:<a href="mailto:nikhil.subscribed@gmail.com">nikhil.subscribed@gmail.com</a>>> schrieb am 16.05.2016 um 14:31 in<br>
> Nachricht<br>
> <CAGNWmJVSye5PJgkdbFAi5AzO+Qq-j=<a href="mailto:2fS1c%2B0rGnqS994vV48w@mail.gmail.com">2fS1c+0rGnqS994vV48w@mail.gmail.com</a><br>
> <mailto:<a href="mailto:2fS1c%252B0rGnqS994vV48w@mail.gmail.com">2fS1c%2B0rGnqS994vV48w@mail.gmail.com</a>>>:<br>
<span class="im HOEnZb">> > Thanks Ken.<br>
> ><br>
> > Could you also respond on the second question?<br>
> ><br>
> >> Also, in /dev/shm I see that it created around 300+ files of<br>
> around<br>
> >> 250 MB.<br>
> >><br>
> >> For e.g.<br>
> >> -rw-rw---- 1 hacluste hacluste 8232 May 6 13:03<br>
> >> qb-cib_rw-response-25035-25038-10-header<br>
> >> -rw-rw---- 1 hacluste hacluste 540672 May 6 13:03<br>
> >> qb-cib_rw-response-25035-25038-10-data<br>
> >> -rw------- 1 hacluste hacluste 8232 May 6 13:03<br>
> >> qb-cib_rw-response-25035-25036-12-header<br>
> >> -rw------- 1 hacluste hacluste 540672 May 6 13:03<br>
> >> qb-cib_rw-response-25035-25036-12-data<br>
> >> And many more..<br>
> >><br>
> >> We have limited space in /dev/shm and all these files are<br>
> filling it<br>
> >> up. Are these all needed? Any way to limit? Do we need to do any<br>
> >> clean-up if pacemaker termination was not graceful? What's the<br>
> > recommended size for this folder for Pacemaker? Our cluster will have<br>
> > maximum 6 nodes.<br>
> ><br>
> > -Regards<br>
> > Nikhil<br>
> ><br>
> > On Sat, May 14, 2016 at 3:11 AM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a><br>
</span><div class="HOEnZb"><div class="h5">> <mailto:<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>>> wrote:<br>
> ><br>
> >> On 05/08/2016 11:19 PM, Nikhil Utane wrote:<br>
> >> > Moving these questions to a different thread.<br>
> >> ><br>
> >> > Hi,<br>
> >> ><br>
> >> > We have limited storage capacity in our system for<br>
> different folders.<br>
> >> > How can I configure to use a different folder for<br>
> /var/lib/pacemaker?<br>
> >><br>
> >> ./configure --localstatedir=/wherever (defaults to /var or<br>
> ${prefix}/var)<br>
> >><br>
> >> That will change everything that normally is placed or looked for<br>
> under<br>
> >> /var (/var/lib/pacemaker, /var/lib/heartbeat, /var/run, etc.).<br>
> >><br>
> >> Note that while ./configure lets you change the location of nearly<br>
> >> everything, /usr/lib/ocf/resource.d is an exception, because it is<br>
> >> specified in the OCF standard.<br>
> >><br>
> >> ><br>
> >> ><br>
> >> > Also, in /dev/shm I see that it created around 300+ files<br>
> of around<br>
> >> > 250 MB.<br>
> >> ><br>
> >> > For e.g.<br>
> >> > -rw-rw---- 1 hacluste hacluste 8232 May 6 13:03<br>
> >> > qb-cib_rw-response-25035-25038-10-header<br>
> >> > -rw-rw---- 1 hacluste hacluste 540672 May 6 13:03<br>
> >> > qb-cib_rw-response-25035-25038-10-data<br>
> >> > -rw------- 1 hacluste hacluste 8232 May 6 13:03<br>
> >> > qb-cib_rw-response-25035-25036-12-header<br>
> >> > -rw------- 1 hacluste hacluste 540672 May 6 13:03<br>
> >> > qb-cib_rw-response-25035-25036-12-data<br>
> >> > And many more..<br>
> >> ><br>
> >> > We have limited space in /dev/shm and all these files are<br>
> filling it<br>
> >> > up. Are these all needed? Any way to limit? Do we need to<br>
> do any<br>
> >> > clean-up if pacemaker termination was not graceful?<br>
> >> ><br>
> >> > -Thanks<br>
> >> > Nikhil<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>