[ClusterLabs] Antw: Re: Using different folder for /var/lib/pacemaker and usage of /dev/shm files

Nikhil Utane nikhil.subscribed at gmail.com
Wed May 18 04:55:24 UTC 2016


Yes, we do have our application using shared memory which is what we see
when the cluster is down.

On Tue, May 17, 2016 at 10:53 PM, Ken Gaillot <kgaillot at redhat.com> wrote:

> On 05/17/2016 12:02 PM, Nikhil Utane wrote:
> > OK. Will do that.
> >
> > Actually I gave the /dev/shm usage when the cluster wasn't up.
> > When it is up, I see it occupies close to 300 MB (it's also the DC).
>
> Hmmm, there should be no usage if the cluster is stopped. Any memory
> used by the cluster will start with "qb-", so anything else is from
> something else.
>
> If all executables using libqb (including corosync and pacemaker) are
> stopped, it's safe to remove any /dev/shm/qb-* files that remain. That
> should be rare, probably only after a core dump or such.
>
> > tmpfs                   500.0M    329.4M    170.6M  66% /dev/shm
> >
> > On another node the same is 115 MB.
> >
> > Anyways, I'll monitor the usage to know what size is needed.
> >
> > Thank you Ken and Ulrich.
> >
> > On Tue, May 17, 2016 at 8:23 PM, Ken Gaillot <kgaillot at redhat.com
> > <mailto:kgaillot at redhat.com>> wrote:
> >
> >     On 05/17/2016 04:07 AM, Nikhil Utane wrote:
> >     > What I would like to understand is how much total shared memory
> >     > (approximately) would Pacemaker need so that accordingly I can
> define
> >     > the partition size. Currently it is 300 MB in our system. I
> recently ran
> >     > into insufficient shared memory issue because of improper
> clean-up. So
> >     > would like to understand how much Pacemaker would need for a 6-node
> >     > cluster so that accordingly I can increase it.
> >
> >     I have no idea :-)
> >
> >     I don't think there's any way to pre-calculate it. The libqb library
> is
> >     the part of the software stack that actually manages the shared
> memory,
> >     but it's used by everything -- corosync (including its cpg and
> >     votequorum components) and each pacemaker daemon.
> >
> >     The size depends directly on the amount of communication activity in
> the
> >     cluster, which is only indirectly related to the number of
> >     nodes/resources/etc., the size of the CIB, etc. A cluster with nodes
> >     joining/leaving frequently and resources moving around a lot will use
> >     more shared memory than a cluster of the same size that's quiet.
> Cluster
> >     options such as cluster-recheck-interval would also matter.
> >
> >     Practically, I think all you can do is simulate expected cluster
> >     configurations and loads, and see what it comes out to be.
> >
> >     > # df -kh
> >     > tmpfs                   300.0M     27.5M    272.5M   9% /dev/shm
> >     >
> >     > Thanks
> >     > Nikhil
> >     >
> >     > On Tue, May 17, 2016 at 12:09 PM, Ulrich Windl
> >     > <Ulrich.Windl at rz.uni-regensburg.de
> >     <mailto:Ulrich.Windl at rz.uni-regensburg.de>
> >     > <mailto:Ulrich.Windl at rz.uni-regensburg.de
> >     <mailto:Ulrich.Windl at rz.uni-regensburg.de>>> wrote:
> >     >
> >     >     Hi!
> >     >
> >     >     One of the main problems I identified with POSIX shared memory
> >     >     (/dev/shm) in Linux is that changes to the shared memory don't
> >     >     affect the i-node, so you cannot tell from a "ls -rtl" which
> >     >     segments are still active and which are not. You can only see
> the
> >     >     creation time.
> >     >
> >     >     Maybe there should be a tool that identifies and cleans up
> obsolete
> >     >     shared memory.
> >     >     I don't understand the part talking about the size of
> /dev/shm: It's
> >     >     shared memory. See "kernel.shmmax" and "kernel.shmall" in you
> sysctl
> >     >     settings (/etc/sysctl.conf).
> >     >
> >     >     Regards,
> >     >     Ulrich
> >     >
> >     >     >>> Nikhil Utane <nikhil.subscribed at gmail.com <mailto:
> nikhil.subscribed at gmail.com>
> >     >     <mailto:nikhil.subscribed at gmail.com
> >     <mailto:nikhil.subscribed at gmail.com>>> schrieb am 16.05.2016 um
> 14:31 in
> >     >     Nachricht
> >     >
> >      <CAGNWmJVSye5PJgkdbFAi5AzO+Qq-j=2fS1c+0rGnqS994vV48w at mail.gmail.com
> >     <mailto:2fS1c%2B0rGnqS994vV48w at mail.gmail.com>
> >     >     <mailto:2fS1c%2B0rGnqS994vV48w at mail.gmail.com
> >     <mailto:2fS1c%252B0rGnqS994vV48w at mail.gmail.com>>>:
> >     >     > Thanks Ken.
> >     >     >
> >     >     > Could you also respond on the second question?
> >     >     >
> >     >     >>     Also, in /dev/shm I see that it created around 300+
> files of
> >     >     around
> >     >     >>     250 MB.
> >     >     >>
> >     >     >>     For e.g.
> >     >     >>     -rw-rw----    1 hacluste hacluste      8232 May  6 13:03
> >     >     >>     qb-cib_rw-response-25035-25038-10-header
> >     >     >>     -rw-rw----    1 hacluste hacluste    540672 May  6 13:03
> >     >     >>     qb-cib_rw-response-25035-25038-10-data
> >     >     >>     -rw-------    1 hacluste hacluste      8232 May  6 13:03
> >     >     >>     qb-cib_rw-response-25035-25036-12-header
> >     >     >>     -rw-------    1 hacluste hacluste    540672 May  6 13:03
> >     >     >>     qb-cib_rw-response-25035-25036-12-data
> >     >     >>     And many more..
> >     >     >>
> >     >     >>     We have limited space in /dev/shm and all these files
> are
> >     >     filling it
> >     >     >>     up. Are these all needed? Any way to limit? Do we need
> to do any
> >     >     >>     clean-up if pacemaker termination was not graceful?
> What's the
> >     >     > recommended size for this folder for Pacemaker? Our cluster
> will have
> >     >     > maximum 6 nodes.
> >     >     >
> >     >     > -Regards
> >     >     > Nikhil
> >     >     >
> >     >     > On Sat, May 14, 2016 at 3:11 AM, Ken Gaillot <
> kgaillot at redhat.com <mailto:kgaillot at redhat.com>
> >     >     <mailto:kgaillot at redhat.com <mailto:kgaillot at redhat.com>>>
> wrote:
> >     >     >
> >     >     >> On 05/08/2016 11:19 PM, Nikhil Utane wrote:
> >     >     >> > Moving these questions to a different thread.
> >     >     >> >
> >     >     >> >     Hi,
> >     >     >> >
> >     >     >> >     We have limited storage capacity in our system for
> >     >     different folders.
> >     >     >> >     How can I configure to use a different folder for
> >     >     /var/lib/pacemaker?
> >     >     >>
> >     >     >> ./configure --localstatedir=/wherever (defaults to /var or
> >     >     ${prefix}/var)
> >     >     >>
> >     >     >> That will change everything that normally is placed or
> >     looked for
> >     >     under
> >     >     >> /var (/var/lib/pacemaker, /var/lib/heartbeat, /var/run,
> etc.).
> >     >     >>
> >     >     >> Note that while ./configure lets you change the location of
> >     nearly
> >     >     >> everything, /usr/lib/ocf/resource.d is an exception,
> >     because it is
> >     >     >> specified in the OCF standard.
> >     >     >>
> >     >     >> >
> >     >     >> >
> >     >     >> >     Also, in /dev/shm I see that it created around 300+
> files
> >     >     of around
> >     >     >> >     250 MB.
> >     >     >> >
> >     >     >> >     For e.g.
> >     >     >> >     -rw-rw----    1 hacluste hacluste      8232 May  6
> 13:03
> >     >     >> >     qb-cib_rw-response-25035-25038-10-header
> >     >     >> >     -rw-rw----    1 hacluste hacluste    540672 May  6
> 13:03
> >     >     >> >     qb-cib_rw-response-25035-25038-10-data
> >     >     >> >     -rw-------    1 hacluste hacluste      8232 May  6
> 13:03
> >     >     >> >     qb-cib_rw-response-25035-25036-12-header
> >     >     >> >     -rw-------    1 hacluste hacluste    540672 May  6
> 13:03
> >     >     >> >     qb-cib_rw-response-25035-25036-12-data
> >     >     >> >     And many more..
> >     >     >> >
> >     >     >> >     We have limited space in /dev/shm and all these files
> are
> >     >     filling it
> >     >     >> >     up. Are these all needed? Any way to limit? Do we
> need to
> >     >     do any
> >     >     >> >     clean-up if pacemaker termination was not graceful?
> >     >     >> >
> >     >     >> >     -Thanks
> >     >     >> >     Nikhil
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160518/19fe7b32/attachment.htm>


More information about the Users mailing list