[ClusterLabs] Antw: crm_report consumes all available RAM

[Lars Ellenberg]

On Tue, Oct 06, 2015 at 11:50:00PM +0200, Jan Pokorný wrote:
> On 06/10/15 10:28 +0200, Dejan Muhamedagic wrote:
> > On Mon, Oct 05, 2015 at 07:00:18PM +0300, Vladislav Bogdanov wrote:
> >> 14.09.2015 02:31, Andrew Beekhof wrote:
> >>> 
> >>>> On 8 Sep 2015, at 10:18 pm, Ulrich Windl <Ulrich.Windl at rz.uni-regensburg.de> wrote:
> >>>> 
> >>>>>>> Vladislav Bogdanov <bubble at hoster-ok.com> schrieb am 08.09.2015 um 14:05 in
> >>>> Nachricht <55EECEFB.8050001 at hoster-ok.com>:
> >>>>> Hi,
> >>>>> 
> >>>>> just discovered very interesting issue.
> >>>>> If there is a system user with very big UID (80000002 in my case),
> >>>>> then crm_report (actually 'grep' it runs) consumes too much RAM.
> >>>>> 
> >>>>> Relevant part of the process tree at that moment looks like (word-wrap off):
> >>>>> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> >>>>> ...
> >>>>> root     25526  0.0  0.0 106364   636 ?        S    12:37   0:00          \_
> >>>>> /bin/sh /usr/sbin/crm_report --dest=/var/log/crm_report -f 0000-01-01 00:00:00
> >>>>> root     25585  0.0  0.0 106364   636 ?        S    12:37   0:00
> >>>>> \_ bash /var/log/crm_report/collector
> >>>>> root     25613  0.0  0.0 106364   152 ?        S    12:37   0:00
> >>>>>     \_ bash /var/log/crm_report/collector
> >>>>> root     25614  0.0  0.0 106364   692 ?        S    12:37   0:00
> >>>>>         \_ bash /var/log/crm_report/collector
> >>>>> root     27965  4.9  0.0 100936   452 ?        S    12:38   0:01
> >>>>>         |   \_ cat /var/log/lastlog
> >>>>> root     27966 23.0 82.9 3248996 1594688 ?     D    12:38   0:08
> >>>>>         |   \_ grep -l -e Starting Pacemaker


Whoa.
grep using up 1.5 gig resident (3.2 gig virtual) still looking for
the first newline.

I suggest in addition to the (good) suggestions so far,
to also set a ulimit.

1) export LC_ALL=C
so grep won't take quadratic time trying to make sure it understands
unicode correctly; yes, I'm sure that bug has been fix on most systems meanwhile...

2)
 ( ulimit -v 100000 ; grep ) 

Usually, even with "very many very long lines",
my grep stays below a few (~3) megabyte.
A limit of 100M seems to be way too much,
but if it thinks it needs that much RAM to find a short string,
then we are very likely not interested in that file.


-- 
: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.