[Pacemaker] RHEL/centos6 - cman+pacemaker cluster with lots of resources - crash

[Nikola Ciprich]

Hi,

as a part of my effort with tuning (and testing) PCMK_ipc_buffer value,
I've get into situation where all pacemaker tools start dying..

My setup:

CentOS 6 + all updates, two nodes CMAN + pacemaker cluster.

I've changed following /etc/sysconfig/pacemaker values (ipc_buffer size
as recommended by pacemaker log):

PCMK_ipc_buffer=13828900
PCMK_ipc_type=shared-mem
PCMK_STACK=cman

I've also exported all those variables using .bash_profile so they're available
to CLI tools.

Then I've populated CIB with 5000 Dummy resources. Since config commit,
all tools start aborting:

cibadmin -Ql
Aborted

Here's the backtrace:

(gdb) bt
#0  0x00007ffe26c21925 in raise () from /lib64/libc.so.6
#1  0x00007ffe26c23105 in abort () from /lib64/libc.so.6
#2  0x00007ffe28c44c4b in crm_abort (file=0x7ffe28c62d95 "ipc.c", function=0x7ffe28c63c10 "crm_ipc_decompress", line=856, 
    assert_condition=0x7ffe28c63110 "(header->size_uncompressed + hdr_offset) >= ipc_buffer_max", do_core=<value optimized out>, do_fork=<value optimized out>)
    at utils.c:1118
#3  0x00007ffe28c4011e in crm_ipc_decompress (client=0x19f6f00) at ipc.c:856
#4  0x00007ffe28c4340b in crm_ipc_read (client=0x19f6f00) at ipc.c:885
#5  0x00007ffe28c5dcbe in mainloop_gio_callback (gio=<value optimized out>, condition=G_IO_IN, data=0x19f5990) at mainloop.c:624
#6  0x00007ffe26517eb2 in g_main_dispatch (context=0x19f5a70) at gmain.c:2149
#7  g_main_context_dispatch (context=0x19f5a70) at gmain.c:2702
#8  0x00007ffe2651bd68 in g_main_context_iterate (context=0x19f5a70, block=1, dispatch=1, self=<value optimized out>) at gmain.c:2780
#9  0x00007ffe2651c275 in g_main_loop_run (loop=0x19fafb0) at gmain.c:2988
#10 0x0000000000402a42 in main (argc=<value optimized out>, argv=<value optimized out>) at cibadmin.c:502

Not sure whether this is of any help, if I could provide more information,
please let me know..

BR

nik

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis at linuxbox.cz
-------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140421/89aa4c88/attachment-0003.sig>