[Pacemaker] Pacemaker 1.1.12 cib testing, crm_mon doesn't work

Andrew Beekhof andrew at beekhof.net
Fri Jun 13 10:38:55 UTC 2014


On 13 Jun 2014, at 7:51 pm, Johan Huysmans <johan.huysmans at inuits.be> wrote:

> I tested with a custom build libqb rpm (libqb-0.17.0-1.15.f8b4.dirty.el6.x86_64) however that didn't solved the issue.
> So for now I will add the export of the PCMK_ipc_buffer to my bash_profile.

Did you rebuild pacemaker once you had the new libqb stuff installed?
The relevant code can't be compiled in unless it has the new libqb API at build time.

> 
> thx
> Johan
> 
> 
> On 13-06-14 11:23, Johan Huysmans wrote:
>> I exported the PCMK_ipc_buffer in my console before running the crm_mon and everything worked.
>> But I see this more as a workaround (could set this in my bash_profile).
>> 
>> I currently have libqb-0.16.0-2.el6.x86_64 installed, 
>> which version should fix this ? 
>> 
>> Thanks!
>> Johan H.
>> 
>> 
>> On 13-06-14 10:41, Andrew Beekhof wrote:
>>> On 13 Jun 2014, at 5:35 pm, Johan Huysmans <johan.huysmans at inuits.be>
>>>  wrote:
>>> 
>>> 
>>>> Hi,
>>>> 
>>>> The PCMK_ipc_buffer was already set to 10000000 (10M).
>>>> 
>>> Try setting that in your environment too (otherwise the CLI tools wont know about the new value).
>>> Or grab a newer libqb which _should_ do the right thing anyway
>>> 
>>> 
>>>> For testing I increased to buffer to 10000000 (100M), without results.
>>>> 
>>>> I decreased the buffer to the default 20480 (20K) as it then shows the suggested value.
>>>> If I leave it running for a couple of minutes I got these suggested values:
>>>> Jun 13 06:33:53 [62206] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (3854984 bytes suggested)
>>>> Jun 13 06:33:54 [62206] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (7709968 bytes suggested)
>>>> Jun 13 06:33:58 [62206] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (15419936 bytes suggested)
>>>> Jun 13 06:35:55 [62206] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (30839872 bytes suggested)
>>>> Jun 13 06:37:56 [62206] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (61679744 bytes suggested)
>>>> Jun 13 07:16:47 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (3952852 bytes suggested)
>>>> Jun 13 07:16:47 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (7905704 bytes suggested)
>>>> Jun 13 07:18:03 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (15811408 bytes suggested)
>>>> Jun 13 07:18:43 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (31622816 bytes suggested)
>>>> Jun 13 07:18:48 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (63245632 bytes suggested)
>>>> Jun 13 07:20:49 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (126491264 bytes suggested)
>>>> Jun 13 07:22:50 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (252982528 bytes suggested)
>>>> Jun 13 07:22:52 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (505965056 bytes suggested)
>>>> Jun 13 07:23:57 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (1011930112 bytes suggested)
>>>> Jun 13 07:24:51 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (2023860224 bytes suggested)
>>>> Jun 13 07:26:52 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (-247246848 bytes suggested)
>>>> Jun 13 07:27:22 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (-494493696 bytes suggested)
>>>> Jun 13 07:29:22 [44112] SRV-5-1        cib:    error: crm_ipc_prepare:     Could not compress the message into less than the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a higher value (-988987392 bytes suggested)
>>>> 
>>>> There is definitely something wrong. Is it the printing of the suggested value or is it something else ?
>>>> 
>>>> If I check the cib.xml files in /var/lib/pacemaker/cib/ all files are a bit smaller then 300K.
>>>> 
>>>> Changing these buffers did not solve my problem not getting results from crm_mon.
>>>> 
>>>> Gr.
>>>> Johan
>>>> 
>>>> 
>>>> On 13-06-14 01:13, Andrew Beekhof wrote:
>>>> 
>>>>> On 12 Jun 2014, at 10:53 pm, Johan Huysmans <johan.huysmans at inuits.be>
>>>>> 
>>>>>  wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> I deployed Pacemaker 1.1.12-rc2 on our platform to test the cib changes.
>>>>>> This was needed on our setup as it contains 6 nodes, 150 resources and the cib process was using lots of cpu.
>>>>>> 
>>>>>> With a limited set of resources (6 nodes, 30 resources) everything worked as expected, including crm_mon.
>>>>>> When loading the complete set of resources we lost the crm_mon functionality on all nodes.
>>>>>> The cluster is running as expected (running all resources) however we don't have any visibility.
>>>>>> 
>>>>>> I noticed that operations performing changes did actually work like (crm resource stop <resourcename>),
>>>>>> but crm resource status didn't work (using crmsh-2.0+git46-1.1.x86_64).
>>>>>> 
>>>>>> I noticed that /dev/shm/qb-cib_ro* files are created, and lsof shows that they are both opened by crm_mon and cib.
>>>>>> 
>>>>>> 
>>>>>> When executing "crm_mon -1" I get following messages in /var/log/messages (and /var/log/pacemaker.log)
>>>>>> Jun 12 12:47:38 [8062] SRV-5-1        cib:   notice: crm_ipcs_sendv:     Response 2 to 0x1810370[17836] (1091618 bytes) failed: Resource temporarily unavailable (-11)
>>>>>> Jun 12 12:47:38 [8062] SRV-5-1        cib:  warning: do_local_notify:     Sync reply to crm_mon failed: No message of desired type
>>>>>> 
>>>>>> 
>>>>>> Restarting the pacemaker and cman service of 1 node didn't solve it.
>>>>>> 
>>>>>> 
>>>>>> What is causing this problem and how can I resolve it ?
>>>>>> 
>>>>>> 
>>>>> Almost certainly you're hitting IPC limits associated with large clusters.
>>>>> 
>>>>> You should be able to tune:
>>>>> 
>>>>> # PCMK_ipc_buffer=20480
>>>>> 
>>>>> In /etc/sysconfig/pacemaker and then restart the cluster.
>>>>> 
>>>>> Note also:
>>>>> 
>>>>> # For non-systemd based systems, prefix 'export' to each enabled line
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Pacemaker mailing list: 
>>>>> 
>>>>> Pacemaker at oss.clusterlabs.org
>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>> 
>>>>> 
>>>>> 
>>>>> Project Home: 
>>>>> 
>>>>> http://www.clusterlabs.org
>>>>> 
>>>>> 
>>>>> Getting started: 
>>>>> 
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> 
>>>>> 
>>>>> Bugs: 
>>>>> 
>>>>> http://bugs.clusterlabs.org
>>>> _______________________________________________
>>>> Pacemaker mailing list: 
>>>> Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>> 
>>>> 
>>>> Project Home: 
>>>> http://www.clusterlabs.org
>>>> 
>>>> Getting started: 
>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> 
>>>> Bugs: 
>>>> http://bugs.clusterlabs.org
>>> 
>>> 
>>> _______________________________________________
>>> Pacemaker mailing list: 
>>> Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>> 
>>> 
>>> Project Home: 
>>> http://www.clusterlabs.org
>>> 
>>> Getting started: 
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> 
>>> Bugs: 
>>> http://bugs.clusterlabs.org
>> 
>> 
>> 
>> _______________________________________________
>> Pacemaker mailing list: 
>> Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> 
>> Project Home: 
>> http://www.clusterlabs.org
>> 
>> Getting started: 
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> 
>> Bugs: 
>> http://bugs.clusterlabs.org
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140613/be1f6dbf/attachment-0004.sig>


More information about the Pacemaker mailing list