[ClusterLabs] PCSD - High Memory Usage

[Tomas Jelinek]

Hi,

We (pcs developers) do not have any statistics regarding pcsd memory 
consumption on a cluster with that many nodes and resources. So I cannot 
confirm if this is normal or not. That being said, it seems to me the 
memory usage is higher than it should be.

Over the time, there has been reports of pcsd consuming unreasonably 
high amounts of memory. These issues are hard to reproduce and they do 
not happen every time so we haven't been able to track them down and 
fix. The problem may not even be in pcsd code itself, it may be buried 
in rubygems, libraries or ruby. (There was a bug in ruby threads causing 
pcsd using 100% CPU.)

We are working on overhauling pcsd to a new architecture while moving 
its codebase from Ruby to Python. The architecture gives us more control 
over pcsd worker threads which should help us lower pcsd memory 
footprint. It is a long term goal, however, definitely not something 
which would be ready in a few months.

For now the only advice I have is to restart pcsd from time to time when 
you think its memory footprint is too high. Or, if you don't use web UI, 
you can stop pcsd completely. The cluster does not depend on pcsd 
running, it is only needed for managing the cluster.

I understand this is far from a solution but at this time I cannot offer 
anything else.


Tomas


Dne 21. 06. 19 v 13:32 Daniel Brant napsal(a):
> Hi,
> 
> I'm running a cluster with 9 active nodes (4 in standby), on this 
> cluster are 258 registered resources. 99% of these are perl and java 
> process ran via multi-target systemd units, 2 are heartbeat IPaddr2 and 
> 1 fence device. I am seeing very high memory usage in the pcsd process, 
> 2.5GB to 2.7GB on most node with one running 3.3GB.
> 
> Is this kind of memory usage to be expected when managing this volume of 
> resources? I am running pacemaker version 1.1.19 on CentOS7.6.1810.
> 
> Any suggestions or advice is greatly appreciated.
> 
> Thanks,
> 
> Danny
> 
> 
> This email is from the Press Association. For more information, see 
> www.pressassociation.com. This email may contain confidential 
> information. Only the addressee is permitted to read, copy, distribute 
> or otherwise use this email or any attachments. If you have received it 
> in error, please contact the sender immediately. Any opinion expressed 
> in this email is personal to the sender and may not reflect the opinion 
> of the Press Association. Any email reply to this address may be subject 
> to interception or monitoring for operational reasons or for lawful 
> business practices.
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
>