[ClusterLabs] Syncing data and reducing CPU utilization of cib process
nikhil.subscribed at gmail.com
Mon Apr 3 02:08:03 EDT 2017
The CIB file is not being updated that often.
I took a packet capture and don't see the node sending any message to other
nodes (other than keep-alives).
What then explains these messages coming every second?
On Sat, Apr 1, 2017 at 1:37 AM, Ken Gaillot <kgaillot at redhat.com> wrote:
> On 03/31/2017 06:44 AM, Nikhil Utane wrote:
> > We are seeing this log in pacemaker.log continuously.
> > Mar 31 17:13:01  0005B932ED72 cib: info:
> > crm_compress_string: Compressed 436756 bytes into 14635 (ratio 29:1) in
> > 284ms
> > This looks to be the reason for high CPU. What does this log indicate?
> If a cluster message is larger than 128KB, pacemaker will compress it
> (using BZ2) before transmitting it across the network to the other
> nodes. This can hit the CPU significantly. Having a large resource
> definition makes such messages more common.
> There are many ways to sync a configuration file between nodes. If the
> configuration rarely changes, a simple rsync cron could do it.
> Specialized tools like lsyncd are more responsive while still having a
> minimal footprint. DRBD or shared storage would be more powerful and
> real-time. If it's a custom app, you could even modify it to use
> something like etcd or a NoSQL db.
> > -Regards
> > Nikhil
> > On Fri, Mar 31, 2017 at 12:08 PM, Nikhil Utane
> > <nikhil.subscribed at gmail.com <mailto:nikhil.subscribed at gmail.com>>
> > Hi,
> > In our current design (which we plan to improve upon) we are using
> > the CIB file to synchronize information across active and standby
> > Basically we want the standby node to take the configuration that
> > was used by the active node so we are adding those as resource
> > attributes. This ensures that when the standby node takes over, it
> > can read all the configuration which will be passed to it as
> > environment variables.
> > Initially we thought the list of configuration parameters will be
> > less and we did some prototyping and saw that there wasn't much of
> > an issue. But now the list has grown it has become close to 300
> > attributes. (I know this is like abusing the feature and we are
> > looking towards doing it the right way).
> > So I have two questions:
> > 1) What is the best way to synchronize such kind of information
> > across nodes in the cluster? DRBD? Anything else that is simpler?
> > For e.g. instead of syncing 300 attributes i could just sync up the
> > path to a file.
> > 2) In the current design, is there anything that I can do to reduce
> > the CPU utilization of cib process? Currently it regularly takes
> > 30-50% of the CPU.
> > Any quick fix that I can do which will bring it down? For e.g.
> > configure how often it synchronizes etc?
> > -Thanks
> > Nikhil
> Users mailing list: Users at clusterlabs.org
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Users