[ClusterLabs Developers] Pacemaker: possibilities of schedulerd optimization

[Denis Koptev]

Hello!

My team uses Pacemaker as a cluster manager. Our configuration is large enough, therefore we have lots of resources and constraints for them.
While using Pacemaker we noticed that its decision making algorithm works slow for large configurations. We gathered some flame graphs to determine where the problem could be.
And according to the flame graph for schedulerd which is attached to this email we can say that the execution of decision making part takes a lot of time.
In particular, stage7 which is responsible for applying ordering constraints and updating actions for all resources is complicated enough.

We did some investigation of Pacemaker code and we have some assumptions how we can potentially speed-up schedulerd.
So I’d very appreciate if you share your thoughts and knowledge on this topic.

The main concern is that currently schedulerd re-calculates all meta-data once notification from controld is received.
This requires querying the entire CIB and analyzing it in schedulerd code applying all constraints from scratch. However, it can be inefficient in case of small CIB changes.
BTW, controld receives CIB diff, but still provides schedulerd with entire CIB.
So, what if we could pass CIB diff to schedulerd instead of the entire CIB and modify schedulerd logic in order to store all previous meta-data and apply diff to it somehow?
Knowing what changed (or what type/part of configuration changed) can potentially allow us to skip processing of some (large or small part of) resources and actions.

Could you tell me please, what do you think about it? Perhaps you already faced this issue and had some ideas or even patches for schedulerd to speed it up.
The version of Pacemaker we us is 2.0.1+20190417.13d370ca9 (release 3.6.1).

Thanks a lot in advance!

--
Denis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/developers/attachments/20200706/6eff9cca/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pacemaker-schedulerd-flamegraph-hw-Feb20.svg
Type: image/svg+xml
Size: 1292476 bytes
Desc: pacemaker-schedulerd-flamegraph-hw-Feb20.svg
URL: <http://lists.clusterlabs.org/pipermail/developers/attachments/20200706/6eff9cca/attachment-0001.svg>