[ClusterLabs] Antw: Re: Antw: Placing resource based on least load on a node

[Bernd]

Am 2018-11-20 09:08, schrieb Michael Schwartzkopff:
> Am 20.11.18 um 08:57 schrieb Ulrich Windl:
>>>>> Michael Schwartzkopff <ms at sys4.de> schrieb am 20.11.2018 um 08:41 
>>>>> in Nachricht
>> <e37932d6-add8-63ce-14c7-dd71df12e203 at sys4.de>:
>>> Am 20.11.18 um 08:35 schrieb Bernd:
>>>> Am 2018-11-20 08:06, schrieb Ulrich Windl:
>>>>>>>> Bernd <bernd at kroenchenstadt.de> schrieb am 20.11.2018 um 07:21 
>>>>>>>> in
>>>>>>>> Nachricht
>>>>> <dbae607c63168d4e14584abfba0b4b1b at kroenchenstadt.de>:
>>>>>> Hi,
>>>>>> 
>>>>>> I'd like to run a certain bunch of cronjobs from time to time on 
>>>>>> the
>>>>>> cluster node (four node cluster) that has the lowest load of all 
>>>>>> four
>>>>>> nodes.
>>>>>> 
>>>>>> The parameters wanted for this system yet to build are
>>>>>> 
>>>>>> * automatic placement on one of the four nodes (i.e., that with 
>>>>>> the
>>>>>> lowest load)
>>>>>> 
>>>>>> * in case a node fails, automatically removed from the cluster
>>>>>> 
>>>>>> * it must only exist a single entity of the cronjob entity running
>>>>>> 
>>>>>> so this really screams for pacemakter being used as foundation.
>>>>>> 
>>>>>> However, I'm not sure how to implement the "put onto node with 
>>>>>> least
>>>>>> load" part. I was thinking to use Node Attributes for that, but I
>>>>>> didn't
>>>>>> find any solution "out of the box" for this. Furthermore, as load 
>>>>>> is a
>>>>>> highly volatile value, how can one make sure that all cronjobs are 
>>>>>> run
>>>>>> to the end without being moved to a node that possibly meanwhile 
>>>>>> got a
>>>>>> lower load than the one executing the jobs?
>>>>> Hi!
>>>>> 
>>>>> Actually I think the last one is the easiest (assuming the cron 
>>>>> jobs
>>>>> do not need any resources that are moved): Once a cron job is 
>>>>> started,
>>>>> it will run until it ends, whether it's crontab has been moved or 
>>>>> not.
>>>>> 
>>>>> Despite of that I think cluster software is not ideal when you
>>>>> actually need load-balancing software.
>>>>> 
>>>>> Regards,
>>>>> Ulrich
>>>> The only resource(s) existing would be the cron "runner".
>>>> 
>>>> The point about load balancing is true, yes... so, any idea what to
>>>> use instead? Is there already a tool or framework for solving a
>>>> problem like this available or do I have to start from scratch? Not
>>>> that I'd be too lazy, but what's the use of reinventing the wheel
>>>> repeatedly...? ;)
>>>> 
>>>> Regards,
>>>> 
>>>> Bernd
>>>> _______________________________________________
>>>> Users mailing list: Users at clusterlabs.org
>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>> 
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started: 
>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>> 
>>> hi,
>>> 
>>> 
>>> I solved this problem years ago. I used the utilization attribute. 
>>> But
>>> you can use any attribute. You have to write an agent that measures 
>>> the
>>> CPU load every X minutes and updates the attribute.  Now you just 
>>> have
>>> to add a location constraint, that starts the resource on the node 
>>> with
>>> the "best" attribute value. The "best" could be lowest CPU usage or 
>>> most
>>> free RAM or whatever you want.
>>> 
>>> 
>>> The disadvantage of this solution is that the cluster (i.e. 
>>> pacemaker)
>>> has to recalculate the scores every time you update your attribute. 
>>> That
>>> causes additional load. If you have many resources the interdepend 
>>> that
>>> additional load may be not negligible.
>> Hi!
>> 
>> Question on this: Is the cluster clever to check only updates of 
>> attributes that some rule actually uses, or does it re-evaluate 
>> everything when any attribute changed?
>> 
> Everytime. That is what causes the load.

My thought was to update a variable stored per node which contains the 
value of the system load avg over the last 15 minutes, which is 
extremely easy to gather. Based on that, crmd could to its job. Every 
ten minutes would be more than sufficient, as it's not a real cluster 
needed here. (Well, this seems to be an extreme rare use case, though.)

Bernd