[ClusterLabs Developers] Proposed future feature: multiple notification scripts

[Ken Gaillot]

On 12/12/2015 12:26 PM, Jan Pokorný wrote:
> On 07/12/15 09:41 +1100, Andrew Beekhof wrote:
>>> On 5 Dec 2015, at 4:22 AM, Ken Gaillot <kgaillot at redhat.com>
>>> wrote: On 12/04/2015 10:59 AM, Jan Pokorný wrote:
>>>> (would the exec mechanism apply some kind of rate-limiting to
>>>> prevent exhaustion?), but what about
>>> 
>>> There is no rate limiting on the Pacemaker end. If there winds
>>> up being a big demand for it, we can look into it, but that is
>>> more likely to be useful within the script itself (a script
>>> that notifies another service of status changes likely does not
>>> want rate limiting, but an SMS notifier sure might).
>> 
>> agreed. this belongs in the scripts or some intermediate party.
> 
> More deeper reasoning behind the rate limiting question was that
> if a fork-exec-forget style is to be applied, you can easily lose 
> the event ordering (be it due to a bad scheduler, high load or not
> covering the machine with a tin foil hat) which may or may not be
> acceptable in some cases.

There was a recent pull request to address exactly that problem.
However due to time pressures, I'm going to wait until 1.1.14 is out
to give notifications more attention.

The first thing that comes to mind is that, even if Pacemaker ensures
notifications are initiated in order, one particular run of the script
could still take longer than the next one, so the actual notifications
might be sent out of order anyway. Waiting for one notification to
complete before sending the next would be too prone to backlogs. I
haven't put much thought into it yet, but I'm thinking one option
would be for Pacemaker to provide a serial number, and let the script
handle any serialization it needs.