[ClusterLabs Developers] Proposed future feature: multiple notification scripts

[Ken Gaillot]

On 12/03/2015 10:58 AM, Klaus Wenninger wrote:
> On 12/03/2015 05:39 PM, Jan Pokorný wrote:
>> On 03/12/15 17:06 +0100, Klaus Wenninger wrote:
>>> On 12/03/2015 04:45 PM, Jan Pokorný wrote:
>>>> What I am up to is a proposal of an alternative/parallel mechanism
>>>> that better fits the asymmetric (and asynchronous from cluster life
>>>> cycle POV) use cases: old good drop-in files.  There would simply
>>>> be a dedicated directory (say /usr/share/pacemaker/notify.d) where
>>>> the software interested in notifications would craft it's own
>>>> listener script (or a symlink thereof), script is then discovered
>>>> by Pacemaker upon subsequent dir rescan or inotify event, done.

I wouldn't want to introduce a dependency on inotify (especially since
we support non-Linux OSes), and scanning a directory is an expensive
operation. We definitely wouldn't want to do rescan for every cluster
event. Possibly every recheck-interval would be OK. So one question is
how do we trigger a rescan, and whether that might lead to confusion as
to why a drop-in isn't (immediately) working.

But I do think it's an interesting approach worth exploring.

>>>> --> no configuration needed (or is external to the CIB, or is
>>>>     interspersed in a non-invasive way there), install and go
>>>>
>>>> --> it has local-only effect, equally as is local the installation
>>>>     of the respective software utilizing notifications
>>>>     (and as is local handling of the notifications!)
>>>>
>>> Why a dedicated directory so that you can't see in the cib that
>>> some kind of notification is enabled?
>> But you cannot see it truthfully in CIB either!
>>
>> You can only see a pointer that something could be actually run to
>> propagate notifications, but without any guarantees that script
>> actually exist across the nodes, you are at the same level
>> of oblivion until more investigation performed.
> But at least you can assure that nothing unknown is run if you see that
> it is disabled ;-)
> And by e.g. using a path that is proprietary to your cluster you can
> prevent that some
> rpm is putting scripts to the default location (well not that they are
> put there but at
> least they wouldn't have any effect).
>>> If we make the drop-in-directory configurable via the cib alternatively
>>> to the script or even in addition to it this would add transparency.
>> Transparency is hard to achieve here directly.  Configuration tools could
>> give you insights on the notification agents in use regardless the method.

Advantages for the CIB approach:

* It is indeed more transparent. Looking at the cluster configuration,
you can know exactly what scripts might be called, and who might get
notified.

* GUI cluster front-ends can be used to enable/disable scripts and
recipients (once support is added).

* Per-script timeouts are easy.

* I envision eventually enabling rules, so for example, different
recipients could be notified depending on the time of day. A script
could implement that internally, but rules are already a somewhat
familiar mechanism for doing such things in pacemaker, and it would not
have to be reimplemented by every script.

On the other hand, a drop-in directory would require no significant
changes to CIB parsing, and configuration would be trivial.

I don't see being node-specific as a big advantage. I think that would
be more error-prone than helpful, and if really desired, could be done
with the CIB approach anyway. It's easy enough for pacemaker to execute
a script only if it exists.

Either way, I think notifications should be disabled by default, so no
external scripts can possibly be run unless the user explicitly asks for
it. So if we used a drop-in directory, we should have a cluster property
for notification-agent-dir that defaults to empty (disabled), and can be
set to the directory of the user's choice to enable it.

> If we want more then the notification-agent could have a
> monitoring-interface that is
> called regularly as with RAs and fencing-RAs. Excuse me if I have
> overseen this
> consideration already coming up in the thread.

We want notification agents to be as simple as possible, and preferably
backward-compatible with existing scripts used with the ClusterMon
resource. So I think we'd want to stay away from requiring metadata and
monitor commands, etc.

>>> Or have the path setable with a default like your suggestion and an
>>> on/off parameter...
>> All up to the consideration.