[ClusterLabs Developers] Proposed future feature: multiple notification scripts

[Jan Pokorný]

On 03/12/15 12:56 -0600, Ken Gaillot wrote:
> On 12/03/2015 10:58 AM, Klaus Wenninger wrote:
>> On 12/03/2015 05:39 PM, Jan Pokorný wrote:
>>> On 03/12/15 17:06 +0100, Klaus Wenninger wrote:
>>>> On 12/03/2015 04:45 PM, Jan Pokorný wrote:
>>>>> What I am up to is a proposal of an alternative/parallel mechanism
>>>>> that better fits the asymmetric (and asynchronous from cluster life
>>>>> cycle POV) use cases: old good drop-in files.  There would simply
>>>>> be a dedicated directory (say /usr/share/pacemaker/notify.d) where
>>>>> the software interested in notifications would craft it's own
>>>>> listener script (or a symlink thereof), script is then discovered
>>>>> by Pacemaker upon subsequent dir rescan or inotify event, done.
> 
> I wouldn't want to introduce a dependency on inotify (especially since
> we support non-Linux OSes), and scanning a directory is an expensive
> operation. We definitely wouldn't want to do rescan for every cluster
> event. Possibly every recheck-interval would be OK. So one question is
> how do we trigger a rescan, and whether that might lead to confusion as
> to why a drop-in isn't (immediately) working.
> 
> But I do think it's an interesting approach worth exploring.
> 
>>>>> --> no configuration needed (or is external to the CIB, or is
>>>>>     interspersed in a non-invasive way there), install and go
>>>>> 
>>>>> --> it has local-only effect, equally as is local the installation
>>>>>     of the respective software utilizing notifications
>>>>>     (and as is local handling of the notifications!)
>>>>> 
>>>> Why a dedicated directory so that you can't see in the cib that
>>>> some kind of notification is enabled?
>>> But you cannot see it truthfully in CIB either!
>>> 
>>> You can only see a pointer that something could be actually run to
>>> propagate notifications, but without any guarantees that script
>>> actually exist across the nodes, you are at the same level
>>> of oblivion until more investigation performed.
>> But at least you can assure that nothing unknown is run if you see that
>> it is disabled ;-)
>> And by e.g. using a path that is proprietary to your cluster you can
>> prevent that some
>> rpm is putting scripts to the default location (well not that they are
>> put there but at
>> least they wouldn't have any effect).
>>>> If we make the drop-in-directory configurable via the cib alternatively
>>>> to the script or even in addition to it this would add transparency.
>>> Transparency is hard to achieve here directly.  Configuration tools could
>>> give you insights on the notification agents in use regardless the method.
> 
> Advantages for the CIB approach:
> 
> * It is indeed more transparent. Looking at the cluster configuration,
> you can know exactly what scripts might be called, and who might get
> notified.
> 
> * GUI cluster front-ends can be used to enable/disable scripts and
> recipients (once support is added).

Ditto drop-in approach with some more complexity put on these.

> * Per-script timeouts are easy.
> 
> * I envision eventually enabling rules, so for example, different
> recipients could be notified depending on the time of day. A script
> could implement that internally, but rules are already a somewhat
> familiar mechanism for doing such things in pacemaker, and it would not
> have to be reimplemented by every script.

This is where CIB approach would become really appealing in my book.

> On the other hand, a drop-in directory would require no significant
> changes to CIB parsing, and configuration would be trivial.
> 
> I don't see being node-specific as a big advantage. I think that would
> be more error-prone than helpful, and if really desired, could be done
> with the CIB approach anyway. It's easy enough for pacemaker to execute
> a script only if it exists.
> 
> Either way, I think notifications should be disabled by default, so no
> external scripts can possibly be run unless the user explicitly asks for
> it. So if we used a drop-in directory, we should have a cluster property
> for notification-agent-dir that defaults to empty (disabled), and can be
> set to the directory of the user's choice to enable it.

Sounds reasonable.

>> If we want more then the notification-agent could have a
>> monitoring-interface that is
>> called regularly as with RAs and fencing-RAs. Excuse me if I have
>> overseen this
>> consideration already coming up in the thread.
> 
> We want notification agents to be as simple as possible, and preferably
> backward-compatible with existing scripts used with the ClusterMon
> resource. So I think we'd want to stay away from requiring metadata and
> monitor commands, etc.

Agreed on this, no extra interaction should be necessary.
Notification is a one-way information propagation.


Btw. was any other architectural approach considered?  It's sad that
multiplatform IPC, which is what might be better to handle more or less
continuous one-way stream of updates (would the exec mechanism apply
some kind of rate-limiting to prevent exhaustion?), but what about
long-lived unix sockets or named pipes?  Especially the latter might be
doable in the agent just in shell + coreutils and other basic tooling
and might play well with file discovery approach.

-- 
Jan (Poki)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/developers/attachments/20151204/7619700d/attachment-0004.sig>