[ClusterLabs Developers] Proposed future feature: multiple notification scripts

[Jan Pokorný]

On 02/12/15 17:23 -0600, Ken Gaillot wrote:
> This will be of interest to cluster front-end developers and anyone who
> needs event notifications ...
> 
> One of the new features in Pacemaker 1.1.14 will be built-in
> notifications of cluster events, as described by Andrew Beekhof on That
> Cluster Guy blog:
> http://blog.clusterlabs.org/blog/2015/reliable-notifications/
> 
> For a future version, we're considering extending that to allow multiple
> notification scripts, each with multiple recipients. This would require
> a significant change in the CIB. Instead of a simple cluster property,
> our current idea is a new configuration section in the CIB, probably
> along these lines:
> 
> <configuration>
>    <!-- usual crm_config etc. here -->
> 
>    <!-- this is the new section -->
>    <notifications>
> 
>       <!-- each script would be in a notify element -->
>       <notify id="notify-1" path="/my/script.sh" timeout="30s">
> 
>          <recipient id="recipient-1" value="me at example.com" />
>          <!-- etc. for multiple recipients -->
> 
>       </notify>
> 
>       <!-- etc. for multiple scripts -->
> 
>    </notifications>
> </configuration>
> 
> 
> The recipient values would be passed to the script as command-line
> arguments (ex. "/my/script.sh me at example.com").

Just thinking out loud, Pacemaker is well adapted to cope with
asymmetric/heterogenous nodes (incl. user-assisted optimizations
like with non-default "resource-discovery" property of a location
contraint, for instance).

Setting notifications universally for all nodes may be desired
in some scenarios, but may not be optimal if nodes may diverge,
or will for sure:

(1) the script may not be distributed across all the nodes
    - or (1b) it is located at the shared storage that will become
      available later during cluster life cycle because it is
      a subject of cluster service management as well

(2) one intentionally wants to run the notification mechanism
    on a subset of nodes

Note also that once you have the responsibility to distribute the
script on your own, you can use the same distribution mechanism to
share your configuration for this script, as an alternative to using
"value" attribute in the above proposal (and again, this way, you
are free to have an asymmetric configuration).  There are tons
of cases like that and one has to deal with that already (some RAs,
file with secret for Corosync, ...).

What I am up to is a proposal of an alternative/parallel mechanism
that better fits the asymmetric (and asynchronous from cluster life
cycle POV) use cases: old good drop-in files.  There would simply
be a dedicated directory (say /usr/share/pacemaker/notify.d) where
the software interested in notifications would craft it's own
listener script (or a symlink thereof), script is then discovered
by Pacemaker upon subsequent dir rescan or inotify event, done.

--> no configuration needed (or is external to the CIB, or is
    interspersed in a non-invasive way there), install and go

--> it has local-only effect, equally as is local the installation
    of the respective software utilizing notifications
    (and as is local handling of the notifications!)

> For backward compatibility, the (brand new!) notification-agent and
> notification-recipient cluster properties would be kept as deprecated
> shortcuts for a single notify script and recipient.
> 
> Also for backward compatibility, the first recipient would be passed to
> the script as the CRM_notify_recipient environment variable.
> 
> This proposal came about because the new notification capability has
> turned out to be useful enough that people sometimes want to use it for
> multiple purposes, e.g. email an administrator, and notify some software
> that an event occurred.

The proposal might be useful especially for the latter.

> Trying to fit unrelated actions in one notification script (or a
> script that calls multiple other scripts) has obvious pitfalls, so
> this would make it easier on sysadmins.
> 
> Another advantage will be a configurable timeout (1.1.14 will have a
> hardcoded 5-minute timeout for notification scripts).

There may be catch-all configurable global default that would be
applied also for drop-in files (replicating metadata framework
in the notification scripts sounds like over-engineering).

> The crm_attribute command and the various cluster front-ends would need
> to be modified to handle the new configuration syntax.
> 
> This is all in the idea stage (development is still a ways off), so any
> comments, suggestions, criticisms, etc. are welcome.

In the same spirit, please comment on this associated idea.

-- 
Jan (Poki)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/developers/attachments/20151203/e88f7144/attachment-0004.sig>