[ClusterLabs Developers] Proposed future feature: multiple notification scripts

[Andrew Beekhof]

> On 4 Dec 2015, at 2:45 AM, Jan Pokorný <jpokorny at redhat.com> wrote:
> 
> On 02/12/15 17:23 -0600, Ken Gaillot wrote:
>> This will be of interest to cluster front-end developers and anyone who
>> needs event notifications ...
>> 
>> One of the new features in Pacemaker 1.1.14 will be built-in
>> notifications of cluster events, as described by Andrew Beekhof on That
>> Cluster Guy blog:
>> http://blog.clusterlabs.org/blog/2015/reliable-notifications/
>> 
>> For a future version, we're considering extending that to allow multiple
>> notification scripts, each with multiple recipients. This would require
>> a significant change in the CIB. Instead of a simple cluster property,
>> our current idea is a new configuration section in the CIB, probably
>> along these lines:
>> 
>> <configuration>
>>   <!-- usual crm_config etc. here -->
>> 
>>   <!-- this is the new section -->
>>   <notifications>
>> 
>>      <!-- each script would be in a notify element -->
>>      <notify id="notify-1" path="/my/script.sh" timeout="30s">
>> 
>>         <recipient id="recipient-1" value="me at example.com" />
>>         <!-- etc. for multiple recipients -->
>> 
>>      </notify>
>> 
>>      <!-- etc. for multiple scripts -->
>> 
>>   </notifications>
>> </configuration>
>> 
>> 
>> The recipient values would be passed to the script as command-line
>> arguments (ex. "/my/script.sh me at example.com").
> 
> Just thinking out loud, Pacemaker is well adapted to cope with
> asymmetric/heterogenous nodes (incl. user-assisted optimizations
> like with non-default "resource-discovery" property of a location
> contraint, for instance).
> 
> Setting notifications universally for all nodes may be desired
> in some scenarios, but may not be optimal if nodes may diverge,

Correct always wins over optimal.

I’d not be optimising around scripts that only apply to specific resources that also don’t run everywhere - at most you waste a few cycles.
If that ever becomes a real issue we can add a filter to the notify block.

Far worse is if a service can run somewhere new and you forgot to copy the script across…
The knowledge doesn’t exist to report that as a problem. 

The common scenario will be feeding fencing events into things like galera or nova and sending via different transports, like SNMP, SMS, email.
Particularly sending SNMP alerts into a fully fledged monitoring and alerts system that finds duplicates and does advanced filtering.
We do not and should not be trying to reimplement that.

> or will for sure:
> 
> (1) the script may not be distributed across all the nodes

Thats a bug, not a feature.

>    - or (1b) it is located at the shared storage that will become
>      available later during cluster life cycle because it is
>      a subject of cluster service management as well

How will that script send a notification that the shared storage is no longer available?

> 
> (2) one intentionally wants to run the notification mechanism
>    on a subset of nodes

Can you explain to me when that would be a good idea?
Particularly when those nodes are the only remaining survivors (which you can’t know isn’t the case).
If we don’t care about the services on those nodes, why did we make them HA?

> 
> Note also that once you have the responsibility to distribute the
> script on your own, you can use the same distribution mechanism to
> share your configuration for this script, as an alternative to using
> "value" attribute in the above proposal

So instead of using a standard pool of agents and pcs to set a value, I get to maintain two sets of files on every node in the cluster?
And this is supposed to be a feature?

> (and again, this way, you
> are free to have an asymmetric configuration).  There are tons
> of cases like that and one has to deal with that already (some RAs,
> file with secret for Corosync, ...).
> 
> What I am up to is a proposal of an alternative/parallel mechanism
> that better fits the asymmetric (and asynchronous from cluster life
> cycle POV) use cases: old good drop-in files.  There would simply
> be a dedicated directory (say /usr/share/pacemaker/notify.d) where
> the software interested in notifications would craft it's own
> listener script (or a symlink thereof), script is then discovered
> by Pacemaker upon subsequent dir rescan or inotify event, done.
> 
> --> no configuration needed (or is external to the CIB, or is
>    interspersed in a non-invasive way there), install and go
> 
> --> it has local-only effect, equally as is local the installation
>    of the respective software utilizing notifications
>    (and as is local handling of the notifications!)

Still not a feature.

> 
>> For backward compatibility, the (brand new!) notification-agent and
>> notification-recipient cluster properties would be kept as deprecated
>> shortcuts for a single notify script and recipient.
>> 
>> Also for backward compatibility, the first recipient would be passed to
>> the script as the CRM_notify_recipient environment variable.
>> 
>> This proposal came about because the new notification capability has
>> turned out to be useful enough that people sometimes want to use it for
>> multiple purposes, e.g. email an administrator, and notify some software
>> that an event occurred.
> 
> The proposal might be useful especially for the latter.
> 
>> Trying to fit unrelated actions in one notification script (or a
>> script that calls multiple other scripts) has obvious pitfalls, so
>> this would make it easier on sysadmins.
>> 
>> Another advantage will be a configurable timeout (1.1.14 will have a
>> hardcoded 5-minute timeout for notification scripts).
> 
> There may be catch-all configurable global default that would be
> applied also for drop-in files (replicating metadata framework
> in the notification scripts sounds like over-engineering).
> 
>> The crm_attribute command and the various cluster front-ends would need
>> to be modified to handle the new configuration syntax.
>> 
>> This is all in the idea stage (development is still a ways off), so any
>> comments, suggestions, criticisms, etc. are welcome.
> 
> In the same spirit, please comment on this associated idea.
> 
> -- 
> Jan (Poki)
> _______________________________________________
> Developers mailing list
> Developers at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/developers