[Pacemaker] Alternative communication engine to corosync (etcd/consul/zookeeper/doozerd)

[Patrick Hemmer]

*From: *Andrew Beekhof <andrew at beekhof.net>
*Sent: * 2014-06-20 04:48:25 EDT
*To: *The Pacemaker cluster resource manager <pacemaker at oss.clusterlabs.org>
*Subject: *Re: [Pacemaker] Alternative communication engine to corosync
(etcd/consul/zookeeper/doozerd)

> On 20 Jun 2014, at 2:14 pm, Patrick Hemmer <pacemaker at feystorm.net> wrote:
>
>> After the demise of the old heartbeat service, and the switch to corosync as the primary (sole) method of communication between nodes,
> heartbeat is still supported as a messaging/membership layer
>
>> has there ever been any consideration into using services such as etcd, consul, zookeeper, or doozerd?
> I think most of these didn't exist at the time.
> Do any provide membership too?  What about message ordering?
Yes. N/A (see below on the CPG vs KVS).
>
>> These alternative communication engines offer some stuff that corosync doesn't. One such item etcd & consul offer is dynamic cluster scaling capabilities (nodes can very easily join and leave the cluster). When working in cloud computing, this feature becomes very important. Pcs is somewhat helpful in this regard but it's still nowhere near as capable (plus corosync doesn't have downscaling finished).
>> However one critical difference between these services and corosync is that they are mainly key/value stores, and don't have something like Corosync's CPG.
> Yes, a rather critical difference.
>
>> Though you could probably implement something looking like CPG using a keyvalue store,
> I rather doubt it.  k/v stores and CPG are not very alike from where I'm sitting.
No, they are not alike, but you could implement something looking like CPG.
When a key is created, that's a CPG message. They support atomic
operations (collision checking), so no 2 nodes could update a key at the
same time. So while it's a different underlying system, the end result
is the same, ordered messages.
>> I think pacemaker might be able to use a key/value store natively.
But I wouldn't even bother with hacking a KVS into something like CPG if
it's not needed. I would do it such that the CIB is stored as keys and
values natively. I would even think this is more efficient. I'm not sure
how the CIB is transmitted between nodes, but I think it easiest to just
set a single key when you want to update something like a resource's
last-rc-change value.

>>
>> So, has this ever been considered? How heavily tied is pacemaker to the corosync API? Could that be abstracted out enough to where different communication engines could be implemented?
> It's already abstracted to the point where it supports 4 different messaging/membership options:
> - heartbeat
> - corosync/pacemaker plugin
> - cman
> - corosync 2
But aside from heartbeat, they all use corosync underneath. I wasn't
aware heartbeat was still supported. I assumed it was dropped. Bad
assumption :-)

>
>>
>> -Patrick
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140620/09e33d10/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 600 bytes
Desc: OpenPGP digital signature
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140620/09e33d10/attachment-0003.sig>