[ClusterLabs] Pacemaker with Zookeeper??

[Andrew Beekhof]

tl;dr - dont port pacemaker, use pacemaker-remote instead

On Wed, May 18, 2016 at 5:20 PM, Jan Friesse <jfriesse at redhat.com> wrote:
> Ken Gaillot napsal(a):
>
>> On 05/17/2016 09:54 AM, Digimer wrote:
>>>
>>> On 16/05/16 04:35 AM, Bogdan Dobrelya wrote:
>>>>
>>>> On 05/16/2016 09:23 AM, Jan Friesse wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have an idea: use Pacemaker with Zookeeper (instead of Corosync). Is
>>>>>> it possible?
>>>>>> Is there any examination about that?
>>>>
>>>>
>>>> Indeed, would be *great* to have a Pacemaker based control plane on top
>>>> of other "pluggable" distributed KVS & messaging systems, for example
>>>> etcd as well :)
>>>> I'm looking forward to joining any dev efforts around that, although I'm
>>>> not a Java or Go developer.
>>>
>>>
>>> Part of open source is the freedom to do whatever you want, of course.
>>>
>>> Let me ask though; What problems would zookeeper, etcd or other systems
>>> solve that can't be solved in corosync?
>>>
>>> I ask because the HA community just finished a multi-year effort to
>>> merge different projects into one common HA stack. This has a lot of
>>> benefits to the user base, not least of which is lack of confusion.
>>>
>>> Strikes me that the significant time investment in supporting a new
>>> comms layer would be much more beneficially spent on improving the
>>> existing stack.
>>>
>>> Again, anyone is free to do whatever they want... I just don't see the
>>> motivator personally.
>>>
>>> digimer
>>
>>
>> I see one big difference that is both a strength and a weakness: these
>> other packages have a much wider user base beyond the HA cluster use
>> case. The strength is that there will be many more developers working to
>> fix bugs, add features, etc. The weakness is that most of those
>
>
> This is exactly what I was thinking about during 2.x developement. If
> replacement of Corosync wouldn't make more sense than continue developing of
> Corosync. I was able to accept implementing some features. Sadly, there was
> exactly ONE project which would be able to replace corosync (Spread toolkit)
> which is even less widespread than Corosync.
>
> From my point of view, replacement of corosync must be (at least) able to:
> - Work without quorum

Agreed. It's a non-starter if messaging stops when quorum is lost.

> - Support 2 node clusters
> - Allow multiple links (something like RRP)

Doesn't bonding (and the imminent arrival of knet) make this somewhat optional?

> - Don't include SPOF (so nothing like configuration stored on one node only
> and/or different machine on network)

There can be subtle variations on this.  The pattern in OpenStack is
to have a "management node".
Which sounds like a SPOF but they also require that the service be
able to function without it.  So its a grey area.

> - Provide EVS/VS

Pacemaker could live without this.  Heartbeat didn't provide it either.

> - Provide something like qdevice

Or the ability to create it.  In fairness, Pacemaker has gotten by for
a long long time without it :-)


It would be nice to be considered for the kinds of scaled deployments
that kubernetes and etcd are built for because that's where all the
excitement and mindshare is.  Zookeeper was one of the options I
thought of too, however realistically Pacemaker is not what those
folks are looking for.  At those scales our stack's smarts take a back
seat to the idea that there are so many copies that dozens can die and
the only recovery you need is to maybe start some more copies (because
with so many, there is always a master around somewhere).


For those of us with a need to scale _and_ an appreciation of "real"
resource orchestration, I would argue that a better architecture is a
small traditional cluster managing a much larger pool of
pacemaker-remote nodes.  Putting effort into making that really shine
(especially since its pretty solid already) is likely to have a better
payoff than porting to another messaging layer.


>
> Both zookeeper and etcd builds on top of quite simple to understand
> membership mechanism (zookeeper = elected master, something like amoeba,
> etcd = raft), what's nice, because it means more contributors. Sadly bare
> metal HA must work even in situations where "simple" quorum is not enough.
>
>
>
>> developers are ignorant of HA clustering and could easily cause more
>> problems for the HA use case than they fix.
>>
>> Another potential benefit is the familiarity factor -- people are more
>> comfortable with things they recognize from somewhere else. So it might
>> help Pacemaker adoption, especially in the communities that already use
>> these packages.
>>
>> I'm not aware of any technical advantages, and I wouldn't expect any,
>> given corosync's long HA focus.
>>
>>>>>  From my point of view (and yes, I'm biased), biggest problem of
>>>>> Zookeper
>>>>> is need to have quorum
>>>>>
>>>>> (https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_designing).
>>>>> Direct consequence is inability to tolerate one node failure in 2 node
>>>>> cluster -> no 2 node clusters (and such deployment is extremely
>>>>> popular). Also Corosync can operate completely without quorum.
>>>>>
>>>>> Regards,
>>>>>    Honza
>>>>>
>>>>>>
>>>>>> Thanks for your help!
>>>>>> Hai Nguyen
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org