<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 04/12/2018 04:37 AM, David Hunt
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAAGW8eu4Ca+twOV5bRWSu16N9U83k=-jJcrL0eqhjqF4jWPAAQ@mail.gmail.com">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Thanks Guys,<br>
<br>
</div>
</div>
Ideally I would like to have event driven (rather than
slower polled) inputs into pacemaker to quickly trigger
the fall over. I assume adding event driven inputs to
pacemaker isn't straightforward? If it was possible to
add event inputs to pacemaker is pacemaker itself fast
enough? Or is it also going to be relatively slow to
switch?<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
I'm not aware of any systematic delays on top of what we discussed.<br>
The time the rule-engine will need to calculate a transition will<br>
of course depend on the complexity of your cluster and the<br>
CPU-power you have available.<br>
<br>
I've mentioned the delay of the DC reelection but what you<br>
might consider as well in your calculations is fencing.<br>
If you are using physical fencing-devices it depends on how<br>
quickly these will react and give feedback. You might be able<br>
to boost that by e.g. already keeping a control connection<br>
to a fencing device open.<br>
<br>
Regards,<br>
Klaus<br>
<br>
<blockquote type="cite"
cite="mid:CAAGW8eu4Ca+twOV5bRWSu16N9U83k=-jJcrL0eqhjqF4jWPAAQ@mail.gmail.com">
<div dir="ltr">
<div>
<div>
<div>
<div><br>
</div>
</div>
It would seem based on this discussion it may still work
still work to use pacemaker & corosync for initial setup
& handle services which can handle a slower switch over
time. For our services that require a much faster switch
over time it would appear we need something propriety.<br>
<br>
</div>
Regards<br>
</div>
David<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 12 April 2018 at 02:56, Klaus
Wenninger <span dir="ltr"><<a
href="mailto:kwenning@redhat.com" target="_blank"
moz-do-not-send="true">kwenning@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb">
<div class="h5">On 04/11/2018 10:44 AM, Jan Friesse wrote:<br>
> David,<br>
><br>
>> Hi,<br>
>><br>
>> We are planning on creating a HA product in an
active/standby<br>
>> configuration<br>
>> whereby the standby unit needs to take over
from the active unit very<br>
>> fast<br>
>> (<50ms including all services restored).<br>
>><br>
>> We are able to do very fast signaling (say
1000Hz) between the two<br>
>> units to<br>
>> detect failures so detecting a failure isn't
really an issue.<br>
>><br>
>> Pacemaker looks to be a very useful piece of
software for managing<br>
>> resources so rather than roll our own it would
make sense to reuse<br>
>> pacemaker.<br>
>><br>
>> So my initial questions are:<br>
>><br>
>> 1. Do people think pacemaker is the right
thing to use? Everything I<br>
>> read seem to be talking about multiple
seconds for failure<br>
>> detection etc.<br>
>> Feature wise it looks pretty similar to
what we would want.<br>
>> 2. Has anyone done anything similar to
this?<br>
>> 3. Any pointers on where/how to add
additional failure detection<br>
>> inputs<br>
>> to pacemaker?<br>
>> 4.<br>
>> 5. For a new design would you go with
pacemaker+corosync,<br>
>> pacemaker+corosync+knet or something
different?<br>
>><br>
><br>
><br>
> I will just share my point of view about Corosync
side.<br>
><br>
> Corosync is using it's own mechanism for detecting
failure, based on<br>
> token rotation. Default timeout for detecting lost
of token is 1<br>
> second, so detecting failure takes hugely more than
50ms. It can be<br>
> lowered, but that is not really tested.<br>
><br>
> That means it's not currently possible to use
different signaling<br>
> mechanism without significant Corosync change.<br>
><br>
> So I don't think Corosync can be really used for
described scenario.<br>
><br>
> Honza<br>
<br>
</div>
</div>
On the other hand if a fail-over is triggered by loosing a
node or anything<br>
that is being detected by corosync this is probably already
the fast-path<br>
in a pacemaker-cluster.<br>
<br>
Detection of other types of failures (like a resource
failing on<br>
an otherwise functional node) is probably even way slower.<br>
When a failure is detected by corosync, pacemaker has some
kind of<br>
an event driven way to react on that.<br>
We even have to add some delay to the mere corosync
detection time<br>
mentioned by Honza as pacemaker will have to run e.g. a
selection<br>
cycle for the designated coordinator to be able to do
decisions again.<br>
<br>
For other failures the base principle is rather probing a
resource at a<br>
fixed rate (multiple seconds usually) for detection of
failures instead<br>
of an event-driven mechanism.<br>
There might be trickery possible though using attributes to
achieve<br>
event-driven-like reaction on certain failures. But I
haven't done<br>
anything concrete to exploit these possibilities. Others
might have<br>
more info (which I personally would be interested in as well
;-) ).<br>
<br>
Approaches to realize event-driven mechanisms for
resource-failure-<br>
detection are under investigation/development
(systemd-resources,<br>
IP resources sitting on interfaces, ...) but afaik there is
nothing<br>
available out of the box by now.<br>
<br>
Having that all said I can add some personal experiences
from<br>
having implemented an embedded product based on a<br>
pacemaker-cluster myself in the past:<br>
<br>
As reaction time based on pacemaker would be too slow for
e.g.<br>
many communication-protocols (e.g. things like SIP) or
realtime-<br>
streams it seems advisable to solve these issues on the<br>
application-layer inside a service (respectively distributed
service<br>
in a cluster).<br>
Pacemaker and it's decision engine can then be used to bring<br>
up this distributed service in a cluster in some kind of an
ordered<br>
way.<br>
Any additional services that might be less demanding
regarding<br>
switch-over timeout can be made available via pacemaker<br>
directly.<br>
<br>
Otherwise pacemaker configuration is very flexible so that
you<br>
can implement merely anything. It might be advisable to
avoid<br>
certain approaches which are common in cases where a cluster<br>
is operated by somebody who can be informed quickly and<br>
has to react under certain SLAs. Thinking of e.g. fencing a
node<br>
to be switched off instead of rebooting it might not be
desirable<br>
with kind of an appliance that is expected to just sit there
and<br>
work without merely any admin effort/expense at all.<br>
But that is of course just an example and configuration
(incl.<br>
configuration concept) has to be tailored to your
requirements.<br>
<br>
Regards,<br>
Klaus<br>
<div class="HOEnZb">
<div class="h5"> <br>
><br>
>><br>
>> Thanks<br>
>><br>
>> David<br>
>><br>
>><br>
>><br>
>> ______________________________<wbr>_________________<br>
>> Users mailing list: <a
href="mailto:Users@clusterlabs.org"
moz-do-not-send="true">Users@clusterlabs.org</a><br>
>> <a
href="https://lists.clusterlabs.org/mailman/listinfo/users"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
>><br>
>> Project Home: <a
href="http://www.clusterlabs.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://www.clusterlabs.org</a><br>
>> Getting started: <a
href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
>> Bugs: <a href="http://bugs.clusterlabs.org"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://bugs.clusterlabs.org</a><br>
>><br>
><br>
> ______________________________<wbr>_________________<br>
> Users mailing list: <a
href="mailto:Users@clusterlabs.org"
moz-do-not-send="true">Users@clusterlabs.org</a><br>
> <a
href="https://lists.clusterlabs.org/mailman/listinfo/users"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.clusterlabs.org</a><br>
> Getting started: <a
href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://bugs.clusterlabs.org</a><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<p><br>
</p>
<pre class="moz-signature" cols="72">
</pre>
</body>
</html>