<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 7/20/20 10:34 AM, Andrei Borzenkov
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAA91j0XLH6C+754FjCAfqrdZhD9GxGV2mTD2uFHOUDu+fAvPVA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default"
style="font-family:arial,sans-serif"><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Jul 20, 2020 at
10:56 AM Klaus Wenninger <<a
href="mailto:kwenning@redhat.com" moz-do-not-send="true">kwenning@redhat.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">On 7/19/20 8:55 AM,
Strahil Nikolov wrote:<br>
> My understanding is that fence_xvm is reaching each
Hypervisour via multicast (otherwise why multicast ?)...
yet I could be simply fooling myself.<br>
Not as far as I know. Otherwise external/libvirt would be
able to<br>
do the trick as well.<br>
afaik it should be able to cope with multiple hosts
listening on<br>
the same multicast-address.</blockquote>
<div><br>
</div>
<div>
<div style="font-family:arial,sans-serif"
class="gmail_default">No, it cannot. fence_virtd connects
back to fence_xvm (or fence_virt) client (using its source
address) and actual fencing request is then sent via this
point to point channel. fence_xvm accepts only the first
connection request. So even if multiple fecne_virtd will
receive the initial multicast, only one of them will be
able to connect to fence_xvm that sent this multicast.</div>
<div style="font-family:arial,sans-serif"
class="gmail_default"><br>
</div>
<div style="font-family:arial,sans-serif"
class="gmail_default">fence_virtd itself is dummy
middleman. It accepts messages from fence_xvm/fence_virtd
via the defined listener, connects back, receives actual
fencing request and forwards it to the configured backend
plugin. It performs no check itself whether the initial
message is applicable - only the backend plugin knows what
VM it can handle. And even if it did, because information
is received only after reverse connection is established,
it is already too late anyway.<br>
</div>
</div>
<div>
<div style="font-family:arial,sans-serif"
class="gmail_default">So if you are using muticast and
your backend can only handle VM on local host, it is
unpredictable which host will make reverse connection.
Which is why RH KB article describes configuration with
unique multicast address for each host. At which point you
can simply use unicast address and spare all troubles
associated with multicast.</div>
<div style="font-family:arial,sans-serif"
class="gmail_default"><br>
</div>
<div style="font-family:arial,sans-serif"
class="gmail_default">I do not see any use case for
multicast. Either backend can fence every VM (i.e. backend
is capable of contacting host that runs VM that has to be
fenced) in which case you can simply contact local
fence_virtd via e.g. vsock listener, or backend can handle
only local VM (libvirt backend) in which case you must
contact host where VM is running via unique address.<br>
</div>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"> But there were at least
issues<br>
with that - don't remember the details but remember a
discussion<br>
on some mailinglist - which is why I had suggested - as Reid
just<br>
repeated - quite in the beginning of this thread to go with
a setup<br>
that has 2 fencing-resources, 2 fence_virtd services
listening on<br>
different multicast addresses ...<br>
</blockquote>
<div><br>
</div>
<div>
<div style="font-family:arial,sans-serif"
class="gmail_default">At which point you can just use tcp
listener with normal unicast.</div>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
The cpg-configuration sounds interesting as well. Haven't
used<br>
it or looked into the details. Would be interested to hear
about<br>
how that works.<br>
</blockquote>
<div><br>
</div>
<div>
<div style="font-family:arial,sans-serif"
class="gmail_default">It maintains a registry of VM
location (each fence_virtd polls local hypervisor at
regular intervals) and forwards fencing request to
appropriate host via corosync interconnect. It is also the
only backend that can handle host failure - if it is known
that host left cluster, any VM on this host is considered
fenced by definition.</div>
<div style="font-family:arial,sans-serif"
class="gmail_default"><br>
</div>
<div style="font-family:arial,sans-serif"
class="gmail_default">It requires that hosts are
configured in pacemaker cluster themselves (to handle host
outage it must be properly fenced).<br>
</div>
</div>
</div>
</div>
</blockquote>
That sounds definitely interesting.<br>
Are you saying that the hosts have to be pacemaker-nodes as well?<br>
Otherwise we might be able to just add them to corosync and
configure<br>
them not to vote on quorum ...<br>
... the same knet might then even be used to connect the bridges<br>
on the hosts with each other on layer-2 ...<br>
<blockquote type="cite"
cite="mid:CAA91j0XLH6C+754FjCAfqrdZhD9GxGV2mTD2uFHOUDu+fAvPVA@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
><br>
> If the VMs are behind NAT, I think that the simplest
way to STONITH is to use SBD over iSCSI. <br>
> Yet, my KVM knowledge is limited and I didn't see any
proof that I'm right (libvirt network was in NAT mode) or
wrong (VMs using Host's bond in a bridged network).<br>
><br>
> Best Regards,<br>
> Strahil Nikolov<br>
><br>
> На 19 юли 2020 г. 9:45:29 GMT+03:00, Andrei Borzenkov
<<a href="mailto:arvidjaar@gmail.com" target="_blank"
moz-do-not-send="true">arvidjaar@gmail.com</a>> написа:<br>
>> 18.07.2020 03:36, Reid Wahl пишет:<br>
>>> I'm not sure that the libvirt backend is
intended to be used in this<br>
>> way,<br>
>>> with multiple hosts using the same multicast
address. From the<br>
>>> fence_virt.conf man page:<br>
>>><br>
>>> ~~~<br>
>>> BACKENDS<br>
>>> libvirt<br>
>>> The libvirt plugin is the simplest
plugin. It is used<br>
>> in<br>
>>> environments where routing fencing requests
between multiple hosts is<br>
>> not<br>
>>> required, for example by a user running a
cluster of virtual<br>
>>> machines on a single desktop computer.<br>
>>> libvirt-qmf<br>
>>> The libvirt-qmf plugin acts as a QMFv2
Console to the<br>
>> libvirt-qmf<br>
>>> daemon in order to route fencing requests over
AMQP to the<br>
>> appropriate<br>
>>> computer.<br>
>>> cpg<br>
>>> The cpg plugin uses corosync CPG and
libvirt to track virtual<br>
>>> machines and route fencing requests to the
appropriate computer.<br>
>>> ~~~<br>
>>><br>
>>> I'm not an expert on fence_xvm or libvirt. It's
possible that this is<br>
>> a<br>
>>> viable configuration with the libvirt backend.<br>
>>><br>
>>> However, when users want to configure fence_xvm
for multiple hosts<br>
>> with the<br>
>>> libvirt backend, I have typically seen them
configure multiple<br>
>> fence_xvm<br>
>>> devices (one per host) and configure a
different multicast address on<br>
>> each<br>
>>> host.<br>
>>><br>
>>> If you have a Red Hat account, see also:<br>
>>> - <a
href="https://access.redhat.com/solutions/2386421"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://access.redhat.com/solutions/2386421</a><br>
>> What's the point in using multicast listener if
every host will have<br>
>> unique multicast address and there will be separate
stonith agent for<br>
>> each host using this unique address? That's not
what everyone expects<br>
>> seeing "multicast" as communication protocol.<br>
>><br>
>> This is serious question. If intention is to avoid
TCP overhead, why<br>
>> not<br>
>> simply use UDP with unique address? Or is single
multicast address<br>
>> still<br>
>> possible and this article describes "what I once
set up and it worked<br>
>> for me" and not "how it is designed to work"?<br>
>><br>
>> Also what is not clear - which fence_virtd instance
on host will be<br>
>> contacted by stonith agent on cluster node? I.e.
consider<br>
>><br>
>> three hosts host1, host2, host3<br>
>> three VM vm1, vm2, vm3 each active on corresponding
host<br>
>><br>
>> vm1 on host1 want to fence vm3 on host3. Will it<br>
>> a) contact fence_virtd on host1 and fence_virtd on
host1 will forward<br>
>> request to host3? Or<br>
>> b) is it mandatory for vm1 to have connectivity to
fence_virtd on<br>
>> host3?<br>
>><br>
>> If we combine existence of local-only listeners
(like serial or vsock)<br>
>> and distributed backend (like cpg) it strongly
suggests that vm1<br>
>> -(listener)-> host1 -(backend)-> host3 ->
-(fence)->vm3 is possible.<br>
>><br>
>> If each cluster node always directly contacts
fence_virtd on *target*<br>
>> host then libvirt backend is still perfectly usable
for multi-host<br>
>> configuration as every fence_virtd will only ever
fence local VM.<br>
>><br>
>> Is there any high level architecture overview (may
be presentation from<br>
>> some conference)?<br>
>><br>
>><br>
>> _______________________________________________<br>
>> Manage your subscription:<br>
>> <a
href="https://lists.clusterlabs.org/mailman/listinfo/users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
>><br>
>> ClusterLabs home: <a
href="https://www.clusterlabs.org/" rel="noreferrer"
target="_blank" moz-do-not-send="true">https://www.clusterlabs.org/</a><br>
> _______________________________________________<br>
> Manage your subscription:<br>
> <a
href="https://lists.clusterlabs.org/mailman/listinfo/users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
><br>
> ClusterLabs home: <a
href="https://www.clusterlabs.org/" rel="noreferrer"
target="_blank" moz-do-not-send="true">https://www.clusterlabs.org/</a><br>
<br>
</blockquote>
</div>
</div>
</blockquote>
<br>
</body>
</html>