<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:arial,sans-serif"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 20, 2020 at 10:56 AM Klaus Wenninger <<a href="mailto:kwenning@redhat.com">kwenning@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 7/19/20 8:55 AM, Strahil Nikolov wrote:<br>
> My understanding is that fence_xvm is reaching each Hypervisour via multicast (otherwise why multicast ?)... yet I could be simply fooling myself.<br>
Not as far as I know. Otherwise external/libvirt would be able to<br>
do the trick as well.<br>
afaik it should be able to cope with multiple hosts listening on<br>
the same multicast-address.</blockquote><div><br></div><div><div style="font-family:arial,sans-serif" class="gmail_default">No, it cannot. fence_virtd connects back to fence_xvm (or fence_virt) client (using its source address) and actual fencing request is then sent via this point to point channel. fence_xvm accepts only the first connection request. So even if multiple fecne_virtd will receive the initial multicast, only one of them will be able to connect to fence_xvm that sent this multicast.</div><div style="font-family:arial,sans-serif" class="gmail_default"><br></div><div style="font-family:arial,sans-serif" class="gmail_default">fence_virtd itself is dummy middleman. It accepts messages from fence_xvm/fence_virtd via the defined listener, connects back, receives actual fencing request and forwards it to the configured backend plugin. It performs no check itself whether the initial message is applicable - only the backend plugin knows what VM it can handle. And even if it did, because information is received only after reverse connection is established, it is already too late anyway.<br></div></div><div><div style="font-family:arial,sans-serif" class="gmail_default"></div><div style="font-family:arial,sans-serif" class="gmail_default">So if you are using muticast and your backend can only handle VM on local host, it is unpredictable which host will make reverse connection. Which is why RH KB article describes configuration with unique multicast address for each host. At which point you can simply use unicast address and spare all troubles associated with multicast.</div><div style="font-family:arial,sans-serif" class="gmail_default"><br></div><div style="font-family:arial,sans-serif" class="gmail_default">I do not see any use case for multicast. Either backend can fence every VM (i.e. backend is capable of contacting host that runs VM that has to be fenced) in which case you can simply contact local fence_virtd via e.g. vsock listener, or backend can handle only local VM (libvirt backend) in which case you must contact host where VM is running via unique address.<br></div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> But there were at least issues<br>
with that - don't remember the details but remember a discussion<br>
on some mailinglist - which is why I had suggested - as Reid just<br>
repeated - quite in the beginning of this thread to go with a setup<br>
that has 2 fencing-resources, 2 fence_virtd services listening on<br>
different multicast addresses ...<br></blockquote><div><br></div><div><div style="font-family:arial,sans-serif" class="gmail_default">At which point you can just use tcp listener with normal unicast.</div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
The cpg-configuration sounds interesting as well. Haven't used<br>
it or looked into the details. Would be interested to hear about<br>
how that works.<br></blockquote><div><br></div><div><div style="font-family:arial,sans-serif" class="gmail_default">It maintains a registry of VM location (each fence_virtd polls local hypervisor at regular intervals) and forwards fencing request to appropriate host via corosync interconnect. It is also the only backend that can handle host failure - if it is known that host left cluster, any VM on this host is considered fenced by definition.</div><div style="font-family:arial,sans-serif" class="gmail_default"><br></div><div style="font-family:arial,sans-serif" class="gmail_default">It requires that hosts are configured in pacemaker cluster themselves (to handle host outage it must be properly fenced).<br></div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
><br>
> If the VMs are behind NAT, I think that the simplest way to STONITH is to use SBD over iSCSI. <br>
> Yet, my KVM knowledge is limited and I didn't see any proof that I'm right (libvirt network was in NAT mode) or wrong (VMs using Host's bond in a bridged network).<br>
><br>
> Best Regards,<br>
> Strahil Nikolov<br>
><br>
> На 19 юли 2020 г. 9:45:29 GMT+03:00, Andrei Borzenkov <<a href="mailto:arvidjaar@gmail.com" target="_blank">arvidjaar@gmail.com</a>> написа:<br>
>> 18.07.2020 03:36, Reid Wahl пишет:<br>
>>> I'm not sure that the libvirt backend is intended to be used in this<br>
>> way,<br>
>>> with multiple hosts using the same multicast address. From the<br>
>>> fence_virt.conf man page:<br>
>>><br>
>>> ~~~<br>
>>> BACKENDS<br>
>>> libvirt<br>
>>> The libvirt plugin is the simplest plugin. It is used<br>
>> in<br>
>>> environments where routing fencing requests between multiple hosts is<br>
>> not<br>
>>> required, for example by a user running a cluster of virtual<br>
>>> machines on a single desktop computer.<br>
>>> libvirt-qmf<br>
>>> The libvirt-qmf plugin acts as a QMFv2 Console to the<br>
>> libvirt-qmf<br>
>>> daemon in order to route fencing requests over AMQP to the<br>
>> appropriate<br>
>>> computer.<br>
>>> cpg<br>
>>> The cpg plugin uses corosync CPG and libvirt to track virtual<br>
>>> machines and route fencing requests to the appropriate computer.<br>
>>> ~~~<br>
>>><br>
>>> I'm not an expert on fence_xvm or libvirt. It's possible that this is<br>
>> a<br>
>>> viable configuration with the libvirt backend.<br>
>>><br>
>>> However, when users want to configure fence_xvm for multiple hosts<br>
>> with the<br>
>>> libvirt backend, I have typically seen them configure multiple<br>
>> fence_xvm<br>
>>> devices (one per host) and configure a different multicast address on<br>
>> each<br>
>>> host.<br>
>>><br>
>>> If you have a Red Hat account, see also:<br>
>>> - <a href="https://access.redhat.com/solutions/2386421" rel="noreferrer" target="_blank">https://access.redhat.com/solutions/2386421</a><br>
>> What's the point in using multicast listener if every host will have<br>
>> unique multicast address and there will be separate stonith agent for<br>
>> each host using this unique address? That's not what everyone expects<br>
>> seeing "multicast" as communication protocol.<br>
>><br>
>> This is serious question. If intention is to avoid TCP overhead, why<br>
>> not<br>
>> simply use UDP with unique address? Or is single multicast address<br>
>> still<br>
>> possible and this article describes "what I once set up and it worked<br>
>> for me" and not "how it is designed to work"?<br>
>><br>
>> Also what is not clear - which fence_virtd instance on host will be<br>
>> contacted by stonith agent on cluster node? I.e. consider<br>
>><br>
>> three hosts host1, host2, host3<br>
>> three VM vm1, vm2, vm3 each active on corresponding host<br>
>><br>
>> vm1 on host1 want to fence vm3 on host3. Will it<br>
>> a) contact fence_virtd on host1 and fence_virtd on host1 will forward<br>
>> request to host3? Or<br>
>> b) is it mandatory for vm1 to have connectivity to fence_virtd on<br>
>> host3?<br>
>><br>
>> If we combine existence of local-only listeners (like serial or vsock)<br>
>> and distributed backend (like cpg) it strongly suggests that vm1<br>
>> -(listener)-> host1 -(backend)-> host3 -> -(fence)->vm3 is possible.<br>
>><br>
>> If each cluster node always directly contacts fence_virtd on *target*<br>
>> host then libvirt backend is still perfectly usable for multi-host<br>
>> configuration as every fence_virtd will only ever fence local VM.<br>
>><br>
>> Is there any high level architecture overview (may be presentation from<br>
>> some conference)?<br>
>><br>
>><br>
>> _______________________________________________<br>
>> Manage your subscription:<br>
>> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
>><br>
>> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
> _______________________________________________<br>
> Manage your subscription:<br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
><br>
> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
<br>
</blockquote></div></div>