[ClusterLabs] Still Beginner STONITH Problem
Klaus Wenninger
kwenning at redhat.com
Wed Jul 15 09:30:42 EDT 2020
On 7/15/20 3:15 PM, Strahil Nikolov wrote:
> If it is created by libvirt - this is NAT and you will never receive output from the other host.
And twice the same subnet behind NAT is probably giving
issues at other places as well.
And if using DHCP you have to at least enforce that both sides
don't go for the same IP at least.
But all no explanation why it doesn't work on the same host.
Which is why I was asking for running the service on the
bridge to check if that would work at least. So that we
can go forward step by step.
Is pacemaker/corosync/knet btw. using the same interfaces/IPs?
Klaus
>
> Best Regards,
> Strahil Nikolov
>
> На 15 юли 2020 г. 15:05:48 GMT+03:00, "stefan.schmitz at farmpartner-tec.com" <stefan.schmitz at farmpartner-tec.com> написа:
>> Hello,
>>
>> Am 15.07.2020 um 13:42 Strahil Nikolov wrote:
>>> By default libvirt is using NAT and not routed network - in such
>> case, vm1 won't receive data from host2.
>>> Can you provide the Networks' xml ?
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>> # cat default.xml
>> <network>
>> <name>default</name>
>> <bridge name="virbr0"/>
>> <forward/>
>> <ip address="192.168.122.1" netmask="255.255.255.0">
>> <dhcp>
>> <range start="192.168.122.2" end="192.168.122.254"/>
>> </dhcp>
>> </ip>
>> </network>
>>
>> I just checked this and the file is identical on both hosts.
>>
>> kind regards
>> Stefan Schmitz
>>
>>
>>> На 15 юли 2020 г. 13:19:59 GMT+03:00, Klaus Wenninger
>> <kwenning at redhat.com> написа:
>>>> On 7/15/20 11:42 AM, stefan.schmitz at farmpartner-tec.com wrote:
>>>>> Hello,
>>>>>
>>>>>
>>>>> Am 15.07.2020 um 06:32 Strahil Nikolov wrote:
>>>>>> How did you configure the network on your ubuntu 20.04 Hosts ? I
>>>>>> tried to setup bridged connection for the test setup , but
>>>> obviously
>>>>>> I'm missing something.
>>>>>>
>>>>>> Best Regards,
>>>>>> Strahil Nikolov
>>>>>>
>>>>> on the hosts (CentOS) the bridge config looks like that.The
>> bridging
>>>>> and configuration is handled by the virtualization software:
>>>>>
>>>>> # cat ifcfg-br0
>>>>> DEVICE=br0
>>>>> TYPE=Bridge
>>>>> BOOTPROTO=static
>>>>> ONBOOT=yes
>>>>> IPADDR=192.168.1.21
>>>>> NETMASK=255.255.0.0
>>>>> GATEWAY=192.168.1.1
>>>>> NM_CONTROLLED=no
>>>>> IPV6_AUTOCONF=yes
>>>>> IPV6_DEFROUTE=yes
>>>>> IPV6_PEERDNS=yes
>>>>> IPV6_PEERROUTES=yes
>>>>> IPV6_FAILURE_FATAL=no
>>>>>
>>>>>
>>>>>
>>>>> Am 15.07.2020 um 09:50 Klaus Wenninger wrote:
>>>>>> Guess it is not easy to have your servers connected physically for
>>>> a
>>>>> try.
>>>>>> But maybe you can at least try on one host to have virt_fenced &
>> VM
>>>>>> on the same bridge - just to see if that basic pattern is working.
>>>>> I am not sure if I understand you correctly. What do you by having
>>>>> them on the same bridge? The bridge device is configured on the
>> host
>>>>> by the virtualization software.
>>>> I meant to check out which bridge the interface of the VM is
>> enslaved
>>>> to and to use that bridge as interface in /etc/fence_virt.conf.
>>>> Get me right - just for now - just to see if it is working for this
>> one
>>>> host and the corresponding guest.
>>>>>
>>>>>> Well maybe still sbdy in the middle playing IGMPv3 or the request
>>>> for
>>>>>> a certain source is needed to shoot open some firewall or
>>>> switch-tables.
>>>>> I am still waiting for the final report from our Data Center techs.
>> I
>>>>> hope that will clear up somethings.
>>>>>
>>>>>
>>>>> Additionally I have just noticed that apparently since switching
>>>> from
>>>>> IGMPv3 to IGMPv2 and back the command "fence_xvm -a 225.0.0.12 -o
>>>>> list" is no completely broken.
>>>>> Before that switch this command at least returned the local VM. Now
>>>> it
>>>>> returns:
>>>>> Timed out waiting for response
>>>>> Operation failed
>>>>>
>>>>> I am a bit confused by that, because all we did was running
>> commands
>>>>> like "sysctl -w net.ipv4.conf.all.force_igmp_version =" with the
>>>>> different Version umbers and #cat /proc/net/igmp shows that V3 is
>>>> used
>>>>> again on every device just like before...?!
>>>>>
>>>>> kind regards
>>>>> Stefan Schmitz
>>>>>
>>>>>
>>>>>> На 14 юли 2020 г. 11:06:42 GMT+03:00,
>>>>>> "stefan.schmitz at farmpartner-tec.com"
>>>>>> <stefan.schmitz at farmpartner-tec.com> написа:
>>>>>>> Hello,
>>>>>>>
>>>>>>>
>>>>>>> Am 09.07.2020 um 19:10 Strahil Nikolov wrote:
>>>>>>>> Have you run 'fence_virtd -c' ?
>>>>>>> Yes I had run that on both Hosts. The current config looks like
>>>> that
>>>>>>> and
>>>>>>> is identical on both.
>>>>>>>
>>>>>>> cat fence_virt.conf
>>>>>>> fence_virtd {
>>>>>>> listener = "multicast";
>>>>>>> backend = "libvirt";
>>>>>>> module_path = "/usr/lib64/fence-virt";
>>>>>>> }
>>>>>>>
>>>>>>> listeners {
>>>>>>> multicast {
>>>>>>> key_file = "/etc/cluster/fence_xvm.key";
>>>>>>> address = "225.0.0.12";
>>>>>>> interface = "bond0";
>>>>>>> family = "ipv4";
>>>>>>> port = "1229";
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> backends {
>>>>>>> libvirt {
>>>>>>> uri = "qemu:///system";
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> The situation is still that no matter on what host I issue the
>>>>>>> "fence_xvm -a 225.0.0.12 -o list" command, both guest systems
>>>> receive
>>>>>>> the traffic. The local guest, but also the guest on the other
>> host.
>>>> I
>>>>>>> reckon that means the traffic is not filtered by any network
>>>> device,
>>>>>>> like switches or firewalls. Since the guest on the other host
>>>> receives
>>>>>>> the packages, the traffic must reach te physical server and
>>>>>>> networkdevice and is then routed to the VM on that host.
>>>>>>> But still, the traffic is not shown on the host itself.
>>>>>>>
>>>>>>> Further the local firewalls on both hosts are set to let each and
>>>> every
>>>>>>> traffic pass. Accept to any and everything. Well at least as far
>> as
>>>> I
>>>>>>> can see.
>>>>>>>
>>>>>>>
>>>>>>> Am 09.07.2020 um 22:34 Klaus Wenninger wrote:
>>>>>>>> makes me believe that
>>>>>>>> the whole setup doesn't lookas I would have
>>>>>>>> expected (bridges on each host where theguest
>>>>>>>> has a connection to and where ethernet interfaces
>>>>>>>> that connect the 2 hosts are part of as well
>>>>>>> On each physical server the networkcards are bonded to achieve
>>>> failure
>>>>>>> safety (bond0). The guest are connected over a bridge(br0) but
>>>>>>> apparently our virtualization softrware creates an own device
>> named
>>>>>>> after the guest (kvm101.0).
>>>>>>> There is no direct connection between the servers, but as I said
>>>>>>> earlier, the multicast traffic does reach the VMs so I assume
>> there
>>>> is
>>>>>>> no problem with that.
>>>>>>>
>>>>>>>
>>>>>>> Am 09.07.2020 um 20:18 Vladislav Bogdanov wrote:
>>>>>>>> First, you need to ensure that your switch (or all switches in
>> the
>>>>>>>> path) have igmp snooping enabled on host ports (and probably
>>>>>>>> interconnects along the path between your hosts).
>>>>>>>>
>>>>>>>> Second, you need an igmp querier to be enabled somewhere near
>>>> (better
>>>>>>>> to have it enabled on a switch itself). Please verify that you
>> see
>>>>>>> its
>>>>>>>> queries on hosts.
>>>>>>>>
>>>>>>>> Next, you probably need to make your hosts to use IGMPv2 (not 3)
>>>> as
>>>>>>>> many switches still can not understand v3. This is doable by
>>>> sysctl,
>>>>>>>> find on internet, there are many articles.
>>>>>>>
>>>>>>> I have send an query to our Data center Techs who are analyzing
>>>> this
>>>>>>> and
>>>>>>> were already on it analyzing if multicast Traffic is somewhere
>>>> blocked
>>>>>>> or hindered. So far the answer is, "multicast ist explictly
>> allowed
>>>> in
>>>>>>> the local network and no packets are filtered or dropped". I am
>>>> still
>>>>>>> waiting for a final report though.
>>>>>>>
>>>>>>> In the meantime I have switched IGMPv3 to IGMPv2 on every
>> involved
>>>>>>> server, hosts and guests via the mentioned sysctl. The switching
>>>> itself
>>>>>>> was successful, according to "cat /proc/net/igmp" but sadly did
>> not
>>>>>>> better the behavior. It actually led to that no VM received the
>>>>>>> multicast traffic anymore too.
>>>>>>>
>>>>>>> kind regards
>>>>>>> Stefan Schmitz
>>>>>>>
>>>>>>>
>>>>>>> Am 09.07.2020 um 22:34 schrieb Klaus Wenninger:
>>>>>>>> On 7/9/20 5:17 PM, stefan.schmitz at farmpartner-tec.com wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>>> Well, theory still holds I would say.
>>>>>>>>>>
>>>>>>>>>> I guess that the multicast-traffic from the other host
>>>>>>>>>> or the guestsdoesn't get to the daemon on the host.
>>>>>>>>>> Can't you just simply check if there are any firewall
>>>>>>>>>> rules configuredon the host kernel?
>>>>>>>>> I hope I did understand you corretcly and you are referring to
>>>>>>> iptables?
>>>>>>>> I didn't say iptables because it might have been
>>>>>>>> nftables - but yesthat is what I was referring to.
>>>>>>>> Guess to understand the config the output is
>>>>>>>> lacking verbositybut it makes me believe that
>>>>>>>> the whole setup doesn't lookas I would have
>>>>>>>> expected (bridges on each host where theguest
>>>>>>>> has a connection to and where ethernet interfaces
>>>>>>>> that connect the 2 hosts are part of as well -
>>>>>>>> everythingconnected via layer 2 basically).
>>>>>>>>> Here is the output of the current rules. Besides the IP of the
>>>> guest
>>>>>>>>> the output is identical on both hosts:
>>>>>>>>>
>>>>>>>>> # iptables -S
>>>>>>>>> -P INPUT ACCEPT
>>>>>>>>> -P FORWARD ACCEPT
>>>>>>>>> -P OUTPUT ACCEPT
>>>>>>>>>
>>>>>>>>> # iptables -L
>>>>>>>>> Chain INPUT (policy ACCEPT)
>>>>>>>>> target prot opt source destination
>>>>>>>>>
>>>>>>>>> Chain FORWARD (policy ACCEPT)
>>>>>>>>> target prot opt source destination
>>>>>>>>> SOLUSVM_TRAFFIC_IN all -- anywhere anywhere
>>>>>>>>> SOLUSVM_TRAFFIC_OUT all -- anywhere anywhere
>>>>>>>>>
>>>>>>>>> Chain OUTPUT (policy ACCEPT)
>>>>>>>>> target prot opt source destination
>>>>>>>>>
>>>>>>>>> Chain SOLUSVM_TRAFFIC_IN (1 references)
>>>>>>>>> target prot opt source destination
>>>>>>>>> all -- anywhere 192.168.1.14
>>>>>>>>>
>>>>>>>>> Chain SOLUSVM_TRAFFIC_OUT (1 references)
>>>>>>>>> target prot opt source destination
>>>>>>>>> all -- 192.168.1.14 anywhere
>>>>>>>>>
>>>>>>>>> kind regards
>>>>>>>>> Stefan Schmitz
>>>>>>>>>
>>>>>>>>>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
More information about the Users
mailing list