[ClusterLabs] Still Beginner STONITH Problem
Strahil Nikolov
hunter86_bg at yahoo.com
Wed Jul 15 09:15:15 EDT 2020
If it is created by libvirt - this is NAT and you will never receive output from the other host.
Best Regards,
Strahil Nikolov
На 15 юли 2020 г. 15:05:48 GMT+03:00, "stefan.schmitz at farmpartner-tec.com" <stefan.schmitz at farmpartner-tec.com> написа:
>Hello,
>
>Am 15.07.2020 um 13:42 Strahil Nikolov wrote:
>> By default libvirt is using NAT and not routed network - in such
>case, vm1 won't receive data from host2.
>>
>> Can you provide the Networks' xml ?
>>
>> Best Regards,
>> Strahil Nikolov
>>
>
># cat default.xml
><network>
> <name>default</name>
> <bridge name="virbr0"/>
> <forward/>
> <ip address="192.168.122.1" netmask="255.255.255.0">
> <dhcp>
> <range start="192.168.122.2" end="192.168.122.254"/>
> </dhcp>
> </ip>
></network>
>
>I just checked this and the file is identical on both hosts.
>
>kind regards
>Stefan Schmitz
>
>
>> На 15 юли 2020 г. 13:19:59 GMT+03:00, Klaus Wenninger
><kwenning at redhat.com> написа:
>>> On 7/15/20 11:42 AM, stefan.schmitz at farmpartner-tec.com wrote:
>>>> Hello,
>>>>
>>>>
>>>> Am 15.07.2020 um 06:32 Strahil Nikolov wrote:
>>>>> How did you configure the network on your ubuntu 20.04 Hosts ? I
>>>>> tried to setup bridged connection for the test setup , but
>>> obviously
>>>>> I'm missing something.
>>>>>
>>>>> Best Regards,
>>>>> Strahil Nikolov
>>>>>
>>>>
>>>> on the hosts (CentOS) the bridge config looks like that.The
>bridging
>>>> and configuration is handled by the virtualization software:
>>>>
>>>> # cat ifcfg-br0
>>>> DEVICE=br0
>>>> TYPE=Bridge
>>>> BOOTPROTO=static
>>>> ONBOOT=yes
>>>> IPADDR=192.168.1.21
>>>> NETMASK=255.255.0.0
>>>> GATEWAY=192.168.1.1
>>>> NM_CONTROLLED=no
>>>> IPV6_AUTOCONF=yes
>>>> IPV6_DEFROUTE=yes
>>>> IPV6_PEERDNS=yes
>>>> IPV6_PEERROUTES=yes
>>>> IPV6_FAILURE_FATAL=no
>>>>
>>>>
>>>>
>>>> Am 15.07.2020 um 09:50 Klaus Wenninger wrote:
>>>>> Guess it is not easy to have your servers connected physically for
>>> a
>>>> try.
>>>>> But maybe you can at least try on one host to have virt_fenced &
>VM
>>>>> on the same bridge - just to see if that basic pattern is working.
>>>>
>>>> I am not sure if I understand you correctly. What do you by having
>>>> them on the same bridge? The bridge device is configured on the
>host
>>>> by the virtualization software.
>>> I meant to check out which bridge the interface of the VM is
>enslaved
>>> to and to use that bridge as interface in /etc/fence_virt.conf.
>>> Get me right - just for now - just to see if it is working for this
>one
>>> host and the corresponding guest.
>>>>
>>>>
>>>>> Well maybe still sbdy in the middle playing IGMPv3 or the request
>>> for
>>>>> a certain source is needed to shoot open some firewall or
>>> switch-tables.
>>>>
>>>> I am still waiting for the final report from our Data Center techs.
>I
>>>> hope that will clear up somethings.
>>>>
>>>>
>>>> Additionally I have just noticed that apparently since switching
>>> from
>>>> IGMPv3 to IGMPv2 and back the command "fence_xvm -a 225.0.0.12 -o
>>>> list" is no completely broken.
>>>> Before that switch this command at least returned the local VM. Now
>>> it
>>>> returns:
>>>> Timed out waiting for response
>>>> Operation failed
>>>>
>>>> I am a bit confused by that, because all we did was running
>commands
>>>> like "sysctl -w net.ipv4.conf.all.force_igmp_version =" with the
>>>> different Version umbers and #cat /proc/net/igmp shows that V3 is
>>> used
>>>> again on every device just like before...?!
>>>>
>>>> kind regards
>>>> Stefan Schmitz
>>>>
>>>>
>>>>> На 14 юли 2020 г. 11:06:42 GMT+03:00,
>>>>> "stefan.schmitz at farmpartner-tec.com"
>>>>> <stefan.schmitz at farmpartner-tec.com> написа:
>>>>>> Hello,
>>>>>>
>>>>>>
>>>>>> Am 09.07.2020 um 19:10 Strahil Nikolov wrote:
>>>>>>> Have you run 'fence_virtd -c' ?
>>>>>> Yes I had run that on both Hosts. The current config looks like
>>> that
>>>>>> and
>>>>>> is identical on both.
>>>>>>
>>>>>> cat fence_virt.conf
>>>>>> fence_virtd {
>>>>>> listener = "multicast";
>>>>>> backend = "libvirt";
>>>>>> module_path = "/usr/lib64/fence-virt";
>>>>>> }
>>>>>>
>>>>>> listeners {
>>>>>> multicast {
>>>>>> key_file = "/etc/cluster/fence_xvm.key";
>>>>>> address = "225.0.0.12";
>>>>>> interface = "bond0";
>>>>>> family = "ipv4";
>>>>>> port = "1229";
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> backends {
>>>>>> libvirt {
>>>>>> uri = "qemu:///system";
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> The situation is still that no matter on what host I issue the
>>>>>> "fence_xvm -a 225.0.0.12 -o list" command, both guest systems
>>> receive
>>>>>> the traffic. The local guest, but also the guest on the other
>host.
>>> I
>>>>>> reckon that means the traffic is not filtered by any network
>>> device,
>>>>>> like switches or firewalls. Since the guest on the other host
>>> receives
>>>>>> the packages, the traffic must reach te physical server and
>>>>>> networkdevice and is then routed to the VM on that host.
>>>>>> But still, the traffic is not shown on the host itself.
>>>>>>
>>>>>> Further the local firewalls on both hosts are set to let each and
>>> every
>>>>>>
>>>>>> traffic pass. Accept to any and everything. Well at least as far
>as
>>> I
>>>>>> can see.
>>>>>>
>>>>>>
>>>>>> Am 09.07.2020 um 22:34 Klaus Wenninger wrote:
>>>>>>> makes me believe that
>>>>>>> the whole setup doesn't lookas I would have
>>>>>>> expected (bridges on each host where theguest
>>>>>>> has a connection to and where ethernet interfaces
>>>>>>> that connect the 2 hosts are part of as well
>>>>>>
>>>>>> On each physical server the networkcards are bonded to achieve
>>> failure
>>>>>> safety (bond0). The guest are connected over a bridge(br0) but
>>>>>> apparently our virtualization softrware creates an own device
>named
>>>>>> after the guest (kvm101.0).
>>>>>> There is no direct connection between the servers, but as I said
>>>>>> earlier, the multicast traffic does reach the VMs so I assume
>there
>>> is
>>>>>> no problem with that.
>>>>>>
>>>>>>
>>>>>> Am 09.07.2020 um 20:18 Vladislav Bogdanov wrote:
>>>>>>> First, you need to ensure that your switch (or all switches in
>the
>>>>>>> path) have igmp snooping enabled on host ports (and probably
>>>>>>> interconnects along the path between your hosts).
>>>>>>>
>>>>>>> Second, you need an igmp querier to be enabled somewhere near
>>> (better
>>>>>>> to have it enabled on a switch itself). Please verify that you
>see
>>>>>> its
>>>>>>> queries on hosts.
>>>>>>>
>>>>>>> Next, you probably need to make your hosts to use IGMPv2 (not 3)
>>> as
>>>>>>> many switches still can not understand v3. This is doable by
>>> sysctl,
>>>>>>> find on internet, there are many articles.
>>>>>>
>>>>>>
>>>>>> I have send an query to our Data center Techs who are analyzing
>>> this
>>>>>> and
>>>>>> were already on it analyzing if multicast Traffic is somewhere
>>> blocked
>>>>>> or hindered. So far the answer is, "multicast ist explictly
>allowed
>>> in
>>>>>> the local network and no packets are filtered or dropped". I am
>>> still
>>>>>> waiting for a final report though.
>>>>>>
>>>>>> In the meantime I have switched IGMPv3 to IGMPv2 on every
>involved
>>>>>> server, hosts and guests via the mentioned sysctl. The switching
>>> itself
>>>>>>
>>>>>> was successful, according to "cat /proc/net/igmp" but sadly did
>not
>>>>>> better the behavior. It actually led to that no VM received the
>>>>>> multicast traffic anymore too.
>>>>>>
>>>>>> kind regards
>>>>>> Stefan Schmitz
>>>>>>
>>>>>>
>>>>>> Am 09.07.2020 um 22:34 schrieb Klaus Wenninger:
>>>>>>> On 7/9/20 5:17 PM, stefan.schmitz at farmpartner-tec.com wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>> Well, theory still holds I would say.
>>>>>>>>>
>>>>>>>>> I guess that the multicast-traffic from the other host
>>>>>>>>> or the guestsdoesn't get to the daemon on the host.
>>>>>>>>> Can't you just simply check if there are any firewall
>>>>>>>>> rules configuredon the host kernel?
>>>>>>>>
>>>>>>>> I hope I did understand you corretcly and you are referring to
>>>>>> iptables?
>>>>>>> I didn't say iptables because it might have been
>>>>>>> nftables - but yesthat is what I was referring to.
>>>>>>> Guess to understand the config the output is
>>>>>>> lacking verbositybut it makes me believe that
>>>>>>> the whole setup doesn't lookas I would have
>>>>>>> expected (bridges on each host where theguest
>>>>>>> has a connection to and where ethernet interfaces
>>>>>>> that connect the 2 hosts are part of as well -
>>>>>>> everythingconnected via layer 2 basically).
>>>>>>>> Here is the output of the current rules. Besides the IP of the
>>> guest
>>>>>>>> the output is identical on both hosts:
>>>>>>>>
>>>>>>>> # iptables -S
>>>>>>>> -P INPUT ACCEPT
>>>>>>>> -P FORWARD ACCEPT
>>>>>>>> -P OUTPUT ACCEPT
>>>>>>>>
>>>>>>>> # iptables -L
>>>>>>>> Chain INPUT (policy ACCEPT)
>>>>>>>> target prot opt source destination
>>>>>>>>
>>>>>>>> Chain FORWARD (policy ACCEPT)
>>>>>>>> target prot opt source destination
>>>>>>>> SOLUSVM_TRAFFIC_IN all -- anywhere anywhere
>>>>>>>> SOLUSVM_TRAFFIC_OUT all -- anywhere anywhere
>>>>>>>>
>>>>>>>> Chain OUTPUT (policy ACCEPT)
>>>>>>>> target prot opt source destination
>>>>>>>>
>>>>>>>> Chain SOLUSVM_TRAFFIC_IN (1 references)
>>>>>>>> target prot opt source destination
>>>>>>>> all -- anywhere 192.168.1.14
>>>>>>>>
>>>>>>>> Chain SOLUSVM_TRAFFIC_OUT (1 references)
>>>>>>>> target prot opt source destination
>>>>>>>> all -- 192.168.1.14 anywhere
>>>>>>>>
>>>>>>>> kind regards
>>>>>>>> Stefan Schmitz
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
More information about the Users
mailing list