[ClusterLabs] Bug pacemaker with multiple IP
Reid Wahl
nwahl at redhat.com
Tue Dec 20 14:34:21 EST 2022
On Tue, Dec 20, 2022 at 6:25 AM Thomas CAS <tcas at ikoula.com> wrote:
>
> Hello Ken,
>
> Thanks for your answer.
> There was no update running at the time of the bug, which is why I thought that having too many IPs caused this type of error.
> The /usr/sbin/ip executable was not being modified either.
>
> We have many clusters, and only this one has so many IPs and this problem.
How often does this happen, and is it reliably reproducible under any
circumstances? Any antivirus software running? It'd be nice to check
something like lsof or strace while it's happening, but that may not
be feasible if it's sporadic; running those at every monitor would
generate lots of logs.
AFAICT, having multiple processes execute (or read) the `ip` binary
simultaneously *shouldn't* cause problems, as long as nothing opens it
for write.
>
> Best regards,
>
> Thomas Cas | Technicien du support infogérance
> PHONE : +33 3 51 25 23 26 WEB : www.ikoula.com/en
> IKOULA Data Center 34 rue Pont Assy - 51100 Reims - FRANCE
> Before printing this letter, think about the impact on the environment!
>
> -----Message d'origine-----
> De : Ken Gaillot <kgaillot at redhat.com>
> Envoyé : lundi 19 décembre 2022 22:08
> À : Cluster Labs - All topics related to open-source clustering welcomed <users at clusterlabs.org>
> Cc : Service Infogérance <infogerance at ikoula.com>
> Objet : Re: [ClusterLabs] Bug pacemaker with multiple IP
>
> [Vous ne recevez pas souvent de courriers de kgaillot at redhat.com. Découvrez pourquoi ceci est important à https://aka.ms/LearnAboutSenderIdentification ]
>
> On Mon, 2022-12-19 at 09:48 +0000, Thomas CAS wrote:
> > Hello Clusterlabs,
> >
> > I would like to report a bug on Pacemaker with the "IPaddr2"
> > resource:
> >
> > OS: Debian 10
> > Kernel: Linux wd-websqlng01 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1
> > (2021-09-29) x86_64 GNU/Linux
> > Pacemaker version: 2.0.1-5+deb10u2
> >
> > You will find the configuration of our cluster with 2 nodes attached.
> >
> > Bug :
> >
> > We have several IP configured in the cluster configuration (12)
> > Sometimes the cluster is unstable with the following errors in the
> > pacemaker logs:
> >
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 232_monitor_10000:28835:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
>
> This doesn't sound like a bug in the agent; "Text file busy" suggests that the system "ip" command is being modified while the command is running. Is a software update happening when the problem occurs?
>
> I'm not sure whether there's some other situation that could cause that error, but simply executing the command a bunch of times simultaneously shouldn't cause it as far as I know.
>
> If simultaneous monitors is somehow causing the problem, you should be able to work around it by using different intervals for different monitors.
>
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 239_monitor_10000:28877:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 239_monitor_10000:28877:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 234_monitor_10000:28830:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 231_monitor_10000:28900:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 231_monitor_10000:28900:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 235_monitor_10000:28905:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 235_monitor_10000:28905:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 237_monitor_10000:28890:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 237_monitor_10000:28890:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 238_monitor_10000:28876:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP-
> > 238_monitor_10000:28876:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP_monitor_10000:28880:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> > Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd [5079]
> > (operation_finished) notice: NGINX-VIP_monitor_10000:28880:stderr [
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> > /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> >
> > The reason is that there are a lot of IPs configured and if the
> > monitors take place at the same time it causes this type of error.
> >
> > Best regards,
> >
> > Thomas Cas | Technicien du support infogérance
> > PHONE : +33 3 51 25 23 26 WEB : https://fra01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ikoula.com%2Fen&data=05%7C01%7Ctcas%40ikoula.com%7C541f4960600340f90a2c08dae20511fc%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638070808660951911%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=AvF%2BGT8DihOOHblzsGKGBjuvdg46GVaP%2FyBhu9OFdKs%3D&reserved=0
> > IKOULA Data Center 34 rue Pont Assy - 51100 Reims - FRANCE Before
> > printing this letter, think about the impact on the environment!
> >
> >
> >
> >
> > _______________________________________________
> > Manage your subscription:
> > https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> > s.clusterlabs.org%2Fmailman%2Flistinfo%2Fusers&data=05%7C01%7Ctcas%40i
> > koula.com%7C541f4960600340f90a2c08dae20511fc%7Ccb7a4a4ea7f747cc931f80d
> > b4a66f1c7%7C0%7C0%7C638070808660951911%7CUnknown%7CTWFpbGZsb3d8eyJWIjo
> > iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> > 7C%7C&sdata=U9osKXkKgjcqp6PN0%2F%2FB%2BzZyX0JMe6WMqRPVDTEGyWg%3D&reser
> > ved=0
> >
> > ClusterLabs home:
> > https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> > clusterlabs.org%2F&data=05%7C01%7Ctcas%40ikoula.com%7C541f4960600340f9
> > 0a2c08dae20511fc%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C63807080
> > 8660951911%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI
> > iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FfODTlNES3onDk
> > %2FfLgs6bWR2iikLdfqx7ePxzZfR%2BIU%3D&reserved=0
> --
> Ken Gaillot <kgaillot at redhat.com>
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
--
Regards,
Reid Wahl (He/Him)
Senior Software Engineer, Red Hat
RHEL High Availability - Pacemaker
More information about the Users
mailing list