[ClusterLabs] Bug pacemaker with multiple IP

[Thomas CAS]

Hello Ken,

Thanks for your answer.
There was no update running at the time of the bug, which is why I thought that having too many IPs caused this type of error.
The /usr/sbin/ip executable was not being modified either.

We have many clusters, and only this one has so many IPs and this problem.

Best regards,

Thomas Cas  |  Technicien du support infogérance
PHONE : +33 3 51 25 23 26       WEB : www.ikoula.com/en
IKOULA Data Center 34 rue Pont Assy - 51100 Reims - FRANCE
Before printing this letter, think about the impact on the environment!

-----Message d'origine-----
De : Ken Gaillot <kgaillot at redhat.com> 
Envoyé : lundi 19 décembre 2022 22:08
À : Cluster Labs - All topics related to open-source clustering welcomed <users at clusterlabs.org>
Cc : Service Infogérance <infogerance at ikoula.com>
Objet : Re: [ClusterLabs] Bug pacemaker with multiple IP

[Vous ne recevez pas souvent de courriers de kgaillot at redhat.com. Découvrez pourquoi ceci est important à https://aka.ms/LearnAboutSenderIdentification ]

On Mon, 2022-12-19 at 09:48 +0000, Thomas CAS wrote:
> Hello Clusterlabs,
>
> I would like to report a bug on Pacemaker with the "IPaddr2"
> resource:
>
> OS: Debian 10
> Kernel: Linux wd-websqlng01 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1
> (2021-09-29) x86_64 GNU/Linux
> Pacemaker version: 2.0.1-5+deb10u2
>
> You will find the configuration of our cluster with 2 nodes attached.
>
> Bug :
>
> We have several IP configured in the cluster configuration (12) 
> Sometimes the cluster is unstable with the following errors in the 
> pacemaker logs:
>
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 232_monitor_10000:28835:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]

This doesn't sound like a bug in the agent; "Text file busy" suggests that the system "ip" command is being modified while the command is running. Is a software update happening when the problem occurs?

I'm not sure whether there's some other situation that could cause that error, but simply executing the command a bunch of times simultaneously shouldn't cause it as far as I know.

If simultaneous monitors is somehow causing the problem, you should be able to work around it by using different intervals for different monitors.

> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 239_monitor_10000:28877:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 239_monitor_10000:28877:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 234_monitor_10000:28830:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 231_monitor_10000:28900:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 231_monitor_10000:28900:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 235_monitor_10000:28905:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 235_monitor_10000:28905:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 237_monitor_10000:28890:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 237_monitor_10000:28890:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 238_monitor_10000:28876:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP-
> 238_monitor_10000:28876:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP_monitor_10000:28880:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 709:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
> Dec 18 21:07:51 **SENSITIVEDATA** pacemaker-execd     [5079]
> (operation_finished)   notice: NGINX-VIP_monitor_10000:28880:stderr [
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: 1:
> /usr/lib/ocf/resource.d/heartbeat/IPaddr2: ip: Text file busy ]
>
> The reason is that there are a lot of IPs configured and if the 
> monitors take place at the same time it causes this type of error.
>
> Best regards,
>
>  Thomas Cas  |  Technicien du support infogérance
>  PHONE : +33 3 51 25 23 26       WEB : https://fra01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ikoula.com%2Fen&data=05%7C01%7Ctcas%40ikoula.com%7C541f4960600340f90a2c08dae20511fc%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638070808660951911%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=AvF%2BGT8DihOOHblzsGKGBjuvdg46GVaP%2FyBhu9OFdKs%3D&reserved=0
>  IKOULA Data Center 34 rue Pont Assy - 51100 Reims - FRANCE  Before 
> printing this letter, think about the impact on the environment!
>
>
>
>
> _______________________________________________
> Manage your subscription:
> https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.clusterlabs.org%2Fmailman%2Flistinfo%2Fusers&data=05%7C01%7Ctcas%40i
> koula.com%7C541f4960600340f90a2c08dae20511fc%7Ccb7a4a4ea7f747cc931f80d
> b4a66f1c7%7C0%7C0%7C638070808660951911%7CUnknown%7CTWFpbGZsb3d8eyJWIjo
> iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%
> 7C%7C&sdata=U9osKXkKgjcqp6PN0%2F%2FB%2BzZyX0JMe6WMqRPVDTEGyWg%3D&reser
> ved=0
>
> ClusterLabs home: 
> https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> clusterlabs.org%2F&data=05%7C01%7Ctcas%40ikoula.com%7C541f4960600340f9
> 0a2c08dae20511fc%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C63807080
> 8660951911%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI
> iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FfODTlNES3onDk
> %2FfLgs6bWR2iikLdfqx7ePxzZfR%2BIU%3D&reserved=0
--
Ken Gaillot <kgaillot at redhat.com>