[ClusterLabs] [Problem] The `attrd` process may encounter an IPC communication error, rendering it unable to communicate and triggering a node reboot.
renayama19661014 at ybb.ne.jp
renayama19661014 at ybb.ne.jp
Tue Oct 27 09:18:32 UTC 2026
Hi All,
We performed repeated cycles of starting and stopping nodes, as well as migrating resources, within a 5-node RHEL 10.2 cluster. The resources used were simple configurations (such as "Dummy" resources) with multiple attributes assigned to them.
----
[root at rh102-04 ~]# pcs status --full
Cluster name: test10-cluster-0
Cluster Summary:
* Stack: corosync (Pacemaker is running)
* Current DC: rh102-03 (3) (version 3.0.1-5.el10-be343ca) - partition with quorum
* Last updated: Thu Oct 22 17:33:18 2026 on rh102-04
* Last change: Thu Oct 22 17:31:55 2026 by root via root on rh102-04
* 5 nodes configured
* 34 resource instances configured
Node List:
* Node rh102-01 (1): online, feature set 3.20.1
* Node rh102-02 (2): online, feature set 3.20.1
* Node rh102-03 (3): online, feature set 3.20.1
* Node rh102-04 (4): online, feature set 3.20.1
* Node rh102-05 (5): online, feature set 3.20.1
Full List of Resources:
* Resource Group: primary-group1:
* ipaddr_gwf_101 (ocf:heartbeat:Dummy): Started rh102-01
* ipaddr_dbf_101 (ocf:heartbeat:Dummy): Started rh102-01
* ipaddr_api_101 (ocf:heartbeat:Dummy): Started rh102-01
* ipaddr_kafka_101 (ocf:heartbeat:Dummy): Started rh102-01
* ipaddr_unyo_101 (ocf:heartbeat:Dummy): Started rh102-01
* fs_share_101 (ocf:heartbeat:Dummy): Started rh102-01
* mw_jp1base_101 (ocf:heartbeat:Dummy): Started rh102-01
* mw_jp1ajs3_101 (ocf:heartbeat:Dummy): Started rh102-01
* Resource Group: primary-group2:
* ipaddr_gwf_102 (ocf:heartbeat:Dummy): Started rh102-02
* ipaddr_dbf_102 (ocf:heartbeat:Dummy): Started rh102-02
* ipaddr_api_102 (ocf:heartbeat:Dummy): Started rh102-02
* ipaddr_kafka_102 (ocf:heartbeat:Dummy): Started rh102-02
* ipaddr_unyo_102 (ocf:heartbeat:Dummy): Started rh102-02
* fs_share_102 (ocf:heartbeat:Dummy): Started rh102-02
* mw_jp1base_102 (ocf:heartbeat:Dummy): Started rh102-02
* mw_jp1ajs3_102 (ocf:heartbeat:Dummy): Started rh102-02
* Resource Group: primary-group3:
* ipaddr_gwf_103 (ocf:heartbeat:Dummy): Started rh102-03
* ipaddr_dbf_103 (ocf:heartbeat:Dummy): Started rh102-03
* ipaddr_api_103 (ocf:heartbeat:Dummy): Started rh102-03
* ipaddr_kafka_103 (ocf:heartbeat:Dummy): Started rh102-03
* ipaddr_unyo_103 (ocf:heartbeat:Dummy): Started rh102-03
* fs_share_103 (ocf:heartbeat:Dummy): Started rh102-03
* mw_jp1base_103 (ocf:heartbeat:Dummy): Started rh102-03
* mw_jp1ajs3_103 (ocf:heartbeat:Dummy): Started rh102-03
* Clone Set: ping-clone [ping]:
* ping (ocf:pacemaker:ping): Started rh102-04
* ping (ocf:pacemaker:ping): Started rh102-03
* ping (ocf:pacemaker:ping): Started rh102-05
* ping (ocf:pacemaker:ping): Started rh102-02
* ping (ocf:pacemaker:ping): Started rh102-01
* fence1-vmware_rest (stonith:fence_vmware_rest): Started rh102-05
* fence2-vmware_rest (stonith:fence_vmware_rest): Started rh102-03
* fence3-vmware_rest (stonith:fence_vmware_rest): Started rh102-05
* fence4-vmware_rest (stonith:fence_vmware_rest): Started rh102-02
* fence5-vmware_rest (stonith:fence_vmware_rest): Started rh102-01
Node Attributes:
* Node: rh102-01 (1):
* ping-status : 1
* test1 : 100
* test10 : 100
* test2 : 100
* test3 : 100
* test4 : 100
* test5 : 100
* test6 : 100
* test7 : 100
* test8 : 100
* test9 : 100
* Node: rh102-02 (2):
* ping-status : 1
* test1 : 100
* test10 : 100
* test2 : 100
* test3 : 100
* test4 : 100
* test5 : 100
* test6 : 100
* test7 : 100
* test8 : 100
* test9 : 100
* Node: rh102-03 (3):
* ping-status : 1
* test1 : 100
* test10 : 100
* test2 : 100
* test3 : 100
* test4 : 100
* test5 : 100
* test6 : 100
* test7 : 100
* test8 : 100
* test9 : 100
* Node: rh102-04 (4):
* ping-status : 1
* Node: rh102-05 (5):
* ping-status : 1
* test1 : 100
* test10 : 100
* test2 : 100
* test3 : 100
* test4 : 100
* test5 : 100
* test6 : 100
* test7 : 100
* test8 : 100
* test9 : 100
Migration Summary:
Tickets:
PCSD Status:
rh102-01: Online
rh102-02: Online
rh102-03: Online
rh102-04: Online
rh102-05: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
----
When these operations—stopping nodes and migrating resources—are repeated, the `attrd` process occasionally encounters the following error (though the issue is not consistently reproducible):
- (pcmk__ipc_send_iov) notice: Response 45877 to pid 4671 failed: Resource temporarily unavailable | bytes=xxx rc=-11 ipcs=xxxxx
----
Oct 24 07:14:30.196 rh102-03 pacemaker-controld [4671] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Oct 24 07:14:35.080 rh102-03 pacemaker-based [4666] (cib_process_ping) info: Reporting our current digest to rh102-03: a584dfda8bbde766eccd6586c5c6d4bd for 100000.111769.0
Oct 24 07:15:03.108 rh102-03 pacemaker-controld [4671] (pcmk__update_peer_expected) info: handle_request: Node rh102-04[4] - expected state is now down (was member)Oct 24 07:15:03.108 rh102-03 pacemaker-controld [4671] (handle_shutdown_request) info: Creating shutdown request for rh102-04 (state=S_IDLE)
Oct 24 07:15:03.108 rh102-03 pacemaker-attrd [4669] (pcmk__ipc_send_iov) notice: Response 45877 to pid 4671 failed: Resource temporarily unavailable | bytes=129 rc=-11 ipcs=0x563355afbed0
Oct 24 07:15:03.108 rh102-03 pacemaker-attrd [4669] (pcmk__ipc_send_iov) notice: Response 45877 to pid 4671 failed: Resource temporarily unavailable | bytes=129 rc=-11 ipcs=0x563355afbed0
(snip)
Oct 24 07:15:38.074 rh102-03 pacemaker-attrd [4669] (pcmk__ipc_send_iov) notice: Response 45877 to pid 4671 failed: Resource temporarily unavailable | bytes=129 rc=-11 ipcs=0x563355afbed0
Oct 24 07:15:38.074 rh102-03 pacemakerd [4665] (pcmk__ipc_is_authentic_process_active) info: Could not connect to attrd IPC: timeout
Oct 24 07:15:38.074 rh102-03 pacemakerd [4665] (check_next_subdaemon) crit: Subdaemon pacemaker-attrd[4669] is unresponsive to IPC after 5 attempts and will now be killed
Oct 24 07:15:38.074 rh102-03 pacemakerd [4665] (stop_child) notice: Stopping subdaemon pacemaker-attrd | via signal 9 to process 4669
Oct 24 07:15:38.076 rh102-03 pacemaker-controld [4671] (crm_ipc_read) error: Connection to attrd IPC failedOct 24 07:15:38.076 rh102-03 pacemaker-controld [4671] (mainloop_gio_callback) error: Connection to attrd closed | client=0x559026a87310 condition=25
Oct 24 07:15:38.077 rh102-03 pacemakerd [4665] (pcmk_child_exit) warning: pacemaker-attrd[4669] terminated with signal 9 (Killed)
Oct 24 07:15:38.085 rh102-03 pacemakerd [4665] (pcmk__panic) emerg: Panicking the system directly: Subdaemon failed
Oct 24 07:15:38.087 rh102-03 pacemaker-controld [4671] (ipc_post_disconnect) info: Disconnected from attribute manager
Oct 24 07:15:38.092 rh102-03 pacemaker-based [4666] (cib_perform_op) info: Diff: --- 100000.111769.0 2
Oct 24 07:15:38.092 rh102-03 pacemaker-based [4666] (cib_perform_op) info: Diff: +++ 100000.111769.1 4e936a0f049b5317527184de9940fb9b
Oct 24 07:15:38.092 rh102-03 pacemaker-based [4666] (cib_perform_op) info: -- /cib/status/node_state[@id='3']/transient_attributes[@id='3']
Oct 24 07:15:38.092 rh102-03 pacemaker-based [4666] (cib_perform_op) info: + /cib: @num_updates=1
Oct 24 07:15:38 ping(ping)[731043]: WARNING: Could not update ping-status = 1: rc=102
(snip)
----
Consequently, if `fail_fast` is enabled, `pacemakerd` terminates the `attrd` process, which triggers a node reboot.
It is highly problematic that such an issue can occur simply by performing routine operations like stopping nodes or migrating resources in an otherwise stable cluster. Could there be an underlying issue with the IPC communication used by `attrd`?
Best Regards,
Hideo Yamauchi.
More information about the Users
mailing list