[ClusterLabs] [Problem] The `attrd` process may encounter an IPC communication error (`pcmk__ipc_send_iov`), rendering it unable to communicate and triggering a restart.
renayama19661014 at ybb.ne.jp
renayama19661014 at ybb.ne.jp
Wed Oct 28 04:47:26 UTC 2026
Hi Chris,
Hi All,
Thanks for the comment.
> So the question is: what's controld doing that's keeping it from responding to attrd?
Looking at the subsequent logs, `attrd_updater` from the `ping` resource is failing, which suggests that `attrd` is not responding.
`controld`(4671) itself appears to be running.
----
Oct 24 07:15:38.105 rh102-03 pacemaker-execd [4668] (log_op_output) info: ping_monitor_10000[731043] error output [ attrd_updater: Could not update ping-status=1: Connection reset by peer (104) ]
Oct 24 07:15:38.110 rh102-03 pacemaker-controld [4671] (log_executor_event) notice: Result of monitor operation for ping on rh102-03: Error occurred | graph action unconfirmed; call=108 key=ping_monitor_10000 rc=1
Oct 24 07:15:38.110 rh102-03 pacemaker-controld [4671] (log_executor_event) notice: ping_monitor_10000 at rh102-03 output [ attrd_updater: Could not update ping-status=1: Connection reset by peer (104) ]
Oct 24 07:15:38.112 rh102-03 pacemaker-based [4666] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='rh102-03']/transient_attributes: OK (rc=0, origin=rh102-05/pacemaker-attrd/77237, version=100000.111769.1)
Oct 24 07:15:38.112 rh102-03 pacemaker-based [4666] (forward_request) info: Forwarding cib_modify operation for section status to all (origin=rh102-03/pacemaker-controld/38017)
Oct 24 07:15:38.114 rh102-03 pacemaker-controld [4671] (abort_transition_graph) info: Transition 6764 aborted by deletion of transient_attributes[@id='3']: Transient attribute change | cib=100000.111769.1 source=abort_unless_down:142 path=/cib/status/node_state[@id='3']/transient_attributes[@id='3'] complete=true
Oct 24 07:15:38.114 rh102-03 pacemaker-controld [4671] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graphOct 24 07:15:38.115 rh102-03 pacemaker-based [4666] (cib_perform_op) info: Diff: --- 100000.111769.1 2
Oct 24 07:15:38.115 rh102-03 pacemaker-based [4666] (cib_perform_op) info: Diff: +++ 100000.111769.2 (null)Oct 24 07:15:38.115 rh102-03 pacemaker-based [4666] (cib_perform_op) info: + /cib: @num_updates=2
Oct 24 07:15:38.115 rh102-03 pacemaker-based [4666] (cib_perform_op) info: + /cib/status/node_state[@id='3']: @crm-debug-origin=controld_update_resource_history
Oct 24 07:15:38.115 rh102-03 pacemaker-based [4666] (cib_perform_op) info: ++ /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='ping']: <lrm_rsc_op id="ping_last_failure_0" operation_key="ping_monitor_10000" operation="monitor" crm-debug-origin="controld_update_resource_history" crm_feature_set="3.20.1" transition-key="171:1:0:6417ad8d-8516-427c-9141-01252845936f" transition-magic="0:1;171:1:0:6417ad8d-8516-427c-9141-01252845936f" exit-reason="" on_node="rh102-
----
> The first thing that comes to mind to me is the bug fixed by 39f99f1b.
> Does the commit message on that seem like it might be relevant to your setup?
Looking at the source code for Pacemaker included in RHEL 10.2, it appears that this fix(https://github.com/ClusterLabs/pacemaker/commit/39f99f1b) is not included.
In RHEL 10.2, the patch `006-fewer_messages.patch` has been applied to address the relevant section.
----
@controld_messages.c
(snip)
if (input == I_PE_CALC) {
GList *ele = NULL;
if (prepend) {
ele = g_list_first(controld_globals.fsa_message_queue);
} else {
ele = g_list_last(controld_globals.fsa_message_queue);
}
if (ele != NULL) {
fsa_data_t *message = (fsa_data_t *) ele->data;
if (message->fsa_input == I_PE_CALC) {
crm_debug("%s item in fsa queue is I_PE_CALC, not adding another",
(prepend ? "First" : "Last"));
return;
}
}
}
(snip)
----
> In particular, do you have a lot of resources or nodes that might be generating a lot of transitions all at once?
As previously mentioned, the setup consists of 5 nodes, with approximately 40 resources (such as "dummy" resources). Note that resource placement is determined based on utilization.
Furthermore, this issue is extremely difficult to reproduce; under normal circumstances, this error does not occur.
----
Cluster name: test10-cluster-0
Cluster Summary:
* Stack: corosync (Pacemaker is running)
* Current DC: rh102-03 (3) (version 3.0.1-5.el10-be343ca) - partition with quorum
* Last updated: Thu Oct 22 17:33:18 2026 on rh102-04
* Last change: Thu Oct 22 17:31:55 2026 by root via root on rh102-04
* 5 nodes configured
* 34 resource instances configured
Node List:
* Node rh102-01 (1): online, feature set 3.20.1
* Node rh102-02 (2): online, feature set 3.20.1
* Node rh102-03 (3): online, feature set 3.20.1
* Node rh102-04 (4): online, feature set 3.20.1
* Node rh102-05 (5): online, feature set 3.20.1
Full List of Resources:
* Resource Group: primary-group1:
* ipaddr_gwf_101 (ocf:heartbeat:Dummy): Started rh102-01
* ipaddr_dbf_101 (ocf:heartbeat:Dummy): Started rh102-01
* ipaddr_api_101 (ocf:heartbeat:Dummy): Started rh102-01
* ipaddr_kafka_101 (ocf:heartbeat:Dummy): Started rh102-01
* ipaddr_unyo_101 (ocf:heartbeat:Dummy): Started rh102-01
* fs_share_101 (ocf:heartbeat:Dummy): Started rh102-01
* mw_jp1base_101 (ocf:heartbeat:Dummy): Started rh102-01
* mw_jp1ajs3_101 (ocf:heartbeat:Dummy): Started rh102-01
* Resource Group: primary-group2:
* ipaddr_gwf_102 (ocf:heartbeat:Dummy): Started rh102-02
* ipaddr_dbf_102 (ocf:heartbeat:Dummy): Started rh102-02
* ipaddr_api_102 (ocf:heartbeat:Dummy): Started rh102-02
* ipaddr_kafka_102 (ocf:heartbeat:Dummy): Started rh102-02
* ipaddr_unyo_102 (ocf:heartbeat:Dummy): Started rh102-02
* fs_share_102 (ocf:heartbeat:Dummy): Started rh102-02
* mw_jp1base_102 (ocf:heartbeat:Dummy): Started rh102-02
* mw_jp1ajs3_102 (ocf:heartbeat:Dummy): Started rh102-02
* Resource Group: primary-group3:
* ipaddr_gwf_103 (ocf:heartbeat:Dummy): Started rh102-03
* ipaddr_dbf_103 (ocf:heartbeat:Dummy): Started rh102-03
* ipaddr_api_103 (ocf:heartbeat:Dummy): Started rh102-03
* ipaddr_kafka_103 (ocf:heartbeat:Dummy): Started rh102-03
* ipaddr_unyo_103 (ocf:heartbeat:Dummy): Started rh102-03
* fs_share_103 (ocf:heartbeat:Dummy): Started rh102-03
* mw_jp1base_103 (ocf:heartbeat:Dummy): Started rh102-03
* mw_jp1ajs3_103 (ocf:heartbeat:Dummy): Started rh102-03
* Clone Set: ping-clone [ping]:
* ping (ocf:pacemaker:ping): Started rh102-04
* ping (ocf:pacemaker:ping): Started rh102-03
* ping (ocf:pacemaker:ping): Started rh102-05
* ping (ocf:pacemaker:ping): Started rh102-02
* ping (ocf:pacemaker:ping): Started rh102-01
* fence1-vmware_rest (stonith:fence_vmware_rest): Started rh102-05
* fence2-vmware_rest (stonith:fence_vmware_rest): Started rh102-03
* fence3-vmware_rest (stonith:fence_vmware_rest): Started rh102-05
* fence4-vmware_rest (stonith:fence_vmware_rest): Started rh102-02
* fence5-vmware_rest (stonith:fence_vmware_rest): Started rh102-01
Node Attributes:
(snip)
----
Many thanks,
Hideo Yamauchi.
More information about the Users
mailing list