[ClusterLabs] Pacemaker 3.0.3 controld SIGSEGV on repeated resource start-failure → cluster shutdown (not seen on 2.1.6)
S Sathish S
s.s.sathish at ericsson.com
Wed Nov 4 04:39:55 UTC 2026
Hi Team,
We found one observation where a single OCF resource that keeps failing to start eventually crashes pacemaker-controld (SIGSEGV) and then brings the entire cluster down(pacemaker service went down) with a fatal error (CRM_EX_FATAL / exit 100) on Pacemaker 3.0.3. The exact same condition on Pacemaker 2.1.6 does NOT crash controld - the resource is simply marked FAILED and the cluster stays up. This looks like a regression in the 3.0.x controller / IPC event-handling path.
To compare behavior, we intentionally kept one resource (an OCF agent whose start consistently fails) in a permanent start-failure state.
Environment:
Affected (crashes):
Pacemaker 3.0.3-1.el8 (pacemakerd --version: "Pacemaker 3.0.3-1.el8")
pcs 0.12.3.1-1.el8, corosync 3.1.10-1.el8
RHEL 8.10 (Ootpa), kernel 4.18.0-553.174.1.el8_10.x86_64
Single-node cluster (node is DC), stonith configured.
Not affected (survives identical trigger):
Pacemaker 2.1.6-1.el8, pcs 0.10.16-1.el8, corosync 3.1.7-1.el8
Same OS, same resource, same resource configuration.
Symptom / impact:
* A single resource that repeatedly fails 'start' drives Pacemaker into a continuous stop -> start -> stop recovery loop (roughly every 30s in our setup).
* After ~2 days / ~2300 failed start attempts, pacemaker-controld crashed with signal 11 (SIGSEGV), and because the controld respawn budget was exhausted, pacemakerd declared a fatal failure and shut the whole stack down, and it stays down.
* Result: total loss of the cluster manager on the node; all managed processes left running with no cluster manager controlling them; manual pacemaker restart was required.
Timeline (affected node):
* 2026-10-28 14:00 CET (approx) Target OCF resource intentionally driven into a permanent start-failure state; recovery loop begins (every 'start' fails, status 1, ~20s timeout)
* 2026-10-30 14:53:48 CET pacemaker-controld fatal failure -> CRM_EX_FATAL cluster shutdown. ExecMainStatus=100, ActiveState=inactive/dead.
[root at testnode-1 testuser]# systemctl status pacemaker
● pacemaker.service - Pacemaker High Availability Cluster Manager
Loaded: loaded (/usr/lib/systemd/system/pacemaker.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Fri 2026-10-30 14:53:48 CET; 4 days ago
Docs: man:pacemakerd
https://clusterlabs.org/pacemaker/doc/
Process: 62993 ExecStart=/usr/sbin/pacemakerd (code=exited, status=100)
Main PID: 62993 (code=exited, status=100)
Tasks: 121
Memory: 9.4G
CGroup: /system.slice/pacemaker.service
├─79496 /opt/java//jre-17-openjdk/bin/java -server -XX:+UseG1GC -XX:G1ReservePercent=30 -XX:+AlwaysPreTouch -XX:+UseTransparentHugePages -XX:ConcGCThreads=2 -XX:ParallelGCT…
├─79894 /opt/java//jre-17-openjdk/bin/java -Djava.util.logging.config.file=/opt/webstart//conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLog…
└─96139 /usr/sbin/crm_mon -d -i 15 -E /opt/occ/CXP_902_0588_R20CT801/tools/PCSESA.sh --output-as=html --output-to=/tmp/ClusterMon_SNMP_testuser-1.html
Oct 30 14:53:48 testnode-1 systemd[1]: pacemaker.service: Succeeded.
Logs:
The crash is consistently preceded by pacemaker-execd failing to deliver a resource action event to controld over IPC. Full shutdown sequence (2026-10-30 14:53:48, from pacemaker.log):
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk__ipc_is_authentic_process_active) info: Could not connect to attrd IPC: Connection reset by peer
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (check_next_subdaemon) notice: Subdaemon pacemaker-attrd[62998] is unresponsive to IPC after 1 attempt (will recheck later)
Oct 30 14:53:48 testnode-1 crm_mon [96138] (crm_ipc_read) error: Connection to cib_rw IPC failed
Oct 30 14:53:48 testnode-1 crm_mon [96138] (mainloop_gio_callback) error: Connection to cib_rw closed | client=0x55b2ed79dd80 condition=17
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (cancel_recurring_action) info: Cancelling ocf operation ESAPMA_OCC_monitor_10000
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (qb_ipcs_event_sendv) warning: new_event_notification (/dev/shm/qb-62997-63000-6-7AAcJe/qb): Bad file descriptor (9)
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (pcmk__ipc_send_xml) notice: IPC message to pid 63000 failed: Bad file descriptor | rc=9
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (send_client_notify) warning: Could not notify client crmd: Bad file descriptor | rc=9
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk_child_exit) warning: Shutting cluster down because pacemaker-controld[63000] had fatal failure
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (cancel_recurring_action) info: Cancelling ocf operation ESASMA_testnode-1_monitor_10000
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk_shutdown_worker) notice: Shutting down Pacemaker
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (stop_child) notice: Stopping subdaemon pacemaker-schedulerd | via signal 15 to process 62999
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (cancel_recurring_action) info: Cancelling ocf operation MANAGER_OCC_monitor_10000
Oct 30 14:53:48 testnode-1 pacemaker-execd [62997] (cancel_recurring_action) info: Cancelling ocf operation ESASMS_testnode-1_monitor_10000
Oct 30 14:53:48 testnode-1 pacemakerd [62993] (pcmk_child_exit) error: pacemaker-attrd[62998] exited with status 102 (Not connected)
The IPC failure pattern that immediately precedes the crash is: execd tries to notify controld of a resource action result (the repeated start/stop churn), controld's IPC endpoint is already gone (Bad file descriptor), and controld then crashes with SIGSEGV. Since the respawn budget was exhausted, pacemakerd treated this as fatal and shut the whole stack down (attrd exiting with status 102 "Not connected" is a side effect of controld's IPC teardown, not an independent fault).
Expected Behavior:
A resource that repeatedly fails to start should be handled via migration-threshold / on-fail (marked FAILED / stopped) without crashing pacemaker-controld or forcing a whole-cluster fatal shutdown - i.e. the Pacemaker 2.1.6 behavior we observe on the comparison node.
Resource configuration (same on both nodes)
--------------------------------------------------------------------------------
migration-threshold=5 failure-timeout=120s priority=70
start: timeout=120s on-fail=restart
monitor: interval=10s timeout=120s on-fail=restart
stop: timeout=120s on-fail=block
Question:
1. Is this a known issue / already fixed after 3.0.3 in the controller or IPC (libcrmcommon mainloop_gio / event-notification) code?
2. Is there a recommended configuration workaround to avoid the fatal path while a fix is pending (e.g. tuning around resource failure handling)?
Attachments we provided for working and non-working node for pacemaker logs.
Pacemaker:3.0.3 --> testnode-1_pacemaker_not-working.log
Pacemaker:2.1.6 --> testnode-2_pacemaker-working.log and testnode-2_pacemaker_rotated-working.log
Thanks and Regards,
S Sathish S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20261104/a7011048/attachment-0001.htm>
More information about the Users
mailing list