[ClusterLabs] pacemaker&corosync services shutting off after the network cable is disconnected

[john tillman]

Thank you for the response, Chris.

fence-remote-without-quorum -> thanks for the info and pointing out my 
mistaken usage.

I could send along a pacemaker log file.  It's 753 lines and about 116K 
bytes.  I'm not sure that's appropriate here.  Let me know if I'm wrong 
about that and I can send it as an attachment.

Spot checking through it I can point out some highlights I see.  Maybe 
that can be a starting point for further questions?  Apologies for the 
long email here.

pacemaker.log: (paraphrased)

I see what seems like normal "I'm disconnected from the rest of the 
cluster" messages.
I see the resources being stopped, as I expect.
After the resources are all stopped I see this:

Aug 06 21:20:40.479 db pacemaker-schedulerd[499615] 
(pcmk__log_transition_summary) 	warning: Calculated transition 1 (with 
warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-21.bz2

The above is the last message in the log from the pacemaker-schedulerd 
service.  It was immediately followed in the log by:

Aug 06 21:20:42.770 db pacemakerd          [1883] 
(pcmk__ipc_is_authentic_process_active) 	info: Could not connect to 
pengine IPC: timeout
Aug 06 21:20:42.770 db pacemakerd          [1883] (check_next_subdaemon) 
	notice: pacemaker-schedulerd[499615] is unresponsive to IPC after 1 
attempt
Aug 06 21:20:44.465 db pacemaker-based     [1997] (cib_process_ping) 
	info: Reporting our current digest to 11.11.11.40: 
986a4c7499f43eb71799560915856ada for 0.23.22
Aug 06 21:20:49.770 db pacemakerd          [1883] 
(pcmk__ipc_is_authentic_process_active) 	info: Could not connect to 
pengine IPC: timeout
Aug 06 21:20:49.770 db pacemakerd          [1883] (check_next_subdaemon) 
	notice: pacemaker-schedulerd[499615] is unresponsive to IPC after 2 
attempts
Aug 06 21:20:56.770 db pacemakerd          [1883] 
(pcmk__ipc_is_authentic_process_active) 	info: Could not connect to 
pengine IPC: timeout

There are more of those attempts repeated.  Then it later changes 
slightly to:

Aug 06 21:24:34.771 db pacemakerd          [1883] 
(pcmk__ipc_is_authentic_process_active) 	info: Could not connect to crmd 
IPC: Connection refused
Aug 06 21:24:41.770 db pacemakerd          [1883] 
(pcmk__ipc_is_authentic_process_active) 	info: Could not connect to 
pengine IPC: timeout
Aug 06 21:24:41.770 db pacemakerd          [1883] (check_next_subdaemon) 
	crit: pacemaker-schedulerd[499615] is unresponsive to IPC after 20 
attempts and will now be killed
Aug 06 21:24:41.770 db pacemakerd          [1883] (stop_child) 	notice: 
Stopping pacemaker-schedulerd | sent signal 9 to process 499615

There are many more of these printed in the log.  Note it changed from 
"notice: " to "crit: " and signal 9 is now sent.  In the log are 80 or 
so attempts to kill the process with signal 9.  They are ignored.  
Finally, about 12 minutes later:

Aug 06 21:36:15.522 db pacemakerd          [1883] (pcmk_child_exit) 
	warning: pacemaker-schedulerd[499615] terminated with signal 9 (Killed)

Once the scheduler service is finally killed, the rest of pacemaker's 
services start shutting down, one by one, with a signal 15 until they 
are all gone and the pacemaker.log file is quiet.

Does any of this help?  To me, the scheduler becoming unresponsive is 
the initial failure.  Is there any way to produce more logging from the 
scheduled service?

Best regards,
-John


On 2026-09-09 14:53, Chris Lumens via Users wrote:
>> I saw a new property fence-remote-without-quorum and set it 'false'. 
>> Read it might take of this situation, but it did not.  And stonith is 
>> disabled.
> 
> If you have fencing disabled, that new property wouldn't do anything. 
> In general, that property is to take care of a niche case that I expect 
> very few people to ever run into.
>> Any suggestions would be appreciated.
>> 
>> If you would like more information, please let me know what I can 
>> provide.
> We'd need to see why the system services are shutting down.  Are they 
> crashing for some reason?  You could use journalctl to dig through the 
> system log around the time it happens and see what's going on.  Looking 
> at /var/log/pacemaker between the time you pull the cable and the 
> services stop would also be useful.
> 
> From there, maybe we can track down what might be happening and if it's 
> a bug or if it's some intended behavior.  As for the latter, nothing is 
> coming to mind at the moment.
> 
> - Chris
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/