[ClusterLabs] Pacemaker fatal shutdown

[Ken Gaillot]

On Wed, 2023-07-19 at 23:49 +0530, Priyanka Balotra wrote:
> Hi All, 
> I am using SLES 15 SP4. One of the nodes of the cluster is brought
> down and boot up after sometime. Pacemaker service came up first but
> later it faced a fatal shutdown. Due to that crm service is down. 
> 
> The logs from /var/log/pacemaker.pacemaker.log are as follows:
> 
> Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956]
> (pcmk_child_exit)        warning: Shutting cluster down because
> pacemaker-controld[15962] had fatal failure

The interesting messages will be before this. The ones with "pacemaker-
controld" will be the most relevant, at least initially.

> Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956]
> (pcmk_shutdown_worker)   notice: Shutting down Pacemaker
> Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956]
> (pcmk_shutdown_worker)   debug: pacemaker-controld confirmed stopped
> Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956] (stop_child)  
>   notice: Stopping pacemaker-schedulerd | sent signal 15 to process
> 15961
> Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> (crm_signal_dispatch)    notice: Caught 'Terminated' signal | 15
> (invoking handler)
> Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> (qb_ipcs_us_withdraw)    info: withdrawing server sockets
> Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> (qb_ipcs_unref)  debug: qb_ipcs_unref() - destroying
> Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> (crm_xml_cleanup)        info: Cleaning up memory from libxml2
> Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] (crm_exit)    
>   info: Exiting pacemaker-schedulerd | with status 0
> Jul 17 14:18:20.093 FILE-2 pacemaker-based     [15957]
> (qb_ipcs_event_sendv)    debug: new_event_notification (/dev/shm/qb-
> 15957-15962-12-RDPw6O/qb): Broken pipe (32)
> Jul 17 14:18:20.093 FILE-2 pacemaker-based     [15957]
> (cib_notify_send_one)    warning: Could not notify client crmd:
> Broken pipe | id=e29d175e-7e91-4b6a-bffb-fabfdd7a33bf
> Jul 17 14:18:20.093 FILE-2 pacemaker-based     [15957]
> (cib_process_request)    info: Completed cib_delete operation for
> section //node_state[@uname='FILE-2']/*: OK (rc=0, origin=FILE-
> 6/crmd/74, version=0.24.75)
> Jul 17 14:18:20.093 FILE-2 pacemaker-fenced    [15958]
> (xml_patch_version_check)        debug: Can apply patch 0.24.75 to
> 0.24.74
> Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956]
> (pcmk_child_exit)        info: pacemaker-schedulerd[15961] exited
> with status 0 (OK)
> Jul 17 14:18:20.093 FILE-2 pacemaker-based     [15957]
> (cib_process_request)    info: Completed cib_modify operation for
> section status: OK (rc=0, origin=FILE-6/crmd/75, version=0.24.75)
> Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956]
> (pcmk_shutdown_worker)   debug: pacemaker-schedulerd confirmed
> stopped
> Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956] (stop_child)  
>   notice: Stopping pacemaker-attrd | sent signal 15 to process 15960
> Jul 17 14:18:20.093 FILE-2 pacemaker-attrd     [15960]
> (crm_signal_dispatch)    notice: Caught 'Terminated' signal | 15
> (invoking handler)
> 
> Could you please help me understand the issue here.
> 
> Regards
> Priyanka
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
-- 
Ken Gaillot <kgaillot at redhat.com>