[ClusterLabs] [EXT] Pacemaker fatal shutdown

[Windl, Ulrich]

Hi!

I guess the interesting log lines are before "fatal failure". Also: Did you install the current updates? Some configuration details would be interesting, like at least the output of "crm_mon -1Arfj".
And, of course, you cannot use the crm shell on a node where pacemaker isn't running (if that was your question).
Also: Did you contact SUSE support?

Kind regards,
Ulrich Windl

-----Original Message-----
From: Users <users-bounces at clusterlabs.org> On Behalf Of Priyanka Balotra
Sent: Wednesday, July 19, 2023 8:20 PM
To: Cluster Labs - All topics related to open-source clustering welcomed <users at clusterlabs.org>
Subject: [EXT] [ClusterLabs] Pacemaker fatal shutdown

Hi All, 
I am using SLES 15 SP4. One of the nodes of the cluster is brought down and boot up after sometime. Pacemaker service came up first but later it faced a fatal shutdown. Due to that crm service is down. 

The logs from /var/log/pacemaker.pacemaker.log are as follows:

Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956] (pcmk_child_exit)        warning: Shutting cluster down because pacemaker-controld[15962] had fatal failure
Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956] (pcmk_shutdown_worker)   notice: Shutting down Pacemaker
Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956] (pcmk_shutdown_worker)   debug: pacemaker-controld confirmed stopped
Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956] (stop_child)     notice: Stopping pacemaker-schedulerd | sent signal 15 to process 15961
Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] (crm_signal_dispatch)    notice: Caught 'Terminated' signal | 15 (invoking handler)
Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] (qb_ipcs_us_withdraw)    info: withdrawing server sockets
Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] (qb_ipcs_unref)  debug: qb_ipcs_unref() - destroying
Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] (crm_xml_cleanup)        info: Cleaning up memory from libxml2
Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] (crm_exit)       info: Exiting pacemaker-schedulerd | with status 0
Jul 17 14:18:20.093 FILE-2 pacemaker-based     [15957] (qb_ipcs_event_sendv)    debug: new_event_notification (/dev/shm/qb-15957-15962-12-RDPw6O/qb): Broken pipe (32)
Jul 17 14:18:20.093 FILE-2 pacemaker-based     [15957] (cib_notify_send_one)    warning: Could not notify client crmd: Broken pipe | id=e29d175e-7e91-4b6a-bffb-fabfdd7a33bf
Jul 17 14:18:20.093 FILE-2 pacemaker-based     [15957] (cib_process_request)    info: Completed cib_delete operation for section //node_state[@uname='FILE-2']/*: OK (rc=0, origin=FILE-6/crmd/74, version=0.24.75)
Jul 17 14:18:20.093 FILE-2 pacemaker-fenced    [15958] (xml_patch_version_check)        debug: Can apply patch 0.24.75 to 0.24.74
Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956] (pcmk_child_exit)        info: pacemaker-schedulerd[15961] exited with status 0 (OK)
Jul 17 14:18:20.093 FILE-2 pacemaker-based     [15957] (cib_process_request)    info: Completed cib_modify operation for section status: OK (rc=0, origin=FILE-6/crmd/75, version=0.24.75)
Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956] (pcmk_shutdown_worker)   debug: pacemaker-schedulerd confirmed stopped
Jul 17 14:18:20.093 FILE-2 pacemakerd          [15956] (stop_child)     notice: Stopping pacemaker-attrd | sent signal 15 to process 15960
Jul 17 14:18:20.093 FILE-2 pacemaker-attrd     [15960] (crm_signal_dispatch)    notice: Caught 'Terminated' signal | 15 (invoking handler)



Could you please help me understand the issue here.

Regards
Priyanka