[ClusterLabs] Pacemaker fatal shutdown
Klaus Wenninger
kwenning at redhat.com
Mon Jul 24 04:39:52 EDT 2023
Well I guess the comment in the code explains it quite well:
/* We were notified of our own fencing. Most likely, either fencing
was
* misconfigured, or fabric fencing that doesn't cut cluster
* communication is in use.
*
* Either way, shutting down the local host is a good idea, to
require
* administrator intervention. Also, other nodes would otherwise
likely
* set our status to lost because of the fencing callback and
discard
* our subsequent election votes as "not part of our cluster".
*/
Basically meaning that if you are there to hear about your own fencing then
something
is wrong. Although iirc that has been seen in the past if a node rebooted
really quickly.
Might as well be some kind of race in a startup-fencing (fence all nodes
not seen
on startup within a certain time everybody is waiting to see each other)
scenario.
But probably still an issue with nodes not properly seeing each other ...
To be able to tell more we'd probably need to know more about your
fencing-setup.
Is that a cluster that has been fired up for the first time or has it been
working before?
Regards,
Klaus
On Mon, Jul 24, 2023 at 7:27 AM Priyanka Balotra <
priyanka.14balotra at gmail.com> wrote:
> Gentle Reminder!
>
> On Fri, Jul 21, 2023 at 10:06 AM Priyanka Balotra <
> priyanka.14balotra at gmail.com> wrote:
>
>>
>> Hi All,
>> Any updates on this issue?
>>
>> Regards
>> Priyanka
>>
>> On Thu, 20 Jul 2023 at 12:43 PM, Priyanka Balotra <
>> priyanka.14balotra at gmail.com> wrote:
>>
>>> What I mainly want to understand is that:
>>> - why "fatal failure" is coming
>>> - why does pacemaker not start on the node after a node boots followed
>>> by "pacemaker fatal failure" .
>>> - How can this be handled?
>>>
>>> Thanks
>>> Priyanka
>>>
>>> On Thu, Jul 20, 2023 at 12:41 PM Priyanka Balotra <
>>> priyanka.14balotra at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Here are FILE-6 logs:
>>>>
>>>> 65710:Jul 17 14:16:51.517 FILE-6 pacemaker-controld [19415]
>>>> (throttle_mode) debug: Current load is 0.760000 across 10 core(s)
>>>> 65711:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (throttle_update) debug: Node FILE-2 has negligible load and supports at
>>>> most 20 jobs; new job limit 20
>>>> 65712:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (handle_request) debug: The throttle changed. Trigger a graph.
>>>> 65713:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00020000 (new_actions)
>>>> for controller set by s_crmd_fsa:198
>>>> 65714:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_JOIN_REQUEST: [ state=S_INTEGRATION
>>>> cause=C_HA_MESSAGE origin=route_message ]
>>>> 65715:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00020000 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65716:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (do_dc_join_filter_offer) debug: Accepting join-1 request from FILE-2 |
>>>> ref=join_request-crmd-1689603392-8
>>>> 65717:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__update_peer_expected) info: do_dc_join_filter_offer: Node
>>>> FILE-2[2] - expected state is now member (was (null))
>>>> 65718:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (do_dc_join_filter_offer) debug: 2 nodes currently integrated in join-1
>>>> 65719:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (check_join_state) debug: join-1: Integration of 2 peers complete |
>>>> state=S_INTEGRATION for=do_dc_join_filter_offer
>>>> 65720:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00040000 (new_actions)
>>>> for controller set by s_crmd_fsa:198
>>>> 65721:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_INTEGRATED: [ state=S_INTEGRATION
>>>> cause=C_FSA_INTERNAL origin=check_join_state ]
>>>> 65722:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (do_state_transition) info: State transition S_INTEGRATION ->
>>>> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL
>>>> origin=check_join_state
>>>> 65723:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00000020
>>>> (A_INTEGRATE_TIMER_STOP) for controller set by do_state_transition:559
>>>> 65724:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00000040
>>>> (A_FINALIZE_TIMER_START) for controller set by do_state_transition:563
>>>> 65725:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00000200
>>>> (A_DC_TIMER_STOP) for controller set by do_state_transition:569
>>>> 65726:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (do_state_transition) debug: All cluster nodes (2) responded to join
>>>> offer
>>>> 65727:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000200 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65728:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000020 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65729:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000040 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65730:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (controld_start_timer) debug: Started Finalization Timer (inject
>>>> I_ELECTION if pops after 1800000ms, source=119)
>>>> 65731:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00040000 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65732:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (do_dc_join_finalize) debug: Finalizing join-1 for 2 nodes (sync'ing
>>>> from local CIB)
>>>> 65733:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (do_dc_join_finalize) debug: Requested CIB version <generation_tuple
>>>> crm_feature_set="3.11.0" validate-with="pacemaker-3.7" epoch="24"
>>>> num_updates="72" admin_epoch="0" cib-last-written="Thu Jul 13 13:11:46
>>>> 2023" update-origin="FILE-1" update-client="cibadmin" update-user="root"
>>>> have-quorum="1" dc-uuid="6"/>
>>>> 65734:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-6=integrated
>>>> 65735:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-2=integrated
>>>> 65736:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-3=confirmed
>>>> 65737:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-1=none
>>>> 65738:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-5=confirmed
>>>> 65739:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-4=confirmed
>>>> 65740:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
>>>> (fsa_data->actions) for controller set by s_crmd_fsa:193
>>>> 65741:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (new_actions) for controller set by s_crmd_fsa:198
>>>> 65742:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
>>>> state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
>>>> 65743:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65744:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415] (do_log)
>>>> info: Input I_WAIT_FOR_EVENT received in state S_FINALIZE_JOIN from
>>>> do_te_invoke
>>>> 65745:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415] (do_log)
>>>> debug: do_log <create_request_adv origin="do_cl_join_query" t="crmd"
>>>> version="3.11.0" subt="request" reference="join_announce-crmd-1689603376-2"
>>>> crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd" src="FILE-1"/>
>>>> 65746:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65747:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
>>>> source=do_te_invoke:135 complete=false
>>>> 65748:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (register_fsa_input_adv) debug: Stalling the FSA pending further input:
>>>> source=do_te_invoke cause=C_HA_MESSAGE data=0x55c6194ed4c0 queue=0
>>>> 65749:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Exiting the FSA: queue=1, fsa_actions=0x0,
>>>> stalled=true
>>>> 65750:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (fsa_dump_queue) debug: queue[0.72]: input I_WAIT_FOR_EVENT raised by
>>>> do_te_invoke(0x55c619869580.1) (cause=C_HA_MESSAGE)
>>>> 65751:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
>>>> (fsa_data->actions) for controller set by s_crmd_fsa:193
>>>> 65752:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (new_actions) for controller set by s_crmd_fsa:198
>>>> 65753:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
>>>> state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
>>>> 65754:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65755:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415] (do_log)
>>>> info: Input I_WAIT_FOR_EVENT received in state S_FINALIZE_JOIN from
>>>> do_te_invoke
>>>> 65756:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415] (do_log)
>>>> debug: do_log <create_request_adv origin="do_cl_join_query" t="crmd"
>>>> version="3.11.0" subt="request" reference="join_announce-crmd-1689603376-2"
>>>> crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd" src="FILE-1"/>
>>>> 65757:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65758:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
>>>> source=do_te_invoke:135 complete=false
>>>> 65759:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (register_fsa_input_adv) debug: Stalling the FSA pending further input:
>>>> source=do_te_invoke cause=C_HA_MESSAGE data=0x55c619869580 queue=0
>>>> 65760:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Exiting the FSA: queue=1, fsa_actions=0x0,
>>>> stalled=true
>>>> 65761:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (fsa_dump_queue) debug: queue[0.73]: input I_WAIT_FOR_EVENT raised by
>>>> do_te_invoke(0x55c6194ed4c0.1) (cause=C_HA_MESSAGE)
>>>> 65762:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__execute_graph) debug: Transition 0 (Complete=33, Pending=2,
>>>> Fired=0, Skipped=0, Incomplete=24,
>>>> Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
>>>> 65764:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (check_join_state) debug: join-1: Still waiting on 2 integrated
>>>> nodes | state=S_FINALIZE_JOIN for=finalize_sync_callback
>>>> 65765:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-6=integrated
>>>> 65766:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-2=integrated
>>>> 65767:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-3=confirmed
>>>> 65768:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-1=none
>>>> 65769:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-5=confirmed
>>>> 65770:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-4=confirmed
>>>> 65771:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (finalize_sync_callback) debug: Notifying 2 nodes of join-1 results
>>>> 65772:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (finalize_join_for) debug: Acknowledging join-1 request from FILE-6
>>>> 65773:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
>>>> (finalize_join_for) debug: Acknowledging join-1 request from FILE-2
>>>> 65776:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (handle_request) debug: Raising I_JOIN_RESULT: join-1
>>>> 65777:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
>>>> (fsa_data->actions) for controller set by s_crmd_fsa:193
>>>> 65778:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (new_actions) for controller set by s_crmd_fsa:198
>>>> 65779:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
>>>> state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
>>>> 65780:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65781:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415] (do_log)
>>>> info: Input I_WAIT_FOR_EVENT received in state S_FINALIZE_JOIN from
>>>> do_te_invoke
>>>> 65782:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415] (do_log)
>>>> debug: do_log <create_request_adv origin="do_cl_join_query" t="crmd"
>>>> version="3.11.0" subt="request" reference="join_announce-crmd-1689603376-2"
>>>> crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd" src="FILE-1"/>
>>>> 65783:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65784:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
>>>> source=do_te_invoke:135 complete=false
>>>> 65785:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (register_fsa_input_adv) debug: Stalling the FSA pending further input:
>>>> source=do_te_invoke cause=C_HA_MESSAGE data=0x55c6194ed4c0 queue=1
>>>> 65786:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Exiting the FSA: queue=2, fsa_actions=0x0,
>>>> stalled=true
>>>> 65787:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (fsa_dump_queue) debug: queue[0.74]: input I_JOIN_RESULT raised by
>>>> route_message(0x55c619861a90.1) (cause=C_HA_MESSAGE)
>>>> 65788:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (fsa_dump_queue) debug: queue[1.75]: input I_WAIT_FOR_EVENT raised by
>>>> do_te_invoke(0x55c61986ed80.1) (cause=C_HA_MESSAGE)
>>>> 65789:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__execute_graph) debug: Transition 0 (Complete=33, Pending=2,
>>>> Fired=0, Skipped=0, Incomplete=24,
>>>> Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
>>>> 65792:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00880000 (new_actions)
>>>> for controller set by s_crmd_fsa:198
>>>> 65793:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_JOIN_RESULT: [ state=S_FINALIZE_JOIN
>>>> cause=C_HA_MESSAGE origin=route_message ]
>>>> 65794:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00800000 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65795:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource stonith-sbd after monitor op complete (interval=0)
>>>> 65796:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource FILE_Filesystem after monitor op complete (interval=0)
>>>> 65797:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource Service_pfile after monitor op complete (interval=0)
>>>> 65798:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource Service_Postgresql after monitor op complete (interval=0)
>>>> 65799:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource Service_esm_primary after monitor op complete (interval=0)
>>>> 65800:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource Service_Postgrest after monitor op complete (interval=0)
>>>> 65801:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource IP_Floating after monitor op complete (interval=0)
>>>> 65802:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource Shared_Cluster_Backup after monitor op complete (interval=0)
>>>> 65803:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (do_cl_join_finalize_respond) debug: Confirming join-1: sending local
>>>> operation history to FILE-6
>>>> 65804:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00080000 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65805:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (do_dc_join_ack) debug: Ignoring 'join_ack_nack' message from FILE-6
>>>> while waiting for 'join_confirm'
>>>> 65806:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
>>>> (fsa_data->actions) for controller set by s_crmd_fsa:193
>>>> 65807:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (new_actions) for controller set by s_crmd_fsa:198
>>>> 65808:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
>>>> state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
>>>> 65809:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65810:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415] (do_log)
>>>> info: Input I_WAIT_FOR_EVENT received in state S_FINALIZE_JOIN from
>>>> do_te_invoke
>>>> 65811:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415] (do_log)
>>>> debug: do_log <create_request_adv origin="do_cl_join_query" t="crmd"
>>>> version="3.11.0" subt="request" reference="join_announce-crmd-1689603376-2"
>>>> crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd" src="FILE-1"/>
>>>> 65812:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65813:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
>>>> source=do_te_invoke:135 complete=false
>>>> 65814:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (register_fsa_input_adv) debug: Stalling the FSA pending further input:
>>>> source=do_te_invoke cause=C_HA_MESSAGE data=0x55c61986ed80 queue=1
>>>> 65815:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Exiting the FSA: queue=2, fsa_actions=0x0,
>>>> stalled=true
>>>> 65816:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (fsa_dump_queue) debug: queue[0.76]: input I_JOIN_RESULT raised by
>>>> route_message(0x55c619871630.1) (cause=C_HA_MESSAGE)
>>>> 65817:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (fsa_dump_queue) debug: queue[1.77]: input I_WAIT_FOR_EVENT raised by
>>>> do_te_invoke(0x55c619861a90.1) (cause=C_HA_MESSAGE)
>>>> 65818:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__execute_graph) debug: Transition 0 (Complete=33, Pending=2,
>>>> Fired=0, Skipped=0, Incomplete=24,
>>>> Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
>>>> 65821:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00880000 (new_actions)
>>>> for controller set by s_crmd_fsa:198
>>>> 65822:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_JOIN_RESULT: [ state=S_FINALIZE_JOIN
>>>> cause=C_HA_MESSAGE origin=route_message ]
>>>> 65823:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00800000 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65824:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00080000 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65825:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (controld_delete_node_state) info: Deleting resource history for node
>>>> FILE-2 (via CIB call 71) | xpath=//node_state[@uname='FILE-2']/lrm
>>>> 65826:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (do_dc_join_ack) debug: Updating node history for FILE-2 from join-1
>>>> confirmation (via CIB call 72)
>>>> 65827:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
>>>> (fsa_data->actions) for controller set by s_crmd_fsa:193
>>>> 65828:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (new_actions) for controller set by s_crmd_fsa:198
>>>> 65829:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
>>>> state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
>>>> 65830:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65831:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415] (do_log)
>>>> info: Input I_WAIT_FOR_EVENT received in state S_FINALIZE_JOIN from
>>>> do_te_invoke
>>>> 65832:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415] (do_log)
>>>> debug: do_log <create_request_adv origin="do_cl_join_query" t="crmd"
>>>> version="3.11.0" subt="request" reference="join_announce-crmd-1689603376-2"
>>>> crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd" src="FILE-1"/>
>>>> 65833:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65834:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
>>>> source=do_te_invoke:135 complete=false
>>>> 65835:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (register_fsa_input_adv) debug: Stalling the FSA pending further input:
>>>> source=do_te_invoke cause=C_HA_MESSAGE data=0x55c619861a90 queue=1
>>>> 65836:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Exiting the FSA: queue=2, fsa_actions=0x0,
>>>> stalled=true
>>>> 65837:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (fsa_dump_queue) debug: queue[0.78]: input I_JOIN_RESULT raised by
>>>> route_message(0x55c6198798d0.1) (cause=C_HA_MESSAGE)
>>>> 65838:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (fsa_dump_queue) debug: queue[1.79]: input I_WAIT_FOR_EVENT raised by
>>>> do_te_invoke(0x55c619871630.1) (cause=C_HA_MESSAGE)
>>>> 65839:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__execute_graph) debug: Transition 0 (Complete=33, Pending=2,
>>>> Fired=0, Skipped=0, Incomplete=24,
>>>> Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
>>>> 65851:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
>>>> (cib_delete_callback) debug: Deletion of resource history for node
>>>> FILE-2 (via CIB call 71) succeeded
>>>> 65861:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
>>>> (te_update_diff) debug: Processing (cib_modify) diff: 0.24.72 -> 0.24.73
>>>> (S_FINALIZE_JOIN)
>>>> 65862:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
>>>> (join_update_complete_callback) debug: join-1 node history update (via
>>>> CIB call 72) complete
>>>> 65863:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
>>>> (check_join_state) debug: join-1: Still waiting on 1 finalized node
>>>> | state=S_FINALIZE_JOIN for=join_update_complete_callback
>>>> 65864:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-6=finalized
>>>> 65865:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-2=confirmed
>>>> 65866:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-3=confirmed
>>>> 65867:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-1=none
>>>> 65868:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-5=confirmed
>>>> 65869:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
>>>> (crmd_join_phase_log) debug: join-1: FILE-4=confirmed
>>>> 65876:Jul 17 14:17:21.517 FILE-6 pacemaker-controld [19415]
>>>> (throttle_cib_load) debug: cib load: 0.001000 (3 ticks in 30s)
>>>> 65877:Jul 17 14:17:21.517 FILE-6 pacemaker-controld [19415]
>>>> (throttle_mode) debug: Current load is 0.960000 across 10 core(s)
>>>> 65878:Jul 17 14:17:51.517 FILE-6 pacemaker-controld [19415]
>>>> (throttle_cib_load) debug: cib load: 0.000333 (1 ticks in 30s)
>>>> 65879:Jul 17 14:17:51.517 FILE-6 pacemaker-controld [19415]
>>>> (throttle_mode) debug: Current load is 0.580000 across 10 core(s)
>>>> 65883:Jul 17 14:18:20.085 FILE-6 pacemaker-fenced [19411]
>>>> (process_remote_stonith_exec) debug: Finalizing action 'reboot'
>>>> targeting FILE-2 on behalf of pacemaker-controld.19415 at FILE-6: OK |
>>>> rc=0 id=4e523b34
>>>> 65884:Jul 17 14:18:20.085 FILE-6 pacemaker-fenced [19411]
>>>> (remote_op_done) notice: Operation 'reboot' targeting FILE-2 by FILE-4
>>>> for pacemaker-controld.19415 at FILE-6: OK | id=4e523b34
>>>> 65886:Jul 17 14:18:20.085 FILE-6 pacemaker-controld [19415]
>>>> (tengine_stonith_callback) notice: Stonith operation
>>>> 3/63:0:0:232e6505-2e98-4a79-b6ce-5f26d9cba645: OK (0)
>>>> 65887:Jul 17 14:18:20.085 FILE-6 pacemaker-controld [19415]
>>>> (tengine_stonith_callback) info: Stonith operation 3 for FILE-2
>>>> passed
>>>> 65888:Jul 17 14:18:20.085 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__update_peer_expected) info: crmd_peer_down: Node FILE-2[2] -
>>>> expected state is now down (was member)
>>>> 65889:Jul 17 14:18:20.085 FILE-6 pacemaker-controld [19415]
>>>> (send_stonith_update) debug: Sending fencing update 73 for FILE-2
>>>> 65890:Jul 17 14:18:20.085 FILE-6 pacemaker-controld [19415]
>>>> (controld_delete_node_state) info: Deleting all state for node FILE-2
>>>> (via CIB call 74) | xpath=//node_state[@uname='FILE-2']/*
>>>> 65892:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
>>>> (exec_alert_list) info: Sending fencing alert via pf-ha-alert to (null)
>>>> 65896:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
>>>> (tengine_stonith_notify) notice: Peer FILE-2 was terminated (reboot) by
>>>> FILE-4 on behalf of pacemaker-controld.19415: OK | initiator=FILE-6
>>>> ref=4e523b34-dcb1-40bc-a296-5e984b4e6b00
>>>> 65897:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
>>>> (send_stonith_update) debug: Sending fencing update 75 for FILE-2
>>>> 65898:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
>>>> (controld_delete_node_state) info: Deleting all state for node FILE-2
>>>> (via CIB call 76) | xpath=//node_state[@uname='FILE-2']/*
>>>> 65899:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__execute_graph) debug: Transition 0 (Complete=34, Pending=1,
>>>> Fired=0, Skipped=0, Incomplete=24,
>>>> Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
>>>> 65907:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
>>>> (te_update_diff) debug: Processing (cib_modify) diff: 0.24.73 -> 0.24.74
>>>> (S_FINALIZE_JOIN)
>>>> 65908:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
>>>> (cib_fencing_updated) info: Fencing update 73 for FILE-2: complete
>>>> 65916:Jul 17 14:18:20.093 FILE-6 pacemaker-controld [19415]
>>>> (te_update_diff) debug: Processing (cib_delete) diff: 0.24.74 -> 0.24.75
>>>> (S_FINALIZE_JOIN)
>>>> 65919:Jul 17 14:18:20.093 FILE-6 pacemaker-controld [19415]
>>>> (match_down_event) debug: Shutdown action 63
>>>> (stonith-FILE-2-reboot) found for node 2
>>>> 65920:Jul 17 14:18:20.093 FILE-6 pacemaker-controld [19415]
>>>> (cib_delete_callback) debug: Deletion of all state for node FILE-2
>>>> (via CIB call 74) succeeded
>>>> 65921:Jul 17 14:18:20.093 FILE-6 pacemaker-controld [19415]
>>>> (cib_fencing_updated) info: Fencing update 75 for FILE-2: complete
>>>> 65924:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (cib_delete_callback) debug: Deletion of all state for node FILE-2
>>>> (via CIB call 76) succeeded
>>>> 65927:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (node_left) info: Group crmd event 5: FILE-2 (node 2 pid 15962) left
>>>> for unknown reason
>>>> 65928:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (crm_update_peer_proc) info: node_left: Node FILE-2[2] - corosync-cpg
>>>> is now offline
>>>> 65929:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (peer_update_callback) info: Node FILE-2 is no longer a peer | DC=true
>>>> old=0x4000000 new=0x0000000
>>>> 65930:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (controld_delete_node_state) info: Deleting transient attributes for
>>>> node FILE-2 (via CIB call 77) |
>>>> xpath=//node_state[@uname='FILE-2']/transient_attributes
>>>> 65932:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (match_down_event) debug: Shutdown action 63
>>>> (stonith-FILE-2-reboot) found for node 2
>>>> 65933:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk_cpg_membership) info: Group crmd event 5: FILE-3 (node 3 pid
>>>> 19250) is member
>>>> 65934:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk_cpg_membership) info: Group crmd event 5: FILE-4 (node 4 pid
>>>> 19122) is member
>>>> 65935:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk_cpg_membership) info: Group crmd event 5: FILE-5 (node 5 pid
>>>> 19273) is member
>>>> 65936:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk_cpg_membership) info: Group crmd event 5: FILE-6 (node 6 pid
>>>> 19415) is member
>>>> 65938:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00880000 (new_actions)
>>>> for controller set by s_crmd_fsa:198
>>>> 65939:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_JOIN_RESULT: [ state=S_FINALIZE_JOIN
>>>> cause=C_HA_MESSAGE origin=route_message ]
>>>> 65940:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00800000 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65941:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00080000 (an_action)
>>>> for controller cleared by do_fsa_action:108
>>>> 65942:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (controld_delete_node_state) info: Deleting resource history for node
>>>> FILE-6 (via CIB call 79) | xpath=//node_state[@uname='FILE-6']/lrm
>>>> 65943:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource stonith-sbd after monitor op complete (interval=0)
>>>> 65945:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource FILE_Filesystem after monitor op complete (interval=0)
>>>> 65946:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource Service_pfile after monitor op complete (interval=0)
>>>> 65947:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource Service_Postgresql after monitor op complete (interval=0)
>>>> 65948:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource Service_esm_primary after monitor op complete (interval=0)
>>>> 65949:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource Service_Postgrest after monitor op complete (interval=0)
>>>> 65950:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource IP_Floating after monitor op complete (interval=0)
>>>> 65951:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__create_history_xml) debug: build_active_RAs: Updating
>>>> resource Shared_Cluster_Backup after monitor op complete (interval=0)
>>>> 65952:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (do_dc_join_ack) debug: Updating local node history for join-1 from query
>>>> result (via CIB call 80)
>>>> 65954:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
>>>> (fsa_data->actions) for controller set by s_crmd_fsa:193
>>>> 65955:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (new_actions) for controller set by s_crmd_fsa:198
>>>> 65956:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
>>>> state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
>>>> 65957:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65958:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415] (do_log)
>>>> info: Input I_WAIT_FOR_EVENT received in state S_FINALIZE_JOIN from
>>>> do_te_invoke
>>>> 65959:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415] (do_log)
>>>> debug: do_log <create_request_adv origin="do_cl_join_query" t="crmd"
>>>> version="3.11.0" subt="request" reference="join_announce-crmd-1689603376-2"
>>>> crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd" src="FILE-1"/>
>>>> 65960:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65961:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
>>>> source=do_te_invoke:135 complete=false
>>>> 65962:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (register_fsa_input_adv) debug: Stalling the FSA pending further input:
>>>> source=do_te_invoke cause=C_HA_MESSAGE data=0x55c619871630 queue=0
>>>> 65963:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Exiting the FSA: queue=1, fsa_actions=0x0,
>>>> stalled=true
>>>> 65964:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (fsa_dump_queue) debug: queue[0.80]: input I_WAIT_FOR_EVENT raised by
>>>> do_te_invoke(0x55c6198798d0.1) (cause=C_HA_MESSAGE)
>>>> 65966:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__execute_graph) debug: Transition 0 (Complete=34, Pending=1,
>>>> Fired=0, Skipped=0, Incomplete=24,
>>>> Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
>>>> 65967:Jul 17 14:18:20.097 FILE-6 pacemaker-fenced [19411]
>>>> (process_remote_stonith_exec) debug: Finalizing action 'reboot'
>>>> targeting FILE-1 on behalf of pacemaker-controld.19415 at FILE-6: OK |
>>>> rc=0 id=446afc42
>>>> 65968:Jul 17 14:18:20.097 FILE-6 pacemaker-fenced [19411]
>>>> (remote_op_done) notice: Operation 'reboot' targeting FILE-1 by FILE-5
>>>> for pacemaker-controld.19415 at FILE-6: OK | id=446afc42
>>>> 65970:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (tengine_stonith_callback) notice: Stonith operation
>>>> 4/62:0:0:232e6505-2e98-4a79-b6ce-5f26d9cba645: OK (0)
>>>> 65971:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (tengine_stonith_callback) info: Stonith operation 4 for FILE-1
>>>> passed
>>>> 65972:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__update_peer_expected) info: crmd_peer_down: Node FILE-1[1] -
>>>> expected state is now down (was pending)
>>>> 65973:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (send_stonith_update) debug: Sending fencing update 81 for FILE-1
>>>> 65974:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (controld_delete_node_state) info: Deleting all state for node FILE-1
>>>> (via CIB call 82) | xpath=//node_state[@uname='FILE-1']/*
>>>> 65975:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
>>>> (exec_alert_list) info: Sending fencing alert via pf-ha-alert to (null)
>>>> 65979:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (tengine_stonith_notify) notice: Peer FILE-1 was terminated (reboot) by
>>>> FILE-5 on behalf of pacemaker-controld.19415: OK | initiator=FILE-6
>>>> ref=446afc42-b46e-47af-9fac-0fa87c1c5e57
>>>> 65980:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (send_stonith_update) debug: Sending fencing update 83 for FILE-1
>>>> 65982:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (controld_delete_node_state) info: Deleting all state for node FILE-1
>>>> (via CIB call 84) | xpath=//node_state[@uname='FILE-1']/*
>>>> 65983:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (cib_delete_callback) debug: Deletion of transient attributes for node
>>>> FILE-2 (via CIB call 77) succeeded
>>>> 65984:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__execute_graph) notice: Transition 0 (Complete=35, Pending=0,
>>>> Fired=0, Skipped=3, Incomplete=24,
>>>> Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
>>>> 65985:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (te_graph_trigger) debug: Transition 0 is now complete
>>>> 65986:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (notify_crmd) debug: Processing transition completion in state
>>>> S_FINALIZE_JOIN
>>>> 65987:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (notify_crmd) debug: Transition 0 status: restart - Node join
>>>> 65988:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
>>>> (fsa_data->actions) for controller set by s_crmd_fsa:193
>>>> 65989:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (new_actions) for controller set by s_crmd_fsa:198
>>>> 65990:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
>>>> state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
>>>> 65991:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65992:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415] (do_log)
>>>> info: Input I_WAIT_FOR_EVENT received in state S_FINALIZE_JOIN from
>>>> do_te_invoke
>>>> 65993:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415] (do_log)
>>>> debug: do_log <create_request_adv origin="do_cl_join_query" t="crmd"
>>>> version="3.11.0" subt="request" reference="join_announce-crmd-1689603376-2"
>>>> crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd" src="FILE-1"/>
>>>> 65994:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
>>>> (an_action) for controller cleared by do_fsa_action:108
>>>> 65995:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
>>>> source=do_te_invoke:135 complete=true
>>>> 65996:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (s_crmd_fsa) debug: Processing I_PE_CALC: [ state=S_FINALIZE_JOIN
>>>> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
>>>> 66024:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
>>>> (cib_delete_callback) debug: Deletion of resource history for node
>>>> FILE-6 (via CIB call 79) succeeded
>>>> 66063:Jul 17 14:18:20.105 FILE-6 pacemaker-controld [19415]
>>>> (join_update_complete_callback) debug: join-1 node history update (via
>>>> CIB call 80) complete
>>>> 66064:Jul 17 14:18:20.105 FILE-6 pacemaker-controld [19415]
>>>> (check_join_state) debug: join-1: Complete | state=S_FINALIZE_JOIN
>>>> for=join_update_complete_callback
>>>> 66068:Jul 17 14:18:20.105 FILE-6 pacemaker-controld [19415]
>>>> (pcmk__set_flags_as) debug: FSA action flags 0x800400000000
>>>> (new_actions) for controller set by s_crmd_fsa:198
>>>>
>>>> Thanks
>>>> Priyanka
>>>>
>>>> On Thu, Jul 20, 2023 at 11:53 AM Reid Wahl <nwahl at redhat.com> wrote:
>>>>
>>>>> On Wed, Jul 19, 2023 at 8:33 PM Priyanka Balotra
>>>>> <priyanka.14balotra at gmail.com> wrote:
>>>>> >
>>>>> > Sure,
>>>>> > Here are the logs:
>>>>> >
>>>>> >
>>>>> > 63138:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (post_cache_update) debug: Updated cache after membership event 44.
>>>>> > 63139:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x200000000
>>>>> (A_ELECTION_CHECK) for controller set by post_cache_update:81
>>>>> > 63140:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000002 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63141:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (do_started) info: Delaying start, Config not read (0000000000000040)
>>>>> > 63142:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (register_fsa_input_adv) debug: Stalling the FSA pending further input:
>>>>> source=do_started cause=C_FSA_INTERNAL data=(nil) queue=0
>>>>> > 63143:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00000002
>>>>> (with_actions) for controller set by register_fsa_input_adv:88
>>>>> > 63144:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (s_crmd_fsa) debug: Exiting the FSA: queue=0,
>>>>> fsa_actions=0x200000002, stalled=true
>>>>> > 63145:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (config_query_callback) debug: Call 3 : Parsing CIB options
>>>>> > 63146:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (config_query_callback) debug: Shutdown escalation occurs if DC has not
>>>>> responded to request in 1200000ms
>>>>> > 63147:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (config_query_callback) debug: Re-run scheduler after 900000ms of
>>>>> inactivity
>>>>> > 63148:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pe_unpack_alerts) debug: Alert pf-ha-alert:
>>>>> path=/usr/lib/ocf/resource.d/pacemaker/pf_ha_alert.sh timeout=30000ms
>>>>> tstamp-format='%H:%M:%S.%06N' 0 vars
>>>>> > 63149:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000002 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63150:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (do_started) debug: Init server comms
>>>>> > 63151:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcs_us_publish) info: server name: crmd
>>>>> > 63152:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (do_started) notice: Pacemaker controller successfully started and
>>>>> accepting connections
>>>>> > 63153:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x200000000 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63154:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (do_election_check) debug: Ignoring election check because we are
>>>>> not in an election
>>>>> > 63155:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000100100
>>>>> (new_actions) for controller set by s_crmd_fsa:198
>>>>> > 63156:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (s_crmd_fsa) debug: Processing I_PENDING: [ state=S_STARTING
>>>>> cause=C_FSA_INTERNAL origin=do_started ]
>>>>> > 63157:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
>>>>> (an_action) for controller cleared by do_fsa_action:108
>>>>> > 63158:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (do_log) info: Input I_PENDING received in state S_STARTING from
>>>>> do_started
>>>>> > 63159:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (do_state_transition) notice: State transition S_STARTING -> S_PENDING
>>>>> | input=I_PENDING cause=C_FSA_INTERNAL origin=do_started
>>>>> > 63160:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00000020
>>>>> (A_INTEGRATE_TIMER_STOP) for controller set by do_state_transition:559
>>>>> > 63161:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00000080
>>>>> (A_FINALIZE_TIMER_STOP) for controller set by do_state_transition:565
>>>>> > 63162:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000020 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63163:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000080 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63164:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00100000 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63165:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
>>>>> (do_cl_join_query) debug: Querying for a DC
>>>>> > 63166:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000100 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63167:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
>>>>> (controld_start_timer) debug: Started Election Trigger (inject
>>>>> I_DC_TIMEOUT if pops after 20000ms, source=18)
>>>>> > 63168:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
>>>>> (stonith_api_signon) debug: Attempting fencer connection by
>>>>> pacemaker-controld with mainloop
>>>>> > 63175:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:131085; real_size:135168;
>>>>> rb->word_size:33792
>>>>> > 63176:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:131085; real_size:135168;
>>>>> rb->word_size:33792
>>>>> > 63177:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:131085; real_size:135168;
>>>>> rb->word_size:33792
>>>>> > 63178:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (stonith_command) debug: Processing register 8 from client
>>>>> pacemaker-controld.15962 with call options 0x00000000
>>>>> > 63179:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (stonith_command) debug: Processed register from client
>>>>> pacemaker-controld.15962: OK (rc=0)
>>>>> > 63180:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
>>>>> (stonith_api_signon) debug: Connection to fencer by
>>>>> pacemaker-controld succeeded (registration token:
>>>>> 5552b1b4-f725-46ac-b239-e404cadd8d94)
>>>>> > 63181:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (stonith_command) debug: Processing st_notify 9 from client
>>>>> pacemaker-controld.15962 with call options 0x00000000
>>>>> > 63182:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (handle_request) debug: Enabling st_notify_disconnect callbacks for
>>>>> client pacemaker-controld.15962
>>>>> > 63183:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (stonith_command) debug: Processed st_notify from client
>>>>> pacemaker-controld.15962: OK (rc=0)
>>>>> > 63184:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (stonith_command) debug: Processing st_notify 10 from client
>>>>> pacemaker-controld.15962 with call options 0x00000000
>>>>> > 63185:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (handle_request) debug: Enabling st_notify_fence callbacks for client
>>>>> pacemaker-controld.15962
>>>>> > 63186:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (stonith_command) debug: Processed st_notify from client
>>>>> pacemaker-controld.15962: OK (rc=0)
>>>>> > 63187:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (stonith_command) debug: Processing st_notify 11 from client
>>>>> pacemaker-controld.15962 with call options 0x00000000
>>>>> > 63188:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (handle_request) debug: Enabling st_notify_history_synced callbacks for
>>>>> client pacemaker-controld.15962
>>>>> > 63189:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
>>>>> (stonith_command) debug: Processed st_notify from client
>>>>> pacemaker-controld.15962: OK (rc=0)
>>>>> > 63190:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
>>>>> (te_trigger_stonith_history_sync) info: Fence history will be synchronized
>>>>> cluster-wide within 30 seconds
>>>>> > 63191:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
>>>>> (te_connect_stonith) notice: Fencer successfully connected
>>>>> > 63192:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962]
>>>>> (quorum_notification_cb) info: Quorum retained | membership=48 members=5
>>>>> > 63193:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962]
>>>>> (quorum_notification_cb) debug: Member[0] 2
>>>>> > 63194:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962]
>>>>> (quorum_notification_cb) debug: Member[1] 4
>>>>> > 63195:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63196:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63197:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63198:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63199:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-e4qK7U/qb-request-cmap-header
>>>>> > 63200:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-e4qK7U/qb-response-cmap-header
>>>>> > 63201:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-e4qK7U/qb-event-cmap-header
>>>>> > 63202:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 4
>>>>> > 63203:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 4
>>>>> > 63204:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63205:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63206:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63209:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63210:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-YYxILU/qb-request-cmap-header
>>>>> > 63211:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-YYxILU/qb-response-cmap-header
>>>>> > 63212:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-YYxILU/qb-event-cmap-header
>>>>> > 63213:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 4
>>>>> > 63214:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
>>>>> (quorum_notification_cb) info: Obtaining name for new node 4
>>>>> > 63218:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63222:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63225:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63240:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63241:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-Cy8QVV/qb-request-cmap-header
>>>>> > 63242:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-Cy8QVV/qb-response-cmap-header
>>>>> > 63243:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-Cy8QVV/qb-event-cmap-header
>>>>> > 63244:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 4
>>>>> > 63245:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 4
>>>>> > 63246:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
>>>>> (quorum_notification_cb) debug: Member[2] 3
>>>>> > 63259:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63265:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63267:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63298:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63299:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-34-0DHKhX/qb-request-cmap-header
>>>>> > 63300:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-34-0DHKhX/qb-response-cmap-header
>>>>> > 63301:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-34-0DHKhX/qb-event-cmap-header
>>>>> > 63302:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 3
>>>>> > 63303:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 3
>>>>> > 63307:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63313:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63320:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63351:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63352:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-34-V0bQlV/qb-request-cmap-header
>>>>> > 63353:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-34-V0bQlV/qb-response-cmap-header
>>>>> > 63355:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-34-V0bQlV/qb-event-cmap-header
>>>>> > 63356:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 3
>>>>> > 63357:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
>>>>> (quorum_notification_cb) info: Obtaining name for new node 3
>>>>> > 63365:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63372:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63374:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63415:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63416:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-34-EAFzTX/qb-request-cmap-header
>>>>> > 63417:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-34-EAFzTX/qb-response-cmap-header
>>>>> > 63418:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-34-EAFzTX/qb-event-cmap-header
>>>>> > 63419:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 3
>>>>> > 63420:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 3
>>>>> > 63421:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
>>>>> (quorum_notification_cb) debug: Member[3] 6
>>>>> > 63425:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63426:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63427:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63479:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63480:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-33-q3mFYU/qb-request-cmap-header
>>>>> > 63481:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-33-q3mFYU/qb-response-cmap-header
>>>>> > 63482:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-33-q3mFYU/qb-event-cmap-header
>>>>> > 63483:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 6
>>>>> > 63484:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 6
>>>>> > 63485:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63486:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63487:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63490:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63491:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-EcEbfV/qb-request-cmap-header
>>>>> > 63492:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-EcEbfV/qb-response-cmap-header
>>>>> > 63493:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-EcEbfV/qb-event-cmap-header
>>>>> > 63494:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 6
>>>>> > 63495:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
>>>>> (quorum_notification_cb) info: Obtaining name for new node 6
>>>>> > 63499:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63502:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63505:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63508:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63509:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-fLk4xW/qb-request-cmap-header
>>>>> > 63510:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-fLk4xW/qb-response-cmap-header
>>>>> > 63511:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-fLk4xW/qb-event-cmap-header
>>>>> > 63512:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 6
>>>>> > 63513:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 6
>>>>> > 63514:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
>>>>> (quorum_notification_cb) debug: Member[4] 5
>>>>> > 63517:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63518:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63521:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63528:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63529:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-ushXmW/qb-request-cmap-header
>>>>> > 63530:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-ushXmW/qb-response-cmap-header
>>>>> > 63531:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-ushXmW/qb-event-cmap-header
>>>>> > 63532:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 5
>>>>> > 63533:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 5
>>>>> > 63534:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63535:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63536:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63537:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63538:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-x3qVkW/qb-request-cmap-header
>>>>> > 63539:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-x3qVkW/qb-response-cmap-header
>>>>> > 63540:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-x3qVkW/qb-event-cmap-header
>>>>> > 63541:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 5
>>>>> > 63542:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
>>>>> (quorum_notification_cb) info: Obtaining name for new node 5
>>>>> > 63543:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63544:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63545:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63546:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63547:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-gUNSFU/qb-request-cmap-header
>>>>> > 63548:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-gUNSFU/qb-response-cmap-header
>>>>> > 63549:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-gUNSFU/qb-event-cmap-header
>>>>> > 63550:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 5
>>>>> > 63551:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 5
>>>>> > 63552:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (update_peer_state_iter) notice: Node (null) state is now lost | nodeid=1
>>>>> previous=member source=pcmk__reap_unseen_nodes
>>>>> > 63553:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (post_cache_update) debug: Updated cache after membership event 48.
>>>>> > 63554:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x200000000
>>>>> (A_ELECTION_CHECK) for controller set by post_cache_update:81
>>>>> > 63555:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x200000000 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63556:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (do_election_check) debug: Ignoring election check because we are
>>>>> not in an election
>>>>> > 63557:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk_cpg_membership) info: Group crmd event 0: node 2 pid 15962
>>>>> joined via cpg_join
>>>>> > 63558:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk_cpg_membership) info: Group crmd event 0: FILE-2 (node 2 pid
>>>>> 15962) is member
>>>>> > 63559:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63560:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63561:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63564:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63565:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-5PH1gV/qb-request-cmap-header
>>>>> > 63566:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-5PH1gV/qb-response-cmap-header
>>>>> > 63567:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-5PH1gV/qb-event-cmap-header
>>>>> > 63568:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 3
>>>>> > 63569:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 3
>>>>> > 63570:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk_cpg_membership) info: Group crmd event 0: peer node (node 3 pid
>>>>> 19250) is member
>>>>> > 63571:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[3] -
>>>>> corosync-cpg is now online
>>>>> > 63572:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
>>>>> (peer_update_callback) debug: Sending hello to node 3 so that it learns
>>>>> our node name
>>>>> > 63573:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63574:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63575:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63576:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63577:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-QATDEV/qb-request-cmap-header
>>>>> > 63578:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-QATDEV/qb-response-cmap-header
>>>>> > 63579:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-QATDEV/qb-event-cmap-header
>>>>> > 63580:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 4
>>>>> > 63581:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 4
>>>>> > 63582:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk_cpg_membership) info: Group crmd event 0: peer node (node 4 pid
>>>>> 19122) is member
>>>>> > 63583:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[4] -
>>>>> corosync-cpg is now online
>>>>> > 63584:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
>>>>> (peer_update_callback) debug: Sending hello to node 4 so that it learns
>>>>> our node name
>>>>> > 63585:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63586:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63587:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63588:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63589:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-TVzR1T/qb-request-cmap-header
>>>>> > 63590:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-TVzR1T/qb-response-cmap-header
>>>>> > 63591:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-TVzR1T/qb-event-cmap-header
>>>>> > 63592:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 5
>>>>> > 63593:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 5
>>>>> > 63594:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk_cpg_membership) info: Group crmd event 0: peer node (node 5 pid
>>>>> 19273) is member
>>>>> > 63595:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[5] -
>>>>> corosync-cpg is now online
>>>>> > 63596:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
>>>>> (peer_update_callback) debug: Sending hello to node 5 so that it learns
>>>>> our node name
>>>>> > 63597:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63598:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63599:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
>>>>> rb->word_size:263168
>>>>> > 63600:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
>>>>> > 63601:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-8LRaoV/qb-request-cmap-header
>>>>> > 63602:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-8LRaoV/qb-response-cmap-header
>>>>> > 63603:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (qb_rb_close_helper) debug: Closing ringbuffer:
>>>>> /dev/shm/qb-13142-15962-31-8LRaoV/qb-event-cmap-header
>>>>> > 63604:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__corosync_name) info: Unable to get node name for nodeid 6
>>>>> > 63605:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (get_node_name) notice: Could not obtain a node name for corosync node
>>>>> with id 6
>>>>> > 63606:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk_cpg_membership) info: Group crmd event 0: peer node (node 6 pid
>>>>> 19415) is member
>>>>> > 63607:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[6] -
>>>>> corosync-cpg is now online
>>>>> > 63608:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (peer_update_callback) debug: Sending hello to node 6 so that it learns
>>>>> our node name
>>>>> > 63609:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (get_xpath_object) debug: No match for //st_notify_history_synced
>>>>> in /notify
>>>>> > 63610:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (stonith_api_del_notification) debug: Removing callback for
>>>>> st_notify_history_synced events
>>>>> > 63611:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958]
>>>>> (stonith_command) debug: Processing st_notify 12 from client
>>>>> pacemaker-controld.15962 with call options 0x00000000
>>>>> > 63612:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958]
>>>>> (handle_request) debug: Disabling st_notify_history_synced callbacks for
>>>>> client pacemaker-controld.15962
>>>>> > 63613:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958]
>>>>> (stonith_command) debug: Processed st_notify from client
>>>>> pacemaker-controld.15962: OK (rc=0)
>>>>> > 63614:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (tengine_stonith_history_synced) debug: Fence-history synced - cancel all
>>>>> timers
>>>>> > 63615:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (crm_get_peer) info: Node 4 is now known as FILE-4
>>>>> > 63616:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
>>>>> (update_peer_uname) warning: Node names with capitals are
>>>>> discouraged, consider changing 'FILE-4'
>>>>> > 63617:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
>>>>> (peer_update_callback) info: Cluster node FILE-4 is now member
>>>>> > 63618:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
>>>>> (crm_get_peer) info: Node 3 is now known as FILE-3
>>>>> > 63619:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
>>>>> (update_peer_uname) warning: Node names with capitals are
>>>>> discouraged, consider changing 'FILE-3'
>>>>> > 63620:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
>>>>> (peer_update_callback) info: Cluster node FILE-3 is now member
>>>>> > 63621:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
>>>>> (crm_get_peer) info: Node 5 is now known as FILE-5
>>>>> > 63622:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
>>>>> (update_peer_uname) warning: Node names with capitals are
>>>>> discouraged, consider changing 'FILE-5'
>>>>> > 63623:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
>>>>> (peer_update_callback) info: Cluster node FILE-5 is now member
>>>>> > 63640:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (crm_get_peer) info: Node 6 is now known as FILE-6
>>>>> > 63641:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (update_peer_uname) warning: Node names with capitals are
>>>>> discouraged, consider changing 'FILE-6'
>>>>> > 63642:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (peer_update_callback) info: Cluster node FILE-6 is now member
>>>>> > 63643:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (handle_request) debug: Raising I_JOIN_OFFER: join-1
>>>>> > 63644:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00400200 (new_actions)
>>>>> for controller set by s_crmd_fsa:198
>>>>> > 63645:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (s_crmd_fsa) debug: Processing I_JOIN_OFFER: [ state=S_PENDING
>>>>> cause=C_HA_MESSAGE origin=route_message ]
>>>>> > 63646:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000200 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63647:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00400000 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63648:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (update_dc) info: Set DC to FILE-6 (3.11.0)
>>>>> > 63649:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__update_peer_expected) info: update_dc: Node FILE-6[6] -
>>>>> expected state is now member (was (null))
>>>>> > 63650:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00000200
>>>>> (A_DC_TIMER_STOP) for controller set by do_cl_join_offer_respond:147
>>>>> > 63651:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000200 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63788:Jul 17 14:16:32.884 FILE-2 pacemaker-controld [15962]
>>>>> (do_cib_replaced) debug: Updating the CIB after a replace: DC=false
>>>>> > 63811:Jul 17 14:16:32.892 FILE-2 pacemaker-controld [15962]
>>>>> (join_query_callback) debug: Respond to join offer join-1 from FILE-6
>>>>> > 63819:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__procfs_pid_of) info: Found pacemaker-based active as process
>>>>> 15957
>>>>> > 63820:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
>>>>> (throttle_cib_load) debug: Init 6 + 2 ticks at 1689603415 (100 tps)
>>>>> > 63821:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
>>>>> (throttle_mode) debug: Current load is 0.980000 across 10 core(s)
>>>>> > 63822:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
>>>>> (throttle_send_command) info: New throttle mode: negligible load (was
>>>>> undetermined)
>>>>> > 63823:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
>>>>> (throttle_update) debug: Node FILE-2 has negligible load and supports at
>>>>> most 20 jobs; new job limit 20
>>>>> > 63824:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (handle_request) debug: Raising I_JOIN_RESULT: join-1
>>>>> > 63825:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00800000 (new_actions)
>>>>> for controller set by s_crmd_fsa:198
>>>>> > 63826:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (s_crmd_fsa) debug: Processing I_JOIN_RESULT: [ state=S_PENDING
>>>>> cause=C_HA_MESSAGE origin=route_message ]
>>>>> > 63827:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00800000 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63828:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (do_cl_join_finalize_respond) debug: Confirming join-1: sending local
>>>>> operation history to FILE-6
>>>>> > 63829:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000200
>>>>> (new_actions) for controller set by s_crmd_fsa:198
>>>>> > 63830:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (s_crmd_fsa) debug: Processing I_NOT_DC: [ state=S_PENDING
>>>>> cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
>>>>> > 63831:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
>>>>> (an_action) for controller cleared by do_fsa_action:108
>>>>> > 63832:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (do_log) info: Input I_NOT_DC received in state S_PENDING from
>>>>> do_cl_join_finalize_respond
>>>>> > 63833:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (do_state_transition) notice: State transition S_PENDING -> S_NOT_DC |
>>>>> input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond
>>>>> > 63834:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00000020
>>>>> (A_INTEGRATE_TIMER_STOP) for controller set by do_state_transition:559
>>>>> > 63835:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__set_flags_as) debug: FSA action flags 0x00000080
>>>>> (A_FINALIZE_TIMER_STOP) for controller set by do_state_transition:565
>>>>> > 63836:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000200 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63837:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000020 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63838:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
>>>>> (pcmk__clear_flags_as) debug: FSA action flags 0x00000080 (an_action)
>>>>> for controller cleared by do_fsa_action:108
>>>>> > 63863:Jul 17 14:17:25.073 FILE-2 pacemaker-controld [15962]
>>>>> (throttle_cib_load) debug: cib load: 0.000667 (2 ticks in 30s)
>>>>> > 63864:Jul 17 14:17:25.073 FILE-2 pacemaker-controld [15962]
>>>>> (throttle_mode) debug: Current load is 0.650000 across 10 core(s)
>>>>> > 63865:Jul 17 14:17:55.073 FILE-2 pacemaker-controld [15962]
>>>>> (throttle_cib_load) debug: cib load: 0.000333 (1 ticks in 30s)
>>>>> > 63866:Jul 17 14:17:55.073 FILE-2 pacemaker-controld [15962]
>>>>> (throttle_mode) debug: Current load is 0.850000 across 10 core(s)
>>>>> > 63868:Jul 17 14:18:20.085 FILE-2 pacemaker-fenced [15958]
>>>>> (process_remote_stonith_exec) debug: Finalizing action 'reboot'
>>>>> targeting FILE-2 on behalf of pacemaker-controld.19415 at FILE-6: OK |
>>>>> rc=0 id=4e523b34
>>>>> > 63869:Jul 17 14:18:20.085 FILE-2 pacemaker-fenced [15958]
>>>>> (remote_op_done) notice: Operation 'reboot' targeting FILE-2 by FILE-4
>>>>> for pacemaker-controld.19415 at FILE-6: OK | id=4e523b34
>>>>> > 63872:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
>>>>> (exec_alert_list) info: Sending fencing alert via pf-ha-alert to (null)
>>>>> > 63875:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
>>>>> (tengine_stonith_notify) crit: We were allegedly just fenced by FILE-4
>>>>> for FILE-6!
>>>>> > 63876:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
>>>>> (crm_xml_cleanup) info: Cleaning up memory from libxml2
>>>>> > 63877:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
>>>>> (crm_exit) info: Exiting pacemaker-controld | with status 100
>>>>> > 63900:Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
>>>>> (pcmk_child_exit) warning: Shutting cluster down because
>>>>> pacemaker-controld[15962] had fatal failure
>>>>> > 63902:Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
>>>>> (pcmk_shutdown_worker) debug: pacemaker-controld confirmed stopped
>>>>> > 63956:Jul 17 14:18:20.101 FILE-2 pacemaker-fenced [15958]
>>>>> (process_remote_stonith_exec) debug: Finalizing action 'reboot'
>>>>> targeting FILE-1 on behalf of pacemaker-controld.19415 at FILE-6: OK |
>>>>> rc=0 id=446afc42
>>>>> > 63957:Jul 17 14:18:20.101 FILE-2 pacemaker-fenced [15958]
>>>>> (remote_op_done) notice: Operation 'reboot' targeting FILE-1 by FILE-5
>>>>> for pacemaker-controld.19415 at FILE-6: OK | id=446afc42>
>>>>> > Thanks
>>>>> > Priyanka
>>>>>
>>>>> Hi, node FILE-6 requested that node FILE-2 be fenced by node FILE-4.
>>>>> FILE-2's controller daemon received notification that it was being
>>>>> fenced, and it shut down. You'd want to check the logs on FILE-6 to
>>>>> determine why FILE-2 was fenced.
>>>>>
>>>>> >
>>>>> > On Thu, Jul 20, 2023 at 12:07 AM Ken Gaillot <kgaillot at redhat.com>
>>>>> wrote:
>>>>> >>
>>>>> >> On Wed, 2023-07-19 at 23:49 +0530, Priyanka Balotra wrote:
>>>>> >> > Hi All,
>>>>> >> > I am using SLES 15 SP4. One of the nodes of the cluster is brought
>>>>> >> > down and boot up after sometime. Pacemaker service came up first
>>>>> but
>>>>> >> > later it faced a fatal shutdown. Due to that crm service is down.
>>>>> >> >
>>>>> >> > The logs from /var/log/pacemaker.pacemaker.log are as follows:
>>>>> >> >
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
>>>>> >> > (pcmk_child_exit) warning: Shutting cluster down because
>>>>> >> > pacemaker-controld[15962] had fatal failure
>>>>> >>
>>>>> >> The interesting messages will be before this. The ones with
>>>>> "pacemaker-
>>>>> >> controld" will be the most relevant, at least initially.
>>>>> >>
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
>>>>> >> > (pcmk_shutdown_worker) notice: Shutting down Pacemaker
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
>>>>> >> > (pcmk_shutdown_worker) debug: pacemaker-controld confirmed
>>>>> stopped
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
>>>>> (stop_child)
>>>>> >> > notice: Stopping pacemaker-schedulerd | sent signal 15 to
>>>>> process
>>>>> >> > 15961
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
>>>>> >> > (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15
>>>>> >> > (invoking handler)
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
>>>>> >> > (qb_ipcs_us_withdraw) info: withdrawing server sockets
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
>>>>> >> > (qb_ipcs_unref) debug: qb_ipcs_unref() - destroying
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
>>>>> >> > (crm_xml_cleanup) info: Cleaning up memory from libxml2
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] (crm_exit)
>>>>> >> > info: Exiting pacemaker-schedulerd | with status 0
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
>>>>> >> > (qb_ipcs_event_sendv) debug: new_event_notification
>>>>> (/dev/shm/qb-
>>>>> >> > 15957-15962-12-RDPw6O/qb): Broken pipe (32)
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
>>>>> >> > (cib_notify_send_one) warning: Could not notify client crmd:
>>>>> >> > Broken pipe | id=e29d175e-7e91-4b6a-bffb-fabfdd7a33bf
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
>>>>> >> > (cib_process_request) info: Completed cib_delete operation for
>>>>> >> > section //node_state[@uname='FILE-2']/*: OK (rc=0, origin=FILE-
>>>>> >> > 6/crmd/74, version=0.24.75)
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-fenced [15958]
>>>>> >> > (xml_patch_version_check) debug: Can apply patch 0.24.75 to
>>>>> >> > 0.24.74
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
>>>>> >> > (pcmk_child_exit) info: pacemaker-schedulerd[15961] exited
>>>>> >> > with status 0 (OK)
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
>>>>> >> > (cib_process_request) info: Completed cib_modify operation for
>>>>> >> > section status: OK (rc=0, origin=FILE-6/crmd/75, version=0.24.75)
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
>>>>> >> > (pcmk_shutdown_worker) debug: pacemaker-schedulerd confirmed
>>>>> >> > stopped
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
>>>>> (stop_child)
>>>>> >> > notice: Stopping pacemaker-attrd | sent signal 15 to process
>>>>> 15960
>>>>> >> > Jul 17 14:18:20.093 FILE-2 pacemaker-attrd [15960]
>>>>> >> > (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15
>>>>> >> > (invoking handler)
>>>>> >> >
>>>>> >> > Could you please help me understand the issue here.
>>>>> >> >
>>>>> >> > Regards
>>>>> >> > Priyanka
>>>>> >> > _______________________________________________
>>>>> >> > Manage your subscription:
>>>>> >> > https://lists.clusterlabs.org/mailman/listinfo/users
>>>>> >> >
>>>>> >> > ClusterLabs home: https://www.clusterlabs.org/
>>>>> >> --
>>>>> >> Ken Gaillot <kgaillot at redhat.com>
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> Manage your subscription:
>>>>> >> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>> >>
>>>>> >> ClusterLabs home: https://www.clusterlabs.org/
>>>>> >
>>>>> > _______________________________________________
>>>>> > Manage your subscription:
>>>>> > https://lists.clusterlabs.org/mailman/listinfo/users
>>>>> >
>>>>> > ClusterLabs home: https://www.clusterlabs.org/
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Reid Wahl (He/Him)
>>>>> Senior Software Engineer, Red Hat
>>>>> RHEL High Availability - Pacemaker
>>>>>
>>>>> _______________________________________________
>>>>> Manage your subscription:
>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> ClusterLabs home: https://www.clusterlabs.org/
>>>>>
>>>> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20230724/f074bca3/attachment-0001.htm>
More information about the Users
mailing list