[ClusterLabs] Pacemaker fatal shutdown
Priyanka Balotra
priyanka.14balotra at gmail.com
Wed Jul 19 23:32:33 EDT 2023
Sure,
Here are the logs:
63138:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(post_cache_update) debug: Updated cache after membership event 44.
63139:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x200000000
(A_ELECTION_CHECK) for controller set by post_cache_update:81
63140:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00000002 (an_action)
for controller cleared by do_fsa_action:108
63141:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (do_started)
info: Delaying start, Config not read (0000000000000040)
63142:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(register_fsa_input_adv) debug: Stalling the FSA pending further input:
source=do_started cause=C_FSA_INTERNAL data=(nil) queue=0
63143:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x00000002
(with_actions) for controller set by register_fsa_input_adv:88
63144:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (s_crmd_fsa)
debug: Exiting the FSA: queue=0, fsa_actions=0x200000002, stalled=true
63145:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(config_query_callback) debug: Call 3 : Parsing CIB options
63146:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(config_query_callback) debug: Shutdown escalation occurs if DC has not
responded to request in 1200000ms
63147:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(config_query_callback) debug: Re-run scheduler after 900000ms of
inactivity
63148:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pe_unpack_alerts) debug: Alert pf-ha-alert:
path=/usr/lib/ocf/resource.d/pacemaker/pf_ha_alert.sh timeout=30000ms
tstamp-format='%H:%M:%S.%06N' 0 vars
63149:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00000002 (an_action)
for controller cleared by do_fsa_action:108
63150:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (do_started)
debug: Init server comms
63151:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(qb_ipcs_us_publish) info: server name: crmd
63152:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (do_started)
notice: Pacemaker controller successfully started and accepting
connections
63153:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x200000000 (an_action)
for controller cleared by do_fsa_action:108
63154:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(do_election_check) debug: Ignoring election check because we are
not in an election
63155:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x1000000000100100
(new_actions) for controller set by s_crmd_fsa:198
63156:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (s_crmd_fsa)
debug: Processing I_PENDING: [ state=S_STARTING cause=C_FSA_INTERNAL
origin=do_started ]
63157:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
(an_action) for controller cleared by do_fsa_action:108
63158:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962] (do_log)
info: Input I_PENDING received in state S_STARTING from do_started
63159:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(do_state_transition) notice: State transition S_STARTING -> S_PENDING
| input=I_PENDING cause=C_FSA_INTERNAL origin=do_started
63160:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x00000020
(A_INTEGRATE_TIMER_STOP) for controller set by do_state_transition:559
63161:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x00000080
(A_FINALIZE_TIMER_STOP) for controller set by do_state_transition:565
63162:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00000020 (an_action)
for controller cleared by do_fsa_action:108
63163:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00000080 (an_action)
for controller cleared by do_fsa_action:108
63164:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00100000 (an_action)
for controller cleared by do_fsa_action:108
63165:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
(do_cl_join_query) debug: Querying for a DC
63166:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00000100 (an_action)
for controller cleared by do_fsa_action:108
63167:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
(controld_start_timer) debug: Started Election Trigger (inject
I_DC_TIMEOUT if pops after 20000ms, source=18)
63168:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
(stonith_api_signon) debug: Attempting fencer connection by
pacemaker-controld with mainloop
63175:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:131085; real_size:135168; rb->word_size:33792
63176:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:131085; real_size:135168; rb->word_size:33792
63177:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:131085; real_size:135168; rb->word_size:33792
63178:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(stonith_command) debug: Processing register 8 from client
pacemaker-controld.15962 with call options 0x00000000
63179:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(stonith_command) debug: Processed register from client
pacemaker-controld.15962: OK (rc=0)
63180:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
(stonith_api_signon) debug: Connection to fencer by
pacemaker-controld succeeded (registration token:
5552b1b4-f725-46ac-b239-e404cadd8d94)
63181:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(stonith_command) debug: Processing st_notify 9 from client
pacemaker-controld.15962 with call options 0x00000000
63182:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(handle_request) debug: Enabling st_notify_disconnect callbacks for
client pacemaker-controld.15962
63183:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(stonith_command) debug: Processed st_notify from client
pacemaker-controld.15962: OK (rc=0)
63184:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(stonith_command) debug: Processing st_notify 10 from client
pacemaker-controld.15962 with call options 0x00000000
63185:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(handle_request) debug: Enabling st_notify_fence callbacks for client
pacemaker-controld.15962
63186:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(stonith_command) debug: Processed st_notify from client
pacemaker-controld.15962: OK (rc=0)
63187:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(stonith_command) debug: Processing st_notify 11 from client
pacemaker-controld.15962 with call options 0x00000000
63188:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(handle_request) debug: Enabling st_notify_history_synced callbacks for
client pacemaker-controld.15962
63189:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
(stonith_command) debug: Processed st_notify from client
pacemaker-controld.15962: OK (rc=0)
63190:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
(te_trigger_stonith_history_sync) info: Fence history will be synchronized
cluster-wide within 30 seconds
63191:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
(te_connect_stonith) notice: Fencer successfully connected
63192:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962]
(quorum_notification_cb) info: Quorum retained | membership=48 members=5
63193:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962]
(quorum_notification_cb) debug: Member[0] 2
63194:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962]
(quorum_notification_cb) debug: Member[1] 4
63195:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63196:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63197:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63198:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63199:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-e4qK7U/qb-request-cmap-header
63200:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-e4qK7U/qb-response-cmap-header
63201:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-e4qK7U/qb-event-cmap-header
63202:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 4
63203:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 4
63204:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63205:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63206:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63209:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63210:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-YYxILU/qb-request-cmap-header
63211:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-YYxILU/qb-response-cmap-header
63212:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-YYxILU/qb-event-cmap-header
63213:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 4
63214:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
(quorum_notification_cb) info: Obtaining name for new node 4
63218:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63222:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63225:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63240:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63241:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-Cy8QVV/qb-request-cmap-header
63242:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-Cy8QVV/qb-response-cmap-header
63243:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-Cy8QVV/qb-event-cmap-header
63244:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 4
63245:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 4
63246:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
(quorum_notification_cb) debug: Member[2] 3
63259:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63265:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63267:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63298:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63299:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-34-0DHKhX/qb-request-cmap-header
63300:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-34-0DHKhX/qb-response-cmap-header
63301:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-34-0DHKhX/qb-event-cmap-header
63302:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 3
63303:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 3
63307:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63313:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63320:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63351:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63352:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-34-V0bQlV/qb-request-cmap-header
63353:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-34-V0bQlV/qb-response-cmap-header
63355:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-34-V0bQlV/qb-event-cmap-header
63356:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 3
63357:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
(quorum_notification_cb) info: Obtaining name for new node 3
63365:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63372:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63374:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63415:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63416:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-34-EAFzTX/qb-request-cmap-header
63417:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-34-EAFzTX/qb-response-cmap-header
63418:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-34-EAFzTX/qb-event-cmap-header
63419:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 3
63420:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 3
63421:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
(quorum_notification_cb) debug: Member[3] 6
63425:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63426:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63427:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63479:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63480:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-33-q3mFYU/qb-request-cmap-header
63481:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-33-q3mFYU/qb-response-cmap-header
63482:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-33-q3mFYU/qb-event-cmap-header
63483:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 6
63484:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 6
63485:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63486:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63487:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63490:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63491:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-EcEbfV/qb-request-cmap-header
63492:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-EcEbfV/qb-response-cmap-header
63493:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-EcEbfV/qb-event-cmap-header
63494:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 6
63495:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
(quorum_notification_cb) info: Obtaining name for new node 6
63499:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63502:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63505:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63508:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63509:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-fLk4xW/qb-request-cmap-header
63510:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-fLk4xW/qb-response-cmap-header
63511:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-fLk4xW/qb-event-cmap-header
63512:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 6
63513:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 6
63514:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
(quorum_notification_cb) debug: Member[4] 5
63517:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63518:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63521:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63528:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63529:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-ushXmW/qb-request-cmap-header
63530:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-ushXmW/qb-response-cmap-header
63531:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-ushXmW/qb-event-cmap-header
63532:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 5
63533:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 5
63534:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63535:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63536:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63537:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63538:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-x3qVkW/qb-request-cmap-header
63539:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-x3qVkW/qb-response-cmap-header
63540:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-x3qVkW/qb-event-cmap-header
63541:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 5
63542:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
(quorum_notification_cb) info: Obtaining name for new node 5
63543:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63544:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63545:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63546:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63547:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-gUNSFU/qb-request-cmap-header
63548:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-gUNSFU/qb-response-cmap-header
63549:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-gUNSFU/qb-event-cmap-header
63550:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 5
63551:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 5
63552:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(update_peer_state_iter) notice: Node (null) state is now lost | nodeid=1
previous=member source=pcmk__reap_unseen_nodes
63553:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(post_cache_update) debug: Updated cache after membership event 48.
63554:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x200000000
(A_ELECTION_CHECK) for controller set by post_cache_update:81
63555:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x200000000 (an_action)
for controller cleared by do_fsa_action:108
63556:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(do_election_check) debug: Ignoring election check because we are
not in an election
63557:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(pcmk_cpg_membership) info: Group crmd event 0: node 2 pid 15962
joined via cpg_join
63558:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
(pcmk_cpg_membership) info: Group crmd event 0: FILE-2 (node 2 pid
15962) is member
63559:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63560:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63561:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63564:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63565:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-5PH1gV/qb-request-cmap-header
63566:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-5PH1gV/qb-response-cmap-header
63567:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-5PH1gV/qb-event-cmap-header
63568:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 3
63569:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 3
63570:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
(pcmk_cpg_membership) info: Group crmd event 0: peer node (node 3 pid
19250) is member
63571:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
(crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[3] -
corosync-cpg is now online
63572:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
(peer_update_callback) debug: Sending hello to node 3 so that it learns
our node name
63573:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63574:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63575:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63576:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63577:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-QATDEV/qb-request-cmap-header
63578:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-QATDEV/qb-response-cmap-header
63579:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-QATDEV/qb-event-cmap-header
63580:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 4
63581:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 4
63582:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
(pcmk_cpg_membership) info: Group crmd event 0: peer node (node 4 pid
19122) is member
63583:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
(crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[4] -
corosync-cpg is now online
63584:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
(peer_update_callback) debug: Sending hello to node 4 so that it learns
our node name
63585:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63586:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63587:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63588:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63589:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-TVzR1T/qb-request-cmap-header
63590:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-TVzR1T/qb-response-cmap-header
63591:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-TVzR1T/qb-event-cmap-header
63592:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 5
63593:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 5
63594:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
(pcmk_cpg_membership) info: Group crmd event 0: peer node (node 5 pid
19273) is member
63595:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
(crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[5] -
corosync-cpg is now online
63596:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
(peer_update_callback) debug: Sending hello to node 5 so that it learns
our node name
63597:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63598:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63599:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] (qb_rb_open_2)
debug: shm size:1048589; real_size:1052672; rb->word_size:263168
63600:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
63601:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-8LRaoV/qb-request-cmap-header
63602:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-8LRaoV/qb-response-cmap-header
63603:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(qb_rb_close_helper) debug: Closing ringbuffer:
/dev/shm/qb-13142-15962-31-8LRaoV/qb-event-cmap-header
63604:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(pcmk__corosync_name) info: Unable to get node name for nodeid 6
63605:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(get_node_name) notice: Could not obtain a node name for corosync node
with id 6
63606:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(pcmk_cpg_membership) info: Group crmd event 0: peer node (node 6 pid
19415) is member
63607:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(crm_update_peer_proc) info: pcmk_cpg_membership: Node (null)[6] -
corosync-cpg is now online
63608:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(peer_update_callback) debug: Sending hello to node 6 so that it learns
our node name
63609:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(get_xpath_object) debug: No match for //st_notify_history_synced
in /notify
63610:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(stonith_api_del_notification) debug: Removing callback for
st_notify_history_synced events
63611:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958]
(stonith_command) debug: Processing st_notify 12 from client
pacemaker-controld.15962 with call options 0x00000000
63612:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958]
(handle_request) debug: Disabling st_notify_history_synced callbacks for
client pacemaker-controld.15962
63613:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958]
(stonith_command) debug: Processed st_notify from client
pacemaker-controld.15962: OK (rc=0)
63614:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(tengine_stonith_history_synced) debug: Fence-history synced - cancel all
timers
63615:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962] (crm_get_peer)
info: Node 4 is now known as FILE-4
63616:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
(update_peer_uname) warning: Node names with capitals are
discouraged, consider changing 'FILE-4'
63617:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
(peer_update_callback) info: Cluster node FILE-4 is now member
63618:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962] (crm_get_peer)
info: Node 3 is now known as FILE-3
63619:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
(update_peer_uname) warning: Node names with capitals are
discouraged, consider changing 'FILE-3'
63620:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
(peer_update_callback) info: Cluster node FILE-3 is now member
63621:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962] (crm_get_peer)
info: Node 5 is now known as FILE-5
63622:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
(update_peer_uname) warning: Node names with capitals are
discouraged, consider changing 'FILE-5'
63623:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
(peer_update_callback) info: Cluster node FILE-5 is now member
63640:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] (crm_get_peer)
info: Node 6 is now known as FILE-6
63641:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
(update_peer_uname) warning: Node names with capitals are
discouraged, consider changing 'FILE-6'
63642:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
(peer_update_callback) info: Cluster node FILE-6 is now member
63643:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
(handle_request) debug: Raising I_JOIN_OFFER: join-1
63644:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x00400200 (new_actions)
for controller set by s_crmd_fsa:198
63645:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] (s_crmd_fsa)
debug: Processing I_JOIN_OFFER: [ state=S_PENDING cause=C_HA_MESSAGE
origin=route_message ]
63646:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00000200 (an_action)
for controller cleared by do_fsa_action:108
63647:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00400000 (an_action)
for controller cleared by do_fsa_action:108
63648:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962] (update_dc)
info: Set DC to FILE-6 (3.11.0)
63649:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
(pcmk__update_peer_expected) info: update_dc: Node FILE-6[6] -
expected state is now member (was (null))
63650:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x00000200
(A_DC_TIMER_STOP) for controller set by do_cl_join_offer_respond:147
63651:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00000200 (an_action)
for controller cleared by do_fsa_action:108
63788:Jul 17 14:16:32.884 FILE-2 pacemaker-controld [15962]
(do_cib_replaced) debug: Updating the CIB after a replace: DC=false
63811:Jul 17 14:16:32.892 FILE-2 pacemaker-controld [15962]
(join_query_callback) debug: Respond to join offer join-1 from FILE-6
63819:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
(pcmk__procfs_pid_of) info: Found pacemaker-based active as process
15957
63820:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
(throttle_cib_load) debug: Init 6 + 2 ticks at 1689603415 (100 tps)
63821:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
(throttle_mode) debug: Current load is 0.980000 across 10 core(s)
63822:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
(throttle_send_command) info: New throttle mode: negligible load (was
undetermined)
63823:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
(throttle_update) debug: Node FILE-2 has negligible load and supports at
most 20 jobs; new job limit 20
63824:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(handle_request) debug: Raising I_JOIN_RESULT: join-1
63825:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x00800000 (new_actions)
for controller set by s_crmd_fsa:198
63826:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] (s_crmd_fsa)
debug: Processing I_JOIN_RESULT: [ state=S_PENDING cause=C_HA_MESSAGE
origin=route_message ]
63827:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00800000 (an_action)
for controller cleared by do_fsa_action:108
63828:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(do_cl_join_finalize_respond) debug: Confirming join-1: sending local
operation history to FILE-6
63829:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x1000000000000200
(new_actions) for controller set by s_crmd_fsa:198
63830:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] (s_crmd_fsa)
debug: Processing I_NOT_DC: [ state=S_PENDING cause=C_HA_MESSAGE
origin=do_cl_join_finalize_respond ]
63831:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000000
(an_action) for controller cleared by do_fsa_action:108
63832:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962] (do_log)
info: Input I_NOT_DC received in state S_PENDING from
do_cl_join_finalize_respond
63833:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(do_state_transition) notice: State transition S_PENDING -> S_NOT_DC |
input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond
63834:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x00000020
(A_INTEGRATE_TIMER_STOP) for controller set by do_state_transition:559
63835:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(pcmk__set_flags_as) debug: FSA action flags 0x00000080
(A_FINALIZE_TIMER_STOP) for controller set by do_state_transition:565
63836:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00000200 (an_action)
for controller cleared by do_fsa_action:108
63837:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00000020 (an_action)
for controller cleared by do_fsa_action:108
63838:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
(pcmk__clear_flags_as) debug: FSA action flags 0x00000080 (an_action)
for controller cleared by do_fsa_action:108
63863:Jul 17 14:17:25.073 FILE-2 pacemaker-controld [15962]
(throttle_cib_load) debug: cib load: 0.000667 (2 ticks in 30s)
63864:Jul 17 14:17:25.073 FILE-2 pacemaker-controld [15962]
(throttle_mode) debug: Current load is 0.650000 across 10 core(s)
63865:Jul 17 14:17:55.073 FILE-2 pacemaker-controld [15962]
(throttle_cib_load) debug: cib load: 0.000333 (1 ticks in 30s)
63866:Jul 17 14:17:55.073 FILE-2 pacemaker-controld [15962]
(throttle_mode) debug: Current load is 0.850000 across 10 core(s)
63868:Jul 17 14:18:20.085 FILE-2 pacemaker-fenced [15958]
(process_remote_stonith_exec) debug: Finalizing action 'reboot'
targeting FILE-2 on behalf of pacemaker-controld.19415 at FILE-6: OK | rc=0
id=4e523b34
63869:Jul 17 14:18:20.085 FILE-2 pacemaker-fenced [15958]
(remote_op_done) notice: Operation 'reboot' targeting FILE-2 by FILE-4
for pacemaker-controld.19415 at FILE-6: OK | id=4e523b34
63872:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
(exec_alert_list) info: Sending fencing alert via pf-ha-alert to (null)
63875:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
(tengine_stonith_notify) crit: We were allegedly just fenced by FILE-4
for FILE-6!
63876:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
(crm_xml_cleanup) info: Cleaning up memory from libxml2
63877:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962] (crm_exit)
info: Exiting pacemaker-controld | with status 100
63900:Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
(pcmk_child_exit) warning: Shutting cluster down because
pacemaker-controld[15962] had fatal failure
63902:Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
(pcmk_shutdown_worker) debug: pacemaker-controld confirmed stopped
63956:Jul 17 14:18:20.101 FILE-2 pacemaker-fenced [15958]
(process_remote_stonith_exec) debug: Finalizing action 'reboot'
targeting FILE-1 on behalf of pacemaker-controld.19415 at FILE-6: OK | rc=0
id=446afc42
63957:Jul 17 14:18:20.101 FILE-2 pacemaker-fenced [15958]
(remote_op_done) notice: Operation 'reboot' targeting FILE-1 by FILE-5
for pacemaker-controld.19415 at FILE-6: OK | id=446afc42
Thanks
Priyanka
On Thu, Jul 20, 2023 at 12:07 AM Ken Gaillot <kgaillot at redhat.com> wrote:
> On Wed, 2023-07-19 at 23:49 +0530, Priyanka Balotra wrote:
> > Hi All,
> > I am using SLES 15 SP4. One of the nodes of the cluster is brought
> > down and boot up after sometime. Pacemaker service came up first but
> > later it faced a fatal shutdown. Due to that crm service is down.
> >
> > The logs from /var/log/pacemaker.pacemaker.log are as follows:
> >
> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > (pcmk_child_exit) warning: Shutting cluster down because
> > pacemaker-controld[15962] had fatal failure
>
> The interesting messages will be before this. The ones with "pacemaker-
> controld" will be the most relevant, at least initially.
>
> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > (pcmk_shutdown_worker) notice: Shutting down Pacemaker
> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > (pcmk_shutdown_worker) debug: pacemaker-controld confirmed stopped
> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] (stop_child)
> > notice: Stopping pacemaker-schedulerd | sent signal 15 to process
> > 15961
> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> > (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15
> > (invoking handler)
> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> > (qb_ipcs_us_withdraw) info: withdrawing server sockets
> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> > (qb_ipcs_unref) debug: qb_ipcs_unref() - destroying
> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> > (crm_xml_cleanup) info: Cleaning up memory from libxml2
> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961] (crm_exit)
> > info: Exiting pacemaker-schedulerd | with status 0
> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
> > (qb_ipcs_event_sendv) debug: new_event_notification (/dev/shm/qb-
> > 15957-15962-12-RDPw6O/qb): Broken pipe (32)
> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
> > (cib_notify_send_one) warning: Could not notify client crmd:
> > Broken pipe | id=e29d175e-7e91-4b6a-bffb-fabfdd7a33bf
> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
> > (cib_process_request) info: Completed cib_delete operation for
> > section //node_state[@uname='FILE-2']/*: OK (rc=0, origin=FILE-
> > 6/crmd/74, version=0.24.75)
> > Jul 17 14:18:20.093 FILE-2 pacemaker-fenced [15958]
> > (xml_patch_version_check) debug: Can apply patch 0.24.75 to
> > 0.24.74
> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > (pcmk_child_exit) info: pacemaker-schedulerd[15961] exited
> > with status 0 (OK)
> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
> > (cib_process_request) info: Completed cib_modify operation for
> > section status: OK (rc=0, origin=FILE-6/crmd/75, version=0.24.75)
> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > (pcmk_shutdown_worker) debug: pacemaker-schedulerd confirmed
> > stopped
> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956] (stop_child)
> > notice: Stopping pacemaker-attrd | sent signal 15 to process 15960
> > Jul 17 14:18:20.093 FILE-2 pacemaker-attrd [15960]
> > (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15
> > (invoking handler)
> >
> > Could you please help me understand the issue here.
> >
> > Regards
> > Priyanka
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users
> >
> > ClusterLabs home: https://www.clusterlabs.org/
> --
> Ken Gaillot <kgaillot at redhat.com>
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20230720/410aa91b/attachment-0001.htm>
More information about the Users
mailing list