[ClusterLabs] CRIT: Emergency, Shutdown: Master Control process died.
Daniel Krambrock
danielk_lists at z9d.de
Fri Apr 24 08:02:00 UTC 2015
Hi all,
two days ago we had another heartbeat crash, this time on our
backup-node. Error seams to be the same: 'Program terminated with signal
24, CPU time limit exceeded.'
I have attached a backtrace and parts of the ha-debug file. Please let
me know what i can do for further debugging.
Thanks, daniel
-------------- next part --------------
root at s4b:~# gdb /usr/lib/heartbeat/heartbeat /coredump/core.1429658181.heartbeat.3515_0-0
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/lib/heartbeat/heartbeat...(no debugging symbols found)...done.
[New LWP 3515]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `heartbeat: master control pro'.
Program terminated with signal 24, CPU time limit exceeded.
#0 0x00007f8cceec57db in g_private_get () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
(gdb) bt
#0 0x00007f8cceec57db in g_private_get () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#1 0x00007f8cceea3301 in g_slice_alloc () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007f8ccee86513 in g_list_append () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x00007f8ccf56582b in ipc_bufpool_update () from /usr/lib/libplumb.so.2
#4 0x00007f8ccf561744 in ?? () from /usr/lib/libplumb.so.2
#5 0x00007f8ccf56124f in ?? () from /usr/lib/libplumb.so.2
#6 0x00007f8ccf55c808 in G_CH_check_int () from /usr/lib/libplumb.so.2
#7 0x00007f8ccee8913b in g_main_context_check () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#8 0x00007f8ccee895c2 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#9 0x00007f8ccee89a82 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#10 0x000000000040a7f4 in main ()
/var/log/ha-debug:
Apr 22 01:16:21 s4b cib: [3561]: info: ha_msg_dispatch: Lost connection to heartbeat service.
Apr 22 01:16:21 s4b attrd: [3564]: info: ha_msg_dispatch: Lost connection to heartbeat service.
Apr 22 01:16:21 s4b crmd: [3565]: info: ha_msg_dispatch: Lost connection to heartbeat service.
Apr 22 01:16:21 s4b ccm: [3560]: info: log-rotate detected on logfile /var/log/ha-debug
Apr 22 01:16:21 s4b ccm: [3560]: ERROR: Lost connection to heartbeat service. Need to bail out.
Apr 22 01:16:21 s4b stonith-ng: [3563]: info: log-rotate detected on logfile /var/log/ha-debug
Apr 22 01:16:21 s4b stonith-ng: [3563]: info: ha_msg_dispatch: Lost connection to heartbeat service.
Apr 22 01:16:21 s4b cib: [3561]: info: mem_handle_func:IPC broken, ccm is dead before the client!
Apr 22 01:16:21 s4b crmd: [3565]: info: mem_handle_func:IPC broken, ccm is dead before the client!
Apr 22 01:16:21 s4b cib: [3561]: ERROR: cib_ccm_dispatch: CCM connection appears to have failed: rc=-1.
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: ccm_dispatch: CCM connection appears to have failed: rc=-1.
Apr 22 01:16:21 s4b cib: [3561]: ERROR: cib_ccm_dispatch: Exiting to recover from CCM connection failure
Apr 22 01:16:21 s4b crmd: [3565]: debug: s_crmd_fsa: Processing I_ERROR: [ state=S_IDLE cause=C_CCM_CALLBACK origin=ccm_dispatch ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_ERROR
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: do_log: FSA: Input I_ERROR from ccm_dispatch() received in state S_IDLE
Apr 22 01:16:21 s4b crmd: [3565]: notice: do_state_transition: State transition S_IDLE -> S_RECOVERY [ input=I_ERROR cause=C_CCM_CALLBACK o
rigin=ccm_dispatch ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_DC_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_INTEGRATE_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_FINALIZE_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_RECOVER
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: do_recover: Action A_RECOVER (0000000001000000) not supported
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_DC_RELEASE
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_dc_release: Releasing the role of DC
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_ELECTION_START
Apr 22 01:16:21 s4b crmd: [3565]: WARN: do_election_vote: Not voting in election, we're in state S_RECOVERY
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_DC_RELEASED
Apr 22 01:16:21 s4b crmd: [3565]: info: do_dc_release: DC role released
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_PE_STOP
Apr 22 01:16:21 s4b crmd: [3565]: info: stop_subsystem: Sent -TERM to pengine: [15560]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_TE_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: cib_client_del_notify_callback: Removing callback for cib_diff_notify events
Apr 22 01:16:21 s4b crmd: [3565]: info: do_te_control: Transitioner is now inactive
Apr 22 01:16:21 s4b crmd: [3565]: debug: s_crmd_fsa: Processing I_TERMINATE: [ state=S_RECOVERY cause=C_FSA_INTERNAL origin=do_recover ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_ERROR
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: do_log: FSA: Input I_TERMINATE from do_recover() received in state S_RECOVERY
Apr 22 01:16:21 s4b crmd: [3565]: notice: do_state_transition: State transition S_RECOVERY -> S_TERMINATE [ input=I_TERMINATE cause=C_FSA_I
NTERNAL origin=do_recover ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_DC_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_INTEGRATE_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_FINALIZE_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_SHUTDOWN
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Terminating the pengine
Apr 22 01:16:21 s4b crmd: [3565]: info: stop_subsystem: Sent -TERM to pengine: [15560]
Apr 22 01:16:21 s4b pengine: [15560]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Waiting for subsystems to exit
Apr 22 01:16:21 s4b crmd: [3565]: debug: register_fsa_input_adv: Stalling the FSA pending further input: cause=C_FSA_INTERNAL
Apr 22 01:16:21 s4b crmd: [3565]: WARN: register_fsa_input_adv: do_shutdown stalled the FSA with pending inputs
Apr 22 01:16:21 s4b crmd: [3565]: debug: fsa_dump_queue: queue[0(4074)]: input I_PENDING raised by do_election_vote() (cause=C_FSA_INTERN
AL)
Apr 22 01:16:21 s4b crmd: [3565]: debug: fsa_dump_queue: queue[1(4075)]: input I_RELEASE_SUCCESS raised by do_dc_release() (cause=C_FS
A_INTERNAL)
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: All subsystems stopped, continuing
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Disconnecting STONITH...
Apr 22 01:16:21 s4b crmd: [3565]: debug: stonith_api_signoff: Signing out of the STONITH Service
Apr 22 01:16:21 s4b stonith-ng: [3563]: debug: xmlfromIPC: Peer disconnected
Apr 22 01:16:21 s4b stonith-ng: [3563]: info: ha_msg_dispatch: Lost connection to heartbeat service.
Apr 22 01:16:21 s4b cib: [3561]: info: mem_handle_func:IPC broken, ccm is dead before the client!
Apr 22 01:16:21 s4b crmd: [3565]: info: mem_handle_func:IPC broken, ccm is dead before the client!
Apr 22 01:16:21 s4b cib: [3561]: ERROR: cib_ccm_dispatch: CCM connection appears to have failed: rc=-1.
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: ccm_dispatch: CCM connection appears to have failed: rc=-1.
Apr 22 01:16:21 s4b cib: [3561]: ERROR: cib_ccm_dispatch: Exiting to recover from CCM connection failure
Apr 22 01:16:21 s4b crmd: [3565]: debug: s_crmd_fsa: Processing I_ERROR: [ state=S_IDLE cause=C_CCM_CALLBACK origin=ccm_dispatch ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_ERROR
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: do_log: FSA: Input I_ERROR from ccm_dispatch() received in state S_IDLE
Apr 22 01:16:21 s4b crmd: [3565]: notice: do_state_transition: State transition S_IDLE -> S_RECOVERY [ input=I_ERROR cause=C_CCM_CALLBACK o
rigin=ccm_dispatch ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_DC_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_INTEGRATE_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_FINALIZE_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_RECOVER
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: do_recover: Action A_RECOVER (0000000001000000) not supported
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_DC_RELEASE
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_dc_release: Releasing the role of DC
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_ELECTION_START
Apr 22 01:16:21 s4b crmd: [3565]: WARN: do_election_vote: Not voting in election, we're in state S_RECOVERY
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_DC_RELEASED
Apr 22 01:16:21 s4b crmd: [3565]: info: do_dc_release: DC role released
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_PE_STOP
Apr 22 01:16:21 s4b crmd: [3565]: info: stop_subsystem: Sent -TERM to pengine: [15560]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_TE_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: cib_client_del_notify_callback: Removing callback for cib_diff_notify events
Apr 22 01:16:21 s4b crmd: [3565]: info: do_te_control: Transitioner is now inactive
Apr 22 01:16:21 s4b crmd: [3565]: debug: s_crmd_fsa: Processing I_TERMINATE: [ state=S_RECOVERY cause=C_FSA_INTERNAL origin=do_recover ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_ERROR
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: do_log: FSA: Input I_TERMINATE from do_recover() received in state S_RECOVERY
Apr 22 01:16:21 s4b crmd: [3565]: notice: do_state_transition: State transition S_RECOVERY -> S_TERMINATE [ input=I_TERMINATE cause=C_FSA_I
NTERNAL origin=do_recover ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_DC_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_INTEGRATE_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_FINALIZE_TIMER_STOP
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_SHUTDOWN
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Terminating the pengine
Apr 22 01:16:21 s4b crmd: [3565]: info: stop_subsystem: Sent -TERM to pengine: [15560]
Apr 22 01:16:21 s4b pengine: [15560]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Waiting for subsystems to exit
Apr 22 01:16:21 s4b crmd: [3565]: debug: register_fsa_input_adv: Stalling the FSA pending further input: cause=C_FSA_INTERNAL
Apr 22 01:16:21 s4b crmd: [3565]: WARN: register_fsa_input_adv: do_shutdown stalled the FSA with pending inputs
Apr 22 01:16:21 s4b crmd: [3565]: debug: fsa_dump_queue: queue[0(4074)]: input I_PENDING raised by do_election_vote() (cause=C_FSA_INTERN
AL)
Apr 22 01:16:21 s4b crmd: [3565]: debug: fsa_dump_queue: queue[1(4075)]: input I_RELEASE_SUCCESS raised by do_dc_release() (cause=C_FS
A_INTERNAL)
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: All subsystems stopped, continuing
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Disconnecting STONITH...
Apr 22 01:16:21 s4b crmd: [3565]: debug: stonith_api_signoff: Signing out of the STONITH Service
Apr 22 01:16:21 s4b stonith-ng: [3563]: debug: xmlfromIPC: Peer disconnected
Apr 22 01:16:21 s4b crmd: [3565]: info: tengine_stonith_connection_destroy: Fencing daemon disconnected
Apr 22 01:16:21 s4b stonith-ng: [3563]: info: cib_native_msgready: Lost connection to the CIB service [3561].
Apr 22 01:16:21 s4b attrd: [3564]: debug: xmlfromIPC: Peer disconnected
Apr 22 01:16:21 s4b stonith-ng: [3563]: CRIT: cib_native_dispatch: Lost connection to the CIB service [3561/callback].
Apr 22 01:16:21 s4b attrd: [3564]: info: cib_native_msgready: Lost connection to the CIB service [3561].
Apr 22 01:16:21 s4b crmd: [3565]: debug: s_crmd_fsa: Exiting the FSA: queue=2, fsa_actions=0x200042070000008, stalled=true
Apr 22 01:16:21 s4b stonith-ng: [3563]: CRIT: cib_native_dispatch: Lost connection to the CIB service [3561/command].
Apr 22 01:16:21 s4b attrd: [3564]: CRIT: cib_native_dispatch: Lost connection to the CIB service [3561/callback].
Apr 22 01:16:21 s4b crmd: [3565]: debug: fsa_dump_queue: queue[0(4074)]: input I_PENDING raised by do_election_vote() (cause=C_FSA_INTERNAL)
Apr 22 01:16:21 s4b attrd: [3564]: CRIT: cib_native_dispatch: Lost connection to the CIB service [3561/command].
Apr 22 01:16:21 s4b crmd: [3565]: debug: fsa_dump_queue: queue[1(4075)]: input I_RELEASE_SUCCESS raised by do_dc_release() (cause=C_FSA_INTERNAL)
Apr 22 01:16:21 s4b attrd: [3564]: ERROR: attrd_cib_connection_destroy: Connection to the CIB terminated...
Apr 22 01:16:21 s4b crmd: [3565]: debug: s_crmd_fsa: Processing I_PENDING: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_election_vote ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_WARN
Apr 22 01:16:21 s4b crmd: [3565]: WARN: do_log: FSA: Input I_PENDING from do_election_vote() received in state S_TERMINATE
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_SHUTDOWN
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Terminating the pengine
Apr 22 01:16:21 s4b crmd: [3565]: info: stop_subsystem: Sent -TERM to pengine: [15560]
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Waiting for subsystems to exit
Apr 22 01:16:21 s4b crmd: [3565]: debug: register_fsa_input_adv: Stalling the FSA pending further input: cause=C_FSA_INTERNAL
Apr 22 01:16:21 s4b crmd: [3565]: WARN: register_fsa_input_adv: do_shutdown stalled the FSA with pending inputs
Apr 22 01:16:21 s4b crmd: [3565]: debug: fsa_dump_queue: queue[0(4075)]: input I_RELEASE_SUCCESS raised by do_dc_release() (cause=C_FSA_INTERNAL)
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: All subsystems stopped, continuing
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Disconnecting STONITH...
Apr 22 01:16:21 s4b crmd: [3565]: debug: stonith_api_signoff: Signing out of the STONITH Service
Apr 22 01:16:21 s4b crmd: [3565]: debug: s_crmd_fsa: Exiting the FSA: queue=1, fsa_actions=0x200042070000008, stalled=true
Apr 22 01:16:21 s4b crmd: [3565]: debug: fsa_dump_queue: queue[0(4075)]: input I_RELEASE_SUCCESS raised by do_dc_release() (cause=C_FSA_INTERNAL)
Apr 22 01:16:21 s4b crmd: [3565]: debug: xmlfromIPC: Peer disconnected
Apr 22 01:16:21 s4b crmd: [3565]: info: cib_native_msgready: Lost connection to the CIB service [3561].
Apr 22 01:16:21 s4b crmd: [3565]: CRIT: cib_native_dispatch: Lost connection to the CIB service [3561/callback].
Apr 22 01:16:21 s4b crmd: [3565]: CRIT: cib_native_dispatch: Lost connection to the CIB service [3561/command].
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: crmd_cib_connection_destroy: Connection to the CIB terminated...
Apr 22 01:16:21 s4b crmd: [3565]: debug: s_crmd_fsa: Processing I_RELEASE_SUCCESS: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_dc_release ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_WARN
Apr 22 01:16:21 s4b crmd: [3565]: WARN: do_log: FSA: Input I_RELEASE_SUCCESS from do_dc_release() received in state S_TERMINATE
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_SHUTDOWN
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Terminating the pengine
Apr 22 01:16:21 s4b crmd: [3565]: info: stop_subsystem: Sent -TERM to pengine: [15560]
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Waiting for subsystems to exit
Apr 22 01:16:21 s4b crmd: [3565]: debug: register_fsa_input_adv: Stalling the FSA pending further input: cause=C_FSA_INTERNAL
Apr 22 01:16:21 s4b crmd: [3565]: WARN: register_fsa_input_adv: do_shutdown stalled the FSA with pending inputs
Apr 22 01:16:21 s4b crmd: [3565]: debug: fsa_dump_queue: queue[0(4078)]: input I_ERROR raised by crmd_cib_connection_destroy() (cause=C_FSA_INTERNAL)
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: All subsystems stopped, continuing
Apr 22 01:16:21 s4b crmd: [3565]: info: do_shutdown: Disconnecting STONITH...
Apr 22 01:16:21 s4b crmd: [3565]: debug: stonith_api_signoff: Signing out of the STONITH Service
Apr 22 01:16:21 s4b crmd: [3565]: debug: s_crmd_fsa: Exiting the FSA: queue=1, fsa_actions=0x200042070000008, stalled=true
Apr 22 01:16:21 s4b crmd: [3565]: debug: fsa_dump_queue: queue[0(4078)]: input I_ERROR raised by crmd_cib_connection_destroy() (cause=C_FSA_INTERNAL)
Apr 22 01:16:21 s4b crmd: [3565]: info: crmdManagedChildDied: Process pengine:[15560] exited (signal=0, exitcode=0)
Apr 22 01:16:21 s4b crmd: [3565]: info: pe_msg_dispatch: Received HUP from pengine:[15560]
Apr 22 01:16:21 s4b crmd: [3565]: info: pe_connection_destroy: Connection to the Policy Engine released
Apr 22 01:16:21 s4b crmd: [3565]: debug: s_crmd_fsa: Processing I_ERROR: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=crmd_cib_connection_destroy ]
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_ERROR
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: do_log: FSA: Input I_ERROR from crmd_cib_connection_destroy() received in state S_TERMINATE
Apr 22 01:16:21 s4b crmd: [3565]: debug: do_fsa_action: actions:trace: // A_EXIT_1
Apr 22 01:16:21 s4b crmd: [3565]: debug: verify_stopped: Checking for active resources before exit
Apr 22 01:16:21 s4b crmd: [3565]: debug: cancel_op: Cancelling op 4321 for drbd0:0 (drbd0:0:4321)
Apr 22 01:16:21 s4b lrmd: [3562]: info: cancel_op: operation monitor[4321] on drbd0:0 for client 3565, its parameters: drbd_resource=[vm1] CRM_meta_role=[Slave] CRM_meta_notify_stop_resource=[ ] drbdconf=[/etc/drbd.conf] CRM_meta_notify_inactive_resource=[drbd0:0 drbd0:1 ] CRM_meta_notify_master_uname=[ ] CRM_meta_timeout=[30000] CRM_meta_name=[monitor] CRM_meta_notify_demote_resource=[ ] CRM_meta_notify_start_resource=[drbd0:0 ] CRM_meta_notify_promote_uname=[ ] crm_feature_set=[3.0.6] CRM_meta_notify=[true] CRM_meta_notify_start_uname=[s4b ] CRM_meta_clo cancelled
Apr 22 01:16:21 s4b lrmd: [3562]: debug: on_msg_cancel_op: operation 4321 cancelled
Apr 22 01:16:21 s4b crmd: [3565]: debug: cancel_op: Op 4321 for drbd0:0 (drbd0:0:4321): cancelled
Apr 22 01:16:21 s4b crmd: [3565]: debug: cancel_op: Cancelling op 9 for pingd_stornet:0 (pingd_stornet:0:9)
Apr 22 01:16:21 s4b lrmd: [3562]: info: cancel_op: operation monitor[9] on pingd_stornet:0 for client 3565, its parameters: options=[-i 2] attempts=[5] CRM_meta_timeout=[30000] multiplier=[100] CRM_meta_name=[monitor] dampen=[22s] name=[pingd_stornet] CRM_meta_clone_node_max=[1] CRM_meta_notify=[false] crm_feature_set=[3.0.6] CRM_meta_clone=[0] host_list=[10.13.0.1] CRM_meta_clone_max=[2] CRM_meta_interval=[11000] CRM_meta_globally_unique=[false] timeout=[1] cancelled
Apr 22 01:16:21 s4b lrmd: [3562]: debug: on_msg_cancel_op: operation 9 cancelled
Apr 22 01:16:21 s4b crmd: [3565]: debug: cancel_op: Op 9 for pingd_stornet:0 (pingd_stornet:0:9): cancelled
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: verify_stopped: Resource pingd_stornet:0 was active at shutdown. You may ignore this error if it is unmanaged.
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: verify_stopped: Resource drbd0:0 was active at shutdown. You may ignore this error if it is unmanaged.
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: do_exit: Performing A_EXIT_1 - forcefully exiting the CRMd
Apr 22 01:16:21 s4b crmd: [3565]: ERROR: do_exit: Could not recover from internal error
Apr 22 01:16:21 s4b crmd: [3565]: debug: free_mem: Number of connected clients: 0
Apr 22 01:16:21 s4b crmd: [3565]: debug: free_mem: Partial destroy: TE
Apr 22 01:16:21 s4b crmd: [3565]: debug: free_mem: Partial destroy: PE
Apr 22 01:16:21 s4b crmd: [3565]: info: crm_xml_cleanup: Cleaning up memory from libxml2
Apr 22 01:16:21 s4b crmd: [3565]: info: do_exit: [crmd] stopped (2)
Apr 22 01:16:21 s4b lrmd: [3562]: debug: on_receive_cmd: the IPC to client [pid:3565] disconnected.
Apr 22 01:16:21 s4b lrmd: [3562]: debug: unregister_client: client crmd [pid:3565] is unregistered
Apr 22 01:16:22 s4b heartbeat: [3550]: info: log-rotate detected on logfile /var/log/ha-debug
Apr 22 01:16:22 s4b heartbeat: [3550]: CRIT: Emergency Shutdown: Master Control process died.
Apr 22 01:16:22 s4b heartbeat: [3550]: CRIT: Killing pid 3515 with SIGTERM
Apr 22 01:16:22 s4b heartbeat: [3550]: CRIT: Killing pid 3551 with SIGTERM
Apr 22 01:16:22 s4b heartbeat: [3550]: CRIT: Killing pid 3552 with SIGTERM
Apr 22 01:16:22 s4b heartbeat: [3550]: CRIT: Killing pid 3553 with SIGTERM
Apr 22 01:16:22 s4b heartbeat: [3550]: CRIT: Killing pid 3554 with SIGTERM
Apr 22 01:16:22 s4b heartbeat: [3550]: CRIT: Killing pid 3555 with SIGTERM
Apr 22 01:16:22 s4b heartbeat: [3550]: CRIT: Killing pid 3556 with SIGTERM
Apr 22 01:16:22 s4b heartbeat: [3550]: CRIT: Emergency Shutdown(MCP dead): Killing ourselves.
Apr 22 01:16:22 s4b heartbeat: [3550]: debug: Process 3550 processing SIGTERM
Apr 22 01:16:22 s4b heartbeat: [3550]: debug: Exiting from pid 3550 [rc=15]
More information about the Users
mailing list