Jun 5 21:58:49 mysqldrupal-n1 crmd: [3151]: info: run_graph: ==================================================== Jun 5 21:58:49 mysqldrupal-n1 crmd: [3151]: notice: run_graph: Transition 100 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-5160.bz2): Complete Jun 5 21:58:49 mysqldrupal-n1 crmd: [3151]: info: te_graph_trigger: Transition 100 is now complete Jun 5 21:58:49 mysqldrupal-n1 crmd: [3151]: info: notify_crmd: Transition 100 status: done - Jun 5 21:58:49 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jun 5 21:58:49 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: Starting PEngine Recheck Timer Jun 5 22:00:38 mysqldrupal-n1 cib: [3147]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jun 5 22:06:12 mysqldrupal-n1 corosync[3139]: [TOTEM ] Process pause detected for 1111 ms, flushing membership messages. Jun 5 22:06:12 mysqldrupal-n1 corosync[3139]: [TOTEM ] Process pause detected for 1112 ms, flushing membership messages. Jun 5 22:06:12 mysqldrupal-n1 corosync[3139]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 8336: memb=2, new=0, lost=0 Jun 5 22:06:12 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: memb: mysqldrupal-n1 1204725952 Jun 5 22:06:12 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: memb: mysqldrupal-n2 1221503168 Jun 5 22:06:12 mysqldrupal-n1 corosync[3139]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 8336: memb=2, new=0, lost=0 Jun 5 22:06:12 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: MEMB: mysqldrupal-n1 1204725952 Jun 5 22:06:12 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: MEMB: mysqldrupal-n2 1221503168 Jun 5 22:06:12 mysqldrupal-n1 corosync[3139]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jun 5 22:06:12 mysqldrupal-n1 corosync[3139]: [MAIN ] Completed service synchronization, ready to provide service. Jun 5 22:10:55 mysqldrupal-n1 corosync[3139]: [TOTEM ] A processor failed, forming new configuration. Jun 5 22:10:55 mysqldrupal-n1 corosync[3139]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 8340: memb=2, new=0, lost=0 Jun 5 22:10:55 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: memb: mysqldrupal-n1 1204725952 Jun 5 22:10:55 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: memb: mysqldrupal-n2 1221503168 Jun 5 22:10:55 mysqldrupal-n1 corosync[3139]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 8340: memb=2, new=0, lost=0 Jun 5 22:10:55 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: MEMB: mysqldrupal-n1 1204725952 Jun 5 22:10:55 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: MEMB: mysqldrupal-n2 1221503168 Jun 5 22:10:55 mysqldrupal-n1 corosync[3139]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jun 5 22:10:55 mysqldrupal-n1 corosync[3139]: [MAIN ] Completed service synchronization, ready to provide service. Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: do_pe_invoke: Query 196: Requesting the current CIB: S_POLICY_ENGINE Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: do_pe_invoke_callback: Invoking the PE: query=196, ref=pe_calc-dc-1275768829-141, seq=8316, quorate=1 Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: notice: unpack_config: On loss of CCM Quorum: Ignore Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: info: determine_online_status: Node mysqldrupal-n2 is online Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: info: determine_online_status: Node mysqldrupal-n1 is online Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: notice: group_print: Resource Group: mysql-grp Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: notice: native_print: MySQL-VIP (ocf::heartbeat:IPaddr2): Started mysqldrupal-n1 Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: notice: native_print: MySQL-DB (ocf::heartbeat:mysql): Started mysqldrupal-n1 Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: notice: clone_print: Clone Set: Fencing Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: notice: short_print: Started: [ mysqldrupal-n2 mysqldrupal-n1 ] Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: notice: LogActions: Leave resource MySQL-VIP (Started mysqldrupal-n1) Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: notice: LogActions: Leave resource MySQL-DB (Started mysqldrupal-n1) Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: notice: LogActions: Leave resource vmware-fencing:0 (Started mysqldrupal-n2) Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: notice: LogActions: Leave resource vmware-fencing:1 (Started mysqldrupal-n1) Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jun 5 22:13:49 mysqldrupal-n1 pengine: [3150]: info: process_pe_message: Transition 101: PEngine Input stored in: /var/lib/pengine/pe-input-5161.bz2 Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: unpack_graph: Unpacked transition 101: 0 actions in 0 synapses Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: do_te_invoke: Processing graph 101 (ref=pe_calc-dc-1275768829-141) derived from /var/lib/pengine/pe-input-5161.bz2 Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: run_graph: ==================================================== Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: notice: run_graph: Transition 101 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-5161.bz2): Complete Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: te_graph_trigger: Transition 101 is now complete Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: notify_crmd: Transition 101 status: done - Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jun 5 22:13:49 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: Starting PEngine Recheck Timer Jun 5 22:15:51 mysqldrupal-n1 corosync[3139]: [TOTEM ] Process pause detected for 1301 ms, flushing membership messages. Jun 5 22:15:51 mysqldrupal-n1 corosync[3139]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 8344: memb=2, new=0, lost=0 Jun 5 22:15:51 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: memb: mysqldrupal-n1 1204725952 Jun 5 22:15:51 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: memb: mysqldrupal-n2 1221503168 Jun 5 22:15:51 mysqldrupal-n1 corosync[3139]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 8344: memb=2, new=0, lost=0 Jun 5 22:15:51 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: MEMB: mysqldrupal-n1 1204725952 Jun 5 22:15:51 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: MEMB: mysqldrupal-n2 1221503168 Jun 5 22:15:51 mysqldrupal-n1 corosync[3139]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jun 5 22:15:51 mysqldrupal-n1 corosync[3139]: [MAIN ] Completed service synchronization, ready to provide service. Jun 5 22:16:18 mysqldrupal-n1 corosync[3139]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 8352: memb=1, new=0, lost=1 Jun 5 22:16:18 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: memb: mysqldrupal-n1 1204725952 Jun 5 22:16:18 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: lost: mysqldrupal-n2 1221503168 Jun 5 22:16:18 mysqldrupal-n1 corosync[3139]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 8352: memb=2, new=1, lost=0 Jun 5 22:16:18 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: NEW: mysqldrupal-n2 1221503168 Jun 5 22:16:18 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: MEMB: mysqldrupal-n1 1204725952 Jun 5 22:16:18 mysqldrupal-n1 corosync[3139]: [pcmk ] info: pcmk_peer_update: MEMB: mysqldrupal-n2 1221503168 Jun 5 22:16:18 mysqldrupal-n1 corosync[3139]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jun 5 22:16:18 mysqldrupal-n1 corosync[3139]: [MAIN ] Completed service synchronization, ready to provide service. Jun 5 22:16:18 mysqldrupal-n1 crmd: [3151]: ERROR: crmd_ha_msg_filter: Another DC detected: mysqldrupal-n2 (op=noop) Jun 5 22:16:18 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: State transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ] Jun 5 22:16:18 mysqldrupal-n1 crmd: [3151]: info: update_dc: Unset DC mysqldrupal-n1 Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_diff: Diff 0.456.1 -> 0.456.2 not applied to 0.455.9: current "epoch" is less than required Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: WARN: cib_server_process_diff: Not requesting full refresh in slave mode. Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_diff: Diff 0.456.2 -> 0.456.3 not applied to 0.455.9: current "epoch" is less than required Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: WARN: cib_server_process_diff: Not requesting full refresh in slave mode. Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_diff: Diff 0.456.3 -> 0.456.4 not applied to 0.455.9: current "epoch" is less than required Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: WARN: cib_server_process_diff: Not requesting full refresh in slave mode. Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_diff: Diff 0.456.4 -> 0.457.1 not applied to 0.455.9: current "epoch" is less than required Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: WARN: cib_server_process_diff: Not requesting full refresh in slave mode. Jun 5 22:16:18 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] Jun 5 22:16:18 mysqldrupal-n1 crmd: [3151]: info: do_dc_takeover: Taking over DC status for this partition Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_readwrite: We are now in R/O mode Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/197, version=0.455.9): ok (rc=0) Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_readwrite: We are now in R/W mode Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/198, version=0.455.9): ok (rc=0) Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/199, version=0.455.9): ok (rc=0) Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/201, version=0.455.9): ok (rc=0) Jun 5 22:16:18 mysqldrupal-n1 cib: [3147]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/203, version=0.455.9): ok (rc=0) Jun 5 22:16:18 mysqldrupal-n1 crmd: [3151]: info: do_dc_join_offer_all: join-4: Waiting on 2 outstanding join acks Jun 5 22:16:18 mysqldrupal-n1 crmd: [3151]: info: ais_dispatch: Membership 8352: quorum retained Jun 5 22:16:18 mysqldrupal-n1 crmd: [3151]: info: crm_ais_dispatch: Setting expected votes to 2 Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/206, version=0.455.9): ok (rc=0) Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: config_query_callback: Checking for expired actions every 900000ms Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: config_query_callback: Sending expected-votes=2 to corosync Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: update_dc: Set DC to mysqldrupal-n1 (3.0.1) Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: check_join_state: do_dc_join_filter_offer: Membership changed since join started: 8316 -> 8352 Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: update_dc: Unset DC mysqldrupal-n1 Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: join_make_offer: Making join offers based on membership 8352 Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: do_dc_join_offer_all: join-5: Waiting on 2 outstanding join acks Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: ais_dispatch: Membership 8352: quorum retained Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: crm_ais_dispatch: Setting expected votes to 2 Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/209, version=0.455.9): ok (rc=0) Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: update_dc: Set DC to mysqldrupal-n1 (3.0.1) Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: do_state_transition: All 2 cluster nodes responded to the join offer. Jun 5 22:16:19 mysqldrupal-n1 crmd: [3151]: info: do_dc_join_finalize: join-5: Syncing the CIB from mysqldrupal-n2 to the rest of the cluster Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: cib_replace_notify: Replaced: 0.455.9 -> 0.457.1 from mysqldrupal-n2 Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 attrd: [3149]: info: do_cib_replaced: Sending full refresh Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 attrd: [3149]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: - Jun 5 22:16:19 mysqldrupal-n1 cib: [3147]: info: log_data_element: cib:diff: -