May 15 13:42:00 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) logd is not running May 15 13:42:00 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) 2013/05/15_13:42:00 WARNING: 192.168.5.1 is inactive May 15 13:42:09 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) logd is not running May 15 13:42:09 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) 2013/05/15_13:42:09 WARNING: 192.168.5.1 is inactive May 15 13:42:18 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) logd is not running May 15 13:42:18 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) 2013/05/15_13:42:18 WARNING: 192.168.5.1 is inactive May 15 13:42:27 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) logd is not running May 15 13:42:27 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) 2013/05/15_13:42:27 WARNING: 192.168.5.1 is inactive May 15 13:42:30 node1 attrd: [27348]: notice: attrd_trigger_update: Sending flush op to all hosts for: pingd (0) May 15 13:42:30 node1 attrd: [27348]: notice: attrd_perform_update: Sent update 238: pingd=0 May 15 13:42:30 node1 crmd: [27349]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=1, tag=nvpair, id=status-f5a576b5-003b-447d-8029-19202823bbfa-pingd, name=pingd, value=0, magic=NA, cib=0.75.78) : Transient attribute: update May 15 13:42:30 node1 crmd: [27349]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] May 15 13:42:30 node1 crmd: [27349]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. May 15 13:42:30 node1 crmd: [27349]: info: do_pe_invoke: Query 362: Requesting the current CIB: S_POLICY_ENGINE May 15 13:42:30 node1 crmd: [27349]: info: do_pe_invoke_callback: Invoking the PE: query=362, ref=pe_calc-dc-1368639750-564, seq=8, quorate=1 May 15 13:42:30 node1 pengine: [6643]: notice: unpack_config: On loss of CCM Quorum: Ignore May 15 13:42:30 node1 pengine: [6643]: notice: unpack_rsc_op: Operation p_drbd:1_last_failure_0 found resource p_drbd:1 active on node2 May 15 13:42:30 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (30s) for p_fs on node2 May 15 13:42:30 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (30s) for p_mysql on node2 May 15 13:42:30 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (30s) for p_drbd:0 on node1 May 15 13:42:30 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (10s) for p_drbd:1 on node2 May 15 13:42:30 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (30s) for p_drbd:0 on node1 May 15 13:42:30 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (10s) for p_drbd:1 on node2 May 15 13:42:30 node1 pengine: [6643]: notice: LogActions: Move p_fs#011(Started node1 -> node2) May 15 13:42:30 node1 pengine: [6643]: notice: LogActions: Move p_mysql#011(Started node1 -> node2) May 15 13:42:30 node1 pengine: [6643]: notice: LogActions: Demote p_drbd:0#011(Master -> Slave node1) May 15 13:42:30 node1 pengine: [6643]: notice: LogActions: Promote p_drbd:1#011(Slave -> Master node2) May 15 13:42:30 node1 pengine: [6643]: notice: LogActions: Leave p_ping:0#011(Started node2) May 15 13:42:30 node1 pengine: [6643]: notice: LogActions: Leave p_ping:1#011(Started node1) May 15 13:42:30 node1 crmd: [27349]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 15 13:42:30 node1 crmd: [27349]: info: unpack_graph: Unpacked transition 56: 40 actions in 40 synapses May 15 13:42:30 node1 crmd: [27349]: info: do_te_invoke: Processing graph 56 (ref=pe_calc-dc-1368639750-564) derived from /var/lib/pengine/pe-input-64.bz2 May 15 13:42:30 node1 crmd: [27349]: info: te_pseudo_action: Pseudo action 23 fired and confirmed May 15 13:42:30 node1 crmd: [27349]: info: te_rsc_command: Initiating action 7: cancel p_drbd:0_monitor_10000 on node1 (local) May 15 13:42:30 node1 lrmd: [18185]: WARN: For LSB init script, no additional parameters are needed. May 15 13:42:30 node1 lrmd: [27346]: info: cancel_op: operation monitor[190] on p_drbd:0 for client 27349, its parameters: CRM_meta_clone=[0] drbd_resource=[r1] CRM_meta_master_node_max=[1] CRM_meta_clone_node_max=[1] CRM_meta_clone_max=[2] CRM_meta_notify=[true] CRM_meta_master_max=[1] CRM_meta_globally_unique=[false] crm_feature_set=[3.0.5] CRM_meta_name=[monitor] CRM_meta_role=[Master] CRM_meta_interval=[10000] CRM_meta_timeout=[20000] cancelled May 15 13:42:30 node1 crmd: [27349]: info: send_direct_ack: ACK'ing resource op p_drbd:0_monitor_10000 from 7:56:0:2fc25787-7f11-4085-bf19-f5297397c859: lrm_invoke-lrmd-1368639750-567 May 15 13:42:30 node1 crmd: [27349]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1368639750-567 from node1 May 15 13:42:30 node1 crmd: [27349]: info: match_graph_event: Action p_drbd:0_monitor_10000 (7) confirmed on node1 (rc=0) May 15 13:42:30 node1 crmd: [27349]: info: te_rsc_command: Initiating action 1: cancel p_drbd:1_monitor_30000 on node2 May 15 13:42:30 node1 pengine: [6643]: notice: process_pe_message: Transition 56: PEngine Input stored in: /var/lib/pengine/pe-input-64.bz2 May 15 13:42:30 node1 crmd: [27349]: info: te_pseudo_action: Pseudo action 54 fired and confirmed May 15 13:42:30 node1 crmd: [27349]: info: process_lrm_event: LRM operation p_drbd:0_monitor_10000 (call=190, status=1, cib-update=0, confirmed=true) Cancelled May 15 13:42:30 node1 crmd: [27349]: info: te_rsc_command: Initiating action 81: notify p_drbd:0_pre_notify_demote_0 on node1 (local) May 15 13:42:30 node1 crmd: [27349]: info: do_lrm_rsc_op: Performing key=81:56:0:2fc25787-7f11-4085-bf19-f5297397c859 op=p_drbd:0_notify_0 ) May 15 13:42:30 node1 lrmd: [27346]: info: rsc:p_drbd:0 notify[195] (pid 18186) May 15 13:42:30 node1 crmd: [27349]: info: te_rsc_command: Initiating action 83: notify p_drbd:1_pre_notify_demote_0 on node2 May 15 13:42:30 node1 lrmd: [27346]: info: operation notify[195] on p_drbd:0 for client 27349: pid 18186 exited with return code 0 May 15 13:42:30 node1 crmd: [27349]: info: send_direct_ack: ACK'ing resource op p_drbd:0_notify_0 from 81:56:0:2fc25787-7f11-4085-bf19-f5297397c859: lrm_invoke-lrmd-1368639750-571 May 15 13:42:30 node1 crmd: [27349]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1368639750-571 from node1 May 15 13:42:30 node1 crmd: [27349]: info: match_graph_event: Action p_drbd:0_notify_0 (81) confirmed on node1 (rc=0) May 15 13:42:30 node1 crmd: [27349]: info: process_lrm_event: LRM operation p_drbd:0_notify_0 (call=195, rc=0, cib-update=0, confirmed=true) ok May 15 13:42:30 node1 crmd: [27349]: info: te_rsc_command: Initiating action 15: stop p_mysql_stop_0 on node1 (local) May 15 13:42:30 node1 lrmd: [27346]: info: cancel_op: operation monitor[191] on p_mysql for client 27349, its parameters: CRM_meta_name=[monitor] binary=[/usr/libexec/mysqld] config=[/drbd/r1/mysql/my.cnf] CRM_meta_timeout=[20000] CRM_meta_interval=[30000] datadir=[/drbd/r1/mysql] crm_feature_set=[3.0.5] cancelled May 15 13:42:30 node1 crmd: [27349]: info: do_lrm_rsc_op: Performing key=15:56:0:2fc25787-7f11-4085-bf19-f5297397c859 op=p_mysql_stop_0 ) May 15 13:42:30 node1 lrmd: [27346]: info: rsc:p_mysql stop[196] (pid 18219) May 15 13:42:30 node1 crmd: [27349]: info: process_lrm_event: LRM operation p_mysql_monitor_30000 (call=191, status=1, cib-update=0, confirmed=true) Cancelled May 15 13:42:31 node1 crmd: [27349]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1368639750-43 from node2 May 15 13:42:31 node1 crmd: [27349]: info: match_graph_event: Action p_drbd:1_monitor_30000 (1) confirmed on node2 (rc=0) May 15 13:42:31 node1 crmd: [27349]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1368639750-44 from node2 May 15 13:42:31 node1 crmd: [27349]: info: match_graph_event: Action p_drbd:1_notify_0 (83) confirmed on node2 (rc=0) May 15 13:42:31 node1 crmd: [27349]: info: te_pseudo_action: Pseudo action 55 fired and confirmed May 15 13:42:31 node1 crmd: [27349]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=0, tag=nvpair, id=status-9b6c298d-2e80-4d2e-a50e-5512a524189e-pingd, name=pingd, value=0, magic=NA, cib=0.75.82) : Transient attribute: update May 15 13:42:31 node1 crmd: [27349]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 May 15 13:42:31 node1 crmd: [27349]: info: update_abort_priority: Abort action done superceeded by restart May 15 13:42:34 node1 lrmd: [27346]: info: RA output: (p_mysql:stop:stderr) logd is not running May 15 13:42:34 node1 lrmd: [27346]: info: RA output: (p_mysql:stop:stderr) 2013/05/15_13:42:34 INFO: MySQL is not running May 15 13:42:34 node1 lrmd: [27346]: info: RA output: (p_mysql:stop:stderr) logd is not running May 15 13:42:34 node1 lrmd: [27346]: info: RA output: (p_mysql:stop:stderr) 2013/05/15_13:42:34 INFO: MySQL is not running May 15 13:42:34 node1 lrmd: [27346]: info: RA output: (p_mysql:stop:stderr) logd is not running May 15 13:42:34 node1 lrmd: [27346]: info: RA output: (p_mysql:stop:stderr) 2013/05/15_13:42:34 INFO: MySQL stopped May 15 13:42:34 node1 lrmd: [27346]: info: operation stop[196] on p_mysql for client 27349: pid 18219 exited with return code 0 May 15 13:42:34 node1 crmd: [27349]: info: process_lrm_event: LRM operation p_mysql_stop_0 (call=196, rc=0, cib-update=365, confirmed=true) ok May 15 13:42:34 node1 crmd: [27349]: info: match_graph_event: Action p_mysql_stop_0 (15) confirmed on node1 (rc=0) May 15 13:42:34 node1 crmd: [27349]: info: run_graph: ==================================================== May 15 13:42:34 node1 crmd: [27349]: notice: run_graph: Transition 56 (Complete=9, Pending=0, Fired=0, Skipped=21, Incomplete=10, Source=/var/lib/pengine/pe-input-64.bz2): Stopped May 15 13:42:34 node1 crmd: [27349]: info: te_graph_trigger: Transition 56 is now complete May 15 13:42:34 node1 crmd: [27349]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] May 15 13:42:34 node1 crmd: [27349]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. May 15 13:42:34 node1 crmd: [27349]: info: do_pe_invoke: Query 366: Requesting the current CIB: S_POLICY_ENGINE May 15 13:42:34 node1 crmd: [27349]: info: do_pe_invoke_callback: Invoking the PE: query=366, ref=pe_calc-dc-1368639754-573, seq=8, quorate=1 May 15 13:42:34 node1 pengine: [6643]: notice: unpack_config: On loss of CCM Quorum: Ignore May 15 13:42:34 node1 pengine: [6643]: notice: unpack_rsc_op: Operation p_drbd:1_last_failure_0 found resource p_drbd:1 active on node2 May 15 13:42:34 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (30s) for p_mysql on node1 May 15 13:42:34 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (10s) for p_drbd:0 on node1 May 15 13:42:34 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (30s) for p_drbd:1 on node2 May 15 13:42:34 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (10s) for p_drbd:0 on node1 May 15 13:42:34 node1 pengine: [6643]: notice: RecurringOp: Start recurring monitor (30s) for p_drbd:1 on node2 May 15 13:42:34 node1 pengine: [6643]: notice: LogActions: Leave p_fs#011(Started node1) May 15 13:42:34 node1 pengine: [6643]: notice: LogActions: Start p_mysql#011(node1) May 15 13:42:34 node1 pengine: [6643]: notice: LogActions: Leave p_drbd:0#011(Master node1) May 15 13:42:34 node1 pengine: [6643]: notice: LogActions: Leave p_drbd:1#011(Slave node2) May 15 13:42:34 node1 pengine: [6643]: notice: LogActions: Leave p_ping:0#011(Started node2) May 15 13:42:34 node1 pengine: [6643]: notice: LogActions: Leave p_ping:1#011(Started node1) May 15 13:42:34 node1 crmd: [27349]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 15 13:42:34 node1 crmd: [27349]: WARN: destroy_action: Cancelling timer for action 7 (src=740) May 15 13:42:34 node1 crmd: [27349]: info: unpack_graph: Unpacked transition 57: 8 actions in 8 synapses May 15 13:42:34 node1 crmd: [27349]: info: do_te_invoke: Processing graph 57 (ref=pe_calc-dc-1368639754-573) derived from /var/lib/pengine/pe-input-65.bz2 May 15 13:42:34 node1 crmd: [27349]: info: te_pseudo_action: Pseudo action 14 fired and confirmed May 15 13:42:34 node1 crmd: [27349]: info: te_rsc_command: Initiating action 10: start p_mysql_start_0 on node1 (local) May 15 13:42:34 node1 crmd: [27349]: info: do_lrm_rsc_op: Performing key=10:57:0:2fc25787-7f11-4085-bf19-f5297397c859 op=p_mysql_start_0 ) May 15 13:42:34 node1 lrmd: [27346]: info: rsc:p_mysql start[197] (pid 18296) May 15 13:42:34 node1 crmd: [27349]: info: te_rsc_command: Initiating action 22: monitor p_drbd:0_monitor_10000 on node1 (local) May 15 13:42:34 node1 crmd: [27349]: info: do_lrm_rsc_op: Performing key=22:57:8:2fc25787-7f11-4085-bf19-f5297397c859 op=p_drbd:0_monitor_10000 ) May 15 13:42:34 node1 lrmd: [27346]: info: rsc:p_drbd:0 monitor[198] (pid 18297) May 15 13:42:34 node1 crmd: [27349]: info: te_rsc_command: Initiating action 26: monitor p_drbd:1_monitor_30000 on node2 May 15 13:42:34 node1 pengine: [6643]: notice: process_pe_message: Transition 57: PEngine Input stored in: /var/lib/pengine/pe-input-65.bz2 May 15 13:42:34 node1 lrmd: [27346]: info: RA output: (p_mysql:start:stderr) logd is not running May 15 13:42:34 node1 lrmd: [27346]: info: RA output: (p_mysql:start:stderr) 2013/05/15_13:42:34 INFO: MySQL is not running May 15 13:42:34 node1 lrmd: [27346]: info: operation monitor[198] on p_drbd:0 for client 27349: pid 18297 exited with return code 8 May 15 13:42:34 node1 crmd: [27349]: info: process_lrm_event: LRM operation p_drbd:0_monitor_10000 (call=198, rc=8, cib-update=367, confirmed=false) master May 15 13:42:34 node1 crmd: [27349]: info: match_graph_event: Action p_drbd:0_monitor_10000 (22) confirmed on node1 (rc=0) May 15 13:42:35 node1 crmd: [27349]: info: match_graph_event: Action p_drbd:1_monitor_30000 (26) confirmed on node2 (rc=0) May 15 13:42:36 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) logd is not running May 15 13:42:36 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) 2013/05/15_13:42:36 WARNING: 192.168.5.1 is inactive May 15 13:42:45 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) logd is not running May 15 13:42:45 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) 2013/05/15_13:42:45 WARNING: 192.168.5.1 is inactive May 15 13:42:54 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) logd is not running May 15 13:42:54 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) 2013/05/15_13:42:54 WARNING: 192.168.5.1 is inactive May 15 13:43:03 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) logd is not running May 15 13:43:03 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) 2013/05/15_13:43:03 WARNING: 192.168.5.1 is inactive May 15 13:43:12 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) logd is not running May 15 13:43:12 node1 lrmd: [27346]: info: RA output: (p_ping:1:monitor:stderr) 2013/05/15_13:43:12 WARNING: 192.168.5.1 is inactive