May 20 12:39:35 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:39:35 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:35 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:39:35 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:39:35 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:35 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:39:35 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:39:35 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:39:35 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:39:35 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:39:35 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:39:35 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:39:35 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:39:35 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:39:35 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:39:35 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:39:35 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:39:35 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:39:35 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_target_monitor_0 failed with rc=5: Preventing iscsi0_target from re-starting on ebony May 20 12:39:35 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun0_monitor_0 failed with rc=5: Preventing iscsi0_lun0 from re-starting on ebony May 20 12:39:35 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun1_monitor_0 failed with rc=5: Preventing iscsi0_lun1 from re-starting on ebony May 20 12:39:35 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun2_monitor_0 failed with rc=5: Preventing iscsi0_lun2 from re-starting on ebony May 20 12:39:35 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:39:35 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:39:35 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:39:35 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:39:35 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:39:35 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:39:35 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:39:35 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:39:35 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:39:35 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:39:35 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:39:35 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:39:35 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:39:35 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:39:35 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:39:35 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:39:35 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:39:35 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:39:35 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:39:35 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:0 ping_nodes:1 ping_nodes:2 ] May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:39:35 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:39:35 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:39:35 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:39:35 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:39:35 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:39:35 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:39:35 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:39:35 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:39:35 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:39:35 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:39:35 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:39:35 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:39:35 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:39:35 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:39:35 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:39:35 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:39:35 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:39:35 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:39:35 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:39:35 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:39:35 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:39:35 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:39:35 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource ping_nodes:0 cannot run anywhere May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource ping_nodes:1 cannot run anywhere May 20 12:39:35 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:39:35 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:0#011(Stopped) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:1#011(Stopped) May 20 12:39:35 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:39:35 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:39:35 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3596: 0 actions in 0 synapses May 20 12:39:35 finley crmd: [25840]: info: do_te_invoke: Processing graph 3596 (ref=pe_calc-dc-1305887975-6249) derived from /var/lib/pengine/pe-input-4755.bz2 May 20 12:39:35 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:39:35 finley crmd: [25840]: notice: run_graph: Transition 3596 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-4755.bz2): Complete May 20 12:39:35 finley crmd: [25840]: info: te_graph_trigger: Transition 3596 is now complete May 20 12:39:35 finley crmd: [25840]: info: notify_crmd: Transition 3596 status: done - May 20 12:39:35 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:39:35 finley crmd: [25840]: info: do_state_transition: Starting PEngine Recheck Timer May 20 12:39:35 finley pengine: [25839]: info: process_pe_message: Transition 3596: PEngine Input stored in: /var/lib/pengine/pe-input-4755.bz2 May 20 12:39:37 finley attrd: [25838]: info: attrd_local_callback: Sending full refresh (origin=crmd) May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_target () May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r1:0 (10000) May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r1:1 () May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r2:0 (INFINITY) May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r2:1 () May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_lun0 (1304204891) May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_lun1 (1304204345) May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_lun2 (1304204850) May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun0 () May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun1 () May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun2 () May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun3 () May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r0:0 (10000) May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r0:1 () May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-ip_virtual0 () May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_target (1304204891) May 20 12:39:37 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-ip_virtual0 (1304204891) May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r3:0 () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r1:1 () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r1:0 (INFINITY) May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r2:0 (1305887962) May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r2:1 () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r3:0 () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r0:0 (INFINITY) May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r3:1 () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r0:1 () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: pingd () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r1:0 (1305887962) May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r1:1 () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r2:0 (10000) May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r2:1 () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r3:0 () May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r0:0 (1305887962) May 20 12:39:38 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r0:1 () May 20 12:39:53 finley cibadmin: [23630]: info: Invoked: cibadmin -Ql May 20 12:39:53 finley cibadmin: [23632]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:53 finley cibadmin: [23634]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:53 finley crm_resource: [23636]: info: Invoked: crm_resource -W -r ip_virtual0 May 20 12:39:53 finley crm_resource: [23636]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:53 finley crm_resource: [23636]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:53 finley crm_resource: [23636]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:53 finley crm_resource: [23636]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:53 finley crm_resource: [23636]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:53 finley crm_resource: [23636]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:53 finley crm_resource: [23636]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:53 finley crm_resource: [23636]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:53 finley crm_resource: [23636]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley cibadmin: [23638]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:54 finley cibadmin: [23640]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:54 finley crm_resource: [23642]: info: Invoked: crm_resource -W -r iscsi0_target May 20 12:39:54 finley crm_resource: [23642]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23642]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23642]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23642]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23642]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23642]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23642]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23642]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23642]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley cibadmin: [23644]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:54 finley cibadmin: [23646]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:54 finley crm_resource: [23648]: info: Invoked: crm_resource -W -r iscsi0_lun0 May 20 12:39:54 finley crm_resource: [23648]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23648]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23648]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23648]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23648]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23648]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23648]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23648]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23648]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley cibadmin: [23650]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:54 finley cibadmin: [23652]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:54 finley crm_resource: [23654]: info: Invoked: crm_resource -W -r iscsi0_lun1 May 20 12:39:54 finley crm_resource: [23654]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23654]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23654]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23654]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23654]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23654]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23654]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23654]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23654]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley cibadmin: [23656]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:54 finley cibadmin: [23658]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:54 finley crm_resource: [23660]: info: Invoked: crm_resource -W -r iscsi0_lun2 May 20 12:39:54 finley crm_resource: [23660]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23660]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23660]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23660]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23660]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23660]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23660]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23660]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley crm_resource: [23660]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:54 finley cibadmin: [23662]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:54 finley cibadmin: [23664]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:54 finley crm_resource: [23666]: info: Invoked: crm_resource -W -r ms_drbd_r0 May 20 12:39:55 finley crm_resource: [23666]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23666]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23666]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23666]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23666]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23666]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23666]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23666]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23666]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley cibadmin: [23668]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:55 finley cibadmin: [23670]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:55 finley crm_resource: [23672]: info: Invoked: crm_resource -W -r ms_drbd_r1 May 20 12:39:55 finley crm_resource: [23672]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23672]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23672]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23672]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23672]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23672]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23672]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23672]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23672]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley cibadmin: [23674]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:55 finley cibadmin: [23676]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:55 finley crm_resource: [23678]: info: Invoked: crm_resource -W -r ms_drbd_r2 May 20 12:39:55 finley crm_resource: [23678]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23678]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23678]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23678]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23678]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23678]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23678]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23678]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23678]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley cibadmin: [23680]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:55 finley cibadmin: [23682]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:55 finley crm_resource: [23684]: info: Invoked: crm_resource -W -r ping_clone May 20 12:39:55 finley crm_resource: [23684]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23684]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23684]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23684]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23684]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23684]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23684]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23684]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23684]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley cibadmin: [23686]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:55 finley cibadmin: [23688]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:55 finley cibadmin: [23690]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:55 finley cibadmin: [23692]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:55 finley crm_resource: [23694]: info: Invoked: crm_resource -W -r ip_virtual0 May 20 12:39:55 finley crm_resource: [23694]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23694]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23694]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23694]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23694]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23694]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23694]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23694]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:55 finley crm_resource: [23694]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley cibadmin: [23696]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:56 finley cibadmin: [23698]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:56 finley crm_resource: [23700]: info: Invoked: crm_resource -W -r iscsi0_target May 20 12:39:56 finley crm_resource: [23700]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23700]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23700]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23700]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23700]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23700]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23700]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23700]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23700]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley cibadmin: [23702]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:56 finley cibadmin: [23704]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:56 finley crm_resource: [23706]: info: Invoked: crm_resource -W -r iscsi0_lun0 May 20 12:39:56 finley crm_resource: [23706]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23706]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23706]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23706]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23706]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23706]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23706]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23706]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23706]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley cibadmin: [23708]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:56 finley cibadmin: [23710]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:56 finley crm_resource: [23712]: info: Invoked: crm_resource -W -r iscsi0_lun1 May 20 12:39:56 finley crm_resource: [23712]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23712]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23712]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23712]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23712]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23712]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23712]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23712]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23712]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley cibadmin: [23714]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:56 finley cibadmin: [23716]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:56 finley crm_resource: [23718]: info: Invoked: crm_resource -W -r iscsi0_lun2 May 20 12:39:56 finley crm_resource: [23718]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23718]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23718]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23718]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23718]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23718]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23718]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23718]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley crm_resource: [23718]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:56 finley cibadmin: [23720]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:56 finley cibadmin: [23722]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:57 finley crm_resource: [23724]: info: Invoked: crm_resource -W -r ms_drbd_r0 May 20 12:39:57 finley crm_resource: [23724]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23724]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23724]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23724]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23724]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23724]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23724]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23724]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23724]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley cibadmin: [23726]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:57 finley cibadmin: [23728]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:57 finley crm_resource: [23731]: info: Invoked: crm_resource -W -r ms_drbd_r1 May 20 12:39:57 finley crm_resource: [23731]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23731]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23731]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23731]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23731]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23731]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23731]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23731]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23731]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley cibadmin: [23733]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:57 finley cibadmin: [23735]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:57 finley crm_resource: [23737]: info: Invoked: crm_resource -W -r ms_drbd_r2 May 20 12:39:57 finley crm_resource: [23737]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23737]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23737]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23737]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23737]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23737]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23737]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23737]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23737]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley cibadmin: [23739]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:57 finley cibadmin: [23741]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:57 finley crm_resource: [23743]: info: Invoked: crm_resource -W -r ping_clone May 20 12:39:57 finley crm_resource: [23743]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23743]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23743]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23743]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23743]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23743]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23743]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23743]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley crm_resource: [23743]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:39:57 finley cibadmin: [23745]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:57 finley cibadmin: [23747]: info: Invoked: cibadmin -Ql -o resources May 20 12:39:58 finley cibadmin: [23749]: info: Invoked: cibadmin -Ql May 20 12:39:59 finley cibadmin: [23832]: info: Invoked: cibadmin -p -R May 20 12:39:59 finley cib: [25836]: info: cib_replace_notify: Replaced: 0.2132.7 -> 0.2133.1 from May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley attrd: [25838]: info: do_cib_replaced: Sending full refresh May 20 12:39:59 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_target () May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:39:59 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_replace for section 'all' (origin=local/cibadmin/2, version=0.2133.1): ok (rc=0) May 20 12:39:59 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r1:0 (10000) May 20 12:39:59 finley crmd: [25840]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change May 20 12:39:59 finley crmd: [25840]: info: need_abort: Aborting on change to admin_epoch May 20 12:39:59 finley crmd: [25840]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] May 20 12:39:59 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:39:59 finley crmd: [25840]: info: do_pe_invoke: Query 5732: Requesting the current CIB: S_POLICY_ENGINE May 20 12:39:59 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ] May 20 12:39:59 finley crmd: [25840]: info: update_dc: Unset DC finley May 20 12:39:59 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r1:1 () May 20 12:39:59 finley cibadmin: [23834]: info: Invoked: cibadmin -Ql May 20 12:39:59 finley cib: [23835]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-46.raw May 20 12:39:59 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/5730, version=0.2133.1): ok (rc=0) May 20 12:39:59 finley crmd: [25840]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] May 20 12:39:59 finley crmd: [25840]: info: do_dc_takeover: Taking over DC status for this partition May 20 12:39:59 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r2:0 (INFINITY) May 20 12:39:59 finley cib: [25836]: info: cib_process_readwrite: We are now in R/O mode May 20 12:39:59 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/5733, version=0.2133.1): ok (rc=0) May 20 12:39:59 finley cib: [25836]: info: cib_process_readwrite: We are now in R/W mode May 20 12:39:59 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/5734, version=0.2133.1): ok (rc=0) May 20 12:39:59 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/5735, version=0.2133.1): ok (rc=0) May 20 12:39:59 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r2:1 () May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:39:59 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:39:59 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5738, version=0.2134.1): ok (rc=0) May 20 12:39:59 finley crmd: [25840]: info: do_dc_join_offer_all: join-34: Waiting on 3 outstanding join acks May 20 12:39:59 finley crmd: [25840]: info: ais_dispatch: Membership 1744: quorum retained May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5741, version=0.2135.1): ok (rc=0) May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_lun0 (1304204891) May 20 12:40:00 finley crmd: [25840]: info: crm_ais_dispatch: Setting expected votes to 3 May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_lun1 (1304204345) May 20 12:40:00 finley crmd: [25840]: info: config_query_callback: Checking for expired actions every 900000ms May 20 12:40:00 finley crmd: [25840]: info: config_query_callback: Sending expected-votes=2 to corosync May 20 12:40:00 finley corosync[25820]: [pcmk ] info: update_expected_votes: Cannot update expected quorum votes 3 -> 2: value cannot be less that the current number of votes May 20 12:40:00 finley crmd: [25840]: info: update_dc: Set DC to finley (3.0.1) May 20 12:40:00 finley crmd: [25840]: info: ais_dispatch: Membership 1744: quorum retained May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:00 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5745, version=0.2136.1): ok (rc=0) May 20 12:40:00 finley crmd: [25840]: info: crm_ais_dispatch: Setting expected votes to 3 May 20 12:40:00 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5748, version=0.2136.1): ok (rc=0) May 20 12:40:00 finley cib: [23835]: info: write_cib_contents: Wrote version 0.2133.0 of the CIB to disk (digest: 62dce2070824941f04a96c8d5be24517) May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_lun2 (1304204850) May 20 12:40:00 finley cib: [23835]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.4ulPLo (digest: /var/lib/heartbeat/crm/cib.vXHHG1) May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun0 () May 20 12:40:00 finley cib: [23837]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-47.raw May 20 12:40:00 finley cib: [23837]: info: write_cib_contents: Wrote version 0.2136.0 of the CIB to disk (digest: 09a935b4d4a24467b9d063a188932566) May 20 12:40:00 finley cib: [23837]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.e4oWKq (digest: /var/lib/heartbeat/crm/cib.xEzVE5) May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun1 () May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun2 () May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun3 () May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r0:0 (10000) May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r0:1 () May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-ip_virtual0 () May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_target (1304204891) May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-ip_virtual0 (1304204891) May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r3:0 () May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r1:1 () May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r1:0 (INFINITY) May 20 12:40:00 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r2:0 (1305887962) May 20 12:40:01 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r2:1 () May 20 12:40:01 finley crmd: [25840]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] May 20 12:40:01 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes responded to the join offer. May 20 12:40:01 finley crmd: [25840]: info: do_dc_join_finalize: join-34: Syncing the CIB from finley to the rest of the cluster May 20 12:40:01 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r3:0 () May 20 12:40:01 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/5749, version=0.2136.1): ok (rc=0) May 20 12:40:01 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r0:0 (INFINITY) May 20 12:40:01 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/5750, version=0.2136.1): ok (rc=0) May 20 12:40:01 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/5751, version=0.2136.1): ok (rc=0) May 20 12:40:01 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/5752, version=0.2136.1): ok (rc=0) May 20 12:40:01 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r3:1 () May 20 12:40:01 finley crmd: [25840]: info: do_dc_join_ack: join-34: Updating node state to member for godfrey May 20 12:40:01 finley crmd: [25840]: info: do_dc_join_ack: join-34: Updating node state to member for finley May 20 12:40:02 finley crmd: [25840]: info: do_dc_join_ack: join-34: Updating node state to member for ebony May 20 12:40:02 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='godfrey']/lrm (origin=local/crmd/5753, version=0.2136.2): ok (rc=0) May 20 12:40:02 finley crmd: [25840]: info: erase_xpath_callback: Deletion of "//node_state[@uname='godfrey']/lrm": ok (rc=0) May 20 12:40:03 finley crmd: [25840]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] May 20 12:40:03 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:40:03 finley crmd: [25840]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date May 20 12:40:03 finley crmd: [25840]: info: crm_update_quorum: Updating quorum status to true (call=5761) May 20 12:40:03 finley crmd: [25840]: info: abort_transition_graph: do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled May 20 12:40:03 finley crmd: [25840]: info: do_pe_invoke: Query 5762: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:03 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='finley']/lrm (origin=local/crmd/5755, version=0.2136.4): ok (rc=0) May 20 12:40:03 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=drbd_r1:0_monitor_0, magic=0:0;12:5:7:b91d50c5-2309-4f96-ae7b-3bf7b28386fa, cib=0.2136.4) : Resource op removal May 20 12:40:03 finley crmd: [25840]: info: erase_xpath_callback: Deletion of "//node_state[@uname='finley']/lrm": ok (rc=0) May 20 12:40:03 finley crmd: [25840]: info: do_pe_invoke: Query 5763: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:03 finley crmd: [25840]: info: te_update_diff: Detected LRM refresh - 15 resources updated: Skipping all resource events May 20 12:40:03 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:227 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.2136.5) : LRM Refresh May 20 12:40:03 finley crmd: [25840]: info: do_pe_invoke: Query 5764: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:03 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ebony']/lrm (origin=local/crmd/5757, version=0.2136.6): ok (rc=0) May 20 12:40:03 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=drbd_r1:0_monitor_0, magic=0:5;22:3579:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2136.6) : Resource op removal May 20 12:40:03 finley crmd: [25840]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ebony']/lrm": ok (rc=0) May 20 12:40:03 finley crmd: [25840]: info: do_pe_invoke: Query 5765: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:03 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () May 20 12:40:03 finley crmd: [25840]: info: te_update_diff: Detected LRM refresh - 9 resources updated: Skipping all resource events May 20 12:40:03 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:227 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.2136.7) : LRM Refresh May 20 12:40:03 finley crmd: [25840]: info: do_pe_invoke: Query 5766: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:03 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/5759, version=0.2136.7): ok (rc=0) May 20 12:40:03 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/5761, version=0.2136.7): ok (rc=0) May 20 12:40:03 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r0:1 () May 20 12:40:04 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: pingd () May 20 12:40:04 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r1:0 (1305887962) May 20 12:40:04 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r1:1 () May 20 12:40:04 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r2:0 (10000) May 20 12:40:04 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r2:1 () May 20 12:40:04 finley crmd: [25840]: info: do_pe_invoke_callback: Invoking the PE: query=5766, ref=pe_calc-dc-1305888004-6259, seq=1744, quorate=1 May 20 12:40:04 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r3:0 () May 20 12:40:04 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r0:0 (1305887962) May 20 12:40:04 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r0:1 () May 20 12:40:04 finley pengine: [25839]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 May 20 12:40:04 finley pengine: [25839]: info: unpack_status: Node godfrey is in standby-mode May 20 12:40:04 finley pengine: [25839]: info: determine_online_status: Node godfrey is standby May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on godfrey May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on godfrey May 20 12:40:04 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:1_stop_0 on godfrey: not configured (6) May 20 12:40:04 finley pengine: [25839]: info: native_add_running: resource drbd_r1:1 isnt managed May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on godfrey May 20 12:40:04 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:1_stop_0 on godfrey: not configured (6) May 20 12:40:04 finley pengine: [25839]: info: native_add_running: resource drbd_r0:1 isnt managed May 20 12:40:04 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:1_stop_0 on godfrey: not configured (6) May 20 12:40:04 finley pengine: [25839]: info: native_add_running: resource drbd_r2:1 isnt managed May 20 12:40:04 finley pengine: [25839]: info: unpack_status: Node finley is in standby-mode May 20 12:40:04 finley pengine: [25839]: info: determine_online_status: Node finley is standby May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on finley May 20 12:40:04 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_demote_0 on finley: not configured (6) May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r1:0 to stop after a failed demote action May 20 12:40:04 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_stop_0 on finley: not configured (6) May 20 12:40:04 finley pengine: [25839]: info: native_add_running: resource drbd_r1:0 isnt managed May 20 12:40:04 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r1:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on finley May 20 12:40:04 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_demote_0 on finley: not configured (6) May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r0:0 to stop after a failed demote action May 20 12:40:04 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_stop_0 on finley: not configured (6) May 20 12:40:04 finley pengine: [25839]: info: native_add_running: resource drbd_r0:0 isnt managed May 20 12:40:04 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r0:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:04 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:1 on finley to drbd_r1:3 (ORPHAN) May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:40:04 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:40:04 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:04 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:40:04 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:40:04 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:04 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:40:04 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:40:04 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:40:04 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:40:04 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:40:04 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:40:04 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_target_monitor_0 failed with rc=5: Preventing iscsi0_target from re-starting on ebony May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun0_monitor_0 failed with rc=5: Preventing iscsi0_lun0 from re-starting on ebony May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun1_monitor_0 failed with rc=5: Preventing iscsi0_lun1 from re-starting on ebony May 20 12:40:04 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun2_monitor_0 failed with rc=5: Preventing iscsi0_lun2 from re-starting on ebony May 20 12:40:04 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:40:04 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:40:04 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:40:04 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:04 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:04 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:04 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:40:04 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:04 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:04 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:40:04 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:40:04 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:04 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:04 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:40:04 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:40:04 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:04 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:04 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:40:04 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:40:04 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:0 ping_nodes:1 ping_nodes:2 ] May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:04 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:04 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:40:04 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:40:04 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:40:04 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:04 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:04 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:40:04 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:40:04 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:40:04 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:04 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:04 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:40:04 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:40:04 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:40:04 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:04 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:04 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:40:04 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:40:04 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:40:04 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:40:04 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:04 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:04 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource ping_nodes:0 cannot run anywhere May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource ping_nodes:1 cannot run anywhere May 20 12:40:04 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:04 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:0#011(Stopped) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:1#011(Stopped) May 20 12:40:04 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:40:04 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:40:04 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3597: 0 actions in 0 synapses May 20 12:40:04 finley crmd: [25840]: info: do_te_invoke: Processing graph 3597 (ref=pe_calc-dc-1305888004-6259) derived from /var/lib/pengine/pe-input-4756.bz2 May 20 12:40:04 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:40:04 finley crmd: [25840]: notice: run_graph: Transition 3597 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-4756.bz2): Complete May 20 12:40:04 finley crmd: [25840]: info: te_graph_trigger: Transition 3597 is now complete May 20 12:40:04 finley crmd: [25840]: info: notify_crmd: Transition 3597 status: done - May 20 12:40:04 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:40:04 finley crmd: [25840]: info: do_state_transition: Starting PEngine Recheck Timer May 20 12:40:04 finley pengine: [25839]: info: process_pe_message: Transition 3597: PEngine Input stored in: /var/lib/pengine/pe-input-4756.bz2 May 20 12:40:04 finley cibadmin: [23914]: info: Invoked: cibadmin -Ql -o nodes May 20 12:40:04 finley cibadmin: [23916]: info: Invoked: cibadmin -Ql -o resources May 20 12:40:04 finley cibadmin: [23918]: info: Invoked: cibadmin -Ql -o resources May 20 12:40:04 finley crm_resource: [23920]: info: Invoked: crm_resource -C -r rg_iscsi -H godfrey May 20 12:40:05 finley crm_resource: [23920]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:05 finley crm_resource: [23920]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:05 finley crm_resource: [23920]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:05 finley crm_resource: [23920]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:05 finley crm_resource: [23920]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:05 finley crm_resource: [23920]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:05 finley crm_resource: [23920]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:05 finley crm_resource: [23920]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:05 finley crm_resource: [23920]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:05 finley cibadmin: [23922]: info: Invoked: cibadmin -Ql -o resources May 20 12:40:05 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='godfrey']//lrm_resource[@id='ip_virtual0'] (origin=godfrey/crmd/50, version=0.2136.8): ok (rc=0) May 20 12:40:05 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=ip_virtual0_monitor_0, magic=0:7;10:3515:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2136.8) : Resource op removal May 20 12:40:05 finley crmd: [25840]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] May 20 12:40:05 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:40:05 finley crmd: [25840]: info: do_pe_invoke: Query 5767: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:05 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:05 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:05 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:05 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:05 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:05 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:05 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:05 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:05 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:05 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:05 finley crmd: [25840]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change May 20 12:40:05 finley crmd: [25840]: info: need_abort: Aborting on change to admin_epoch May 20 12:40:05 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=godfrey/crmd/53, version=0.2137.1): ok (rc=0) May 20 12:40:05 finley crmd: [25840]: info: do_pe_invoke: Query 5768: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:05 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='godfrey']//lrm_resource[@id='iscsi0_target'] (origin=godfrey/crmd/54, version=0.2137.2): ok (rc=0) May 20 12:40:05 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=iscsi0_target_monitor_0, magic=0:7;11:3515:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2137.2) : Resource op removal May 20 12:40:05 finley crmd: [25840]: info: do_pe_invoke: Query 5770: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:05 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=godfrey/crmd/57, version=0.2137.2): ok (rc=0) May 20 12:40:05 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='godfrey']//lrm_resource[@id='iscsi0_lun0'] (origin=godfrey/crmd/58, version=0.2137.3): ok (rc=0) May 20 12:40:05 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=iscsi0_lun0_monitor_0, magic=0:7;12:3515:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2137.3) : Resource op removal May 20 12:40:05 finley crmd: [25840]: info: do_pe_invoke: Query 5771: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:05 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=godfrey/crmd/61, version=0.2137.3): ok (rc=0) May 20 12:40:05 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='godfrey']//lrm_resource[@id='iscsi0_lun1'] (origin=godfrey/crmd/62, version=0.2137.4): ok (rc=0) May 20 12:40:05 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=iscsi0_lun1_monitor_0, magic=0:7;13:3515:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2137.4) : Resource op removal May 20 12:40:05 finley crmd: [25840]: info: do_pe_invoke: Query 5772: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:05 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=godfrey/crmd/65, version=0.2137.4): ok (rc=0) May 20 12:40:05 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='godfrey']//lrm_resource[@id='iscsi0_lun2'] (origin=godfrey/crmd/66, version=0.2137.5): ok (rc=0) May 20 12:40:05 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=iscsi0_lun2_monitor_0, magic=0:7;14:3515:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2137.5) : Resource op removal May 20 12:40:05 finley crmd: [25840]: info: do_pe_invoke: Query 5773: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:05 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=godfrey/crmd/69, version=0.2137.5): ok (rc=0) May 20 12:40:05 finley crmd: [25840]: info: config_query_callback: Checking for expired actions every 900000ms May 20 12:40:05 finley crmd: [25840]: info: config_query_callback: Sending expected-votes=3 to corosync May 20 12:40:05 finley crmd: [25840]: info: ais_dispatch: Membership 1744: quorum retained May 20 12:40:05 finley crmd: [25840]: info: crm_ais_dispatch: Setting expected votes to 3 May 20 12:40:06 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5775, version=0.2137.5): ok (rc=0) May 20 12:40:06 finley crmd: [25840]: info: do_pe_invoke_callback: Invoking the PE: query=5773, ref=pe_calc-dc-1305888006-6261, seq=1744, quorate=1 May 20 12:40:06 finley pengine: [25839]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 May 20 12:40:06 finley pengine: [25839]: info: unpack_status: Node godfrey is in standby-mode May 20 12:40:06 finley pengine: [25839]: info: determine_online_status: Node godfrey is standby May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on godfrey May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on godfrey May 20 12:40:06 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:1_stop_0 on godfrey: not configured (6) May 20 12:40:06 finley pengine: [25839]: info: native_add_running: resource drbd_r1:1 isnt managed May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on godfrey May 20 12:40:06 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:1_stop_0 on godfrey: not configured (6) May 20 12:40:06 finley pengine: [25839]: info: native_add_running: resource drbd_r0:1 isnt managed May 20 12:40:06 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:1_stop_0 on godfrey: not configured (6) May 20 12:40:06 finley pengine: [25839]: info: native_add_running: resource drbd_r2:1 isnt managed May 20 12:40:06 finley pengine: [25839]: info: unpack_status: Node finley is in standby-mode May 20 12:40:06 finley pengine: [25839]: info: determine_online_status: Node finley is standby May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on finley May 20 12:40:06 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_demote_0 on finley: not configured (6) May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r1:0 to stop after a failed demote action May 20 12:40:06 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_stop_0 on finley: not configured (6) May 20 12:40:06 finley pengine: [25839]: info: native_add_running: resource drbd_r1:0 isnt managed May 20 12:40:06 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r1:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on finley May 20 12:40:06 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_demote_0 on finley: not configured (6) May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r0:0 to stop after a failed demote action May 20 12:40:06 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_stop_0 on finley: not configured (6) May 20 12:40:06 finley pengine: [25839]: info: native_add_running: resource drbd_r0:0 isnt managed May 20 12:40:06 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r0:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:06 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:1 on finley to drbd_r1:3 (ORPHAN) May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:40:06 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:40:06 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:06 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:40:06 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:40:06 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:06 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:40:06 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:40:06 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:40:06 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:40:06 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:40:06 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:40:06 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_target_monitor_0 failed with rc=5: Preventing iscsi0_target from re-starting on ebony May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun0_monitor_0 failed with rc=5: Preventing iscsi0_lun0 from re-starting on ebony May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun1_monitor_0 failed with rc=5: Preventing iscsi0_lun1 from re-starting on ebony May 20 12:40:06 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun2_monitor_0 failed with rc=5: Preventing iscsi0_lun2 from re-starting on ebony May 20 12:40:06 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:40:06 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:40:06 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:40:06 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:06 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:06 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:06 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:40:06 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:06 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:06 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:40:06 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:40:06 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:06 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:06 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:40:06 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:40:06 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:06 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:06 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:40:06 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:40:06 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:0 ping_nodes:1 ping_nodes:2 ] May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:06 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:06 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:40:06 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:40:06 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:40:06 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:06 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:06 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:40:06 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:40:06 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:40:06 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:06 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:06 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:40:06 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:40:06 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:40:06 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:06 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:06 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:40:06 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:40:06 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:40:06 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:40:06 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:06 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:06 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource ping_nodes:0 cannot run anywhere May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource ping_nodes:1 cannot run anywhere May 20 12:40:06 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:06 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:0#011(Stopped) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:1#011(Stopped) May 20 12:40:06 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:40:06 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:40:06 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3598: 7 actions in 7 synapses May 20 12:40:06 finley crmd: [25840]: info: do_te_invoke: Processing graph 3598 (ref=pe_calc-dc-1305888006-6261) derived from /var/lib/pengine/pe-input-4757.bz2 May 20 12:40:06 finley crmd: [25840]: info: te_rsc_command: Initiating action 4: monitor ip_virtual0_monitor_0 on godfrey May 20 12:40:06 finley crmd: [25840]: info: te_rsc_command: Initiating action 5: monitor iscsi0_target_monitor_0 on godfrey May 20 12:40:06 finley crmd: [25840]: info: te_rsc_command: Initiating action 6: monitor iscsi0_lun0_monitor_0 on godfrey May 20 12:40:06 finley crmd: [25840]: info: te_rsc_command: Initiating action 7: monitor iscsi0_lun1_monitor_0 on godfrey May 20 12:40:06 finley crmd: [25840]: info: te_rsc_command: Initiating action 8: monitor iscsi0_lun2_monitor_0 on godfrey May 20 12:40:06 finley cib: [23923]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-48.raw May 20 12:40:06 finley pengine: [25839]: info: process_pe_message: Transition 3598: PEngine Input stored in: /var/lib/pengine/pe-input-4757.bz2 May 20 12:40:06 finley cib: [23923]: info: write_cib_contents: Wrote version 0.2137.0 of the CIB to disk (digest: 45a0870cc02ecf5d691d5e62abaa9543) May 20 12:40:06 finley cib: [23923]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.qHJf4A (digest: /var/lib/heartbeat/crm/cib.70byhq) May 20 12:40:06 finley cibadmin: [23925]: info: Invoked: cibadmin -Ql -o resources May 20 12:40:06 finley crmd: [25840]: info: match_graph_event: Action ip_virtual0_monitor_0 (4) confirmed on godfrey (rc=0) May 20 12:40:06 finley crmd: [25840]: info: match_graph_event: Action iscsi0_lun0_monitor_0 (6) confirmed on godfrey (rc=0) May 20 12:40:06 finley crmd: [25840]: info: match_graph_event: Action iscsi0_target_monitor_0 (5) confirmed on godfrey (rc=0) May 20 12:40:06 finley crmd: [25840]: info: match_graph_event: Action iscsi0_lun1_monitor_0 (7) confirmed on godfrey (rc=0) May 20 12:40:06 finley crm_resource: [23927]: info: Invoked: crm_resource -C -r rg_iscsi -H finley May 20 12:40:06 finley crm_resource: [23927]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:06 finley crm_resource: [23927]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:06 finley crm_resource: [23927]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:06 finley crm_resource: [23927]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:06 finley crm_resource: [23927]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:06 finley crm_resource: [23927]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:06 finley crm_resource: [23927]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:06 finley crm_resource: [23927]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:06 finley crm_resource: [23927]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:06 finley crmd: [25840]: info: do_lrm_invoke: Removing resource ip_virtual0 from the LRM May 20 12:40:06 finley crmd: [25840]: info: do_lrm_invoke: Resource 'ip_virtual0' deleted for 23927_crm_resource on finley May 20 12:40:06 finley crmd: [25840]: info: notify_deleted: Notifying 23927_crm_resource on finley that ip_virtual0 was deleted May 20 12:40:06 finley crmd: [25840]: info: send_direct_ack: ACK'ing resource op ip_virtual0_delete_60000 from 0:0:crm-resource-23927: lrm_invoke-lrmd-1305888006-6268 May 20 12:40:06 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='finley']//lrm_resource[@id='ip_virtual0'] (origin=local/crmd/5776, version=0.2137.10): ok (rc=0) May 20 12:40:06 finley crmd: [25840]: info: do_lrm_invoke: Removing resource iscsi0_target from the LRM May 20 12:40:06 finley crmd: [25840]: info: do_lrm_invoke: Resource 'iscsi0_target' deleted for 23927_crm_resource on finley May 20 12:40:06 finley crmd: [25840]: info: notify_deleted: Notifying 23927_crm_resource on finley that iscsi0_target was deleted May 20 12:40:06 finley crmd: [25840]: info: send_direct_ack: ACK'ing resource op iscsi0_target_delete_60000 from 0:0:crm-resource-23927: lrm_invoke-lrmd-1305888006-6269 May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5778, version=0.2138.1): ok (rc=0) May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='finley']//lrm_resource[@id='iscsi0_target'] (origin=local/crmd/5779, version=0.2138.2): ok (rc=0) May 20 12:40:07 finley crmd: [25840]: info: do_lrm_invoke: Removing resource iscsi0_lun0 from the LRM May 20 12:40:07 finley crmd: [25840]: info: do_lrm_invoke: Resource 'iscsi0_lun0' deleted for 23927_crm_resource on finley May 20 12:40:07 finley crmd: [25840]: info: notify_deleted: Notifying 23927_crm_resource on finley that iscsi0_lun0 was deleted May 20 12:40:07 finley crmd: [25840]: info: send_direct_ack: ACK'ing resource op iscsi0_lun0_delete_60000 from 0:0:crm-resource-23927: lrm_invoke-lrmd-1305888007-6270 May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5781, version=0.2138.2): ok (rc=0) May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='finley']//lrm_resource[@id='iscsi0_lun0'] (origin=local/crmd/5782, version=0.2138.3): ok (rc=0) May 20 12:40:07 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=ip_virtual0_monitor_0, magic=0:7;11:3496:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2137.10) : Resource op removal May 20 12:40:07 finley crmd: [25840]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 May 20 12:40:07 finley crmd: [25840]: info: update_abort_priority: Abort action done superceeded by restart May 20 12:40:07 finley crmd: [25840]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=0) : Non-status change May 20 12:40:07 finley crmd: [25840]: info: need_abort: Aborting on change to admin_epoch May 20 12:40:07 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_target_monitor_0, magic=0:7;12:3496:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2138.2) : Resource op removal May 20 12:40:07 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_lun0_monitor_0, magic=0:7;13:3496:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2138.3) : Resource op removal May 20 12:40:07 finley crmd: [25840]: info: do_lrm_invoke: Removing resource iscsi0_lun1 from the LRM May 20 12:40:07 finley crmd: [25840]: info: do_lrm_invoke: Resource 'iscsi0_lun1' deleted for 23927_crm_resource on finley May 20 12:40:07 finley crmd: [25840]: info: notify_deleted: Notifying 23927_crm_resource on finley that iscsi0_lun1 was deleted May 20 12:40:07 finley crmd: [25840]: info: send_direct_ack: ACK'ing resource op iscsi0_lun1_delete_60000 from 0:0:crm-resource-23927: lrm_invoke-lrmd-1305888007-6271 May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5784, version=0.2139.1): ok (rc=0) May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='finley']//lrm_resource[@id='iscsi0_lun1'] (origin=local/crmd/5786, version=0.2139.2): ok (rc=0) May 20 12:40:07 finley crmd: [25840]: info: do_lrm_invoke: Removing resource iscsi0_lun2 from the LRM May 20 12:40:07 finley crmd: [25840]: info: do_lrm_invoke: Resource 'iscsi0_lun2' deleted for 23927_crm_resource on finley May 20 12:40:07 finley crmd: [25840]: info: notify_deleted: Notifying 23927_crm_resource on finley that iscsi0_lun2 was deleted May 20 12:40:07 finley crmd: [25840]: info: send_direct_ack: ACK'ing resource op iscsi0_lun2_delete_60000 from 0:0:crm-resource-23927: lrm_invoke-lrmd-1305888007-6272 May 20 12:40:07 finley cibadmin: [23929]: info: Invoked: cibadmin -Ql -o resources May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5788, version=0.2139.2): ok (rc=0) May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='finley']//lrm_resource[@id='iscsi0_lun2'] (origin=local/crmd/5789, version=0.2139.3): ok (rc=0) May 20 12:40:07 finley crmd: [25840]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=0) : Non-status change May 20 12:40:07 finley crmd: [25840]: info: need_abort: Aborting on change to admin_epoch May 20 12:40:07 finley crmd: [25840]: info: config_query_callback: Checking for expired actions every 900000ms May 20 12:40:07 finley crmd: [25840]: info: config_query_callback: Sending expected-votes=3 to corosync May 20 12:40:07 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_lun1_monitor_0, magic=0:7;14:3496:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2139.2) : Resource op removal May 20 12:40:07 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_lun2_monitor_0, magic=0:7;15:3496:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2139.3) : Resource op removal May 20 12:40:07 finley crmd: [25840]: info: ais_dispatch: Membership 1744: quorum retained May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5791, version=0.2139.4): ok (rc=0) May 20 12:40:07 finley crmd: [25840]: info: crm_ais_dispatch: Setting expected votes to 3 May 20 12:40:07 finley crmd: [25840]: info: match_graph_event: Action iscsi0_lun2_monitor_0 (8) confirmed on godfrey (rc=0) May 20 12:40:07 finley crmd: [25840]: info: config_query_callback: Checking for expired actions every 900000ms May 20 12:40:07 finley crmd: [25840]: info: config_query_callback: Sending expected-votes=3 to corosync May 20 12:40:07 finley crmd: [25840]: info: ais_dispatch: Membership 1744: quorum retained May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5794, version=0.2139.4): ok (rc=0) May 20 12:40:07 finley crmd: [25840]: info: crm_ais_dispatch: Setting expected votes to 3 May 20 12:40:07 finley crmd: [25840]: info: te_rsc_command: Initiating action 3: probe_complete probe_complete on godfrey - no waiting May 20 12:40:07 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:40:07 finley crmd: [25840]: notice: run_graph: Transition 3598 (Complete=6, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-4757.bz2): Stopped May 20 12:40:07 finley crmd: [25840]: info: te_graph_trigger: Transition 3598 is now complete May 20 12:40:07 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:40:07 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:40:07 finley crmd: [25840]: info: do_pe_invoke: Query 5797: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:07 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5796, version=0.2139.4): ok (rc=0) May 20 12:40:07 finley crmd: [25840]: info: do_pe_invoke_callback: Invoking the PE: query=5797, ref=pe_calc-dc-1305888007-6274, seq=1744, quorate=1 May 20 12:40:07 finley pengine: [25839]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 May 20 12:40:07 finley pengine: [25839]: info: unpack_status: Node godfrey is in standby-mode May 20 12:40:07 finley pengine: [25839]: info: determine_online_status: Node godfrey is standby May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on godfrey May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on godfrey May 20 12:40:07 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:1_stop_0 on godfrey: not configured (6) May 20 12:40:07 finley pengine: [25839]: info: native_add_running: resource drbd_r1:1 isnt managed May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on godfrey May 20 12:40:07 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:1_stop_0 on godfrey: not configured (6) May 20 12:40:07 finley pengine: [25839]: info: native_add_running: resource drbd_r0:1 isnt managed May 20 12:40:07 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:1_stop_0 on godfrey: not configured (6) May 20 12:40:07 finley pengine: [25839]: info: native_add_running: resource drbd_r2:1 isnt managed May 20 12:40:07 finley pengine: [25839]: info: unpack_status: Node finley is in standby-mode May 20 12:40:07 finley pengine: [25839]: info: determine_online_status: Node finley is standby May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on finley May 20 12:40:07 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_demote_0 on finley: not configured (6) May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r1:0 to stop after a failed demote action May 20 12:40:07 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_stop_0 on finley: not configured (6) May 20 12:40:07 finley pengine: [25839]: info: native_add_running: resource drbd_r1:0 isnt managed May 20 12:40:07 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r1:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on finley May 20 12:40:07 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_demote_0 on finley: not configured (6) May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r0:0 to stop after a failed demote action May 20 12:40:07 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_stop_0 on finley: not configured (6) May 20 12:40:07 finley pengine: [25839]: info: native_add_running: resource drbd_r0:0 isnt managed May 20 12:40:07 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r0:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:07 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:1 on finley to drbd_r1:3 (ORPHAN) May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:40:07 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:40:07 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:07 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:40:07 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:40:07 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:07 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:40:07 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:40:07 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:40:07 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:40:07 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:40:07 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:40:07 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_target_monitor_0 failed with rc=5: Preventing iscsi0_target from re-starting on ebony May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun0_monitor_0 failed with rc=5: Preventing iscsi0_lun0 from re-starting on ebony May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun1_monitor_0 failed with rc=5: Preventing iscsi0_lun1 from re-starting on ebony May 20 12:40:07 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun2_monitor_0 failed with rc=5: Preventing iscsi0_lun2 from re-starting on ebony May 20 12:40:07 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:40:07 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:40:07 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:40:07 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:07 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:07 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:07 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:40:07 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:07 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:07 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:40:07 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:40:07 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:07 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:07 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:40:07 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:40:07 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:07 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:07 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:40:07 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:40:07 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:0 ping_nodes:1 ping_nodes:2 ] May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:07 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:07 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:40:07 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:40:07 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:40:07 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:07 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:07 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:40:07 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:40:07 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:40:07 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:07 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:07 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:40:07 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:40:07 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:40:07 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:07 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:07 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:40:07 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:40:07 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:40:07 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:40:07 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:07 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:07 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource ping_nodes:0 cannot run anywhere May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource ping_nodes:1 cannot run anywhere May 20 12:40:07 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:07 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:0#011(Stopped) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:1#011(Stopped) May 20 12:40:07 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:40:07 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:40:07 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3599: 7 actions in 7 synapses May 20 12:40:07 finley crmd: [25840]: info: do_te_invoke: Processing graph 3599 (ref=pe_calc-dc-1305888007-6274) derived from /var/lib/pengine/pe-input-4758.bz2 May 20 12:40:07 finley crmd: [25840]: info: te_rsc_command: Initiating action 5: monitor ip_virtual0_monitor_0 on finley (local) May 20 12:40:07 finley crmd: [25840]: info: do_lrm_rsc_op: Performing key=5:3599:7:8f399f14-fe26-4279-b701-1947eb7b322d op=ip_virtual0_monitor_0 ) May 20 12:40:07 finley lrmd: [25837]: info: rsc:ip_virtual0:1150: probe May 20 12:40:07 finley crmd: [25840]: info: te_rsc_command: Initiating action 6: monitor iscsi0_target_monitor_0 on finley (local) May 20 12:40:07 finley crmd: [25840]: info: do_lrm_rsc_op: Performing key=6:3599:7:8f399f14-fe26-4279-b701-1947eb7b322d op=iscsi0_target_monitor_0 ) May 20 12:40:07 finley lrmd: [25837]: info: rsc:iscsi0_target:1151: probe May 20 12:40:07 finley crmd: [25840]: info: te_rsc_command: Initiating action 7: monitor iscsi0_lun0_monitor_0 on finley (local) May 20 12:40:08 finley crmd: [25840]: info: do_lrm_rsc_op: Performing key=7:3599:7:8f399f14-fe26-4279-b701-1947eb7b322d op=iscsi0_lun0_monitor_0 ) May 20 12:40:08 finley lrmd: [25837]: info: rsc:iscsi0_lun0:1152: probe May 20 12:40:08 finley crmd: [25840]: info: te_rsc_command: Initiating action 8: monitor iscsi0_lun1_monitor_0 on finley (local) May 20 12:40:08 finley crmd: [25840]: info: do_lrm_rsc_op: Performing key=8:3599:7:8f399f14-fe26-4279-b701-1947eb7b322d op=iscsi0_lun1_monitor_0 ) May 20 12:40:08 finley lrmd: [25837]: info: rsc:iscsi0_lun1:1153: probe May 20 12:40:08 finley crmd: [25840]: info: te_rsc_command: Initiating action 9: monitor iscsi0_lun2_monitor_0 on finley (local) May 20 12:40:08 finley crmd: [25840]: info: do_lrm_rsc_op: Performing key=9:3599:7:8f399f14-fe26-4279-b701-1947eb7b322d op=iscsi0_lun2_monitor_0 ) May 20 12:40:08 finley crmd: [25840]: info: process_lrm_event: LRM operation ip_virtual0_monitor_0 (call=1150, rc=7, cib-update=5798, confirmed=true) not running May 20 12:40:08 finley cibadmin: [24009]: info: Invoked: cibadmin -Ql -o resources May 20 12:40:08 finley iSCSITarget[23931]: WARNING: Configuration parameter "portals" is not supported by the iSCSI implementation and will be ignored. May 20 12:40:08 finley crmd: [25840]: info: match_graph_event: Action ip_virtual0_monitor_0 (5) confirmed on finley (rc=0) May 20 12:40:08 finley crmd: [25840]: info: process_lrm_event: LRM operation iscsi0_target_monitor_0 (call=1151, rc=7, cib-update=5799, confirmed=true) not running May 20 12:40:08 finley crmd: [25840]: info: process_lrm_event: LRM operation iscsi0_lun1_monitor_0 (call=1153, rc=7, cib-update=5800, confirmed=true) not running May 20 12:40:08 finley crmd: [25840]: info: process_lrm_event: LRM operation iscsi0_lun0_monitor_0 (call=1152, rc=7, cib-update=5801, confirmed=true) not running May 20 12:40:08 finley crmd: [25840]: info: match_graph_event: Action iscsi0_target_monitor_0 (6) confirmed on finley (rc=0) May 20 12:40:08 finley crmd: [25840]: info: match_graph_event: Action iscsi0_lun1_monitor_0 (8) confirmed on finley (rc=0) May 20 12:40:08 finley crmd: [25840]: info: match_graph_event: Action iscsi0_lun0_monitor_0 (7) confirmed on finley (rc=0) May 20 12:40:08 finley pengine: [25839]: info: process_pe_message: Transition 3599: PEngine Input stored in: /var/lib/pengine/pe-input-4758.bz2 May 20 12:40:08 finley cib: [24032]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-49.raw May 20 12:40:08 finley cib: [24032]: info: write_cib_contents: Wrote version 0.2139.0 of the CIB to disk (digest: 18473e9280e9249cb8fca8f80e6e2a28) May 20 12:40:08 finley cib: [24032]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.DkzJiF (digest: /var/lib/heartbeat/crm/cib.RmUvKy) May 20 12:40:08 finley attrd: [25838]: info: attrd_local_callback: Sending full refresh (origin=crmd) May 20 12:40:08 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_target () May 20 12:40:08 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r1:0 (10000) May 20 12:40:08 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r1:1 () May 20 12:40:09 finley lrmd: [25837]: info: rsc:iscsi0_lun2:1154: probe May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r2:0 (INFINITY) May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r2:1 () May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_lun0 (1304204891) May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_lun1 (1304204345) May 20 12:40:09 finley crmd: [25840]: info: process_lrm_event: LRM operation iscsi0_lun2_monitor_0 (call=1154, rc=7, cib-update=5802, confirmed=true) not running May 20 12:40:09 finley crm_resource: [24058]: info: Invoked: crm_resource -C -r rg_iscsi -H ebony May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_lun2 (1304204850) May 20 12:40:09 finley crmd: [25840]: info: match_graph_event: Action iscsi0_lun2_monitor_0 (9) confirmed on finley (rc=0) May 20 12:40:09 finley crmd: [25840]: info: te_rsc_command: Initiating action 4: probe_complete probe_complete on finley (local) - no waiting May 20 12:40:09 finley crmd: [25840]: info: te_pseudo_action: Pseudo action 2 fired and confirmed May 20 12:40:09 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:40:09 finley crmd: [25840]: notice: run_graph: Transition 3599 (Complete=7, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-4758.bz2): Complete May 20 12:40:09 finley crmd: [25840]: info: te_graph_trigger: Transition 3599 is now complete May 20 12:40:09 finley crmd: [25840]: info: notify_crmd: Transition 3599 status: done - May 20 12:40:09 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:40:09 finley crmd: [25840]: info: do_state_transition: Starting PEngine Recheck Timer May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun0 () May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun1 () May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun2 () May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iscsi0_lun3 () May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r0:0 (10000) May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r0:1 () May 20 12:40:09 finley crm_resource: [24058]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:09 finley crm_resource: [24058]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:09 finley crm_resource: [24058]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:09 finley crm_resource: [24058]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:09 finley crm_resource: [24058]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:09 finley crm_resource: [24058]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:09 finley crm_resource: [24058]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:09 finley crm_resource: [24058]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:09 finley crm_resource: [24058]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-ip_virtual0 () May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iscsi0_target (1304204891) May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-ip_virtual0 (1304204891) May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r3:0 () May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r1:1 () May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r1:0 (INFINITY) May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r2:0 (1305887962) May 20 12:40:09 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r2:1 () May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r3:0 () May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r0:0 (INFINITY) May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r3:1 () May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r0:1 () May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: pingd () May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r1:0 (1305887962) May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r1:1 () May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r2:0 (10000) May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd_r2:1 () May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-drbd_r3:0 () May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r0:0 (1305887962) May 20 12:40:10 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-drbd_r0:1 () May 20 12:40:11 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ebony']//lrm_resource[@id='ip_virtual0'] (origin=ebony/crmd/25, version=0.2139.10): ok (rc=0) May 20 12:40:11 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=ip_virtual0_monitor_0, magic=0:7;16:3579:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2139.10) : Resource op removal May 20 12:40:11 finley crmd: [25840]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] May 20 12:40:11 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:40:11 finley crmd: [25840]: info: do_pe_invoke: Query 5803: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:11 finley crmd: [25840]: info: do_pe_invoke_callback: Invoking the PE: query=5803, ref=pe_calc-dc-1305888011-6282, seq=1744, quorate=1 May 20 12:40:11 finley pengine: [25839]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 May 20 12:40:11 finley pengine: [25839]: info: unpack_status: Node godfrey is in standby-mode May 20 12:40:11 finley pengine: [25839]: info: determine_online_status: Node godfrey is standby May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on godfrey May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on godfrey May 20 12:40:11 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:1_stop_0 on godfrey: not configured (6) May 20 12:40:11 finley pengine: [25839]: info: native_add_running: resource drbd_r1:1 isnt managed May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on godfrey May 20 12:40:11 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:1_stop_0 on godfrey: not configured (6) May 20 12:40:11 finley pengine: [25839]: info: native_add_running: resource drbd_r0:1 isnt managed May 20 12:40:11 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:1_stop_0 on godfrey: not configured (6) May 20 12:40:11 finley pengine: [25839]: info: native_add_running: resource drbd_r2:1 isnt managed May 20 12:40:11 finley pengine: [25839]: info: unpack_status: Node finley is in standby-mode May 20 12:40:11 finley pengine: [25839]: info: determine_online_status: Node finley is standby May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on finley May 20 12:40:11 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_demote_0 on finley: not configured (6) May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r1:0 to stop after a failed demote action May 20 12:40:11 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_stop_0 on finley: not configured (6) May 20 12:40:11 finley pengine: [25839]: info: native_add_running: resource drbd_r1:0 isnt managed May 20 12:40:11 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r1:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on finley May 20 12:40:11 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_demote_0 on finley: not configured (6) May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r0:0 to stop after a failed demote action May 20 12:40:11 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_stop_0 on finley: not configured (6) May 20 12:40:11 finley pengine: [25839]: info: native_add_running: resource drbd_r0:0 isnt managed May 20 12:40:11 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r0:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:11 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:1 on finley to drbd_r1:3 (ORPHAN) May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:40:11 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:40:11 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:11 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:40:11 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:40:11 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:11 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:40:11 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:40:11 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:40:11 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:40:11 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:40:11 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:40:11 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_target_monitor_0 failed with rc=5: Preventing iscsi0_target from re-starting on ebony May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun0_monitor_0 failed with rc=5: Preventing iscsi0_lun0 from re-starting on ebony May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun1_monitor_0 failed with rc=5: Preventing iscsi0_lun1 from re-starting on ebony May 20 12:40:11 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun2_monitor_0 failed with rc=5: Preventing iscsi0_lun2 from re-starting on ebony May 20 12:40:11 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:40:11 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:40:11 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:40:11 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:11 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:11 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:11 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:40:11 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:11 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:11 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:40:11 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:40:11 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:11 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:11 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:40:11 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:40:11 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:11 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:11 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:40:11 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:40:11 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:0 ping_nodes:1 ping_nodes:2 ] May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:11 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:11 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:40:11 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:40:11 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:40:11 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:11 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:11 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:40:11 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:40:11 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:40:11 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:11 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:11 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:40:11 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:40:11 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:40:11 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:11 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:11 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:40:11 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:40:11 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:40:11 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:40:11 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:11 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:11 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource ping_nodes:0 cannot run anywhere May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource ping_nodes:1 cannot run anywhere May 20 12:40:11 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:11 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:0#011(Stopped) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:1#011(Stopped) May 20 12:40:11 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:40:11 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:40:11 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3600: 3 actions in 3 synapses May 20 12:40:11 finley crmd: [25840]: info: do_te_invoke: Processing graph 3600 (ref=pe_calc-dc-1305888011-6282) derived from /var/lib/pengine/pe-input-4759.bz2 May 20 12:40:11 finley crmd: [25840]: info: te_rsc_command: Initiating action 6: monitor ip_virtual0_monitor_0 on ebony May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:11 finley crmd: [25840]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=0) : Non-status change May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:11 finley crmd: [25840]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 May 20 12:40:11 finley crmd: [25840]: info: update_abort_priority: Abort action done superceeded by restart May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:11 finley crmd: [25840]: info: need_abort: Aborting on change to admin_epoch May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:11 finley pengine: [25839]: info: process_pe_message: Transition 3600: PEngine Input stored in: /var/lib/pengine/pe-input-4759.bz2 May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:11 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:11 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=ebony/crmd/27, version=0.2140.1): ok (rc=0) May 20 12:40:11 finley crmd: [25840]: info: config_query_callback: Checking for expired actions every 900000ms May 20 12:40:11 finley crmd: [25840]: info: config_query_callback: Sending expected-votes=3 to corosync May 20 12:40:11 finley crmd: [25840]: info: ais_dispatch: Membership 1744: quorum retained May 20 12:40:11 finley crmd: [25840]: info: crm_ais_dispatch: Setting expected votes to 3 May 20 12:40:11 finley cib: [24059]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-50.raw May 20 12:40:11 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5806, version=0.2140.1): ok (rc=0) May 20 12:40:12 finley cib: [24059]: info: write_cib_contents: Wrote version 0.2140.0 of the CIB to disk (digest: c533f0bd0a2955ef02b47ead6a1ff780) May 20 12:40:12 finley cib: [24059]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.850kmL (digest: /var/lib/heartbeat/crm/cib.HSEIRK) May 20 12:40:12 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ebony']//lrm_resource[@id='iscsi0_target'] (origin=ebony/crmd/28, version=0.2140.2): ok (rc=0) May 20 12:40:12 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_target_monitor_0, magic=0:5;17:3579:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2140.2) : Resource op removal May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:12 finley crmd: [25840]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=0) : Non-status change May 20 12:40:12 finley crmd: [25840]: info: need_abort: Aborting on change to admin_epoch May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:12 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:12 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=ebony/crmd/30, version=0.2141.1): ok (rc=0) May 20 12:40:12 finley crmd: [25840]: info: config_query_callback: Checking for expired actions every 900000ms May 20 12:40:12 finley crmd: [25840]: info: config_query_callback: Sending expected-votes=3 to corosync May 20 12:40:12 finley crmd: [25840]: info: ais_dispatch: Membership 1744: quorum retained May 20 12:40:12 finley crmd: [25840]: info: crm_ais_dispatch: Setting expected votes to 3 May 20 12:40:12 finley cib: [24060]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-51.raw May 20 12:40:12 finley cib: [24060]: info: write_cib_contents: Wrote version 0.2141.0 of the CIB to disk (digest: 5da347e40632869a4da2d0b26afa18b9) May 20 12:40:12 finley cib: [24060]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.5H22cM (digest: /var/lib/heartbeat/crm/cib.LWH8yM) May 20 12:40:12 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5809, version=0.2141.1): ok (rc=0) May 20 12:40:12 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ebony']//lrm_resource[@id='iscsi0_lun0'] (origin=ebony/crmd/31, version=0.2141.2): ok (rc=0) May 20 12:40:12 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_lun0_monitor_0, magic=0:5;18:3579:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2141.2) : Resource op removal May 20 12:40:12 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=ebony/crmd/33, version=0.2141.2): ok (rc=0) May 20 12:40:12 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ebony']//lrm_resource[@id='iscsi0_lun1'] (origin=ebony/crmd/34, version=0.2141.3): ok (rc=0) May 20 12:40:12 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_lun1_monitor_0, magic=0:5;19:3579:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2141.3) : Resource op removal May 20 12:40:13 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=ebony/crmd/36, version=0.2141.3): ok (rc=0) May 20 12:40:13 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ebony']//lrm_resource[@id='iscsi0_lun2'] (origin=ebony/crmd/37, version=0.2141.4): ok (rc=0) May 20 12:40:13 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_lun2_monitor_0, magic=0:5;20:3579:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2141.4) : Resource op removal May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:13 finley crmd: [25840]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=0) : Non-status change May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:13 finley crmd: [25840]: info: need_abort: Aborting on change to admin_epoch May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:13 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:13 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=ebony/crmd/39, version=0.2142.1): ok (rc=0) May 20 12:40:13 finley crmd: [25840]: info: config_query_callback: Checking for expired actions every 900000ms May 20 12:40:13 finley crmd: [25840]: info: config_query_callback: Sending expected-votes=3 to corosync May 20 12:40:13 finley crmd: [25840]: info: ais_dispatch: Membership 1744: quorum retained May 20 12:40:13 finley crmd: [25840]: info: crm_ais_dispatch: Setting expected votes to 3 May 20 12:40:13 finley cib: [24061]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-52.raw May 20 12:40:13 finley cib: [24061]: info: write_cib_contents: Wrote version 0.2142.0 of the CIB to disk (digest: 08d1080750f90dde71f60a22bbf7c74c) May 20 12:40:13 finley cib: [24061]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.mf7zyO (digest: /var/lib/heartbeat/crm/cib.xXedgR) May 20 12:40:13 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/5812, version=0.2142.1): ok (rc=0) May 20 12:40:14 finley crmd: [25840]: info: match_graph_event: Action ip_virtual0_monitor_0 (6) confirmed on ebony (rc=0) May 20 12:40:14 finley crmd: [25840]: info: te_rsc_command: Initiating action 5: probe_complete probe_complete on ebony - no waiting May 20 12:40:14 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:40:14 finley crmd: [25840]: notice: run_graph: Transition 3600 (Complete=2, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-4759.bz2): Stopped May 20 12:40:14 finley crmd: [25840]: info: te_graph_trigger: Transition 3600 is now complete May 20 12:40:14 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:40:14 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:40:14 finley crmd: [25840]: info: do_pe_invoke: Query 5813: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:14 finley crmd: [25840]: info: do_pe_invoke_callback: Invoking the PE: query=5813, ref=pe_calc-dc-1305888014-6285, seq=1744, quorate=1 May 20 12:40:14 finley pengine: [25839]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 May 20 12:40:14 finley pengine: [25839]: info: unpack_status: Node godfrey is in standby-mode May 20 12:40:14 finley pengine: [25839]: info: determine_online_status: Node godfrey is standby May 20 12:40:14 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on godfrey May 20 12:40:14 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on godfrey May 20 12:40:14 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:1_stop_0 on godfrey: not configured (6) May 20 12:40:14 finley pengine: [25839]: info: native_add_running: resource drbd_r1:1 isnt managed May 20 12:40:14 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on godfrey May 20 12:40:14 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:1_stop_0 on godfrey: not configured (6) May 20 12:40:14 finley pengine: [25839]: info: native_add_running: resource drbd_r0:1 isnt managed May 20 12:40:14 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:1_stop_0 on godfrey: not configured (6) May 20 12:40:14 finley pengine: [25839]: info: native_add_running: resource drbd_r2:1 isnt managed May 20 12:40:14 finley pengine: [25839]: info: unpack_status: Node finley is in standby-mode May 20 12:40:14 finley pengine: [25839]: info: determine_online_status: Node finley is standby May 20 12:40:14 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on finley May 20 12:40:14 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_demote_0 on finley: not configured (6) May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r1:0 to stop after a failed demote action May 20 12:40:14 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_stop_0 on finley: not configured (6) May 20 12:40:14 finley pengine: [25839]: info: native_add_running: resource drbd_r1:0 isnt managed May 20 12:40:14 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r1:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:14 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on finley May 20 12:40:14 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_demote_0 on finley: not configured (6) May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r0:0 to stop after a failed demote action May 20 12:40:14 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_stop_0 on finley: not configured (6) May 20 12:40:14 finley pengine: [25839]: info: native_add_running: resource drbd_r0:0 isnt managed May 20 12:40:14 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r0:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:14 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:1 on finley to drbd_r1:3 (ORPHAN) May 20 12:40:14 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:40:14 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:40:14 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:14 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:40:14 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:40:14 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:14 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:40:14 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:40:14 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:40:14 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:40:14 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:40:14 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:40:14 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:40:14 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:40:14 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:40:14 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:40:14 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:40:14 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:40:14 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:40:14 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:14 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:14 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:14 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:40:14 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:14 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:14 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:40:14 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:40:14 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:14 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:14 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:40:14 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:40:14 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:14 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:14 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:40:14 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:40:14 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:0 ping_nodes:1 ping_nodes:2 ] May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:14 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:14 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:40:14 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:40:14 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:40:14 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:14 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:14 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:40:14 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:40:14 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:40:14 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:14 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:14 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:40:14 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:40:14 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:40:14 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:14 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:14 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:40:14 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:40:14 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:40:14 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:40:14 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:14 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:14 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource ping_nodes:0 cannot run anywhere May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource ping_nodes:1 cannot run anywhere May 20 12:40:14 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:14 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:0#011(Stopped) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:1#011(Stopped) May 20 12:40:14 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:40:14 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:40:14 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3601: 6 actions in 6 synapses May 20 12:40:14 finley crmd: [25840]: info: do_te_invoke: Processing graph 3601 (ref=pe_calc-dc-1305888014-6285) derived from /var/lib/pengine/pe-input-4760.bz2 May 20 12:40:14 finley crmd: [25840]: info: te_rsc_command: Initiating action 6: monitor iscsi0_target_monitor_0 on ebony May 20 12:40:14 finley crmd: [25840]: info: te_rsc_command: Initiating action 7: monitor iscsi0_lun0_monitor_0 on ebony May 20 12:40:14 finley crmd: [25840]: info: te_rsc_command: Initiating action 8: monitor iscsi0_lun1_monitor_0 on ebony May 20 12:40:14 finley crmd: [25840]: info: te_rsc_command: Initiating action 9: monitor iscsi0_lun2_monitor_0 on ebony May 20 12:40:14 finley pengine: [25839]: info: process_pe_message: Transition 3601: PEngine Input stored in: /var/lib/pengine/pe-input-4760.bz2 May 20 12:40:15 finley crmd: [25840]: WARN: status_from_rc: Action 6 (iscsi0_target_monitor_0) on ebony failed (target: 7 vs. rc: 5): Error May 20 12:40:15 finley crmd: [25840]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_target_monitor_0, magic=0:5;6:3601:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2142.3) : Event failed May 20 12:40:15 finley crmd: [25840]: info: update_abort_priority: Abort priority upgraded from 0 to 1 May 20 12:40:15 finley crmd: [25840]: info: update_abort_priority: Abort action done superceeded by restart May 20 12:40:15 finley crmd: [25840]: info: match_graph_event: Action iscsi0_target_monitor_0 (6) confirmed on ebony (rc=4) May 20 12:40:16 finley crmd: [25840]: WARN: status_from_rc: Action 8 (iscsi0_lun1_monitor_0) on ebony failed (target: 7 vs. rc: 5): Error May 20 12:40:16 finley crmd: [25840]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_lun1_monitor_0, magic=0:5;8:3601:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2142.4) : Event failed May 20 12:40:16 finley crmd: [25840]: info: match_graph_event: Action iscsi0_lun1_monitor_0 (8) confirmed on ebony (rc=4) May 20 12:40:16 finley crmd: [25840]: WARN: status_from_rc: Action 7 (iscsi0_lun0_monitor_0) on ebony failed (target: 7 vs. rc: 5): Error May 20 12:40:16 finley crmd: [25840]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_lun0_monitor_0, magic=0:5;7:3601:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2142.5) : Event failed May 20 12:40:16 finley crmd: [25840]: info: match_graph_event: Action iscsi0_lun0_monitor_0 (7) confirmed on ebony (rc=4) May 20 12:40:16 finley crmd: [25840]: WARN: status_from_rc: Action 9 (iscsi0_lun2_monitor_0) on ebony failed (target: 7 vs. rc: 5): Error May 20 12:40:16 finley crmd: [25840]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=iscsi0_lun2_monitor_0, magic=0:5;9:3601:7:8f399f14-fe26-4279-b701-1947eb7b322d, cib=0.2142.6) : Event failed May 20 12:40:16 finley crmd: [25840]: info: match_graph_event: Action iscsi0_lun2_monitor_0 (9) confirmed on ebony (rc=4) May 20 12:40:16 finley crmd: [25840]: info: te_rsc_command: Initiating action 5: probe_complete probe_complete on ebony - no waiting May 20 12:40:16 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:40:16 finley crmd: [25840]: notice: run_graph: Transition 3601 (Complete=5, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-4760.bz2): Stopped May 20 12:40:16 finley crmd: [25840]: info: te_graph_trigger: Transition 3601 is now complete May 20 12:40:16 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:40:16 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:40:16 finley crmd: [25840]: info: do_pe_invoke: Query 5814: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:16 finley crmd: [25840]: info: do_pe_invoke_callback: Invoking the PE: query=5814, ref=pe_calc-dc-1305888016-6291, seq=1744, quorate=1 May 20 12:40:16 finley pengine: [25839]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 May 20 12:40:16 finley pengine: [25839]: info: unpack_status: Node godfrey is in standby-mode May 20 12:40:16 finley pengine: [25839]: info: determine_online_status: Node godfrey is standby May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on godfrey May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on godfrey May 20 12:40:16 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:1_stop_0 on godfrey: not configured (6) May 20 12:40:16 finley pengine: [25839]: info: native_add_running: resource drbd_r1:1 isnt managed May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on godfrey May 20 12:40:16 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:1_stop_0 on godfrey: not configured (6) May 20 12:40:16 finley pengine: [25839]: info: native_add_running: resource drbd_r0:1 isnt managed May 20 12:40:16 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:1_stop_0 on godfrey: not configured (6) May 20 12:40:16 finley pengine: [25839]: info: native_add_running: resource drbd_r2:1 isnt managed May 20 12:40:16 finley pengine: [25839]: info: unpack_status: Node finley is in standby-mode May 20 12:40:16 finley pengine: [25839]: info: determine_online_status: Node finley is standby May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on finley May 20 12:40:16 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_demote_0 on finley: not configured (6) May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r1:0 to stop after a failed demote action May 20 12:40:16 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_stop_0 on finley: not configured (6) May 20 12:40:16 finley pengine: [25839]: info: native_add_running: resource drbd_r1:0 isnt managed May 20 12:40:16 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r1:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on finley May 20 12:40:16 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_demote_0 on finley: not configured (6) May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r0:0 to stop after a failed demote action May 20 12:40:16 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_stop_0 on finley: not configured (6) May 20 12:40:16 finley pengine: [25839]: info: native_add_running: resource drbd_r0:0 isnt managed May 20 12:40:16 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r0:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:16 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:1 on finley to drbd_r1:3 (ORPHAN) May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:40:16 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:40:16 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:16 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:40:16 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:40:16 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:16 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:40:16 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:40:16 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:40:16 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:40:16 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:40:16 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:40:16 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_target_monitor_0 failed with rc=5: Preventing iscsi0_target from re-starting on ebony May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun1_monitor_0 failed with rc=5: Preventing iscsi0_lun1 from re-starting on ebony May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun0_monitor_0 failed with rc=5: Preventing iscsi0_lun0 from re-starting on ebony May 20 12:40:16 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun2_monitor_0 failed with rc=5: Preventing iscsi0_lun2 from re-starting on ebony May 20 12:40:16 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:40:16 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:40:16 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:40:16 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:16 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:16 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:16 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:40:16 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:16 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:16 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:40:16 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:40:16 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:16 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:16 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:40:16 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:40:16 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:16 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:16 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:40:16 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:40:16 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:0 ping_nodes:1 ping_nodes:2 ] May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:16 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:16 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:40:16 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:40:16 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:40:16 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:16 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:16 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:40:16 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:40:16 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:40:16 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:16 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:16 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:40:16 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:40:16 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:40:16 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:16 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:16 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:40:16 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:40:16 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:40:16 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:40:16 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:16 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:16 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource ping_nodes:0 cannot run anywhere May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource ping_nodes:1 cannot run anywhere May 20 12:40:16 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:16 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:0#011(Stopped) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:1#011(Stopped) May 20 12:40:16 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:40:16 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:40:16 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3602: 0 actions in 0 synapses May 20 12:40:16 finley crmd: [25840]: info: do_te_invoke: Processing graph 3602 (ref=pe_calc-dc-1305888016-6291) derived from /var/lib/pengine/pe-input-4761.bz2 May 20 12:40:16 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:40:16 finley crmd: [25840]: notice: run_graph: Transition 3602 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-4761.bz2): Complete May 20 12:40:16 finley crmd: [25840]: info: te_graph_trigger: Transition 3602 is now complete May 20 12:40:16 finley crmd: [25840]: info: notify_crmd: Transition 3602 status: done - May 20 12:40:16 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:40:16 finley crmd: [25840]: info: do_state_transition: Starting PEngine Recheck Timer May 20 12:40:16 finley pengine: [25839]: info: process_pe_message: Transition 3602: PEngine Input stored in: /var/lib/pengine/pe-input-4761.bz2 May 20 12:40:45 finley crm_attribute: [24087]: info: Invoked: crm_attribute -N finley -n standby -v off -l forever May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:40:45 finley crmd: [25840]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley crmd: [25840]: info: need_abort: Aborting on change to admin_epoch May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley crmd: [25840]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley crmd: [25840]: info: do_pe_invoke: Query 5815: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:40:45 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crm_attribute/4, version=0.2143.1): ok (rc=0) May 20 12:40:45 finley cib: [24088]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-53.raw May 20 12:40:45 finley cib: [24088]: info: write_cib_contents: Wrote version 0.2143.0 of the CIB to disk (digest: 5f4e84be521476dfe79cedbcb9b352d2) May 20 12:40:45 finley cib: [24088]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.VilQKK (digest: /var/lib/heartbeat/crm/cib.HgqJEJ) May 20 12:40:45 finley crmd: [25840]: info: do_pe_invoke_callback: Invoking the PE: query=5815, ref=pe_calc-dc-1305888045-6292, seq=1744, quorate=1 May 20 12:40:45 finley pengine: [25839]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 May 20 12:40:45 finley pengine: [25839]: info: unpack_status: Node godfrey is in standby-mode May 20 12:40:45 finley pengine: [25839]: info: determine_online_status: Node godfrey is standby May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on godfrey May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on godfrey May 20 12:40:45 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:1_stop_0 on godfrey: not configured (6) May 20 12:40:45 finley pengine: [25839]: info: native_add_running: resource drbd_r1:1 isnt managed May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on godfrey May 20 12:40:45 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:1_stop_0 on godfrey: not configured (6) May 20 12:40:45 finley pengine: [25839]: info: native_add_running: resource drbd_r0:1 isnt managed May 20 12:40:45 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:1_stop_0 on godfrey: not configured (6) May 20 12:40:45 finley pengine: [25839]: info: native_add_running: resource drbd_r2:1 isnt managed May 20 12:40:45 finley pengine: [25839]: info: determine_online_status: Node finley is online May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on finley May 20 12:40:45 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_demote_0 on finley: not configured (6) May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r1:0 to stop after a failed demote action May 20 12:40:45 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_stop_0 on finley: not configured (6) May 20 12:40:45 finley pengine: [25839]: info: native_add_running: resource drbd_r1:0 isnt managed May 20 12:40:45 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r1:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on finley May 20 12:40:45 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_demote_0 on finley: not configured (6) May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r0:0 to stop after a failed demote action May 20 12:40:45 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_stop_0 on finley: not configured (6) May 20 12:40:45 finley pengine: [25839]: info: native_add_running: resource drbd_r0:0 isnt managed May 20 12:40:45 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r0:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:45 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:1 on finley to drbd_r1:3 (ORPHAN) May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:40:45 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:40:45 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:45 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:40:45 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:40:45 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:45 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:40:45 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:40:45 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:40:45 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:40:45 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:40:45 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:40:45 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_target_monitor_0 failed with rc=5: Preventing iscsi0_target from re-starting on ebony May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun1_monitor_0 failed with rc=5: Preventing iscsi0_lun1 from re-starting on ebony May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun0_monitor_0 failed with rc=5: Preventing iscsi0_lun0 from re-starting on ebony May 20 12:40:45 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun2_monitor_0 failed with rc=5: Preventing iscsi0_lun2 from re-starting on ebony May 20 12:40:45 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:40:45 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:40:45 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:40:45 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:45 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:45 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:45 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:40:45 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:45 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:45 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:40:45 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:40:45 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:45 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:45 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:40:45 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:40:45 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:45 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:45 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:40:45 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:40:45 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:0 ping_nodes:1 ping_nodes:2 ] May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:45 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:45 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:45 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:40:45 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:40:45 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:40:45 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:45 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:45 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:45 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:40:45 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:40:45 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:40:45 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:45 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:45 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:45 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:40:45 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:40:45 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:40:45 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:45 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:45 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:40:45 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:40:45 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:40:45 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:40:45 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:40:45 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:40:45 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:40:45 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:40:45 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:40:45 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:45 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:45 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:45 finley pengine: [25839]: info: native_color: Resource ping_nodes:1 cannot run anywhere May 20 12:40:45 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:45 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:45 finley pengine: [25839]: notice: RecurringOp: Start recurring monitor (1s) for ping_nodes:0 on finley May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Start ping_nodes:0#011(finley) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:1#011(Stopped) May 20 12:40:45 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:40:45 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:40:45 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3603: 4 actions in 4 synapses May 20 12:40:45 finley crmd: [25840]: info: do_te_invoke: Processing graph 3603 (ref=pe_calc-dc-1305888045-6292) derived from /var/lib/pengine/pe-input-4762.bz2 May 20 12:40:45 finley crmd: [25840]: info: te_pseudo_action: Pseudo action 102 fired and confirmed May 20 12:40:45 finley crmd: [25840]: info: te_rsc_command: Initiating action 100: start ping_nodes:0_start_0 on finley (local) May 20 12:40:45 finley crmd: [25840]: info: do_lrm_rsc_op: Performing key=100:3603:0:8f399f14-fe26-4279-b701-1947eb7b322d op=ping_nodes:0_start_0 ) May 20 12:40:45 finley lrmd: [25837]: info: rsc:ping_nodes:0:1155: start May 20 12:40:45 finley pengine: [25839]: info: process_pe_message: Transition 3603: PEngine Input stored in: /var/lib/pengine/pe-input-4762.bz2 May 20 12:40:46 finley crmd: [25840]: info: process_lrm_event: LRM operation ping_nodes:0_start_0 (call=1155, rc=0, cib-update=5816, confirmed=true) ok May 20 12:40:46 finley crmd: [25840]: info: match_graph_event: Action ping_nodes:0_start_0 (100) confirmed on finley (rc=0) May 20 12:40:46 finley crmd: [25840]: info: te_rsc_command: Initiating action 101: monitor ping_nodes:0_monitor_1000 on finley (local) May 20 12:40:46 finley crmd: [25840]: info: do_lrm_rsc_op: Performing key=101:3603:0:8f399f14-fe26-4279-b701-1947eb7b322d op=ping_nodes:0_monitor_1000 ) May 20 12:40:46 finley lrmd: [25837]: info: rsc:ping_nodes:0:1156: monitor May 20 12:40:46 finley crmd: [25840]: info: te_pseudo_action: Pseudo action 103 fired and confirmed May 20 12:40:47 finley attrd: [25838]: info: attrd_trigger_update: Sending flush op to all hosts for: pingd (20) May 20 12:40:47 finley attrd: [25838]: info: attrd_perform_update: Sent update 4084: pingd=20 May 20 12:40:47 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=finley, magic=NA, cib=0.2143.3) : Transient attribute: update May 20 12:40:47 finley crmd: [25840]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 May 20 12:40:47 finley crmd: [25840]: info: update_abort_priority: Abort action done superceeded by restart May 20 12:40:47 finley crmd: [25840]: info: process_lrm_event: LRM operation ping_nodes:0_monitor_1000 (call=1156, rc=0, cib-update=5817, confirmed=false) ok May 20 12:40:47 finley crmd: [25840]: info: match_graph_event: Action ping_nodes:0_monitor_1000 (101) confirmed on finley (rc=0) May 20 12:40:47 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:40:47 finley crmd: [25840]: notice: run_graph: Transition 3603 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-4762.bz2): Complete May 20 12:40:47 finley crmd: [25840]: info: te_graph_trigger: Transition 3603 is now complete May 20 12:40:47 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:40:47 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:40:47 finley crmd: [25840]: info: do_pe_invoke: Query 5818: Requesting the current CIB: S_POLICY_ENGINE May 20 12:40:48 finley crmd: [25840]: info: do_pe_invoke_callback: Invoking the PE: query=5818, ref=pe_calc-dc-1305888048-6295, seq=1744, quorate=1 May 20 12:40:48 finley pengine: [25839]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 May 20 12:40:48 finley pengine: [25839]: info: unpack_status: Node godfrey is in standby-mode May 20 12:40:48 finley pengine: [25839]: info: determine_online_status: Node godfrey is standby May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on godfrey May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on godfrey May 20 12:40:48 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:1_stop_0 on godfrey: not configured (6) May 20 12:40:48 finley pengine: [25839]: info: native_add_running: resource drbd_r1:1 isnt managed May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on godfrey May 20 12:40:48 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:1_stop_0 on godfrey: not configured (6) May 20 12:40:48 finley pengine: [25839]: info: native_add_running: resource drbd_r0:1 isnt managed May 20 12:40:48 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:1_stop_0 on godfrey: not configured (6) May 20 12:40:48 finley pengine: [25839]: info: native_add_running: resource drbd_r2:1 isnt managed May 20 12:40:48 finley pengine: [25839]: info: determine_online_status: Node finley is online May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on finley May 20 12:40:48 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_demote_0 on finley: not configured (6) May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r1:0 to stop after a failed demote action May 20 12:40:48 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_stop_0 on finley: not configured (6) May 20 12:40:48 finley pengine: [25839]: info: native_add_running: resource drbd_r1:0 isnt managed May 20 12:40:48 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r1:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on finley May 20 12:40:48 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_demote_0 on finley: not configured (6) May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r0:0 to stop after a failed demote action May 20 12:40:48 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_stop_0 on finley: not configured (6) May 20 12:40:48 finley pengine: [25839]: info: native_add_running: resource drbd_r0:0 isnt managed May 20 12:40:48 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r0:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:48 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:1 on finley to drbd_r1:3 (ORPHAN) May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:40:48 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:40:48 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:40:48 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:40:48 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:40:48 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:40:48 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:40:48 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:40:48 finley pengine: [25839]: info: find_clone: Internally renamed ping_nodes:1 on finley to ping_nodes:3 (ORPHAN) May 20 12:40:48 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:40:48 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:40:48 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:40:48 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:40:48 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_target_monitor_0 failed with rc=5: Preventing iscsi0_target from re-starting on ebony May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun1_monitor_0 failed with rc=5: Preventing iscsi0_lun1 from re-starting on ebony May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun0_monitor_0 failed with rc=5: Preventing iscsi0_lun0 from re-starting on ebony May 20 12:40:48 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun2_monitor_0 failed with rc=5: Preventing iscsi0_lun2 from re-starting on ebony May 20 12:40:48 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:40:48 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:40:48 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:40:48 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:48 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:48 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:40:48 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:40:48 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:48 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:48 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:40:48 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:40:48 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:48 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:48 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:40:48 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:40:48 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:40:48 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:40:48 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:40:48 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:40:48 finley pengine: [25839]: notice: short_print: Started: [ finley ] May 20 12:40:48 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:1 ping_nodes:2 ] May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:40:48 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:40:48 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:48 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:40:48 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:40:48 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:40:48 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:40:48 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:48 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:48 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:40:48 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:40:48 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:40:48 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:40:48 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:48 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:48 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:40:48 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:40:48 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:40:48 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:40:48 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:48 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:40:48 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:40:48 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:40:48 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:40:48 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:40:48 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:40:48 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:40:48 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:40:48 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:40:48 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:40:48 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:40:48 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:40:48 finley pengine: [25839]: info: native_color: Resource ping_nodes:1 cannot run anywhere May 20 12:40:48 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:48 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:0#011(Started finley) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:1#011(Stopped) May 20 12:40:48 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:40:48 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:40:48 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3604: 0 actions in 0 synapses May 20 12:40:48 finley crmd: [25840]: info: do_te_invoke: Processing graph 3604 (ref=pe_calc-dc-1305888048-6295) derived from /var/lib/pengine/pe-input-4763.bz2 May 20 12:40:48 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:40:48 finley crmd: [25840]: notice: run_graph: Transition 3604 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-4763.bz2): Complete May 20 12:40:48 finley crmd: [25840]: info: te_graph_trigger: Transition 3604 is now complete May 20 12:40:48 finley crmd: [25840]: info: notify_crmd: Transition 3604 status: done - May 20 12:40:48 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:40:48 finley crmd: [25840]: info: do_state_transition: Starting PEngine Recheck Timer May 20 12:40:48 finley pengine: [25839]: info: process_pe_message: Transition 3604: PEngine Input stored in: /var/lib/pengine/pe-input-4763.bz2 May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: - May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley crmd: [25840]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change May 20 12:41:01 finley crmd: [25840]: info: need_abort: Aborting on change to admin_epoch May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley crmd: [25840]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley crmd: [25840]: info: do_pe_invoke: Query 5819: Requesting the current CIB: S_POLICY_ENGINE May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley cib: [25836]: info: log_data_element: cib:diff: + May 20 12:41:01 finley cib: [25836]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=godfrey/crm_attribute/4, version=0.2144.1): ok (rc=0) May 20 12:41:01 finley cib: [24205]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-54.raw May 20 12:41:01 finley cib: [24205]: info: write_cib_contents: Wrote version 0.2144.0 of the CIB to disk (digest: de656a39b11c05f2ecffafea852a2d51) May 20 12:41:01 finley cib: [24205]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.goj1dd (digest: /var/lib/heartbeat/crm/cib.Bzp5AE) May 20 12:41:01 finley crmd: [25840]: info: do_pe_invoke_callback: Invoking the PE: query=5819, ref=pe_calc-dc-1305888061-6296, seq=1744, quorate=1 May 20 12:41:01 finley pengine: [25839]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 May 20 12:41:01 finley pengine: [25839]: info: determine_online_status: Node godfrey is online May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on godfrey May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on godfrey May 20 12:41:01 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:1_stop_0 on godfrey: not configured (6) May 20 12:41:01 finley pengine: [25839]: info: native_add_running: resource drbd_r1:1 isnt managed May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on godfrey May 20 12:41:01 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:1_stop_0 on godfrey: not configured (6) May 20 12:41:01 finley pengine: [25839]: info: native_add_running: resource drbd_r0:1 isnt managed May 20 12:41:01 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:1_stop_0 on godfrey: not configured (6) May 20 12:41:01 finley pengine: [25839]: info: native_add_running: resource drbd_r2:1 isnt managed May 20 12:41:01 finley pengine: [25839]: info: determine_online_status: Node finley is online May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on finley May 20 12:41:01 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_demote_0 on finley: not configured (6) May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r1:0 to stop after a failed demote action May 20 12:41:01 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_stop_0 on finley: not configured (6) May 20 12:41:01 finley pengine: [25839]: info: native_add_running: resource drbd_r1:0 isnt managed May 20 12:41:01 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r1:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on finley May 20 12:41:01 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_demote_0 on finley: not configured (6) May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r0:0 to stop after a failed demote action May 20 12:41:01 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_stop_0 on finley: not configured (6) May 20 12:41:01 finley pengine: [25839]: info: native_add_running: resource drbd_r0:0 isnt managed May 20 12:41:01 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r0:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:41:01 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:1 on finley to drbd_r1:3 (ORPHAN) May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:41:01 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:41:01 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:41:01 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:41:01 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:41:01 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:41:01 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:41:01 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:41:01 finley pengine: [25839]: info: find_clone: Internally renamed ping_nodes:1 on finley to ping_nodes:3 (ORPHAN) May 20 12:41:01 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:41:01 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:41:01 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:41:01 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:41:01 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_target_monitor_0 failed with rc=5: Preventing iscsi0_target from re-starting on ebony May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun1_monitor_0 failed with rc=5: Preventing iscsi0_lun1 from re-starting on ebony May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun0_monitor_0 failed with rc=5: Preventing iscsi0_lun0 from re-starting on ebony May 20 12:41:01 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun2_monitor_0 failed with rc=5: Preventing iscsi0_lun2 from re-starting on ebony May 20 12:41:01 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:41:01 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:41:01 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:41:01 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:41:01 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:41:01 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:41:01 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:41:01 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:41:01 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:41:01 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:41:01 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:41:01 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:41:01 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:41:01 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:41:01 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:41:01 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:41:01 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:41:01 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:41:01 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:41:01 finley pengine: [25839]: notice: short_print: Started: [ finley ] May 20 12:41:01 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:1 ping_nodes:2 ] May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:41:01 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:41:01 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:41:01 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:41:01 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:41:01 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:41:01 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:41:01 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:41:01 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:41:01 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:41:01 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:41:01 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:41:01 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:41:01 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:41:01 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:41:01 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:41:01 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:41:01 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:41:01 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:41:01 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:41:01 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:41:01 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:41:01 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:41:01 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:41:01 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:41:01 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:41:01 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:41:01 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:41:01 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:41:01 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:41:01 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:41:01 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:41:01 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:41:01 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:41:01 finley pengine: [25839]: notice: RecurringOp: Start recurring monitor (1s) for ping_nodes:1 on godfrey May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:0#011(Started finley) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Start ping_nodes:1#011(godfrey) May 20 12:41:01 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:41:01 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:41:01 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3605: 4 actions in 4 synapses May 20 12:41:01 finley crmd: [25840]: info: do_te_invoke: Processing graph 3605 (ref=pe_calc-dc-1305888061-6296) derived from /var/lib/pengine/pe-input-4764.bz2 May 20 12:41:01 finley pengine: [25839]: info: process_pe_message: Transition 3605: PEngine Input stored in: /var/lib/pengine/pe-input-4764.bz2 May 20 12:41:01 finley crmd: [25840]: info: te_pseudo_action: Pseudo action 105 fired and confirmed May 20 12:41:01 finley crmd: [25840]: info: te_rsc_command: Initiating action 103: start ping_nodes:1_start_0 on godfrey May 20 12:41:02 finley crmd: [25840]: info: match_graph_event: Action ping_nodes:1_start_0 (103) confirmed on godfrey (rc=0) May 20 12:41:02 finley crmd: [25840]: info: te_rsc_command: Initiating action 104: monitor ping_nodes:1_monitor_1000 on godfrey May 20 12:41:02 finley crmd: [25840]: info: te_pseudo_action: Pseudo action 106 fired and confirmed May 20 12:41:03 finley crmd: [25840]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=godfrey, magic=NA, cib=0.2144.3) : Transient attribute: update May 20 12:41:03 finley crmd: [25840]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 May 20 12:41:03 finley crmd: [25840]: info: update_abort_priority: Abort action done superceeded by restart May 20 12:41:03 finley crmd: [25840]: info: match_graph_event: Action ping_nodes:1_monitor_1000 (104) confirmed on godfrey (rc=0) May 20 12:41:03 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:41:03 finley crmd: [25840]: notice: run_graph: Transition 3605 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-4764.bz2): Complete May 20 12:41:03 finley crmd: [25840]: info: te_graph_trigger: Transition 3605 is now complete May 20 12:41:03 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:41:03 finley crmd: [25840]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. May 20 12:41:03 finley crmd: [25840]: info: do_pe_invoke: Query 5820: Requesting the current CIB: S_POLICY_ENGINE May 20 12:41:03 finley crmd: [25840]: info: do_pe_invoke_callback: Invoking the PE: query=5820, ref=pe_calc-dc-1305888063-6299, seq=1744, quorate=1 May 20 12:41:03 finley pengine: [25839]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 May 20 12:41:03 finley pengine: [25839]: info: determine_online_status: Node godfrey is online May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on godfrey May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on godfrey May 20 12:41:03 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:1_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:1_stop_0 on godfrey: not configured (6) May 20 12:41:03 finley pengine: [25839]: info: native_add_running: resource drbd_r1:1 isnt managed May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on godfrey May 20 12:41:03 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:1_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:1_stop_0 on godfrey: not configured (6) May 20 12:41:03 finley pengine: [25839]: info: native_add_running: resource drbd_r0:1 isnt managed May 20 12:41:03 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:1_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:1_stop_0 on godfrey: not configured (6) May 20 12:41:03 finley pengine: [25839]: info: native_add_running: resource drbd_r2:1 isnt managed May 20 12:41:03 finley pengine: [25839]: info: determine_online_status: Node finley is online May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r1:0_monitor_0 found resource drbd_r1:0 active on finley May 20 12:41:03 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_demote_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_demote_0 on finley: not configured (6) May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r1:0 to stop after a failed demote action May 20 12:41:03 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r1:0_stop_0 failed with rc=6: Preventing ms_drbd_r1 from re-starting anywhere in the cluster May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r1:0_stop_0 on finley: not configured (6) May 20 12:41:03 finley pengine: [25839]: info: native_add_running: resource drbd_r1:0 isnt managed May 20 12:41:03 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r1:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r0:0_monitor_0 found resource drbd_r0:0 active on finley May 20 12:41:03 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_demote_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_demote_0 on finley: not configured (6) May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r0:0 to stop after a failed demote action May 20 12:41:03 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r0:0_stop_0 failed with rc=6: Preventing ms_drbd_r0 from re-starting anywhere in the cluster May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r0:0_stop_0 on finley: not configured (6) May 20 12:41:03 finley pengine: [25839]: info: native_add_running: resource drbd_r0:0 isnt managed May 20 12:41:03 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r0:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:41:03 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:1 on finley to drbd_r1:3 (ORPHAN) May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Operation drbd_r2:0_monitor_0 found resource drbd_r2:0 active on finley May 20 12:41:03 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_demote_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_demote_0 on finley: not configured (6) May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Forcing drbd_r2:0 to stop after a failed demote action May 20 12:41:03 finley pengine: [25839]: ERROR: unpack_rsc_op: Hard error - drbd_r2:0_stop_0 failed with rc=6: Preventing ms_drbd_r2 from re-starting anywhere in the cluster May 20 12:41:03 finley pengine: [25839]: WARN: unpack_rsc_op: Processing failed op drbd_r2:0_stop_0 on finley: not configured (6) May 20 12:41:03 finley pengine: [25839]: info: native_add_running: resource drbd_r2:0 isnt managed May 20 12:41:03 finley pengine: [25839]: info: unpack_lrm_rsc_state: drbd_r2:0: Not overwriting calculated next role Stopped with requested next role Master May 20 12:41:03 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:1 on finley to drbd_r0:3 (ORPHAN) May 20 12:41:03 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:1 on finley to drbd_r2:3 (ORPHAN) May 20 12:41:03 finley pengine: [25839]: info: find_clone: Internally renamed ping_nodes:1 on finley to ping_nodes:3 (ORPHAN) May 20 12:41:03 finley pengine: [25839]: info: unpack_status: Node ebony is in standby-mode May 20 12:41:03 finley pengine: [25839]: info: determine_online_status: Node ebony is standby May 20 12:41:03 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r1:0 on ebony to drbd_r1:2 May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r1:0_monitor_0 failed with rc=5: Preventing ms_drbd_r1 from re-starting on ebony May 20 12:41:03 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r2:0 on ebony to drbd_r2:2 May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r2:0_monitor_0 failed with rc=5: Preventing ms_drbd_r2 from re-starting on ebony May 20 12:41:03 finley pengine: [25839]: info: find_clone: Internally renamed drbd_r0:0 on ebony to drbd_r0:2 May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - drbd_r0:0_monitor_0 failed with rc=5: Preventing ms_drbd_r0 from re-starting on ebony May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_target_monitor_0 failed with rc=5: Preventing iscsi0_target from re-starting on ebony May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun1_monitor_0 failed with rc=5: Preventing iscsi0_lun1 from re-starting on ebony May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun0_monitor_0 failed with rc=5: Preventing iscsi0_lun0 from re-starting on ebony May 20 12:41:03 finley pengine: [25839]: notice: unpack_rsc_op: Hard error - iscsi0_lun2_monitor_0 failed with rc=5: Preventing iscsi0_lun2 from re-starting on ebony May 20 12:41:03 finley pengine: [25839]: notice: group_print: Resource Group: rg_iscsi May 20 12:41:03 finley pengine: [25839]: notice: native_print: ip_virtual0#011(ocf::heartbeat:IPaddr2):#011Stopped May 20 12:41:03 finley pengine: [25839]: notice: native_print: iscsi0_target#011(ocf::heartbeat:iSCSITarget):#011Stopped May 20 12:41:03 finley pengine: [25839]: notice: native_print: iscsi0_lun0#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:41:03 finley pengine: [25839]: notice: native_print: iscsi0_lun1#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:41:03 finley pengine: [25839]: notice: native_print: iscsi0_lun2#011(ocf::heartbeat:iSCSILogicalUnit):#011Stopped May 20 12:41:03 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r0 May 20 12:41:03 finley pengine: [25839]: notice: native_print: drbd_r0:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:41:03 finley pengine: [25839]: notice: native_print: drbd_r0:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:41:03 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r0:2 ] May 20 12:41:03 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r1 May 20 12:41:03 finley pengine: [25839]: notice: native_print: drbd_r1:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:41:03 finley pengine: [25839]: notice: native_print: drbd_r1:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:41:03 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r1:2 ] May 20 12:41:03 finley pengine: [25839]: notice: clone_print: Master/Slave Set: ms_drbd_r2 May 20 12:41:03 finley pengine: [25839]: notice: native_print: drbd_r2:0#011(ocf::linbit:drbd):#011Slave finley (unmanaged) FAILED May 20 12:41:03 finley pengine: [25839]: notice: native_print: drbd_r2:1#011(ocf::linbit:drbd):#011Slave godfrey (unmanaged) FAILED May 20 12:41:03 finley pengine: [25839]: notice: short_print: Stopped: [ drbd_r2:2 ] May 20 12:41:03 finley pengine: [25839]: notice: clone_print: Clone Set: ping_clone May 20 12:41:03 finley pengine: [25839]: notice: short_print: Started: [ finley godfrey ] May 20 12:41:03 finley pengine: [25839]: notice: short_print: Stopped: [ ping_nodes:2 ] May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on godfrey May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from godfrey after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r0 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r0 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r1 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r1 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: get_failcount: ms_drbd_r2 has failed INFINITY times on finley May 20 12:41:03 finley pengine: [25839]: WARN: common_apply_stickiness: Forcing ms_drbd_r2 away from finley after 1000000 failures (max=1000000) May 20 12:41:03 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:41:03 finley pengine: [25839]: info: native_color: Resource drbd_r0:2 cannot run anywhere May 20 12:41:03 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:0 allocated to 'nowhere': failed May 20 12:41:03 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r0:1 allocated to 'nowhere': failed May 20 12:41:03 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r0: Rolling back scores from ip_virtual0 May 20 12:41:03 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:41:03 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:41:03 finley pengine: [25839]: info: native_color: Resource drbd_r1:2 cannot run anywhere May 20 12:41:03 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:0 allocated to 'nowhere': failed May 20 12:41:03 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r1:1 allocated to 'nowhere': failed May 20 12:41:03 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r1: Rolling back scores from ip_virtual0 May 20 12:41:03 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:41:03 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:41:03 finley pengine: [25839]: info: native_color: Resource drbd_r2:2 cannot run anywhere May 20 12:41:03 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:0 allocated to 'nowhere': failed May 20 12:41:03 finley pengine: [25839]: info: native_color: Unmanaged resource drbd_r2:1 allocated to 'nowhere': failed May 20 12:41:03 finley pengine: [25839]: info: native_merge_weights: ms_drbd_r2: Rolling back scores from ip_virtual0 May 20 12:41:03 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:41:03 finley pengine: [25839]: info: native_merge_weights: ip_virtual0: Rolling back scores from iscsi0_target May 20 12:41:03 finley pengine: [25839]: info: native_color: Resource ip_virtual0 cannot run anywhere May 20 12:41:03 finley pengine: [25839]: info: native_merge_weights: iscsi0_target: Rolling back scores from iscsi0_lun0 May 20 12:41:03 finley pengine: [25839]: info: native_color: Resource iscsi0_target cannot run anywhere May 20 12:41:03 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun0: Rolling back scores from iscsi0_lun1 May 20 12:41:03 finley pengine: [25839]: info: native_color: Resource iscsi0_lun0 cannot run anywhere May 20 12:41:03 finley pengine: [25839]: info: native_merge_weights: iscsi0_lun1: Rolling back scores from iscsi0_lun2 May 20 12:41:03 finley pengine: [25839]: info: native_color: Resource iscsi0_lun1 cannot run anywhere May 20 12:41:03 finley pengine: [25839]: info: native_color: Resource iscsi0_lun2 cannot run anywhere May 20 12:41:03 finley pengine: [25839]: info: master_color: ms_drbd_r0: Promoted 0 instances of a possible 1 to master May 20 12:41:03 finley pengine: [25839]: info: master_color: ms_drbd_r1: Promoted 0 instances of a possible 1 to master May 20 12:41:03 finley pengine: [25839]: info: master_color: ms_drbd_r2: Promoted 0 instances of a possible 1 to master May 20 12:41:03 finley pengine: [25839]: info: native_color: Resource ping_nodes:2 cannot run anywhere May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r0 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r1 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:41:03 finley pengine: [25839]: ERROR: create_notification_boundaries: Creating boundaries for ms_drbd_r2 May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource ip_virtual0#011(Stopped) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_target#011(Stopped) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun0#011(Stopped) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun1#011(Stopped) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource iscsi0_lun2#011(Stopped) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:0#011(Slave unmanaged) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:1#011(Slave unmanaged) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r0:2#011(Stopped) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:0#011(Slave unmanaged) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:1#011(Slave unmanaged) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r1:2#011(Stopped) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:0#011(Slave unmanaged) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:1#011(Slave unmanaged) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource drbd_r2:2#011(Stopped) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:0#011(Started finley) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:1#011(Started godfrey) May 20 12:41:03 finley pengine: [25839]: notice: LogActions: Leave resource ping_nodes:2#011(Stopped) May 20 12:41:03 finley crmd: [25840]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] May 20 12:41:03 finley crmd: [25840]: info: unpack_graph: Unpacked transition 3606: 0 actions in 0 synapses May 20 12:41:03 finley crmd: [25840]: info: do_te_invoke: Processing graph 3606 (ref=pe_calc-dc-1305888063-6299) derived from /var/lib/pengine/pe-input-4765.bz2 May 20 12:41:03 finley crmd: [25840]: info: run_graph: ==================================================== May 20 12:41:03 finley crmd: [25840]: notice: run_graph: Transition 3606 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-4765.bz2): Complete May 20 12:41:03 finley crmd: [25840]: info: te_graph_trigger: Transition 3606 is now complete May 20 12:41:03 finley crmd: [25840]: info: notify_crmd: Transition 3606 status: done - May 20 12:41:03 finley crmd: [25840]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] May 20 12:41:03 finley crmd: [25840]: info: do_state_transition: Starting PEngine Recheck Timer May 20 12:41:03 finley pengine: [25839]: info: process_pe_message: Transition 3606: PEngine Input stored in: /var/lib/pengine/pe-input-4765.bz2 May 20 12:45:01 finley /USR/SBIN/CRON[25631]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) May 20 12:49:11 finley cib: [25836]: info: cib_stats: Processed 1022 operations (18864.00us average, 3% utilization) in the last 10min