Feb 8 12:58:32 server01 cibadmin: [15012]: info: Invoked: cibadmin -Ql -o nodes Feb 8 12:58:32 server01 cibadmin: [15014]: info: Invoked: cibadmin -Ql -o resources Feb 8 12:58:32 server01 cibadmin: [15016]: info: Invoked: cibadmin -Ql -o resources Feb 8 12:58:32 server01 crm_resource: [15018]: info: Invoked: crm_resource -C -r supervision-grp -H server01 Feb 8 12:58:32 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource fs-data from the LRM Feb 8 12:58:32 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'fs-data' deleted for 15018_crm_resource on server01 Feb 8 12:58:32 server01 crmd: [19470]: info: notify_deleted: Notifying 15018_crm_resource on server01 that fs-data was deleted Feb 8 12:58:32 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op fs-data_delete_60000 from 0:0:crm-resource-15018: lrm_invoke-lrmd-1328702312-302 Feb 8 12:58:32 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource nagios-ip from the LRM Feb 8 12:58:32 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'nagios-ip' deleted for 15018_crm_resource on server01 Feb 8 12:58:32 server01 crmd: [19470]: info: notify_deleted: Notifying 15018_crm_resource on server01 that nagios-ip was deleted Feb 8 12:58:32 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op nagios-ip_delete_60000 from 0:0:crm-resource-15018: lrm_invoke-lrmd-1328702312-303 Feb 8 12:58:32 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource apache2 from the LRM Feb 8 12:58:32 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'apache2' deleted for 15018_crm_resource on server01 Feb 8 12:58:32 server01 crmd: [19470]: info: notify_deleted: Notifying 15018_crm_resource on server01 that apache2 was deleted Feb 8 12:58:32 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op apache2_delete_60000 from 0:0:crm-resource-15018: lrm_invoke-lrmd-1328702312-304 Feb 8 12:58:56 server01 lrmd: [19467]: WARN: drbd-nagios:1:monitor process (PID 15019) timed out (try 1). Killing with signal SIGTERM (15). Feb 8 12:58:56 server01 lrmd: [19467]: WARN: operation monitor[90] on ocf::drbd::drbd-nagios:1 for client 19470, its parameters: CRM_meta_clone=[1] CRM_meta_timeout=[20000] CRM_meta_notify_slave_resource=[ ] CRM_meta_notify_active_resource=[ ] CRM_meta_notify_demote_uname=[ ] drbd_resource=[data] CRM_meta_notify_inactive_resource=[drbd-nagios:1 ] CRM_meta_notify_active_uname=[ ] CRM_meta_master_node_max=[1] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_master_resource=[drbd-nagios:0 ] CRM_meta_clone_node_max=[1] CRM_meta_notify=[true] CRM_meta_no: pid [15019] timed out Feb 8 12:59:04 server01 cib: [19466]: WARN: send_ipc_message: IPC Channel to 15045 is not connected Feb 8 12:59:04 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource nagios from the LRM Feb 8 12:59:04 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'nagios' deleted for 15018_crm_resource on server01 Feb 8 12:59:04 server01 crmd: [19470]: info: notify_deleted: Notifying 15018_crm_resource on server01 that nagios was deleted Feb 8 12:59:04 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op nagios_delete_60000 from 0:0:crm-resource-15018: lrm_invoke-lrmd-1328702344-305 Feb 8 12:59:04 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource pnp from the LRM Feb 8 12:59:04 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'pnp' deleted for 15018_crm_resource on server01 Feb 8 12:59:04 server01 crmd: [19470]: info: notify_deleted: Notifying 15018_crm_resource on server01 that pnp was deleted Feb 8 12:59:04 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op pnp_delete_60000 from 0:0:crm-resource-15018: lrm_invoke-lrmd-1328702344-306 Feb 8 12:59:04 server01 cibadmin: [15047]: info: Invoked: cibadmin -Ql -o resources Feb 8 12:59:05 server01 crmd: [19470]: ERROR: process_lrm_event: LRM operation drbd-nagios:1_monitor_15000 (90) Timed Out (timeout=20000ms) Feb 8 12:59:05 server01 cibadmin: [15049]: info: Invoked: cibadmin -Ql -o resources Feb 8 12:59:05 server01 crm_resource: [15051]: info: Invoked: crm_resource -C -r supervision-grp -H server02 Feb 8 12:59:08 server01 corosync[19441]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 124: memb=1, new=0, lost=1 Feb 8 12:59:08 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: memb: server01 929963018 Feb 8 12:59:08 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: lost: server02 1584274442 Feb 8 12:59:08 server01 corosync[19441]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 124: memb=2, new=1, lost=0 Feb 8 12:59:08 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: NEW: server02 1584274442 Feb 8 12:59:08 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: MEMB: server01 929963018 Feb 8 12:59:08 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: MEMB: server02 1584274442 Feb 8 12:59:08 server01 corosync[19441]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Feb 8 12:59:08 server01 corosync[19441]: [MAIN ] Completed service synchronization, ready to provide service. Feb 8 13:00:05 server01 crmd: [19470]: WARN: cib_rsc_callback: Resource update 415 failed: (rc=-41) Remote node did not respond Feb 8 13:00:05 server01 crm_resource: [15051]: ERROR: resource_ipc_timeout: No messages received in 60 seconds Feb 8 13:00:52 server01 cibadmin: [15605]: info: Invoked: cibadmin -Ql -o nodes Feb 8 13:00:52 server01 crm_resource: [15607]: info: Invoked: crm_resource -M -r supervision-grp --node=server01 Feb 8 13:01:25 server01 cib: [19466]: info: cib_stats: Processed 65 operations (506000.00us average, 5% utilization) in the last 10min Feb 8 13:03:19 server01 cibadmin: [15959]: info: Invoked: cibadmin -Ql Feb 8 13:03:21 server01 cibadmin: [15980]: info: Invoked: cibadmin -Ql Feb 8 13:05:04 server01 puppet-agent[22112]: Finished catalog run in 9.64 seconds Feb 8 13:06:33 server01 corosync[19441]: [TOTEM ] A processor failed, forming new configuration. Feb 8 13:06:36 server01 corosync[19441]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 128: memb=1, new=0, lost=1 Feb 8 13:06:36 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: memb: server01 929963018 Feb 8 13:06:36 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: lost: server02 1584274442 Feb 8 13:06:36 server01 corosync[19441]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 128: memb=1, new=0, lost=0 Feb 8 13:06:36 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: MEMB: server01 929963018 Feb 8 13:06:36 server01 corosync[19441]: [pcmk ] info: ais_mark_unseen_peer_dead: Node server02 was not seen in the previous transition Feb 8 13:06:36 server01 corosync[19441]: [pcmk ] info: update_member: Node 1584274442/server02 is now: lost Feb 8 13:06:36 server01 corosync[19441]: [pcmk ] info: send_member_notification: Sending membership update 128 to 2 children Feb 8 13:06:36 server01 crmd: [19470]: notice: ais_dispatch: Membership 128: quorum lost Feb 8 13:06:36 server01 crmd: [19470]: info: crm_update_peer: Node server02: id=1584274442 state=lost (new) addr=r(0) ip(10.28.110.94) votes=1 born=92 seen=108 proc=00000000000000000000000000013312 Feb 8 13:06:36 server01 crmd: [19470]: info: erase_node_from_join: Removed node server02 from join calculations: welcomed=1 itegrated=0 finalized=0 confirmed=0 Feb 8 13:06:36 server01 crmd: [19470]: WARN: check_dead_member: Our DC node (server02) left the cluster Feb 8 13:06:36 server01 crmd: [19470]: info: do_state_transition: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=check_dead_member ] Feb 8 13:06:36 server01 crmd: [19470]: info: update_dc: Unset DC server02 Feb 8 13:06:36 server01 crmd: [19470]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] Feb 8 13:06:36 server01 corosync[19441]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Feb 8 13:06:36 server01 corosync[19441]: [MAIN ] Completed service synchronization, ready to provide service. Feb 8 13:06:36 server01 cib: [19466]: notice: ais_dispatch: Membership 128: quorum lost Feb 8 13:06:36 server01 cib: [19466]: info: crm_update_peer: Node server02: id=1584274442 state=lost (new) addr=r(0) ip(10.28.110.94) votes=1 born=92 seen=108 proc=00000000000000000000000000013312 Feb 8 13:06:36 server01 crmd: [19470]: info: do_te_control: Registering TE UUID: ada70012-d57b-48f3-9bbb-9c35d4f020f1 Feb 8 13:06:36 server01 crmd: [19470]: WARN: cib_client_add_notify_callback: Callback already present Feb 8 13:06:36 server01 crmd: [19470]: info: set_graph_functions: Setting custom graph functions Feb 8 13:06:36 server01 crmd: [19470]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses Feb 8 13:06:36 server01 crmd: [19470]: info: do_dc_takeover: Taking over DC status for this partition Feb 8 13:06:36 server01 cib: [19466]: info: cib_process_readwrite: We are now in R/W mode Feb 8 13:06:36 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/417, version=0.127.43): ok (rc=0) Feb 8 13:06:36 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/418, version=0.127.43): ok (rc=0) Feb 8 13:06:36 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/420, version=0.127.43): ok (rc=0) Feb 8 13:06:36 server01 crmd: [19470]: info: join_make_offer: Making join offers based on membership 128 Feb 8 13:06:36 server01 crmd: [19470]: info: do_dc_join_offer_all: join-3: Waiting on 1 outstanding join acks Feb 8 13:06:36 server01 crmd: [19470]: info: ais_dispatch: Membership 128: quorum still lost Feb 8 13:06:36 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/422, version=0.127.43): ok (rc=0) Feb 8 13:06:36 server01 crmd: [19470]: info: crm_ais_dispatch: Setting expected votes to 2 Feb 8 13:06:36 server01 crmd: [19470]: info: config_query_callback: Checking for expired actions every 900000ms Feb 8 13:06:36 server01 crmd: [19470]: info: config_query_callback: Sending expected-votes=2 to corosync Feb 8 13:06:36 server01 crmd: [19470]: info: update_dc: Set DC to server01 (3.0.1) Feb 8 13:06:36 server01 crmd: [19470]: info: ais_dispatch: Membership 128: quorum still lost Feb 8 13:06:36 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/425, version=0.127.43): ok (rc=0) Feb 8 13:06:37 server01 crmd: [19470]: info: crm_ais_dispatch: Setting expected votes to 2 Feb 8 13:06:37 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/428, version=0.127.43): ok (rc=0) Feb 8 13:06:37 server01 crmd: [19470]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Feb 8 13:06:37 server01 crmd: [19470]: info: do_state_transition: All 1 cluster nodes responded to the join offer. Feb 8 13:06:37 server01 crmd: [19470]: info: do_dc_join_finalize: join-3: Syncing the CIB from server01 to the rest of the cluster Feb 8 13:06:37 server01 crmd: [19470]: info: te_connect_stonith: Attempting connection to fencing daemon... Feb 8 13:06:37 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/429, version=0.127.43): ok (rc=0) Feb 8 13:06:38 server01 crmd: [19470]: info: te_connect_stonith: Connected Feb 8 13:06:38 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/430, version=0.127.43): ok (rc=0) Feb 8 13:06:38 server01 crmd: [19470]: info: do_dc_join_ack: join-3: Updating node state to member for server01 Feb 8 13:06:38 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']/lrm (origin=local/crmd/431, version=0.127.44): ok (rc=0) Feb 8 13:06:38 server01 crmd: [19470]: info: erase_xpath_callback: Deletion of "//node_state[@uname='server01']/lrm": ok (rc=0) Feb 8 13:06:38 server01 crmd: [19470]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Feb 8 13:06:38 server01 crmd: [19470]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Feb 8 13:06:38 server01 crmd: [19470]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date Feb 8 13:06:38 server01 crmd: [19470]: info: crm_update_quorum: Updating quorum status to false (call=435) Feb 8 13:06:38 server01 crmd: [19470]: info: abort_transition_graph: do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled Feb 8 13:06:38 server01 crmd: [19470]: info: do_pe_invoke: Query 436: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_local_callback: Sending full refresh (origin=crmd) Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-nagios-ip () Feb 8 13:06:38 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/433, version=0.127.45): ok (rc=0) Feb 8 13:06:38 server01 crmd: [19470]: WARN: match_down_event: No match for shutdown action on server02 Feb 8 13:06:38 server01 crmd: [19470]: info: te_update_diff: Stonith/shutdown of server02 not matched Feb 8 13:06:38 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:191 - Triggered transition abort (complete=1, tag=node_state, id=server02, magic=NA, cib=0.127.46) : Node failure Feb 8 13:06:38 server01 crmd: [19470]: info: do_pe_invoke: Query 437: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:06:38 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:06:38 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:06:38 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/435, version=0.128.1): ok (rc=0) Feb 8 13:06:38 server01 crmd: [19470]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change Feb 8 13:06:38 server01 crmd: [19470]: info: need_abort: Aborting on change to have-quorum Feb 8 13:06:38 server01 crmd: [19470]: info: do_pe_invoke: Query 438: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd-nagios:0 () Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd-nagios:1 (10000) Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-apache2 () Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-nagios-ip (1326606464) Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-pnp () Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-pnp (1326606099) Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-nagios (1326606492) Feb 8 13:06:38 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=438, ref=pe_calc-dc-1328702798-314, seq=128, quorate=0 Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-fs-data () Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-apache2 (1326606483) Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-nagios () Feb 8 13:06:38 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Feb 8 13:06:38 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:06:38 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:06:38 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:06:38 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:06:38 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Stopped Feb 8 13:06:38 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:06:38 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Stopped Feb 8 13:06:38 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:06:38 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:06:38 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:06:38 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:06:38 server01 pengine: [19469]: notice: short_print: Stopped: [ drbd-nagios:0 ] Feb 8 13:06:38 server01 pengine: [19469]: info: get_failcount: nagios-ip has failed 2 times on server02 Feb 8 13:06:38 server01 pengine: [19469]: notice: common_apply_stickiness: nagios-ip can fail 999998 more times on server02 before being forced off Feb 8 13:06:38 server01 pengine: [19469]: info: get_failcount: apache2 has failed 1 times on server02 Feb 8 13:06:38 server01 pengine: [19469]: notice: common_apply_stickiness: apache2 can fail 999999 more times on server02 before being forced off Feb 8 13:06:38 server01 pengine: [19469]: info: get_failcount: pnp has failed 1 times on server02 Feb 8 13:06:38 server01 pengine: [19469]: notice: common_apply_stickiness: pnp can fail 999999 more times on server02 before being forced off Feb 8 13:06:38 server01 pengine: [19469]: info: native_color: Resource drbd-nagios:0 cannot run anywhere Feb 8 13:06:38 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:1 (Slave server01) Feb 8 13:06:38 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:06:38 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:1 (Slave server01) Feb 8 13:06:38 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:06:38 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (5s) for nagios-ip on server01 Feb 8 13:06:38 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (10s) for apache2 on server01 Feb 8 13:06:38 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for nagios on server01 Feb 8 13:06:38 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for pnp on server01 Feb 8 13:06:38 server01 pengine: [19469]: info: RecurringOp: Cancelling action drbd-nagios:1_monitor_15000 (Slave vs. Master) Feb 8 13:06:38 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:06:38 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:06:38 server01 pengine: [19469]: info: RecurringOp: Cancelling action drbd-nagios:1_monitor_15000 (Slave vs. Master) Feb 8 13:06:38 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:06:38 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:06:38 server01 cib: [17148]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-19.raw Feb 8 13:06:38 server01 pengine: [19469]: notice: LogActions: Start fs-data (server01) Feb 8 13:06:38 server01 pengine: [19469]: notice: LogActions: Start nagios-ip (server01) Feb 8 13:06:38 server01 pengine: [19469]: notice: LogActions: Start apache2 (server01) Feb 8 13:06:38 server01 pengine: [19469]: notice: LogActions: Start nagios (server01) Feb 8 13:06:38 server01 pengine: [19469]: notice: LogActions: Start pnp (server01) Feb 8 13:06:38 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Stopped) Feb 8 13:06:38 server01 pengine: [19469]: notice: LogActions: Promote drbd-nagios:1 (Slave -> Master server01) Feb 8 13:06:38 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:06:38 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 204: 36 actions in 36 synapses Feb 8 13:06:38 server01 crmd: [19470]: info: do_te_invoke: Processing graph 204 (ref=pe_calc-dc-1328702798-314) derived from /var/lib/pengine/pe-input-6629.bz2 Feb 8 13:06:38 server01 crmd: [19470]: info: te_rsc_command: Initiating action 5: monitor fs-data_monitor_0 on server01 (local) Feb 8 13:06:38 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=5:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=fs-data_monitor_0 ) Feb 8 13:06:38 server01 lrmd: [19467]: info: rsc:fs-data:96: probe Feb 8 13:06:38 server01 crmd: [19470]: info: te_rsc_command: Initiating action 6: monitor nagios-ip_monitor_0 on server01 (local) Feb 8 13:06:38 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=6:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios-ip_monitor_0 ) Feb 8 13:06:38 server01 cib: [17148]: info: write_cib_contents: Wrote version 0.128.0 of the CIB to disk (digest: ab19ced688f210ad136be8cd1e855ff7) Feb 8 13:06:38 server01 lrmd: [19467]: info: rsc:nagios-ip:97: probe Feb 8 13:06:38 server01 crmd: [19470]: info: te_rsc_command: Initiating action 7: monitor apache2_monitor_0 on server01 (local) Feb 8 13:06:38 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=7:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=apache2_monitor_0 ) Feb 8 13:06:38 server01 lrmd: [19467]: info: rsc:apache2:98: probe Feb 8 13:06:38 server01 crmd: [19470]: info: te_rsc_command: Initiating action 8: monitor nagios_monitor_0 on server01 (local) Feb 8 13:06:38 server01 lrmd: [19467]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Feb 8 13:06:38 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=8:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios_monitor_0 ) Feb 8 13:06:38 server01 lrmd: [19467]: info: rsc:nagios:99: probe Feb 8 13:06:38 server01 crmd: [19470]: info: te_rsc_command: Initiating action 9: monitor pnp_monitor_0 on server01 (local) Feb 8 13:06:38 server01 lrmd: [19467]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Feb 8 13:06:38 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=9:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=pnp_monitor_0 ) Feb 8 13:06:38 server01 crmd: [19470]: info: te_rsc_command: Initiating action 1: cancel drbd-nagios:1_monitor_15000 on server01 (local) Feb 8 13:06:38 server01 lrmd: [19467]: info: cancel_op: operation monitor[90] on ocf::drbd::drbd-nagios:1 for client 19470, its parameters: CRM_meta_clone=[1] CRM_meta_timeout=[20000] CRM_meta_notify_slave_resource=[ ] CRM_meta_notify_active_resource=[ ] CRM_meta_notify_demote_uname=[ ] drbd_resource=[data] CRM_meta_notify_inactive_resource=[drbd-nagios:1 ] CRM_meta_notify_active_uname=[ ] CRM_meta_master_node_max=[1] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_master_resource=[drbd-nagios:0 ] CRM_meta_clone_node_max=[1] CRM_meta_notify=[true] CRM_meta_no cancelled Feb 8 13:06:38 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op drbd-nagios:1_monitor_15000 from 1:204:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1: lrm_invoke-lrmd-1328702798-321 Feb 8 13:06:38 server01 crmd: [19470]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1328702798-321 from server01 Feb 8 13:06:38 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_monitor_15000 (1) confirmed on server01 (rc=0) Feb 8 13:06:38 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 41 fired and confirmed Feb 8 13:06:38 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_monitor_15000 (call=90, status=1, cib-update=0, confirmed=true) Cancelled Feb 8 13:06:38 server01 crmd: [19470]: info: te_rsc_command: Initiating action 66: notify drbd-nagios:1_pre_notify_promote_0 on server01 (local) Feb 8 13:06:38 server01 cib: [17148]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.dlO11G (digest: /var/lib/heartbeat/crm/cib.jUUI91) Feb 8 13:06:38 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=66:204:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:06:38 server01 pengine: [19469]: info: process_pe_message: Transition 204: PEngine Input stored in: /var/lib/pengine/pe-input-6629.bz2 Feb 8 13:06:38 server01 crmd: [19470]: info: process_lrm_event: LRM operation fs-data_monitor_0 (call=96, rc=7, cib-update=440, confirmed=true) not running Feb 8 13:06:38 server01 crmd: [19470]: info: match_graph_event: Action fs-data_monitor_0 (5) confirmed on server01 (rc=0) Feb 8 13:06:38 server01 kernel: [5979142.784734] block drbd0: PingAck did not arrive in time. Feb 8 13:06:38 server01 kernel: [5979142.784834] block drbd0: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) Feb 8 13:06:38 server01 kernel: [5979142.784863] block drbd0: asender terminated Feb 8 13:06:38 server01 kernel: [5979142.784866] block drbd0: Terminating drbd0_asender Feb 8 13:06:38 server01 kernel: [5979142.784901] block drbd0: short read receiving data: read 2688 expected 4096 Feb 8 13:06:38 server01 kernel: [5979142.784907] block drbd0: error receiving Data, l: 4120! Feb 8 13:06:38 server01 kernel: [5979142.785174] block drbd0: Connection closed Feb 8 13:06:38 server01 kernel: [5979142.785179] block drbd0: conn( NetworkFailure -> Unconnected ) Feb 8 13:06:38 server01 kernel: [5979142.785193] block drbd0: receiver terminated Feb 8 13:06:38 server01 kernel: [5979142.785195] block drbd0: Restarting drbd0_receiver Feb 8 13:06:38 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios-ip_monitor_0 (call=97, rc=7, cib-update=441, confirmed=true) not running Feb 8 13:06:38 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_monitor_0 (6) confirmed on server01 (rc=0) Feb 8 13:06:38 server01 kernel: [5979142.785213] block drbd0: receiver (re)started Feb 8 13:06:38 server01 kernel: [5979142.785217] block drbd0: conn( Unconnected -> WFConnection ) Feb 8 13:06:39 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios_monitor_0 (call=99, rc=7, cib-update=442, confirmed=true) not running Feb 8 13:06:39 server01 crmd: [19470]: info: match_graph_event: Action nagios_monitor_0 (8) confirmed on server01 (rc=0) Feb 8 13:06:39 server01 crmd: [19470]: info: process_lrm_event: LRM operation apache2_monitor_0 (call=98, rc=0, cib-update=443, confirmed=true) ok Feb 8 13:06:39 server01 crmd: [19470]: WARN: status_from_rc: Action 7 (apache2_monitor_0) on server01 failed (target: 7 vs. rc: 0): Error Feb 8 13:06:39 server01 crmd: [19470]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=apache2_monitor_0, magic=0:0;7:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.128.6) : Event failed Feb 8 13:06:39 server01 crmd: [19470]: info: update_abort_priority: Abort priority upgraded from 0 to 1 Feb 8 13:06:39 server01 crmd: [19470]: info: update_abort_priority: Abort action done superceeded by restart Feb 8 13:06:39 server01 crmd: [19470]: info: match_graph_event: Action apache2_monitor_0 (7) confirmed on server01 (rc=4) Feb 8 13:06:39 server01 lrmd: [19467]: info: rsc:pnp:100: probe Feb 8 13:06:39 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:101: notify Feb 8 13:06:39 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=101, rc=0, cib-update=444, confirmed=true) ok Feb 8 13:06:39 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_pre_notify_promote_0 (66) confirmed on server01 (rc=0) Feb 8 13:06:39 server01 crmd: [19470]: info: process_lrm_event: LRM operation pnp_monitor_0 (call=100, rc=7, cib-update=445, confirmed=true) not running Feb 8 13:06:39 server01 crmd: [19470]: info: match_graph_event: Action pnp_monitor_0 (9) confirmed on server01 (rc=0) Feb 8 13:06:39 server01 crmd: [19470]: info: te_rsc_command: Initiating action 4: probe_complete probe_complete on server01 (local) - no waiting Feb 8 13:06:39 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:06:39 server01 crmd: [19470]: notice: run_graph: Transition 204 (Complete=9, Pending=0, Fired=0, Skipped=23, Incomplete=4, Source=/var/lib/pengine/pe-input-6629.bz2): Stopped Feb 8 13:06:39 server01 crmd: [19470]: info: te_graph_trigger: Transition 204 is now complete Feb 8 13:06:39 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:06:39 server01 crmd: [19470]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Feb 8 13:06:39 server01 crmd: [19470]: info: do_pe_invoke: Query 446: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:06:39 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=446, ref=pe_calc-dc-1328702799-324, seq=128, quorate=0 Feb 8 13:06:39 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:06:39 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:06:39 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:06:39 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:06:39 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:06:39 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Stopped Feb 8 13:06:39 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:06:39 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server01 Feb 8 13:06:39 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:06:39 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:06:39 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:06:39 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:06:39 server01 pengine: [19469]: notice: short_print: Stopped: [ drbd-nagios:0 ] Feb 8 13:06:39 server01 pengine: [19469]: info: get_failcount: nagios-ip has failed 2 times on server02 Feb 8 13:06:39 server01 pengine: [19469]: notice: common_apply_stickiness: nagios-ip can fail 999998 more times on server02 before being forced off Feb 8 13:06:39 server01 pengine: [19469]: info: get_failcount: apache2 has failed 1 times on server02 Feb 8 13:06:39 server01 pengine: [19469]: notice: common_apply_stickiness: apache2 can fail 999999 more times on server02 before being forced off Feb 8 13:06:39 server01 pengine: [19469]: info: get_failcount: pnp has failed 1 times on server02 Feb 8 13:06:39 server01 pengine: [19469]: notice: common_apply_stickiness: pnp can fail 999999 more times on server02 before being forced off Feb 8 13:06:39 server01 pengine: [19469]: info: native_color: Resource drbd-nagios:0 cannot run anywhere Feb 8 13:06:39 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:1 (Slave server01) Feb 8 13:06:39 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:06:39 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:1 (Slave server01) Feb 8 13:06:39 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:06:39 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (5s) for nagios-ip on server01 Feb 8 13:06:39 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (10s) for apache2 on server01 Feb 8 13:06:39 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for nagios on server01 Feb 8 13:06:39 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for pnp on server01 Feb 8 13:06:39 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:06:39 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:06:39 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:06:39 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:06:39 server01 pengine: [19469]: notice: LogActions: Start fs-data (server01) Feb 8 13:06:39 server01 pengine: [19469]: notice: LogActions: Start nagios-ip (server01) Feb 8 13:06:39 server01 pengine: [19469]: notice: LogActions: Restart resource apache2 (Started server01) Feb 8 13:06:39 server01 pengine: [19469]: notice: LogActions: Start nagios (server01) Feb 8 13:06:39 server01 pengine: [19469]: notice: LogActions: Start pnp (server01) Feb 8 13:06:39 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Stopped) Feb 8 13:06:39 server01 pengine: [19469]: notice: LogActions: Promote drbd-nagios:1 (Slave -> Master server01) Feb 8 13:06:39 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:06:39 server01 crmd: [19470]: WARN: destroy_action: Cancelling timer for action 1 (src=667) Feb 8 13:06:39 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 205: 24 actions in 24 synapses Feb 8 13:06:39 server01 crmd: [19470]: info: do_te_invoke: Processing graph 205 (ref=pe_calc-dc-1328702799-324) derived from /var/lib/pengine/pe-input-6630.bz2 Feb 8 13:06:39 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 16 fired and confirmed Feb 8 13:06:39 server01 crmd: [19470]: info: te_rsc_command: Initiating action 7: stop apache2_stop_0 on server01 (local) Feb 8 13:06:39 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=7:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=apache2_stop_0 ) Feb 8 13:06:39 server01 lrmd: [19467]: info: rsc:apache2:102: stop Feb 8 13:06:39 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 36 fired and confirmed Feb 8 13:06:39 server01 crmd: [19470]: info: te_rsc_command: Initiating action 60: notify drbd-nagios:1_pre_notify_promote_0 on server01 (local) Feb 8 13:06:39 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=60:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:06:39 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:103: notify Feb 8 13:06:39 server01 pengine: [19469]: info: process_pe_message: Transition 205: PEngine Input stored in: /var/lib/pengine/pe-input-6630.bz2 Feb 8 13:06:39 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=103, rc=0, cib-update=447, confirmed=true) ok Feb 8 13:06:39 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_pre_notify_promote_0 (60) confirmed on server01 (rc=0) Feb 8 13:06:39 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 37 fired and confirmed Feb 8 13:06:39 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 34 fired and confirmed Feb 8 13:06:39 server01 crmd: [19470]: info: te_rsc_command: Initiating action 21: promote drbd-nagios:1_promote_0 on server01 (local) Feb 8 13:06:39 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=21:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_promote_0 ) Feb 8 13:06:39 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:104: promote Feb 8 13:06:39 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:promote:stdout) Feb 8 13:06:39 server01 kernel: [5979143.821841] block drbd0: role( Secondary -> Primary ) Feb 8 13:06:39 server01 kernel: [5979143.822123] block drbd0: Creating new current UUID Feb 8 13:06:39 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_promote_0 (call=104, rc=0, cib-update=448, confirmed=true) ok Feb 8 13:06:39 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_promote_0 (21) confirmed on server01 (rc=0) Feb 8 13:06:39 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 35 fired and confirmed Feb 8 13:06:39 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 38 fired and confirmed Feb 8 13:06:39 server01 crmd: [19470]: info: te_rsc_command: Initiating action 61: notify drbd-nagios:1_post_notify_promote_0 on server01 (local) Feb 8 13:06:39 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=61:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:06:39 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:105: notify Feb 8 13:06:39 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:notify:stdout) Feb 8 13:06:39 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=105, rc=0, cib-update=449, confirmed=true) ok Feb 8 13:06:39 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_post_notify_promote_0 (61) confirmed on server01 (rc=0) Feb 8 13:06:39 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 39 fired and confirmed Feb 8 13:06:40 server01 apache[17313]: INFO: Killing apache PID 25554 Feb 8 13:06:41 server01 apache[17313]: INFO: Killing apache PID 25554 Feb 8 13:06:42 server01 lrmd: [19467]: info: RA output: (apache2:stop:stderr) kill: 830: No such process Feb 8 13:06:42 server01 apache[17313]: INFO: Killing apache PID 25554 Feb 8 13:06:42 server01 apache[17313]: INFO: apache stopped. Feb 8 13:06:43 server01 crmd: [19470]: info: process_lrm_event: LRM operation apache2_stop_0 (call=102, rc=0, cib-update=450, confirmed=true) ok Feb 8 13:06:43 server01 crmd: [19470]: info: match_graph_event: Action apache2_stop_0 (7) confirmed on server01 (rc=0) Feb 8 13:06:43 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 17 fired and confirmed Feb 8 13:06:43 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 1 fired and confirmed Feb 8 13:06:43 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 14 fired and confirmed Feb 8 13:06:43 server01 crmd: [19470]: info: te_rsc_command: Initiating action 4: start fs-data_start_0 on server01 (local) Feb 8 13:06:43 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=4:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=fs-data_start_0 ) Feb 8 13:06:43 server01 lrmd: [19467]: info: rsc:fs-data:106: start Feb 8 13:06:43 server01 Filesystem[17484]: INFO: Running start for /dev/drbd/by-res/data on /data Feb 8 13:06:43 server01 lrmd: [19467]: info: RA output: (fs-data:start:stderr) FATAL: Module scsi_hostadapter not found. Feb 8 13:06:52 server01 kernel: [5979156.542603] kjournald starting. Commit interval 5 seconds Feb 8 13:06:52 server01 kernel: [5979156.542631] EXT3-fs warning: maximal mount count reached, running e2fsck is recommended Feb 8 13:06:52 server01 kernel: [5979156.550565] EXT3 FS on drbd0, internal journal Feb 8 13:06:52 server01 kernel: [5979156.550711] EXT3-fs: drbd0: 3 orphan inodes deleted Feb 8 13:06:52 server01 kernel: [5979156.550713] EXT3-fs: recovery complete. Feb 8 13:06:52 server01 kernel: [5979156.622776] EXT3-fs: mounted filesystem with writeback data mode. Feb 8 13:06:52 server01 crmd: [19470]: info: process_lrm_event: LRM operation fs-data_start_0 (call=106, rc=0, cib-update=451, confirmed=true) ok Feb 8 13:06:52 server01 crmd: [19470]: info: match_graph_event: Action fs-data_start_0 (4) confirmed on server01 (rc=0) Feb 8 13:06:52 server01 crmd: [19470]: info: te_rsc_command: Initiating action 5: start nagios-ip_start_0 on server01 (local) Feb 8 13:06:52 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=5:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios-ip_start_0 ) Feb 8 13:06:52 server01 lrmd: [19467]: info: rsc:nagios-ip:107: start Feb 8 13:06:52 server01 IPaddr2[17539]: INFO: ip -f inet addr add 10.28.110.108/32 brd 10.28.110.108 dev eth0 Feb 8 13:06:52 server01 IPaddr2[17539]: INFO: ip link set eth0 up Feb 8 13:06:52 server01 IPaddr2[17539]: INFO: /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp/send_arp-10.28.110.108 eth0 10.28.110.108 auto not_used not_used Feb 8 13:06:52 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios-ip_start_0 (call=107, rc=0, cib-update=452, confirmed=true) ok Feb 8 13:06:52 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_start_0 (5) confirmed on server01 (rc=0) Feb 8 13:06:52 server01 crmd: [19470]: info: te_rsc_command: Initiating action 6: monitor nagios-ip_monitor_5000 on server01 (local) Feb 8 13:06:52 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=6:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios-ip_monitor_5000 ) Feb 8 13:06:52 server01 lrmd: [19467]: info: rsc:nagios-ip:108: monitor Feb 8 13:06:52 server01 crmd: [19470]: info: te_rsc_command: Initiating action 8: start apache2_start_0 on server01 (local) Feb 8 13:06:52 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=8:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=apache2_start_0 ) Feb 8 13:06:52 server01 lrmd: [19467]: info: rsc:apache2:109: start Feb 8 13:06:52 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios-ip_monitor_5000 (call=108, rc=0, cib-update=453, confirmed=false) ok Feb 8 13:06:52 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_monitor_5000 (6) confirmed on server01 (rc=0) Feb 8 13:06:52 server01 cibadmin: [17692]: info: Invoked: cibadmin -Ql -o nodes Feb 8 13:06:52 server01 crm_resource: [17701]: info: Invoked: crm_resource -M -r supervision-grp --node=server01 Feb 8 13:06:53 server01 apache[17596]: INFO: apache not running Feb 8 13:06:53 server01 apache[17596]: INFO: waiting for apache /etc/apache2/apache2.conf to come up Feb 8 13:06:54 server01 crmd: [19470]: info: process_lrm_event: LRM operation apache2_start_0 (call=109, rc=0, cib-update=454, confirmed=true) ok Feb 8 13:06:54 server01 crmd: [19470]: info: match_graph_event: Action apache2_start_0 (8) confirmed on server01 (rc=0) Feb 8 13:06:54 server01 crmd: [19470]: info: te_rsc_command: Initiating action 9: monitor apache2_monitor_10000 on server01 (local) Feb 8 13:06:54 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=9:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=apache2_monitor_10000 ) Feb 8 13:06:54 server01 lrmd: [19467]: info: rsc:apache2:110: monitor Feb 8 13:06:54 server01 crmd: [19470]: info: te_rsc_command: Initiating action 10: start nagios_start_0 on server01 (local) Feb 8 13:06:54 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=10:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios_start_0 ) Feb 8 13:06:54 server01 lrmd: [19467]: info: rsc:nagios:111: start Feb 8 13:06:54 server01 lrmd: [17733]: WARN: For LSB init script, no additional parameters are needed. Feb 8 13:06:54 server01 lrmd: [19467]: info: RA output: (nagios:start:stdout) Starting nagios3 monitoring daemon: nagios3 Feb 8 13:06:54 server01 crmd: [19470]: info: process_lrm_event: LRM operation apache2_monitor_10000 (call=110, rc=0, cib-update=455, confirmed=false) ok Feb 8 13:06:54 server01 crmd: [19470]: info: match_graph_event: Action apache2_monitor_10000 (9) confirmed on server01 (rc=0) Feb 8 13:06:55 server01 lrmd: [19467]: info: RA output: (nagios:start:stdout) . Feb 8 13:06:55 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios_start_0 (call=111, rc=0, cib-update=456, confirmed=true) ok Feb 8 13:06:55 server01 crmd: [19470]: info: match_graph_event: Action nagios_start_0 (10) confirmed on server01 (rc=0) Feb 8 13:06:55 server01 crmd: [19470]: info: te_rsc_command: Initiating action 11: monitor nagios_monitor_30000 on server01 (local) Feb 8 13:06:55 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=11:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios_monitor_30000 ) Feb 8 13:06:55 server01 lrmd: [19467]: info: rsc:nagios:112: monitor Feb 8 13:06:55 server01 crmd: [19470]: info: te_rsc_command: Initiating action 12: start pnp_start_0 on server01 (local) Feb 8 13:06:55 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=12:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=pnp_start_0 ) Feb 8 13:06:55 server01 lrmd: [19467]: info: rsc:pnp:113: start Feb 8 13:06:55 server01 lrmd: [17832]: WARN: For LSB init script, no additional parameters are needed. Feb 8 13:06:55 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios_monitor_30000 (call=112, rc=0, cib-update=457, confirmed=false) ok Feb 8 13:06:55 server01 crmd: [19470]: info: match_graph_event: Action nagios_monitor_30000 (11) confirmed on server01 (rc=0) Feb 8 13:06:55 server01 lrmd: [19467]: info: RA output: (pnp:start:stdout) Starting npcd: Feb 8 13:06:55 server01 lrmd: [19467]: info: RA output: (pnp:start:stdout) done. Feb 8 13:06:55 server01 crmd: [19470]: info: process_lrm_event: LRM operation pnp_start_0 (call=113, rc=0, cib-update=458, confirmed=true) ok Feb 8 13:06:55 server01 crmd: [19470]: info: match_graph_event: Action pnp_start_0 (12) confirmed on server01 (rc=0) Feb 8 13:06:55 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 15 fired and confirmed Feb 8 13:06:55 server01 crmd: [19470]: info: te_rsc_command: Initiating action 13: monitor pnp_monitor_30000 on server01 (local) Feb 8 13:06:55 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=13:205:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=pnp_monitor_30000 ) Feb 8 13:06:55 server01 lrmd: [19467]: info: rsc:pnp:114: monitor Feb 8 13:06:55 server01 crmd: [19470]: info: process_lrm_event: LRM operation pnp_monitor_30000 (call=114, rc=0, cib-update=459, confirmed=false) ok Feb 8 13:06:55 server01 crmd: [19470]: info: match_graph_event: Action pnp_monitor_30000 (13) confirmed on server01 (rc=0) Feb 8 13:06:55 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:06:55 server01 crmd: [19470]: notice: run_graph: Transition 205 (Complete=24, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6630.bz2): Complete Feb 8 13:06:55 server01 crmd: [19470]: info: te_graph_trigger: Transition 205 is now complete Feb 8 13:06:55 server01 crmd: [19470]: info: notify_crmd: Transition 205 status: done - Feb 8 13:06:55 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:06:55 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 13:11:25 server01 cib: [19466]: info: cib_stats: Processed 100 operations (3900.00us average, 0% utilization) in the last 10min Feb 8 13:21:25 server01 cib: [19466]: info: cib_stats: Processed 1 operations (0.00us average, 0% utilization) in the last 10min Feb 8 13:21:55 server01 crmd: [19470]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Feb 8 13:21:55 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Feb 8 13:21:55 server01 crmd: [19470]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Feb 8 13:21:55 server01 crmd: [19470]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Feb 8 13:21:55 server01 crmd: [19470]: info: do_pe_invoke: Query 460: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:21:56 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=460, ref=pe_calc-dc-1328703716-338, seq=128, quorate=0 Feb 8 13:21:56 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:21:56 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:21:56 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:21:56 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:21:56 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:21:56 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server01 Feb 8 13:21:56 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server01 Feb 8 13:21:56 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server01 Feb 8 13:21:56 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server01 Feb 8 13:21:56 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server01 Feb 8 13:21:56 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:21:56 server01 pengine: [19469]: notice: short_print: Masters: [ server01 ] Feb 8 13:21:56 server01 pengine: [19469]: notice: short_print: Stopped: [ drbd-nagios:0 ] Feb 8 13:21:56 server01 pengine: [19469]: info: get_failcount: nagios-ip has failed 2 times on server02 Feb 8 13:21:56 server01 pengine: [19469]: notice: common_apply_stickiness: nagios-ip can fail 999998 more times on server02 before being forced off Feb 8 13:21:56 server01 pengine: [19469]: info: get_failcount: apache2 has failed 1 times on server02 Feb 8 13:21:56 server01 pengine: [19469]: notice: common_apply_stickiness: apache2 can fail 999999 more times on server02 before being forced off Feb 8 13:21:56 server01 pengine: [19469]: info: get_failcount: pnp has failed 1 times on server02 Feb 8 13:21:56 server01 pengine: [19469]: notice: common_apply_stickiness: pnp can fail 999999 more times on server02 before being forced off Feb 8 13:21:56 server01 pengine: [19469]: info: native_color: Resource drbd-nagios:0 cannot run anywhere Feb 8 13:21:56 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:1 (Master server01) Feb 8 13:21:56 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:21:56 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:1 (Master server01) Feb 8 13:21:56 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:21:56 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:21:56 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:21:56 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:21:56 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:21:56 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server01) Feb 8 13:21:56 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server01) Feb 8 13:21:56 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server01) Feb 8 13:21:56 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server01) Feb 8 13:21:56 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server01) Feb 8 13:21:56 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Stopped) Feb 8 13:21:56 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Master server01) Feb 8 13:21:56 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:21:56 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 206: 0 actions in 0 synapses Feb 8 13:21:56 server01 crmd: [19470]: info: do_te_invoke: Processing graph 206 (ref=pe_calc-dc-1328703716-338) derived from /var/lib/pengine/pe-input-6631.bz2 Feb 8 13:21:56 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:21:56 server01 crmd: [19470]: notice: run_graph: Transition 206 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6631.bz2): Complete Feb 8 13:21:56 server01 crmd: [19470]: info: te_graph_trigger: Transition 206 is now complete Feb 8 13:21:56 server01 crmd: [19470]: info: notify_crmd: Transition 206 status: done - Feb 8 13:21:56 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:21:56 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 13:21:56 server01 pengine: [19469]: info: process_pe_message: Transition 206: PEngine Input stored in: /var/lib/pengine/pe-input-6631.bz2 Feb 8 13:26:24 server01 kernel: [5980327.407337] block drbd0: Handshake successful: Agreed network protocol version 91 Feb 8 13:26:24 server01 kernel: [5980327.408099] block drbd0: Peer authenticated using 20 bytes of 'sha1' HMAC Feb 8 13:26:24 server01 kernel: [5980327.408107] block drbd0: conn( WFConnection -> WFReportParams ) Feb 8 13:26:24 server01 kernel: [5980327.408170] block drbd0: Starting asender thread (from drbd0_receiver [3710]) Feb 8 13:26:24 server01 kernel: [5980327.408739] block drbd0: data-integrity-alg: Feb 8 13:26:24 server01 kernel: [5980327.408872] block drbd0: drbd_sync_handshake: Feb 8 13:26:24 server01 kernel: [5980327.408875] block drbd0: self A3DC11958F8CB2EB:ADB2239BD54918F8:4E1551459CE58E6A:0BE229AFB4B51397 bits:50511 flags:0 Feb 8 13:26:24 server01 kernel: [5980327.408885] block drbd0: peer ADB2239BD54918F8:0000000000000000:4E1551459CE58E6A:0BE229AFB4B51397 bits:263168 flags:2 Feb 8 13:26:24 server01 kernel: [5980327.408888] block drbd0: uuid_compare()=1 by rule 70 Feb 8 13:26:24 server01 kernel: [5980327.408893] block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate ) Feb 8 13:26:24 server01 kernel: [5980328.331065] block drbd0: conn( WFBitMapS -> SyncSource ) pdsk( UpToDate -> Inconsistent ) Feb 8 13:26:24 server01 kernel: [5980328.331082] block drbd0: Began resync as SyncSource (will sync 1241936 KB [310484 bits set]). Feb 8 13:26:25 server01 crmd: [19470]: notice: ais_dispatch: Membership 132: quorum acquired Feb 8 13:26:25 server01 crmd: [19470]: info: ais_status_callback: status: server02 is now member (was lost) Feb 8 13:26:25 server01 crmd: [19470]: info: crm_update_peer: Node server02: id=1584274442 state=member (new) addr=r(0) ip(10.28.110.94) votes=1 born=92 seen=132 proc=00000000000000000000000000013312 Feb 8 13:26:25 server01 corosync[19441]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 132: memb=1, new=0, lost=0 Feb 8 13:26:25 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: memb: server01 929963018 Feb 8 13:26:25 server01 corosync[19441]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 132: memb=2, new=1, lost=0 Feb 8 13:26:25 server01 corosync[19441]: [pcmk ] info: update_member: Node 1584274442/server02 is now: member Feb 8 13:26:25 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: NEW: server02 1584274442 Feb 8 13:26:25 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: MEMB: server01 929963018 Feb 8 13:26:25 server01 corosync[19441]: [pcmk ] info: pcmk_peer_update: MEMB: server02 1584274442 Feb 8 13:26:25 server01 corosync[19441]: [pcmk ] info: send_member_notification: Sending membership update 132 to 2 children Feb 8 13:26:25 server01 corosync[19441]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Feb 8 13:26:25 server01 corosync[19441]: [pcmk ] info: update_member: 0x21d6590 Node 1584274442 (server02) born on: 132 Feb 8 13:26:25 server01 corosync[19441]: [pcmk ] info: send_member_notification: Sending membership update 132 to 2 children Feb 8 13:26:25 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']/lrm (origin=local/crmd/461, version=0.128.22): ok (rc=0) Feb 8 13:26:25 server01 crmd: [19470]: info: crm_update_quorum: Updating quorum status to true (call=465) Feb 8 13:26:25 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']/transient_attributes (origin=local/crmd/462, version=0.128.23): ok (rc=0) Feb 8 13:26:25 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/463, version=0.128.23): ok (rc=0) Feb 8 13:26:25 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:26:25 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:26:25 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/465, version=0.129.1): ok (rc=0) Feb 8 13:26:25 server01 cib: [19466]: notice: ais_dispatch: Membership 132: quorum acquired Feb 8 13:26:25 server01 cib: [19466]: info: crm_update_peer: Node server02: id=1584274442 state=member (new) addr=r(0) ip(10.28.110.94) votes=1 born=92 seen=132 proc=00000000000000000000000000013312 Feb 8 13:26:25 server01 cib: [19466]: info: ais_dispatch: Membership 132: quorum retained Feb 8 13:26:25 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_sync_one for section 'all' (origin=server02/server02/(null), version=0.129.1): ok (rc=0) Feb 8 13:26:25 server01 crmd: [19470]: info: crm_ais_dispatch: Setting expected votes to 2 Feb 8 13:26:25 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=drbd-nagios:0_monitor_0, magic=0:0;14:3939:7:2997a66f-57bd-4b0c-8001-a3beabe8cbe4, cib=0.128.22) : Resource op removal Feb 8 13:26:25 server01 crmd: [19470]: info: erase_xpath_callback: Deletion of "//node_state[@uname='server02']/lrm": ok (rc=0) Feb 8 13:26:25 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=1, tag=transient_attributes, id=server02, magic=NA, cib=0.128.23) : Transient attribute: removal Feb 8 13:26:25 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/467, version=0.129.1): ok (rc=0) Feb 8 13:26:25 server01 crmd: [19470]: info: erase_xpath_callback: Deletion of "//node_state[@uname='server02']/transient_attributes": ok (rc=0) Feb 8 13:26:25 server01 crmd: [19470]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change Feb 8 13:26:25 server01 crmd: [19470]: info: need_abort: Aborting on change to have-quorum Feb 8 13:26:25 server01 crmd: [19470]: info: ais_dispatch: Membership 132: quorum retained Feb 8 13:26:25 server01 corosync[19441]: [MAIN ] Completed service synchronization, ready to provide service. Feb 8 13:26:25 server01 cib: [15849]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-20.raw Feb 8 13:26:25 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/468, version=0.129.1): ok (rc=0) Feb 8 13:26:25 server01 crmd: [19470]: info: crm_ais_dispatch: Setting expected votes to 2 Feb 8 13:26:25 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] Feb 8 13:26:25 server01 crmd: [19470]: info: do_state_transition: Membership changed: 128 -> 132 - join restart Feb 8 13:26:25 server01 crmd: [19470]: info: do_pe_invoke: Query 472: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:26:25 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=do_state_transition ] Feb 8 13:26:25 server01 crmd: [19470]: info: update_dc: Unset DC server01 Feb 8 13:26:25 server01 crmd: [19470]: info: join_make_offer: Making join offers based on membership 132 Feb 8 13:26:25 server01 crmd: [19470]: info: do_dc_join_offer_all: join-4: Waiting on 2 outstanding join acks Feb 8 13:26:25 server01 crmd: [19470]: info: update_dc: Set DC to server01 (3.0.1) Feb 8 13:26:25 server01 cib: [15849]: info: write_cib_contents: Wrote version 0.129.0 of the CIB to disk (digest: 65dbfc72fb9bf1de5f7d457d2895fc47) Feb 8 13:26:25 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/471, version=0.129.1): ok (rc=0) Feb 8 13:26:25 server01 cib: [15849]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.mFyz09 (digest: /var/lib/heartbeat/crm/cib.FynO6X) Feb 8 13:26:27 server01 crmd: [19470]: info: update_dc: Unset DC server01 Feb 8 13:26:27 server01 crmd: [19470]: info: do_dc_join_offer_all: A new node joined the cluster Feb 8 13:26:27 server01 crmd: [19470]: info: do_dc_join_offer_all: join-5: Waiting on 2 outstanding join acks Feb 8 13:26:27 server01 crmd: [19470]: info: update_dc: Set DC to server01 (3.0.1) Feb 8 13:26:27 server01 crmd: [19470]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Feb 8 13:26:27 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes responded to the join offer. Feb 8 13:26:27 server01 crmd: [19470]: info: do_dc_join_finalize: join-5: Syncing the CIB from server01 to the rest of the cluster Feb 8 13:26:27 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/475, version=0.129.1): ok (rc=0) Feb 8 13:26:27 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/476, version=0.129.1): ok (rc=0) Feb 8 13:26:27 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/477, version=0.129.1): ok (rc=0) Feb 8 13:26:27 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']/transient_attributes (origin=server02/crmd/6, version=0.129.1): ok (rc=0) Feb 8 13:26:27 server01 crmd: [19470]: info: do_dc_join_ack: join-5: Updating node state to member for server02 Feb 8 13:26:27 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']/lrm (origin=local/crmd/478, version=0.129.1): ok (rc=0) Feb 8 13:26:27 server01 crmd: [19470]: info: erase_xpath_callback: Deletion of "//node_state[@uname='server02']/lrm": ok (rc=0) Feb 8 13:26:27 server01 crmd: [19470]: info: do_dc_join_ack: join-5: Updating node state to member for server01 Feb 8 13:26:27 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']/lrm (origin=local/crmd/480, version=0.129.3): ok (rc=0) Feb 8 13:26:27 server01 crmd: [19470]: info: erase_xpath_callback: Deletion of "//node_state[@uname='server01']/lrm": ok (rc=0) Feb 8 13:26:27 server01 crmd: [19470]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Feb 8 13:26:27 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:26:27 server01 crmd: [19470]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date Feb 8 13:26:27 server01 crmd: [19470]: info: crm_update_quorum: Updating quorum status to true (call=484) Feb 8 13:26:27 server01 crmd: [19470]: info: abort_transition_graph: do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled Feb 8 13:26:27 server01 crmd: [19470]: info: do_pe_invoke: Query 485: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_local_callback: Sending full refresh (origin=crmd) Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-nagios-ip () Feb 8 13:26:27 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/482, version=0.129.4): ok (rc=0) Feb 8 13:26:27 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/484, version=0.129.4): ok (rc=0) Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd-nagios:0 () Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd-nagios:1 (10000) Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-apache2 () Feb 8 13:26:27 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=485, ref=pe_calc-dc-1328703987-350, seq=132, quorate=1 Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-nagios-ip (1326606464) Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-pnp () Feb 8 13:26:27 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:26:27 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:26:27 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:26:27 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:26:27 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:26:27 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:26:27 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server01 Feb 8 13:26:27 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server01 Feb 8 13:26:27 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server01 Feb 8 13:26:27 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server01 Feb 8 13:26:27 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server01 Feb 8 13:26:27 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:26:27 server01 pengine: [19469]: notice: short_print: Masters: [ server01 ] Feb 8 13:26:27 server01 pengine: [19469]: notice: short_print: Stopped: [ drbd-nagios:0 ] Feb 8 13:26:27 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:1 (Master server01) Feb 8 13:26:27 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:26:27 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:1 (Master server01) Feb 8 13:26:27 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-pnp (1326606099) Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-nagios (1326606492) Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-fs-data () Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-apache2 (1326606483) Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-nagios () Feb 8 13:26:27 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Feb 8 13:26:27 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:0 on server02 Feb 8 13:26:27 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:27 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:27 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:0 on server02 Feb 8 13:26:27 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:27 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:27 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server01) Feb 8 13:26:27 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server01) Feb 8 13:26:27 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server01) Feb 8 13:26:27 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server01) Feb 8 13:26:27 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server01) Feb 8 13:26:27 server01 pengine: [19469]: notice: LogActions: Start drbd-nagios:0 (server02) Feb 8 13:26:27 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Master server01) Feb 8 13:26:27 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:26:27 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 207: 19 actions in 19 synapses Feb 8 13:26:27 server01 crmd: [19470]: info: do_te_invoke: Processing graph 207 (ref=pe_calc-dc-1328703987-350) derived from /var/lib/pengine/pe-input-6632.bz2 Feb 8 13:26:27 server01 crmd: [19470]: info: te_rsc_command: Initiating action 9: monitor fs-data_monitor_0 on server02 Feb 8 13:26:27 server01 crmd: [19470]: info: te_rsc_command: Initiating action 10: monitor nagios-ip_monitor_0 on server02 Feb 8 13:26:27 server01 crmd: [19470]: info: te_rsc_command: Initiating action 11: monitor apache2_monitor_0 on server02 Feb 8 13:26:27 server01 crmd: [19470]: info: te_rsc_command: Initiating action 12: monitor nagios_monitor_0 on server02 Feb 8 13:26:27 server01 crmd: [19470]: info: te_rsc_command: Initiating action 13: monitor pnp_monitor_0 on server02 Feb 8 13:26:27 server01 crmd: [19470]: info: te_rsc_command: Initiating action 14: monitor drbd-nagios:0_monitor_0 on server02 Feb 8 13:26:27 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 37 fired and confirmed Feb 8 13:26:27 server01 crmd: [19470]: info: te_rsc_command: Initiating action 65: notify drbd-nagios:1_pre_notify_start_0 on server01 (local) Feb 8 13:26:27 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=65:207:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:26:27 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:115: notify Feb 8 13:26:27 server01 crmd: [19470]: info: match_graph_event: Action nagios_monitor_0 (12) confirmed on server02 (rc=0) Feb 8 13:26:27 server01 pengine: [19469]: info: process_pe_message: Transition 207: PEngine Input stored in: /var/lib/pengine/pe-input-6632.bz2 Feb 8 13:26:27 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --rate=100M --al-extents=257 Feb 8 13:26:27 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=115, rc=0, cib-update=486, confirmed=true) ok Feb 8 13:26:27 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_pre_notify_start_0 (65) confirmed on server01 (rc=0) Feb 8 13:26:27 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 38 fired and confirmed Feb 8 13:26:27 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_monitor_0 (10) confirmed on server02 (rc=0) Feb 8 13:26:28 server01 crmd: [19470]: info: match_graph_event: Action fs-data_monitor_0 (9) confirmed on server02 (rc=0) Feb 8 13:26:28 server01 crmd: [19470]: info: match_graph_event: Action apache2_monitor_0 (11) confirmed on server02 (rc=0) Feb 8 13:26:28 server01 crmd: [19470]: info: match_graph_event: Action pnp_monitor_0 (13) confirmed on server02 (rc=0) Feb 8 13:26:28 server01 crmd: [19470]: WARN: status_from_rc: Action 14 (drbd-nagios:0_monitor_0) on server02 failed (target: 7 vs. rc: 0): Error Feb 8 13:26:28 server01 crmd: [19470]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=drbd-nagios:0_monitor_0, magic=0:0;14:207:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.129.11) : Event failed Feb 8 13:26:28 server01 crmd: [19470]: info: update_abort_priority: Abort priority upgraded from 0 to 1 Feb 8 13:26:28 server01 crmd: [19470]: info: update_abort_priority: Abort action done superceeded by restart Feb 8 13:26:28 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_monitor_0 (14) confirmed on server02 (rc=4) Feb 8 13:26:28 server01 crmd: [19470]: info: te_rsc_command: Initiating action 8: probe_complete probe_complete on server02 - no waiting Feb 8 13:26:28 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:26:28 server01 crmd: [19470]: notice: run_graph: Transition 207 (Complete=10, Pending=0, Fired=0, Skipped=4, Incomplete=5, Source=/var/lib/pengine/pe-input-6632.bz2): Stopped Feb 8 13:26:28 server01 crmd: [19470]: info: te_graph_trigger: Transition 207 is now complete Feb 8 13:26:28 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:26:28 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:26:28 server01 crmd: [19470]: info: do_pe_invoke: Query 487: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:26:28 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=487, ref=pe_calc-dc-1328703988-359, seq=132, quorate=1 Feb 8 13:26:28 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:26:28 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:26:28 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:26:28 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:26:28 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:26:28 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:26:28 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:26:28 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server01 Feb 8 13:26:28 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server01 Feb 8 13:26:28 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server01 Feb 8 13:26:28 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server01 Feb 8 13:26:28 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server01 Feb 8 13:26:28 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:26:28 server01 pengine: [19469]: notice: short_print: Masters: [ server01 ] Feb 8 13:26:28 server01 pengine: [19469]: notice: short_print: Slaves: [ server02 ] Feb 8 13:26:28 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:1 (Master server01) Feb 8 13:26:28 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:26:28 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:1 (Master server01) Feb 8 13:26:28 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:26:28 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:0 on server02 Feb 8 13:26:28 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:28 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:28 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:0 on server02 Feb 8 13:26:28 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:28 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:28 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server01) Feb 8 13:26:28 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server01) Feb 8 13:26:28 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server01) Feb 8 13:26:28 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server01) Feb 8 13:26:28 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server01) Feb 8 13:26:28 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Slave server02) Feb 8 13:26:28 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Master server01) Feb 8 13:26:28 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:26:28 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 208: 2 actions in 2 synapses Feb 8 13:26:28 server01 crmd: [19470]: info: do_te_invoke: Processing graph 208 (ref=pe_calc-dc-1328703988-359) derived from /var/lib/pengine/pe-input-6633.bz2 Feb 8 13:26:28 server01 crmd: [19470]: info: te_rsc_command: Initiating action 26: monitor drbd-nagios:0_monitor_15000 on server02 Feb 8 13:26:28 server01 crmd: [19470]: info: te_rsc_command: Initiating action 8: probe_complete probe_complete on server02 - no waiting Feb 8 13:26:28 server01 pengine: [19469]: info: process_pe_message: Transition 208: PEngine Input stored in: /var/lib/pengine/pe-input-6633.bz2 Feb 8 13:26:29 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_monitor_15000 (26) confirmed on server02 (rc=0) Feb 8 13:26:29 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:26:29 server01 crmd: [19470]: notice: run_graph: Transition 208 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6633.bz2): Complete Feb 8 13:26:29 server01 crmd: [19470]: info: te_graph_trigger: Transition 208 is now complete Feb 8 13:26:29 server01 crmd: [19470]: info: notify_crmd: Transition 208 status: done - Feb 8 13:26:29 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:26:29 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 13:26:30 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=server02, magic=NA, cib=0.129.13) : Transient attribute: update Feb 8 13:26:30 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] Feb 8 13:26:30 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:26:30 server01 crmd: [19470]: info: do_pe_invoke: Query 488: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:26:30 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=server02, magic=NA, cib=0.129.14) : Transient attribute: update Feb 8 13:26:30 server01 crmd: [19470]: info: do_pe_invoke: Query 489: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:26:30 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=489, ref=pe_calc-dc-1328703990-362, seq=132, quorate=1 Feb 8 13:26:30 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:26:30 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:26:30 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:26:30 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:26:30 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:26:30 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:26:30 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:26:30 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server01 Feb 8 13:26:30 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server01 Feb 8 13:26:30 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server01 Feb 8 13:26:30 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server01 Feb 8 13:26:30 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server01 Feb 8 13:26:30 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:26:30 server01 pengine: [19469]: notice: short_print: Masters: [ server01 ] Feb 8 13:26:30 server01 pengine: [19469]: notice: short_print: Slaves: [ server02 ] Feb 8 13:26:30 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:26:30 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:26:30 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:26:30 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:26:30 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (5s) for nagios-ip on server02 Feb 8 13:26:30 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (10s) for apache2 on server02 Feb 8 13:26:30 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for nagios on server02 Feb 8 13:26:30 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for pnp on server02 Feb 8 13:26:30 server01 pengine: [19469]: info: RecurringOp: Cancelling action drbd-nagios:0_monitor_15000 (Slave vs. Master) Feb 8 13:26:30 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:26:30 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:30 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:30 server01 pengine: [19469]: info: RecurringOp: Cancelling action drbd-nagios:0_monitor_15000 (Slave vs. Master) Feb 8 13:26:30 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:26:30 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:30 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:26:30 server01 pengine: [19469]: notice: LogActions: Move resource fs-data (Started server01 -> server02) Feb 8 13:26:30 server01 pengine: [19469]: notice: LogActions: Move resource nagios-ip (Started server01 -> server02) Feb 8 13:26:30 server01 pengine: [19469]: notice: LogActions: Move resource apache2 (Started server01 -> server02) Feb 8 13:26:30 server01 pengine: [19469]: notice: LogActions: Move resource nagios (Started server01 -> server02) Feb 8 13:26:30 server01 pengine: [19469]: notice: LogActions: Move resource pnp (Started server01 -> server02) Feb 8 13:26:30 server01 pengine: [19469]: notice: LogActions: Promote drbd-nagios:0 (Slave -> Master server02) Feb 8 13:26:30 server01 pengine: [19469]: notice: LogActions: Demote drbd-nagios:1 (Master -> Slave server01) Feb 8 13:26:30 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:26:30 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 209: 43 actions in 43 synapses Feb 8 13:26:30 server01 crmd: [19470]: info: do_te_invoke: Processing graph 209 (ref=pe_calc-dc-1328703990-362) derived from /var/lib/pengine/pe-input-6634.bz2 Feb 8 13:26:30 server01 pengine: [19469]: info: process_pe_message: Transition 209: PEngine Input stored in: /var/lib/pengine/pe-input-6634.bz2 Feb 8 13:26:30 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 26 fired and confirmed Feb 8 13:26:30 server01 crmd: [19470]: info: te_rsc_command: Initiating action 21: stop pnp_stop_0 on server01 (local) Feb 8 13:26:30 server01 lrmd: [19467]: info: cancel_op: operation monitor[114] on lsb::npcd::pnp for client 19470, its parameters: CRM_meta_name=[monitor] crm_feature_set=[3.0.1] CRM_meta_timeout=[20000] CRM_meta_interval=[30000] cancelled Feb 8 13:26:30 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=21:209:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=pnp_stop_0 ) Feb 8 13:26:30 server01 lrmd: [19467]: info: rsc:pnp:116: stop Feb 8 13:26:30 server01 crmd: [19470]: info: te_rsc_command: Initiating action 5: cancel drbd-nagios:0_monitor_15000 on server02 Feb 8 13:26:30 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 56 fired and confirmed Feb 8 13:26:30 server01 lrmd: [16471]: WARN: For LSB init script, no additional parameters are needed. Feb 8 13:26:30 server01 crmd: [19470]: info: process_lrm_event: LRM operation pnp_monitor_30000 (call=114, status=1, cib-update=0, confirmed=true) Cancelled Feb 8 13:26:30 server01 crmd: [19470]: info: te_rsc_command: Initiating action 75: notify drbd-nagios:0_pre_notify_demote_0 on server02 Feb 8 13:26:30 server01 crmd: [19470]: info: te_rsc_command: Initiating action 77: notify drbd-nagios:1_pre_notify_demote_0 on server01 (local) Feb 8 13:26:30 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=77:209:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:26:30 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:117: notify Feb 8 13:26:30 server01 lrmd: [19467]: info: RA output: (pnp:stop:stdout) Stopping npcd: Feb 8 13:26:30 server01 lrmd: [19467]: info: RA output: (pnp:stop:stdout) . Feb 8 13:26:30 server01 crmd: [19470]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1328703990-4 from server02 Feb 8 13:26:30 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_monitor_15000 (5) confirmed on server02 (rc=0) Feb 8 13:26:30 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=117, rc=0, cib-update=490, confirmed=true) ok Feb 8 13:26:30 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_pre_notify_demote_0 (77) confirmed on server01 (rc=0) Feb 8 13:26:30 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_pre_notify_demote_0 (75) confirmed on server02 (rc=0) Feb 8 13:26:30 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 57 fired and confirmed Feb 8 13:26:31 server01 lrmd: [19467]: info: RA output: (pnp:stop:stdout) done. Feb 8 13:26:31 server01 crmd: [19470]: info: process_lrm_event: LRM operation pnp_stop_0 (call=116, rc=0, cib-update=491, confirmed=true) ok Feb 8 13:26:31 server01 crmd: [19470]: info: match_graph_event: Action pnp_stop_0 (21) confirmed on server01 (rc=0) Feb 8 13:26:31 server01 crmd: [19470]: info: te_rsc_command: Initiating action 18: stop nagios_stop_0 on server01 (local) Feb 8 13:26:31 server01 lrmd: [19467]: info: cancel_op: operation monitor[112] on lsb::nagios3::nagios for client 19470, its parameters: CRM_meta_name=[monitor] crm_feature_set=[3.0.1] CRM_meta_timeout=[20000] CRM_meta_interval=[30000] cancelled Feb 8 13:26:31 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=18:209:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios_stop_0 ) Feb 8 13:26:31 server01 lrmd: [19467]: info: rsc:nagios:118: stop Feb 8 13:26:31 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios_monitor_30000 (call=112, status=1, cib-update=0, confirmed=true) Cancelled Feb 8 13:26:31 server01 lrmd: [16533]: WARN: For LSB init script, no additional parameters are needed. Feb 8 13:26:31 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) Stopping nagios3 monitoring daemon: nagios3 Feb 8 13:26:31 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) Waiting for nagios3 daemon to die.. Feb 8 13:26:32 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:33 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:34 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:35 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:36 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:37 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:38 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:39 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:40 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:41 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:42 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:43 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:44 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:45 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:46 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:46 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) Feb 8 13:26:46 server01 lrmd: [19467]: info: RA output: (nagios:stop:stdout) . Feb 8 13:26:46 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios_stop_0 (call=118, rc=0, cib-update=492, confirmed=true) ok Feb 8 13:26:46 server01 crmd: [19470]: info: match_graph_event: Action nagios_stop_0 (18) confirmed on server01 (rc=0) Feb 8 13:26:46 server01 crmd: [19470]: info: te_rsc_command: Initiating action 15: stop apache2_stop_0 on server01 (local) Feb 8 13:26:46 server01 lrmd: [19467]: info: cancel_op: operation monitor[110] on ocf::apache::apache2 for client 19470, its parameters: configfile=[/etc/apache2/apache2.conf] CRM_meta_depth=[0] depth=[0] crm_feature_set=[3.0.1] port=[443] CRM_meta_name=[monitor] CRM_meta_timeout=[20000] CRM_meta_interval=[10000] cancelled Feb 8 13:26:46 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=15:209:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=apache2_stop_0 ) Feb 8 13:26:46 server01 lrmd: [19467]: info: rsc:apache2:119: stop Feb 8 13:26:46 server01 crmd: [19470]: info: process_lrm_event: LRM operation apache2_monitor_10000 (call=110, status=1, cib-update=0, confirmed=true) Cancelled Feb 8 13:26:47 server01 lrmd: [19467]: info: RA output: (apache2:stop:stderr) kill: 830: No such process Feb 8 13:26:47 server01 apache[16853]: INFO: Killing apache PID 17702 Feb 8 13:26:47 server01 apache[16853]: INFO: apache stopped. Feb 8 13:26:47 server01 crmd: [19470]: info: process_lrm_event: LRM operation apache2_stop_0 (call=119, rc=0, cib-update=493, confirmed=true) ok Feb 8 13:26:47 server01 crmd: [19470]: info: match_graph_event: Action apache2_stop_0 (15) confirmed on server01 (rc=0) Feb 8 13:26:47 server01 crmd: [19470]: info: te_rsc_command: Initiating action 12: stop nagios-ip_stop_0 on server01 (local) Feb 8 13:26:47 server01 lrmd: [19467]: info: cancel_op: operation monitor[108] on ocf::IPaddr2::nagios-ip for client 19470, its parameters: CRM_meta_name=[monitor] cidr_netmask=[32] crm_feature_set=[3.0.1] CRM_meta_timeout=[20000] CRM_meta_interval=[5000] ip=[10.28.110.108] cancelled Feb 8 13:26:47 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=12:209:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios-ip_stop_0 ) Feb 8 13:26:47 server01 lrmd: [19467]: info: rsc:nagios-ip:120: stop Feb 8 13:26:47 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios-ip_monitor_5000 (call=108, status=1, cib-update=0, confirmed=true) Cancelled Feb 8 13:26:47 server01 IPaddr2[16925]: INFO: IP status = ok, IP_CIP= Feb 8 13:26:47 server01 IPaddr2[16925]: INFO: ip -f inet addr delete 10.28.110.108/32 dev eth0 Feb 8 13:26:47 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios-ip_stop_0 (call=120, rc=0, cib-update=494, confirmed=true) ok Feb 8 13:26:47 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_stop_0 (12) confirmed on server01 (rc=0) Feb 8 13:26:47 server01 crmd: [19470]: info: te_rsc_command: Initiating action 10: stop fs-data_stop_0 on server01 (local) Feb 8 13:26:47 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=10:209:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=fs-data_stop_0 ) Feb 8 13:26:47 server01 lrmd: [19467]: info: rsc:fs-data:121: stop Feb 8 13:26:47 server01 Filesystem[16963]: INFO: Running stop for /dev/drbd/by-res/data on /data Feb 8 13:26:47 server01 Filesystem[16963]: INFO: Trying to unmount /data Feb 8 13:26:47 server01 lrmd: [19467]: info: RA output: (fs-data:stop:stderr) umount: /data: device is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) Feb 8 13:26:47 server01 Filesystem[16963]: ERROR: Couldn't unmount /data; trying cleanup with SIGTERM Feb 8 13:26:47 server01 lrmd: [19467]: info: RA output: (fs-data:stop:stderr) /data: Feb 8 13:26:47 server01 lrmd: [19467]: info: RA output: (fs-data:stop:stdout) 14507 15142 Feb 8 13:26:47 server01 Filesystem[16963]: INFO: Some processes on /data were signalled Feb 8 13:27:07 server01 lrmd: [19467]: WARN: fs-data:stop process (PID 16963) timed out (try 1). Killing with signal SIGTERM (15). Feb 8 13:27:07 server01 lrmd: [19467]: WARN: operation stop[121] on ocf::Filesystem::fs-data for client 19470, its parameters: fstype=[ext3] crm_feature_set=[3.0.1] device=[/dev/drbd/by-res/data] CRM_meta_timeout=[20000] directory=[/data/] : pid [16963] timed out Feb 8 13:27:07 server01 crmd: [19470]: ERROR: process_lrm_event: LRM operation fs-data_stop_0 (121) Timed Out (timeout=20000ms) Feb 8 13:27:07 server01 crmd: [19470]: WARN: status_from_rc: Action 10 (fs-data_stop_0) on server01 failed (target: 0 vs. rc: -2): Error Feb 8 13:27:07 server01 crmd: [19470]: WARN: update_failcount: Updating failcount for fs-data on server01 after failed stop: rc=-2 (update=INFINITY, time=1328704027) Feb 8 13:27:07 server01 crmd: [19470]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=fs-data_stop_0, magic=2:-2;10:209:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.129.22) : Event failed Feb 8 13:27:07 server01 crmd: [19470]: info: update_abort_priority: Abort priority upgraded from 0 to 1 Feb 8 13:27:07 server01 crmd: [19470]: info: update_abort_priority: Abort action done superceeded by restart Feb 8 13:27:07 server01 crmd: [19470]: info: match_graph_event: Action fs-data_stop_0 (10) confirmed on server01 (rc=4) Feb 8 13:27:07 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:27:07 server01 crmd: [19470]: notice: run_graph: Transition 209 (Complete=11, Pending=0, Fired=0, Skipped=22, Incomplete=10, Source=/var/lib/pengine/pe-input-6634.bz2): Stopped Feb 8 13:27:07 server01 crmd: [19470]: info: te_graph_trigger: Transition 209 is now complete Feb 8 13:27:07 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:27:07 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:27:07 server01 crmd: [19470]: info: do_pe_invoke: Query 496: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:27:07 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=496, ref=pe_calc-dc-1328704027-371, seq=132, quorate=1 Feb 8 13:27:07 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-fs-data (INFINITY) Feb 8 13:27:07 server01 attrd: [19468]: info: attrd_perform_update: Sent update 371: fail-count-fs-data=INFINITY Feb 8 13:27:07 server01 attrd: [19468]: info: find_hash_entry: Creating hash entry for last-failure-fs-data Feb 8 13:27:07 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-fs-data (1328704027) Feb 8 13:27:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=server01, magic=NA, cib=0.129.23) : Transient attribute: update Feb 8 13:27:07 server01 crmd: [19470]: info: do_pe_invoke: Query 497: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:27:07 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:27:07 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:27:07 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:27:07 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op fs-data_stop_0 on server01: unknown exec error (-2) Feb 8 13:27:07 server01 pengine: [19469]: info: native_add_running: resource fs-data isnt managed Feb 8 13:27:07 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:27:07 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:27:07 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:27:07 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server01 (unmanaged) FAILED Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: notice: short_print: Masters: [ server01 ] Feb 8 13:27:07 server01 pengine: [19469]: notice: short_print: Slaves: [ server02 ] Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Unmanaged resource fs-data allocated to 'nowhere': failed Feb 8 13:27:07 server01 pengine: [19469]: info: native_merge_weights: nagios-ip: Rolling back scores from apache2 Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource nagios-ip cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: native_merge_weights: apache2: Rolling back scores from nagios Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource apache2 cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: native_merge_weights: nagios: Rolling back scores from pnp Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource nagios cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource pnp cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:27:07 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started unmanaged) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Promote drbd-nagios:0 (Slave -> Master server02) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Demote drbd-nagios:1 (Master -> Slave server01) Feb 8 13:27:07 server01 pengine: [19469]: WARN: should_dump_input: Ignoring requirement that fs-data_stop_0 comeplete before supervision-grp_stopped_0: unmanaged failed resources cannot prevent shutdown Feb 8 13:27:07 server01 attrd: [19468]: info: attrd_perform_update: Sent update 374: last-failure-fs-data=1328704027 Feb 8 13:27:07 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=497, ref=pe_calc-dc-1328704027-372, seq=132, quorate=1 Feb 8 13:27:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=server01, magic=NA, cib=0.129.24) : Transient attribute: update Feb 8 13:27:07 server01 crmd: [19470]: info: do_pe_invoke: Query 498: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:27:07 server01 crmd: [19470]: info: handle_response: pe_calc calculation pe_calc-dc-1328704027-371 is obsolete Feb 8 13:27:07 server01 pengine: [19469]: info: process_pe_message: Transition 210: PEngine Input stored in: /var/lib/pengine/pe-input-6635.bz2 Feb 8 13:27:07 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=498, ref=pe_calc-dc-1328704027-373, seq=132, quorate=1 Feb 8 13:27:07 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:27:07 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:27:07 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:27:07 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op fs-data_stop_0 on server01: unknown exec error (-2) Feb 8 13:27:07 server01 pengine: [19469]: info: native_add_running: resource fs-data isnt managed Feb 8 13:27:07 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:27:07 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:27:07 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:27:07 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server01 (unmanaged) FAILED Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: notice: short_print: Masters: [ server01 ] Feb 8 13:27:07 server01 pengine: [19469]: notice: short_print: Slaves: [ server02 ] Feb 8 13:27:07 server01 pengine: [19469]: info: get_failcount: fs-data has failed INFINITY times on server01 Feb 8 13:27:07 server01 pengine: [19469]: WARN: common_apply_stickiness: Forcing fs-data away from server01 after 1000000 failures (max=1000000) Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Unmanaged resource fs-data allocated to 'nowhere': failed Feb 8 13:27:07 server01 pengine: [19469]: info: native_merge_weights: nagios-ip: Rolling back scores from apache2 Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource nagios-ip cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: native_merge_weights: apache2: Rolling back scores from nagios Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource apache2 cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: native_merge_weights: nagios: Rolling back scores from pnp Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource nagios cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource pnp cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:27:07 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started unmanaged) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Promote drbd-nagios:0 (Slave -> Master server02) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Demote drbd-nagios:1 (Master -> Slave server01) Feb 8 13:27:07 server01 pengine: [19469]: WARN: should_dump_input: Ignoring requirement that fs-data_stop_0 comeplete before supervision-grp_stopped_0: unmanaged failed resources cannot prevent shutdown Feb 8 13:27:07 server01 crmd: [19470]: info: handle_response: pe_calc calculation pe_calc-dc-1328704027-372 is obsolete Feb 8 13:27:07 server01 pengine: [19469]: info: process_pe_message: Transition 211: PEngine Input stored in: /var/lib/pengine/pe-input-6636.bz2 Feb 8 13:27:07 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:27:07 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:27:07 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:27:07 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op fs-data_stop_0 on server01: unknown exec error (-2) Feb 8 13:27:07 server01 pengine: [19469]: info: native_add_running: resource fs-data isnt managed Feb 8 13:27:07 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:27:07 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:27:07 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:27:07 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server01 (unmanaged) FAILED Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:27:07 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: notice: short_print: Masters: [ server01 ] Feb 8 13:27:07 server01 pengine: [19469]: notice: short_print: Slaves: [ server02 ] Feb 8 13:27:07 server01 pengine: [19469]: info: get_failcount: fs-data has failed INFINITY times on server01 Feb 8 13:27:07 server01 pengine: [19469]: WARN: common_apply_stickiness: Forcing fs-data away from server01 after 1000000 failures (max=1000000) Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Unmanaged resource fs-data allocated to 'nowhere': failed Feb 8 13:27:07 server01 pengine: [19469]: info: native_merge_weights: nagios-ip: Rolling back scores from apache2 Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource nagios-ip cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: native_merge_weights: apache2: Rolling back scores from nagios Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource apache2 cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: native_merge_weights: nagios: Rolling back scores from pnp Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource nagios cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: native_color: Resource pnp cannot run anywhere Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:27:07 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:27:07 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started unmanaged) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Stopped) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Promote drbd-nagios:0 (Slave -> Master server02) Feb 8 13:27:07 server01 pengine: [19469]: notice: LogActions: Demote drbd-nagios:1 (Master -> Slave server01) Feb 8 13:27:07 server01 pengine: [19469]: WARN: should_dump_input: Ignoring requirement that fs-data_stop_0 comeplete before supervision-grp_stopped_0: unmanaged failed resources cannot prevent shutdown Feb 8 13:27:08 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:27:08 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 212: 30 actions in 30 synapses Feb 8 13:27:08 server01 crmd: [19470]: info: do_te_invoke: Processing graph 212 (ref=pe_calc-dc-1328704027-373) derived from /var/lib/pengine/pe-input-6637.bz2 Feb 8 13:27:08 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 9 fired and confirmed Feb 8 13:27:08 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 51 fired and confirmed Feb 8 13:27:08 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 39 fired and confirmed Feb 8 13:27:08 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 49 fired and confirmed Feb 8 13:27:08 server01 crmd: [19470]: info: te_rsc_command: Initiating action 66: notify drbd-nagios:0_pre_notify_demote_0 on server02 Feb 8 13:27:08 server01 crmd: [19470]: info: te_rsc_command: Initiating action 68: notify drbd-nagios:1_pre_notify_demote_0 on server01 (local) Feb 8 13:27:08 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=68:212:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:27:08 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:122: notify Feb 8 13:27:08 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 47 fired and confirmed Feb 8 13:27:08 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 45 fired and confirmed Feb 8 13:27:08 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 1 fired and confirmed Feb 8 13:27:08 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 10 fired and confirmed Feb 8 13:27:08 server01 pengine: [19469]: info: process_pe_message: Transition 212: PEngine Input stored in: /var/lib/pengine/pe-input-6637.bz2 Feb 8 13:27:08 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=122, rc=0, cib-update=499, confirmed=true) ok Feb 8 13:27:08 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_pre_notify_demote_0 (68) confirmed on server01 (rc=0) Feb 8 13:27:08 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_pre_notify_demote_0 (66) confirmed on server02 (rc=0) Feb 8 13:27:08 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 40 fired and confirmed Feb 8 13:27:08 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 37 fired and confirmed Feb 8 13:27:08 server01 crmd: [19470]: info: te_rsc_command: Initiating action 17: demote drbd-nagios:1_demote_0 on server01 (local) Feb 8 13:27:08 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=17:212:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_demote_0 ) Feb 8 13:27:08 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:123: demote Feb 8 13:27:08 server01 kernel: [5980371.531312] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:08 server01 kernel: [5980371.531462] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:08 server01 kernel: [5980371.531621] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:08 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:08 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:08 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:08 server01 kernel: [5980371.547868] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:08 server01 kernel: [5980371.548000] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:08 server01 kernel: [5980371.548145] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:08 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:08 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:08 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:09 server01 kernel: [5980372.581903] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:09 server01 kernel: [5980372.582053] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:09 server01 kernel: [5980372.582212] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:09 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:09 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:09 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:10 server01 kernel: [5980373.603608] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:10 server01 kernel: [5980373.603741] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:10 server01 kernel: [5980373.603888] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:10 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:10 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:10 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:11 server01 kernel: [5980374.624261] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:11 server01 kernel: [5980374.624393] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:11 server01 kernel: [5980374.624555] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:11 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:11 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:11 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:12 server01 kernel: [5980375.644995] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:12 server01 kernel: [5980375.645125] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:12 server01 kernel: [5980375.645269] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:12 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:12 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:12 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:13 server01 kernel: [5980376.664986] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:13 server01 kernel: [5980376.665131] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:13 server01 kernel: [5980376.665277] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:13 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:13 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:13 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:14 server01 kernel: [5980377.683659] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:14 server01 kernel: [5980377.683796] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:14 server01 kernel: [5980377.683966] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:14 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:14 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:14 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:15 server01 kernel: [5980378.701835] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:15 server01 kernel: [5980378.701971] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:15 server01 kernel: [5980378.702137] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:15 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:15 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:15 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:16 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:16 server01 kernel: [5980379.722500] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:16 server01 kernel: [5980379.722653] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:16 server01 kernel: [5980379.722819] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:16 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:16 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:16 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:16 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:16 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:16 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:16 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:16 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:16 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:16 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:16 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:17 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:17 server01 kernel: [5980380.741804] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:17 server01 kernel: [5980380.741937] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:17 server01 kernel: [5980380.742083] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:17 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:17 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:17 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:17 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:17 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:17 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:17 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:17 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:17 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:17 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:17 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:18 server01 kernel: [5980381.759930] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:18 server01 kernel: [5980381.760068] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:18 server01 kernel: [5980381.760244] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:18 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:18 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:18 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:19 server01 kernel: [5980382.778069] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:19 server01 kernel: [5980382.778213] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:19 server01 kernel: [5980382.778369] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:19 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:19 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:19 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:19 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:19 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:19 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:19 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:19 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:19 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:19 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:19 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:19 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:20 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:20 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:20 server01 kernel: [5980383.796519] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:20 server01 kernel: [5980383.796651] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:20 server01 kernel: [5980383.796797] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:20 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:20 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:20 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:20 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:20 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:20 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:20 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:20 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:20 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:20 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:21 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:21 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:21 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:21 server01 kernel: [5980384.814703] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:21 server01 kernel: [5980384.814841] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:21 server01 kernel: [5980384.815004] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:21 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:21 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:21 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:21 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:21 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:21 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:21 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:21 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:21 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:22 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:22 server01 kernel: [5980385.832077] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:22 server01 kernel: [5980385.832221] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:22 server01 kernel: [5980385.832390] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:22 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:22 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:22 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:22 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:22 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:22 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:22 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:22 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:22 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:22 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:22 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:23 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:23 server01 kernel: [5980386.849045] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:23 server01 kernel: [5980386.849177] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:23 server01 kernel: [5980386.849324] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:23 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:23 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:23 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:23 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:23 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:23 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:23 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:23 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:23 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:23 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:23 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:24 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:24 server01 kernel: [5980387.865477] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:24 server01 kernel: [5980387.865617] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:24 server01 kernel: [5980387.865775] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:24 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:24 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:24 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:24 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:24 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:24 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:24 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:24 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:24 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:24 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:24 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:25 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:25 server01 kernel: [5980388.882976] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:25 server01 kernel: [5980388.883117] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:25 server01 kernel: [5980388.883276] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:25 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:25 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:25 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:25 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:25 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:25 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:25 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:25 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:25 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:25 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:25 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:26 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:26 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:26 server01 kernel: [5980389.899551] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:26 server01 kernel: [5980389.899684] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:26 server01 kernel: [5980389.899831] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:26 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:26 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:26 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:26 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:26 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:26 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:26 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:26 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:26 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:26 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:27 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:27 server01 kernel: [5980390.916891] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:27 server01 kernel: [5980390.917059] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:27 server01 kernel: [5980390.917208] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:27 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Command ' Feb 8 13:27:27 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) drbdsetup Feb 8 13:27:27 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:27 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) 0 Feb 8 13:27:27 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) Feb 8 13:27:27 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) secondary Feb 8 13:27:27 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stderr) ' terminated with exit code 11 Feb 8 13:27:27 server01 drbd[17050]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:27 server01 drbd[17050]: ERROR: data: Exit code 11 Feb 8 13:27:27 server01 drbd[17050]: ERROR: data: Command output: Feb 8 13:27:27 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:demote:stdout) Feb 8 13:27:28 server01 lrmd: [19467]: WARN: drbd-nagios:1:demote process (PID 17050) timed out (try 1). Killing with signal SIGTERM (15). Feb 8 13:27:28 server01 lrmd: [19467]: WARN: operation demote[123] on ocf::drbd::drbd-nagios:1 for client 19470, its parameters: CRM_meta_clone=[1] CRM_meta_notify_slave_resource=[drbd-nagios:0 ] CRM_meta_notify_active_resource=[ ] CRM_meta_notify_demote_uname=[server01 ] drbd_resource=[data] CRM_meta_notify_start_resource=[ ] CRM_meta_master_node_max=[1] CRM_meta_notify_stop_resource=[ ] CRM_meta_notify_master_resource=[drbd-nagios:1 ] CRM_meta_clone_node_max=[1] CRM_meta_clone_max=[2] CRM_meta_notify=[true] CRM_meta_notify_master_uname=[versaille: pid [17050] timed out Feb 8 13:27:28 server01 crmd: [19470]: ERROR: process_lrm_event: LRM operation drbd-nagios:1_demote_0 (123) Timed Out (timeout=20000ms) Feb 8 13:27:28 server01 crmd: [19470]: WARN: status_from_rc: Action 17 (drbd-nagios:1_demote_0) on server01 failed (target: 0 vs. rc: -2): Error Feb 8 13:27:28 server01 crmd: [19470]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=drbd-nagios:1_demote_0, magic=2:-2;17:212:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.129.27) : Event failed Feb 8 13:27:28 server01 crmd: [19470]: info: update_abort_priority: Abort priority upgraded from 0 to 1 Feb 8 13:27:28 server01 crmd: [19470]: info: update_abort_priority: Abort action done superceeded by restart Feb 8 13:27:28 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_demote_0 (17) confirmed on server01 (rc=4) Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 38 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 41 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: te_rsc_command: Initiating action 67: notify drbd-nagios:0_post_notify_demote_0 on server02 Feb 8 13:27:28 server01 crmd: [19470]: info: te_rsc_command: Initiating action 69: notify drbd-nagios:1_post_notify_demote_0 on server01 (local) Feb 8 13:27:28 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=69:212:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:27:28 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:124: notify Feb 8 13:27:28 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_post_notify_demote_0 (67) confirmed on server02 (rc=0) Feb 8 13:27:28 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:notify:stdout) Feb 8 13:27:28 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=124, rc=0, cib-update=501, confirmed=true) ok Feb 8 13:27:28 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_post_notify_demote_0 (69) confirmed on server01 (rc=0) Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 42 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:27:28 server01 crmd: [19470]: notice: run_graph: Transition 212 (Complete=18, Pending=0, Fired=0, Skipped=7, Incomplete=5, Source=/var/lib/pengine/pe-input-6637.bz2): Stopped Feb 8 13:27:28 server01 crmd: [19470]: info: te_graph_trigger: Transition 212 is now complete Feb 8 13:27:28 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:27:28 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:27:28 server01 crmd: [19470]: info: do_pe_invoke: Query 502: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:27:28 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=502, ref=pe_calc-dc-1328704048-379, seq=132, quorate=1 Feb 8 13:27:28 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:27:28 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:27:28 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:27:28 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op fs-data_stop_0 on server01: unknown exec error (-2) Feb 8 13:27:28 server01 pengine: [19469]: info: native_add_running: resource fs-data isnt managed Feb 8 13:27:28 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:27:28 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:27:28 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:27:28 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:27:28 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:27:28 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:27:28 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server01 (unmanaged) FAILED Feb 8 13:27:28 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:27:28 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Stopped Feb 8 13:27:28 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:27:28 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:27:28 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:27:28 server01 pengine: [19469]: notice: native_print: drbd-nagios:1 (ocf::linbit:drbd): Slave server01 FAILED Feb 8 13:27:28 server01 pengine: [19469]: notice: short_print: Slaves: [ server02 ] Feb 8 13:27:28 server01 pengine: [19469]: info: get_failcount: fs-data has failed INFINITY times on server01 Feb 8 13:27:28 server01 pengine: [19469]: WARN: common_apply_stickiness: Forcing fs-data away from server01 after 1000000 failures (max=1000000) Feb 8 13:27:28 server01 pengine: [19469]: info: native_color: Resource drbd-nagios:1 cannot run anywhere Feb 8 13:27:28 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:27:28 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:27:28 server01 pengine: [19469]: info: native_color: Unmanaged resource fs-data allocated to 'nowhere': failed Feb 8 13:27:28 server01 pengine: [19469]: info: native_merge_weights: nagios-ip: Rolling back scores from apache2 Feb 8 13:27:28 server01 pengine: [19469]: info: native_color: Resource nagios-ip cannot run anywhere Feb 8 13:27:28 server01 pengine: [19469]: info: native_merge_weights: apache2: Rolling back scores from nagios Feb 8 13:27:28 server01 pengine: [19469]: info: native_color: Resource apache2 cannot run anywhere Feb 8 13:27:28 server01 pengine: [19469]: info: native_merge_weights: nagios: Rolling back scores from pnp Feb 8 13:27:28 server01 pengine: [19469]: info: native_color: Resource nagios cannot run anywhere Feb 8 13:27:28 server01 pengine: [19469]: info: native_color: Resource pnp cannot run anywhere Feb 8 13:27:28 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:27:28 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:27:28 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:28 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:28 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:28 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:27:28 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started unmanaged) Feb 8 13:27:28 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Stopped) Feb 8 13:27:28 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Stopped) Feb 8 13:27:28 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Stopped) Feb 8 13:27:28 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Stopped) Feb 8 13:27:28 server01 pengine: [19469]: notice: LogActions: Promote drbd-nagios:0 (Slave -> Master server02) Feb 8 13:27:28 server01 pengine: [19469]: notice: LogActions: Stop resource drbd-nagios:1 (server01) Feb 8 13:27:28 server01 pengine: [19469]: WARN: should_dump_input: Ignoring requirement that fs-data_stop_0 comeplete before supervision-grp_stopped_0: unmanaged failed resources cannot prevent shutdown Feb 8 13:27:28 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:27:28 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 213: 26 actions in 26 synapses Feb 8 13:27:28 server01 crmd: [19470]: info: do_te_invoke: Processing graph 213 (ref=pe_calc-dc-1328704048-379) derived from /var/lib/pengine/pe-input-6638.bz2 Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 10 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 48 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 24 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 46 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: te_rsc_command: Initiating action 55: notify drbd-nagios:0_pre_notify_stop_0 on server02 Feb 8 13:27:28 server01 crmd: [19470]: info: te_rsc_command: Initiating action 57: notify drbd-nagios:1_pre_notify_stop_0 on server01 (local) Feb 8 13:27:28 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=57:213:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:27:28 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:125: notify Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 44 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 42 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 11 fired and confirmed Feb 8 13:27:28 server01 pengine: [19469]: info: process_pe_message: Transition 213: PEngine Input stored in: /var/lib/pengine/pe-input-6638.bz2 Feb 8 13:27:28 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=125, rc=0, cib-update=503, confirmed=true) ok Feb 8 13:27:28 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_pre_notify_stop_0 (57) confirmed on server01 (rc=0) Feb 8 13:27:28 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_pre_notify_stop_0 (55) confirmed on server02 (rc=0) Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 25 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 22 fired and confirmed Feb 8 13:27:28 server01 crmd: [19470]: info: te_rsc_command: Initiating action 1: stop drbd-nagios:1_stop_0 on server01 (local) Feb 8 13:27:28 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=1:213:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_stop_0 ) Feb 8 13:27:28 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:126: stop Feb 8 13:27:28 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:28 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:28 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:28 server01 kernel: [5980391.819806] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:28 server01 kernel: [5980391.819937] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:28 server01 kernel: [5980391.820082] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:28 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:28 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:28 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:28 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:28 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:28 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:28 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:28 server01 kernel: [5980391.840640] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:28 server01 kernel: [5980391.840774] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:28 server01 kernel: [5980391.840920] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:28 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:28 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:28 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:28 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:29 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:29 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:29 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:29 server01 kernel: [5980392.861080] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:29 server01 kernel: [5980392.861212] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:29 server01 kernel: [5980392.861359] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:29 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:29 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:29 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:29 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:30 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:30 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:30 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:30 server01 kernel: [5980393.880303] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:30 server01 kernel: [5980393.880436] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:30 server01 kernel: [5980393.880583] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:30 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:30 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:30 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:30 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:31 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:31 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:31 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:31 server01 kernel: [5980394.899796] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:31 server01 kernel: [5980394.899928] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:31 server01 kernel: [5980394.900074] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:31 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:31 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:31 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:31 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:32 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:32 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command ' Feb 8 13:27:32 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:32 server01 kernel: [5980395.921027] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:32 server01 kernel: [5980395.922993] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:32 server01 kernel: [5980395.923148] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:32 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:32 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:32 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:32 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:33 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:33 server01 kernel: [5980396.943925] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:33 server01 kernel: [5980396.944065] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:33 server01 kernel: [5980396.944225] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:33 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command 'drbdsetup Feb 8 13:27:33 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0 secondary' terminated with exit code 11 Feb 8 13:27:33 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:33 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:33 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:33 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:34 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:34 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command 'drbdsetup 0 secondary Feb 8 13:27:34 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) ' terminated with exit code 11 Feb 8 13:27:34 server01 kernel: [5980397.964234] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:34 server01 kernel: [5980397.964396] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:34 server01 kernel: [5980397.964543] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:34 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:34 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:34 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:34 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:35 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:35 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:35 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:35 server01 kernel: [5980398.984669] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:35 server01 kernel: [5980398.984801] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:35 server01 kernel: [5980398.984948] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:35 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:35 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:35 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:35 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:36 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:36 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:36 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:36 server01 kernel: [5980400.003593] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:36 server01 kernel: [5980400.003724] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:36 server01 kernel: [5980400.003871] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:36 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:36 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:36 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:36 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:37 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:37 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:37 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:37 server01 kernel: [5980401.023834] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:37 server01 kernel: [5980401.023965] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:37 server01 kernel: [5980401.024111] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:37 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:37 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:37 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:37 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:38 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:38 server01 kernel: [5980402.045401] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:38 server01 kernel: [5980402.045546] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:38 server01 kernel: [5980402.045704] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:38 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:38 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:38 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:38 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:38 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:38 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:39 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:39 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:39 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:39 server01 kernel: [5980403.065388] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:39 server01 kernel: [5980403.065564] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:39 server01 kernel: [5980403.065725] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:39 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:39 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:39 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:39 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:40 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:40 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:40 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary Feb 8 13:27:40 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) ' terminated with exit code 11 Feb 8 13:27:40 server01 kernel: [5980404.085653] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:40 server01 kernel: [5980404.085785] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:40 server01 kernel: [5980404.085930] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:40 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:40 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:40 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:40 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:41 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:41 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:41 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:41 server01 kernel: [5980405.106627] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:41 server01 kernel: [5980405.106757] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:41 server01 kernel: [5980405.106902] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:41 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:41 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:41 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:41 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:42 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:42 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:42 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:42 server01 kernel: [5980406.125977] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:42 server01 kernel: [5980406.126108] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:42 server01 kernel: [5980406.126255] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:42 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:42 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:42 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:42 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:43 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:43 server01 kernel: [5980407.146937] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:43 server01 kernel: [5980407.147068] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:43 server01 kernel: [5980407.147214] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:43 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:43 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:43 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:43 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:43 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:43 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:44 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:44 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:44 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:44 server01 kernel: [5980408.168214] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:44 server01 kernel: [5980408.168345] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:44 server01 kernel: [5980408.168490] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:44 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:44 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:44 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:44 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:45 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:45 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:45 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:45 server01 kernel: [5980409.188503] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:45 server01 kernel: [5980409.188635] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:45 server01 kernel: [5980409.188780] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:45 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:45 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:45 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:45 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:46 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:46 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:46 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:46 server01 kernel: [5980410.208322] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:46 server01 kernel: [5980410.208453] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:46 server01 kernel: [5980410.208600] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:46 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:46 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:46 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:46 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:47 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:47 server01 kernel: [5980411.229196] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:47 server01 kernel: [5980411.229327] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:47 server01 kernel: [5980411.229486] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:47 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:47 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:47 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:47 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:47 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:47 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:48 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:48 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:48 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary Feb 8 13:27:48 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) ' terminated with exit code 11 Feb 8 13:27:48 server01 kernel: [5980412.248973] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:48 server01 kernel: [5980412.249109] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:48 server01 kernel: [5980412.249267] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:48 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:48 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:48 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:48 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:49 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:49 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command 'drbdsetup 0 secondary Feb 8 13:27:49 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) ' terminated with exit code 11 Feb 8 13:27:49 server01 kernel: [5980413.269214] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:49 server01 kernel: [5980413.269345] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:49 server01 kernel: [5980413.269491] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:49 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:49 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:49 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:49 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:50 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:50 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:50 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:50 server01 kernel: [5980414.290848] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:50 server01 kernel: [5980414.290980] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:50 server01 kernel: [5980414.291127] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:50 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:50 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:50 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:50 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:51 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:51 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:51 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:51 server01 kernel: [5980415.310199] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:51 server01 kernel: [5980415.310332] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:51 server01 kernel: [5980415.310478] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:51 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:51 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:51 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:51 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:52 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:52 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:52 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:52 server01 kernel: [5980416.329410] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:52 server01 kernel: [5980416.329589] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:52 server01 kernel: [5980416.329752] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:52 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:52 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:52 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:52 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:53 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:53 server01 kernel: [5980417.350542] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:53 server01 kernel: [5980417.350689] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:53 server01 kernel: [5980417.350848] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:53 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command ' Feb 8 13:27:53 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:53 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:53 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:53 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:53 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:55 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:55 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:55 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:55 server01 kernel: [5980418.370139] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:55 server01 kernel: [5980418.370271] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:55 server01 kernel: [5980418.370416] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:55 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:55 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:55 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:55 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:56 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:56 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:56 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:56 server01 kernel: [5980419.390219] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:56 server01 kernel: [5980419.390369] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:56 server01 kernel: [5980419.390523] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:56 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:56 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:56 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:56 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:57 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:57 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:57 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:57 server01 kernel: [5980420.410926] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:57 server01 kernel: [5980420.411058] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:57 server01 kernel: [5980420.411203] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:57 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:57 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:57 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:57 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:58 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:58 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:58 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:58 server01 kernel: [5980421.431609] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:58 server01 kernel: [5980421.431741] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:58 server01 kernel: [5980421.431887] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:58 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:58 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:58 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:58 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:27:59 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:27:59 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:27:59 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:27:59 server01 kernel: [5980422.452316] block drbd0: State change failed: Device is held open by someone Feb 8 13:27:59 server01 kernel: [5980422.452458] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:59 server01 kernel: [5980422.452618] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:27:59 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:27:59 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:27:59 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:27:59 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:00 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:00 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command ' Feb 8 13:28:00 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:00 server01 kernel: [5980423.472688] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:00 server01 kernel: [5980423.472819] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:00 server01 kernel: [5980423.472965] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:00 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:00 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:00 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:00 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:01 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:01 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:28:01 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:01 server01 kernel: [5980424.492386] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:01 server01 kernel: [5980424.492516] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:01 server01 kernel: [5980424.492661] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:01 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:01 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:01 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:01 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:02 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:02 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:28:02 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:02 server01 kernel: [5980425.511877] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:02 server01 kernel: [5980425.512007] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:02 server01 kernel: [5980425.512154] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:02 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:02 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:02 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:02 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:03 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:03 server01 kernel: [5980426.536774] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:03 server01 kernel: [5980426.538641] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:03 server01 kernel: [5980426.538806] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:03 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:28:03 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:03 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:03 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:03 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:03 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:04 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:04 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:04 server01 kernel: [5980427.587033] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:04 server01 kernel: [5980427.587207] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:04 server01 kernel: [5980427.587354] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:04 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:04 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:04 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:04 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:05 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:05 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:05 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:05 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:05 server01 kernel: [5980428.615122] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:05 server01 kernel: [5980428.615256] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:05 server01 kernel: [5980428.615403] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:05 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:05 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:06 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:06 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:28:06 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:06 server01 kernel: [5980429.639782] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:06 server01 kernel: [5980429.639920] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:06 server01 kernel: [5980429.640119] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:06 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:06 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:06 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:06 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:07 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:07 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:07 server01 kernel: [5980430.662398] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:07 server01 kernel: [5980430.662530] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:07 server01 kernel: [5980430.662676] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:07 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:07 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:07 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:07 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:08 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:28:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:08 server01 kernel: [5980431.685336] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:08 server01 kernel: [5980431.685469] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:08 server01 kernel: [5980431.685616] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:08 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:08 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:08 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:08 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:09 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:09 server01 kernel: [5980432.707220] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:09 server01 kernel: [5980432.707444] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:09 server01 kernel: [5980432.707619] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:28:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:09 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:09 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:09 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:09 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:10 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:28:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:10 server01 kernel: [5980433.727891] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:10 server01 kernel: [5980433.728023] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:10 server01 kernel: [5980433.728168] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:10 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:10 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:10 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:10 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:11 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:28:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:11 server01 kernel: [5980434.748709] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:11 server01 kernel: [5980434.748848] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:11 server01 kernel: [5980434.749011] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:11 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:11 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:11 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:11 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:12 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:12 server01 kernel: [5980435.775014] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:12 server01 kernel: [5980435.775146] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:12 server01 kernel: [5980435.775292] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:12 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:12 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:12 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:12 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:13 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Command ' Feb 8 13:28:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:13 server01 kernel: [5980436.801708] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:13 server01 kernel: [5980436.801839] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:13 server01 kernel: [5980436.801985] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:13 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:13 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:13 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:13 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:14 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:28:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:14 server01 kernel: [5980437.821546] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:14 server01 kernel: [5980437.821676] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:14 server01 kernel: [5980437.821822] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:14 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:14 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:14 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:14 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:15 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) 0: State change failed: (-12) Device is held open by someone Feb 8 13:28:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stderr) Command 'drbdsetup 0 secondary' terminated with exit code 11 Feb 8 13:28:15 server01 drbd[17479]: ERROR: data: Called drbdadm -c /etc/drbd.conf secondary data Feb 8 13:28:15 server01 drbd[17479]: ERROR: data: Exit code 11 Feb 8 13:28:15 server01 kernel: [5980438.842402] block drbd0: State change failed: Device is held open by someone Feb 8 13:28:15 server01 kernel: [5980438.842533] block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:15 server01 kernel: [5980438.842678] block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r--- } Feb 8 13:28:15 server01 drbd[17479]: ERROR: data: Command output: Feb 8 13:28:15 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:16 server01 kernel: [5980439.403713] block drbd0: Resync done (total 111 sec; paused 0 sec; 11188 K/sec) Feb 8 13:28:16 server01 kernel: [5980439.403721] block drbd0: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate ) Feb 8 13:28:16 server01 drbd[17479]: WARNING: data still Primary, demoting. Feb 8 13:28:16 server01 kernel: [5980439.860909] block drbd0: role( Primary -> Secondary ) Feb 8 13:28:16 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:17 server01 kernel: [5980440.877712] block drbd0: peer( Secondary -> Unknown ) conn( Connected -> Disconnecting ) pdsk( UpToDate -> DUnknown ) Feb 8 13:28:17 server01 kernel: [5980440.877741] block drbd0: short read expecting header on sock: r=-512 Feb 8 13:28:17 server01 kernel: [5980440.877879] block drbd0: asender terminated Feb 8 13:28:17 server01 kernel: [5980440.877882] block drbd0: Terminating drbd0_asender Feb 8 13:28:17 server01 kernel: [5980440.878088] block drbd0: Connection closed Feb 8 13:28:17 server01 kernel: [5980440.878179] block drbd0: conn( Disconnecting -> StandAlone ) Feb 8 13:28:17 server01 kernel: [5980440.878227] block drbd0: receiver terminated Feb 8 13:28:17 server01 kernel: [5980440.878229] block drbd0: Terminating drbd0_receiver Feb 8 13:28:17 server01 kernel: [5980440.878239] block drbd0: disk( UpToDate -> Diskless ) Feb 8 13:28:17 server01 kernel: [5980440.878260] block drbd0: drbd_bm_resize called with capacity == 0 Feb 8 13:28:17 server01 kernel: [5980440.878554] block drbd0: worker terminated Feb 8 13:28:17 server01 kernel: [5980440.878557] block drbd0: Terminating drbd0_worker Feb 8 13:28:17 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:18 server01 kernel: [5980441.905035] block drbd0: State change failed: Disk state is lower than outdated Feb 8 13:28:18 server01 kernel: [5980441.905201] block drbd0: state = { cs:StandAlone ro:Secondary/Unknown ds:Diskless/DUnknown r--- } Feb 8 13:28:18 server01 kernel: [5980441.905343] block drbd0: wanted = { cs:StandAlone ro:Secondary/Unknown ds:Outdated/DUnknown r--- } Feb 8 13:28:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:18 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd-nagios:1 (-INFINITY) Feb 8 13:28:18 server01 attrd: [19468]: info: attrd_perform_update: Sent update 376: master-drbd-nagios:1=-INFINITY Feb 8 13:28:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:18 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=server01, magic=NA, cib=0.129.32) : Transient attribute: update Feb 8 13:28:18 server01 crmd: [19470]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Feb 8 13:28:18 server01 crmd: [19470]: info: update_abort_priority: Abort action done superceeded by restart Feb 8 13:28:18 server01 crm_attribute: [18568]: info: Invoked: crm_attribute -N server01 -n master-drbd-nagios:1 -l reboot -D Feb 8 13:28:18 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd-nagios:1 () Feb 8 13:28:18 server01 attrd: [19468]: info: attrd_perform_update: Sent delete 378: node=server01, attr=master-drbd-nagios:1, id=, set=(null), section=status Feb 8 13:28:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:stop:stdout) Feb 8 13:28:18 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=0, tag=transient_attributes, id=server01, magic=NA, cib=0.129.33) : Transient attribute: removal Feb 8 13:28:18 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_stop_0 (call=126, rc=0, cib-update=504, confirmed=true) ok Feb 8 13:28:18 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_stop_0 (1) confirmed on server01 (rc=0) Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 23 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 26 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_rsc_command: Initiating action 56: notify drbd-nagios:0_post_notify_stop_0 on server02 Feb 8 13:28:18 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=server02, magic=NA, cib=0.129.35) : Transient attribute: update Feb 8 13:28:18 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_post_notify_stop_0 (56) confirmed on server02 (rc=0) Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 27 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:28:18 server01 crmd: [19470]: notice: run_graph: Transition 213 (Complete=16, Pending=0, Fired=0, Skipped=6, Incomplete=4, Source=/var/lib/pengine/pe-input-6638.bz2): Stopped Feb 8 13:28:18 server01 crmd: [19470]: info: te_graph_trigger: Transition 213 is now complete Feb 8 13:28:18 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:28:18 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:28:18 server01 crmd: [19470]: info: do_pe_invoke: Query 505: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:18 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=505, ref=pe_calc-dc-1328704098-384, seq=132, quorate=1 Feb 8 13:28:18 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:28:18 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:28:18 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:28:18 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op fs-data_stop_0 on server01: unknown exec error (-2) Feb 8 13:28:18 server01 pengine: [19469]: info: native_add_running: resource fs-data isnt managed Feb 8 13:28:18 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:28:18 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:28:18 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:28:18 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:28:18 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:28:18 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:28:18 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server01 (unmanaged) FAILED Feb 8 13:28:18 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:28:18 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Stopped Feb 8 13:28:18 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:28:18 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:28:18 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:28:18 server01 pengine: [19469]: notice: short_print: Slaves: [ server02 ] Feb 8 13:28:18 server01 pengine: [19469]: notice: short_print: Stopped: [ drbd-nagios:1 ] Feb 8 13:28:18 server01 pengine: [19469]: info: get_failcount: fs-data has failed INFINITY times on server01 Feb 8 13:28:18 server01 pengine: [19469]: WARN: common_apply_stickiness: Forcing fs-data away from server01 after 1000000 failures (max=1000000) Feb 8 13:28:18 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:28:18 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:28:18 server01 pengine: [19469]: info: native_color: Unmanaged resource fs-data allocated to 'nowhere': failed Feb 8 13:28:18 server01 pengine: [19469]: info: native_merge_weights: nagios-ip: Rolling back scores from apache2 Feb 8 13:28:18 server01 pengine: [19469]: info: native_color: Resource nagios-ip cannot run anywhere Feb 8 13:28:18 server01 pengine: [19469]: info: native_merge_weights: apache2: Rolling back scores from nagios Feb 8 13:28:18 server01 pengine: [19469]: info: native_color: Resource apache2 cannot run anywhere Feb 8 13:28:18 server01 pengine: [19469]: info: native_merge_weights: nagios: Rolling back scores from pnp Feb 8 13:28:18 server01 pengine: [19469]: info: native_color: Resource nagios cannot run anywhere Feb 8 13:28:18 server01 pengine: [19469]: info: native_color: Resource pnp cannot run anywhere Feb 8 13:28:18 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Slave server02) Feb 8 13:28:18 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:28:18 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:28:18 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:18 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:18 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:28:18 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:18 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:18 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started unmanaged) Feb 8 13:28:18 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Stopped) Feb 8 13:28:18 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Stopped) Feb 8 13:28:18 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Stopped) Feb 8 13:28:18 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Stopped) Feb 8 13:28:18 server01 pengine: [19469]: notice: LogActions: Promote drbd-nagios:0 (Slave -> Master server02) Feb 8 13:28:18 server01 pengine: [19469]: notice: LogActions: Start drbd-nagios:1 (server01) Feb 8 13:28:18 server01 pengine: [19469]: WARN: should_dump_input: Ignoring requirement that fs-data_stop_0 comeplete before supervision-grp_stopped_0: unmanaged failed resources cannot prevent shutdown Feb 8 13:28:18 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:28:18 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 214: 29 actions in 29 synapses Feb 8 13:28:18 server01 crmd: [19470]: info: do_te_invoke: Processing graph 214 (ref=pe_calc-dc-1328704098-384) derived from /var/lib/pengine/pe-input-6639.bz2 Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 9 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 49 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 19 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 47 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_rsc_command: Initiating action 54: notify drbd-nagios:0_pre_notify_start_0 on server02 Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 45 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 43 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 1 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 10 fired and confirmed Feb 8 13:28:18 server01 pengine: [19469]: info: process_pe_message: Transition 214: PEngine Input stored in: /var/lib/pengine/pe-input-6639.bz2 Feb 8 13:28:18 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_pre_notify_start_0 (54) confirmed on server02 (rc=0) Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 20 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 17 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_rsc_command: Initiating action 15: start drbd-nagios:1_start_0 on server01 (local) Feb 8 13:28:18 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=15:214:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_start_0 ) Feb 8 13:28:18 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:127: start Feb 8 13:28:18 server01 kernel: [5980442.204420] block drbd0: Starting worker thread (from cqueue [2002]) Feb 8 13:28:18 server01 kernel: [5980442.204544] block drbd0: disk( Diskless -> Attaching ) Feb 8 13:28:18 server01 kernel: [5980442.211338] block drbd0: Found 6 transactions (199 active extents) in activity log. Feb 8 13:28:18 server01 kernel: [5980442.211343] block drbd0: Method to ensure write ordering: barrier Feb 8 13:28:18 server01 kernel: [5980442.211347] block drbd0: Backing device's merge_bvec_fn() = ffffffffa01256d8 Feb 8 13:28:18 server01 kernel: [5980442.211350] block drbd0: max_segment_size ( = BIO size ) = 4096 Feb 8 13:28:18 server01 kernel: [5980442.211354] block drbd0: drbd_bm_resize called with capacity == 335517624 Feb 8 13:28:18 server01 kernel: [5980442.212917] block drbd0: resync bitmap: bits=41939703 words=655308 Feb 8 13:28:18 server01 kernel: [5980442.212922] block drbd0: size = 160 GB (167758812 KB) Feb 8 13:28:18 server01 kernel: [5980442.254069] block drbd0: recounting of set bits took additional 0 jiffies Feb 8 13:28:18 server01 kernel: [5980442.254073] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map. Feb 8 13:28:18 server01 kernel: [5980442.254080] block drbd0: disk( Attaching -> Outdated ) Feb 8 13:28:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:start:stdout) Feb 8 13:28:18 server01 kernel: [5980442.258099] block drbd0: conn( StandAlone -> Unconnected ) Feb 8 13:28:18 server01 kernel: [5980442.258126] block drbd0: Starting receiver thread (from drbd0_worker [18597]) Feb 8 13:28:18 server01 kernel: [5980442.258160] block drbd0: receiver (re)started Feb 8 13:28:18 server01 kernel: [5980442.258165] block drbd0: conn( Unconnected -> WFConnection ) Feb 8 13:28:18 server01 crm_attribute: [18620]: info: Invoked: crm_attribute -N server01 -n master-drbd-nagios:1 -l reboot -D Feb 8 13:28:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:start:stdout) Feb 8 13:28:18 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_start_0 (call=127, rc=0, cib-update=506, confirmed=true) ok Feb 8 13:28:18 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_start_0 (15) confirmed on server01 (rc=0) Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 18 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 21 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_rsc_command: Initiating action 55: notify drbd-nagios:0_post_notify_start_0 on server02 Feb 8 13:28:18 server01 crmd: [19470]: info: te_rsc_command: Initiating action 56: notify drbd-nagios:1_post_notify_start_0 on server01 (local) Feb 8 13:28:18 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=56:214:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:28:18 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:128: notify Feb 8 13:28:18 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --rate=100M --al-extents=257 Feb 8 13:28:18 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=128, rc=0, cib-update=507, confirmed=true) ok Feb 8 13:28:18 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_post_notify_start_0 (56) confirmed on server01 (rc=0) Feb 8 13:28:18 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_post_notify_start_0 (55) confirmed on server02 (rc=0) Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 22 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 31 fired and confirmed Feb 8 13:28:18 server01 crmd: [19470]: info: te_rsc_command: Initiating action 59: notify drbd-nagios:0_pre_notify_promote_0 on server02 Feb 8 13:28:19 server01 crmd: [19470]: info: te_rsc_command: Initiating action 61: notify drbd-nagios:1_pre_notify_promote_0 on server01 (local) Feb 8 13:28:19 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=61:214:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:28:19 server01 kernel: [5980442.357936] block drbd0: Handshake successful: Agreed network protocol version 91 Feb 8 13:28:19 server01 kernel: [5980442.358709] block drbd0: Peer authenticated using 20 bytes of 'sha1' HMAC Feb 8 13:28:19 server01 kernel: [5980442.358717] block drbd0: conn( WFConnection -> WFReportParams ) Feb 8 13:28:19 server01 kernel: [5980442.358729] block drbd0: Starting asender thread (from drbd0_receiver [18608]) Feb 8 13:28:19 server01 kernel: [5980442.358887] block drbd0: data-integrity-alg: Feb 8 13:28:19 server01 kernel: [5980442.359379] block drbd0: drbd_sync_handshake: Feb 8 13:28:19 server01 kernel: [5980442.359382] block drbd0: self A3DC11958F8CB2EA:0000000000000000:E79F0BAF35900E17:ADB2239BD54918F8 bits:0 flags:0 Feb 8 13:28:19 server01 kernel: [5980442.359385] block drbd0: peer A3DC11958F8CB2EA:0000000000000000:E79F0BAF35900E16:ADB2239BD54918F8 bits:0 flags:0 Feb 8 13:28:19 server01 kernel: [5980442.359388] block drbd0: uuid_compare()=0 by rule 40 Feb 8 13:28:19 server01 kernel: [5980442.359391] block drbd0: Implicitly set disk from Outdated to UpToDate Feb 8 13:28:19 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:129: notify Feb 8 13:28:19 server01 kernel: [5980442.359395] block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> Connected ) disk( Outdated -> UpToDate ) pdsk( DUnknown -> UpToDate ) Feb 8 13:28:19 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=129, rc=0, cib-update=508, confirmed=true) ok Feb 8 13:28:19 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_pre_notify_promote_0 (61) confirmed on server01 (rc=0) Feb 8 13:28:19 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_pre_notify_promote_0 (59) confirmed on server02 (rc=0) Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 32 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 29 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: te_rsc_command: Initiating action 14: promote drbd-nagios:0_promote_0 on server02 Feb 8 13:28:19 server01 kernel: [5980443.071433] block drbd0: peer( Secondary -> Primary ) Feb 8 13:28:19 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_promote_0 (14) confirmed on server02 (rc=0) Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 30 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 33 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: te_rsc_command: Initiating action 60: notify drbd-nagios:0_post_notify_promote_0 on server02 Feb 8 13:28:19 server01 crmd: [19470]: info: te_rsc_command: Initiating action 62: notify drbd-nagios:1_post_notify_promote_0 on server01 (local) Feb 8 13:28:19 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=62:214:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_notify_0 ) Feb 8 13:28:19 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:130: notify Feb 8 13:28:19 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: master-drbd-nagios:1 (10000) Feb 8 13:28:19 server01 attrd: [19468]: info: attrd_perform_update: Sent update 383: master-drbd-nagios:1=10000 Feb 8 13:28:19 server01 lrmd: [19467]: info: RA output: (drbd-nagios:1:notify:stdout) Feb 8 13:28:19 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=server01, magic=NA, cib=0.129.44) : Transient attribute: update Feb 8 13:28:19 server01 crmd: [19470]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Feb 8 13:28:19 server01 crmd: [19470]: info: update_abort_priority: Abort action done superceeded by restart Feb 8 13:28:19 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_notify_0 (call=130, rc=0, cib-update=509, confirmed=true) ok Feb 8 13:28:19 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=server02, magic=NA, cib=0.129.45) : Transient attribute: update Feb 8 13:28:19 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:0_post_notify_promote_0 (60) confirmed on server02 (rc=0) Feb 8 13:28:19 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_post_notify_promote_0 (62) confirmed on server01 (rc=0) Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 34 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:28:19 server01 crmd: [19470]: notice: run_graph: Transition 214 (Complete=28, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-6639.bz2): Stopped Feb 8 13:28:19 server01 crmd: [19470]: info: te_graph_trigger: Transition 214 is now complete Feb 8 13:28:19 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:28:19 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:28:19 server01 crmd: [19470]: info: do_pe_invoke: Query 510: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:19 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=510, ref=pe_calc-dc-1328704099-394, seq=132, quorate=1 Feb 8 13:28:19 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:28:19 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:28:19 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:28:19 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op fs-data_stop_0 on server01: unknown exec error (-2) Feb 8 13:28:19 server01 pengine: [19469]: info: native_add_running: resource fs-data isnt managed Feb 8 13:28:19 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:28:19 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:28:19 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server01 Feb 8 13:28:19 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:28:19 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:28:19 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:28:19 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server01 (unmanaged) FAILED Feb 8 13:28:19 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:28:19 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Stopped Feb 8 13:28:19 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:28:19 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:28:19 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:28:19 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 13:28:19 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:28:19 server01 pengine: [19469]: info: get_failcount: fs-data has failed INFINITY times on server01 Feb 8 13:28:19 server01 pengine: [19469]: WARN: common_apply_stickiness: Forcing fs-data away from server01 after 1000000 failures (max=1000000) Feb 8 13:28:19 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:28:19 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:28:19 server01 pengine: [19469]: info: native_color: Unmanaged resource fs-data allocated to 'nowhere': failed Feb 8 13:28:19 server01 pengine: [19469]: info: native_merge_weights: nagios-ip: Rolling back scores from apache2 Feb 8 13:28:19 server01 pengine: [19469]: info: native_color: Resource nagios-ip cannot run anywhere Feb 8 13:28:19 server01 pengine: [19469]: info: native_merge_weights: apache2: Rolling back scores from nagios Feb 8 13:28:19 server01 pengine: [19469]: info: native_color: Resource apache2 cannot run anywhere Feb 8 13:28:19 server01 pengine: [19469]: info: native_merge_weights: nagios: Rolling back scores from pnp Feb 8 13:28:19 server01 pengine: [19469]: info: native_color: Resource nagios cannot run anywhere Feb 8 13:28:19 server01 pengine: [19469]: info: native_color: Resource pnp cannot run anywhere Feb 8 13:28:19 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:28:19 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:28:19 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:28:19 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:19 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:19 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (15s) for drbd-nagios:1 on server01 Feb 8 13:28:19 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:19 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:19 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started unmanaged) Feb 8 13:28:19 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Stopped) Feb 8 13:28:19 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Stopped) Feb 8 13:28:19 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Stopped) Feb 8 13:28:19 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Stopped) Feb 8 13:28:19 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 13:28:19 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 13:28:19 server01 pengine: [19469]: WARN: should_dump_input: Ignoring requirement that fs-data_stop_0 comeplete before supervision-grp_stopped_0: unmanaged failed resources cannot prevent shutdown Feb 8 13:28:19 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:28:19 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 215: 8 actions in 8 synapses Feb 8 13:28:19 server01 crmd: [19470]: info: do_te_invoke: Processing graph 215 (ref=pe_calc-dc-1328704099-394) derived from /var/lib/pengine/pe-input-6640.bz2 Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 9 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 51 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: te_rsc_command: Initiating action 18: monitor drbd-nagios:1_monitor_15000 on server01 (local) Feb 8 13:28:19 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=18:215:0:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=drbd-nagios:1_monitor_15000 ) Feb 8 13:28:19 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:131: monitor Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 49 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 47 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 45 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 1 fired and confirmed Feb 8 13:28:19 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 10 fired and confirmed Feb 8 13:28:19 server01 pengine: [19469]: info: process_pe_message: Transition 215: PEngine Input stored in: /var/lib/pengine/pe-input-6640.bz2 Feb 8 13:28:19 server01 crmd: [19470]: info: process_lrm_event: LRM operation drbd-nagios:1_monitor_15000 (call=131, rc=0, cib-update=511, confirmed=false) ok Feb 8 13:28:19 server01 crmd: [19470]: info: match_graph_event: Action drbd-nagios:1_monitor_15000 (18) confirmed on server01 (rc=0) Feb 8 13:28:19 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:28:19 server01 crmd: [19470]: notice: run_graph: Transition 215 (Complete=8, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6640.bz2): Complete Feb 8 13:28:19 server01 crmd: [19470]: info: te_graph_trigger: Transition 215 is now complete Feb 8 13:28:19 server01 crmd: [19470]: info: notify_crmd: Transition 215 status: done - Feb 8 13:28:19 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:28:19 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 13:28:32 server01 cibadmin: [18810]: info: Invoked: cibadmin -Ql -o nodes Feb 8 13:28:32 server01 cibadmin: [18812]: info: Invoked: cibadmin -Ql -o resources Feb 8 13:28:32 server01 cibadmin: [18814]: info: Invoked: cibadmin -Ql -o resources Feb 8 13:28:33 server01 crm_resource: [18816]: info: Invoked: crm_resource -C -r supervision-grp -H server01 Feb 8 13:28:33 server01 attrd: [19468]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-fs-data () Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource fs-data from the LRM Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'fs-data' deleted for 18816_crm_resource on server01 Feb 8 13:28:33 server01 crmd: [19470]: info: notify_deleted: Notifying 18816_crm_resource on server01 that fs-data was deleted Feb 8 13:28:33 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op fs-data_delete_60000 from 0:0:crm-resource-18816: lrm_invoke-lrmd-1328704113-397 Feb 8 13:28:33 server01 attrd: [19468]: info: attrd_perform_update: Sent delete 386: node=server01, attr=fail-count-fs-data, id=, set=(null), section=status Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']//lrm_resource[@id='fs-data'] (origin=local/crmd/512, version=0.129.49): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource nagios-ip from the LRM Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'nagios-ip' deleted for 18816_crm_resource on server01 Feb 8 13:28:33 server01 crmd: [19470]: info: notify_deleted: Notifying 18816_crm_resource on server01 that nagios-ip was deleted Feb 8 13:28:33 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op nagios-ip_delete_60000 from 0:0:crm-resource-18816: lrm_invoke-lrmd-1328704113-398 Feb 8 13:28:33 server01 attrd: [19468]: info: attrd_perform_update: Sent delete -22: node=server01, attr=fail-count-fs-data, id=, set=(null), section=status Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:28:33 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/514, version=0.130.1): ok (rc=0) Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']//lrm_resource[@id='nagios-ip'] (origin=local/crmd/515, version=0.130.2): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource apache2 from the LRM Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'apache2' deleted for 18816_crm_resource on server01 Feb 8 13:28:33 server01 crmd: [19470]: info: notify_deleted: Notifying 18816_crm_resource on server01 that apache2 was deleted Feb 8 13:28:33 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op apache2_delete_60000 from 0:0:crm-resource-18816: lrm_invoke-lrmd-1328704113-399 Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/517, version=0.130.2): ok (rc=0) Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']//lrm_resource[@id='apache2'] (origin=local/crmd/518, version=0.130.3): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource nagios from the LRM Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'nagios' deleted for 18816_crm_resource on server01 Feb 8 13:28:33 server01 crmd: [19470]: info: notify_deleted: Notifying 18816_crm_resource on server01 that nagios was deleted Feb 8 13:28:33 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op nagios_delete_60000 from 0:0:crm-resource-18816: lrm_invoke-lrmd-1328704113-400 Feb 8 13:28:33 server01 cib: [18817]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-21.raw Feb 8 13:28:33 server01 cib: [18817]: info: write_cib_contents: Wrote version 0.130.0 of the CIB to disk (digest: d1bfcc15c2d9cfcaf551817381e1b302) Feb 8 13:28:33 server01 cib: [18817]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.6gClzW (digest: /var/lib/heartbeat/crm/cib.L3ymex) Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/520, version=0.130.3): ok (rc=0) Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']//lrm_resource[@id='nagios'] (origin=local/crmd/521, version=0.130.4): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource pnp from the LRM Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'pnp' deleted for 18816_crm_resource on server01 Feb 8 13:28:33 server01 crmd: [19470]: info: notify_deleted: Notifying 18816_crm_resource on server01 that pnp was deleted Feb 8 13:28:33 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op pnp_delete_60000 from 0:0:crm-resource-18816: lrm_invoke-lrmd-1328704113-401 Feb 8 13:28:33 server01 cibadmin: [18819]: info: Invoked: cibadmin -Ql -o resources Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/523, version=0.130.4): ok (rc=0) Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']//lrm_resource[@id='pnp'] (origin=local/crmd/524, version=0.130.5): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=fs-data_monitor_0, magic=0:7;5:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.129.49) : Resource op removal Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=1, tag=transient_attributes, id=server01, magic=NA, cib=0.129.50) : Transient attribute: removal Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change Feb 8 13:28:33 server01 crmd: [19470]: info: need_abort: Aborting on change to admin_epoch Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=nagios-ip_monitor_0, magic=0:7;6:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.2) : Resource op removal Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=apache2_monitor_0, magic=0:0;7:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.3) : Resource op removal Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=nagios_monitor_0, magic=0:7;8:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.4) : Resource op removal Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=pnp_monitor_0, magic=0:7;9:204:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.5) : Resource op removal Feb 8 13:28:33 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] Feb 8 13:28:33 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:28:33 server01 crmd: [19470]: info: do_pe_invoke: Query 527: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:33 server01 crmd: [19470]: info: do_pe_invoke: Query 528: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:33 server01 crmd: [19470]: info: do_pe_invoke: Query 529: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:33 server01 crmd: [19470]: info: do_pe_invoke: Query 530: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:33 server01 crmd: [19470]: info: do_pe_invoke: Query 531: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:33 server01 crmd: [19470]: info: do_pe_invoke: Query 532: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:33 server01 crmd: [19470]: info: do_pe_invoke: Query 533: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/526, version=0.130.5): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=533, ref=pe_calc-dc-1328704113-402, seq=132, quorate=1 Feb 8 13:28:33 server01 crmd: [19470]: info: config_query_callback: Checking for expired actions every 900000ms Feb 8 13:28:33 server01 crmd: [19470]: info: config_query_callback: Sending expected-votes=2 to corosync Feb 8 13:28:33 server01 crmd: [19470]: info: ais_dispatch: Membership 132: quorum retained Feb 8 13:28:33 server01 crmd: [19470]: info: crm_ais_dispatch: Setting expected votes to 2 Feb 8 13:28:33 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:28:33 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:28:33 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:28:33 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:28:33 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:28:33 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:28:33 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:28:33 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:28:33 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Stopped Feb 8 13:28:33 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:28:33 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Stopped Feb 8 13:28:33 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:28:33 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:28:33 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:28:33 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 13:28:33 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:28:33 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:28:33 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:28:33 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:28:33 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:28:33 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (5s) for nagios-ip on server02 Feb 8 13:28:33 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (10s) for apache2 on server02 Feb 8 13:28:33 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for nagios on server02 Feb 8 13:28:33 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for pnp on server02 Feb 8 13:28:33 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:33 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:33 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:33 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:33 server01 pengine: [19469]: notice: LogActions: Start fs-data (server02) Feb 8 13:28:33 server01 pengine: [19469]: notice: LogActions: Start nagios-ip (server02) Feb 8 13:28:33 server01 pengine: [19469]: notice: LogActions: Start apache2 (server02) Feb 8 13:28:33 server01 pengine: [19469]: notice: LogActions: Start nagios (server02) Feb 8 13:28:33 server01 pengine: [19469]: notice: LogActions: Start pnp (server02) Feb 8 13:28:33 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 13:28:33 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 13:28:33 server01 cibadmin: [18821]: info: Invoked: cibadmin -Ql -o resources Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/536, version=0.130.5): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:28:33 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 216: 26 actions in 26 synapses Feb 8 13:28:33 server01 crmd: [19470]: info: do_te_invoke: Processing graph 216 (ref=pe_calc-dc-1328704113-402) derived from /var/lib/pengine/pe-input-6641.bz2 Feb 8 13:28:33 server01 crmd: [19470]: info: te_rsc_command: Initiating action 5: monitor fs-data_monitor_0 on server01 (local) Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=5:216:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=fs-data_monitor_0 ) Feb 8 13:28:33 server01 lrmd: [19467]: info: rsc:fs-data:132: probe Feb 8 13:28:33 server01 crmd: [19470]: info: te_rsc_command: Initiating action 6: monitor nagios-ip_monitor_0 on server01 (local) Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=6:216:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios-ip_monitor_0 ) Feb 8 13:28:33 server01 lrmd: [19467]: info: rsc:nagios-ip:133: probe Feb 8 13:28:33 server01 crm_resource: [18823]: info: Invoked: crm_resource -C -r supervision-grp -H server02 Feb 8 13:28:33 server01 crmd: [19470]: info: te_rsc_command: Initiating action 7: monitor apache2_monitor_0 on server01 (local) Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=7:216:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=apache2_monitor_0 ) Feb 8 13:28:33 server01 lrmd: [19467]: info: rsc:apache2:134: probe Feb 8 13:28:33 server01 crmd: [19470]: info: te_rsc_command: Initiating action 8: monitor nagios_monitor_0 on server01 (local) Feb 8 13:28:33 server01 lrmd: [19467]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Feb 8 13:28:33 server01 pengine: [19469]: info: process_pe_message: Transition 216: PEngine Input stored in: /var/lib/pengine/pe-input-6641.bz2 Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=8:216:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios_monitor_0 ) Feb 8 13:28:33 server01 lrmd: [19467]: info: rsc:nagios:135: probe Feb 8 13:28:33 server01 crmd: [19470]: info: te_rsc_command: Initiating action 9: monitor pnp_monitor_0 on server01 (local) Feb 8 13:28:33 server01 lrmd: [19467]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Feb 8 13:28:33 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=9:216:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=pnp_monitor_0 ) Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']//lrm_resource[@id='fs-data'] (origin=server02/crmd/25, version=0.130.6): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios_monitor_0 (call=135, rc=7, cib-update=537, confirmed=true) not running Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=fs-data_monitor_0, magic=0:7;9:207:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.6) : Resource op removal Feb 8 13:28:33 server01 crmd: [19470]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Feb 8 13:28:33 server01 crmd: [19470]: info: update_abort_priority: Abort action done superceeded by restart Feb 8 13:28:33 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios-ip_monitor_0 (call=133, rc=7, cib-update=538, confirmed=true) not running Feb 8 13:28:33 server01 crmd: [19470]: info: process_lrm_event: LRM operation fs-data_monitor_0 (call=132, rc=7, cib-update=539, confirmed=true) not running Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=server02/crmd/27, version=0.130.6): ok (rc=0) Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']//lrm_resource[@id='nagios-ip'] (origin=server02/crmd/28, version=0.130.7): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=nagios-ip_monitor_0, magic=0:7;10:207:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.7) : Resource op removal Feb 8 13:28:33 server01 crmd: [19470]: info: match_graph_event: Action nagios_monitor_0 (8) confirmed on server01 (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_monitor_0 (6) confirmed on server01 (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: match_graph_event: Action fs-data_monitor_0 (5) confirmed on server01 (rc=0) Feb 8 13:28:33 server01 apache[18827]: INFO: apache not running Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=server02/crmd/30, version=0.130.10): ok (rc=0) Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']//lrm_resource[@id='apache2'] (origin=server02/crmd/31, version=0.130.11): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: process_lrm_event: LRM operation apache2_monitor_0 (call=134, rc=7, cib-update=540, confirmed=true) not running Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=apache2_monitor_0, magic=0:7;11:207:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.11) : Resource op removal Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=server02/crmd/33, version=0.130.11): ok (rc=0) Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']//lrm_resource[@id='nagios'] (origin=server02/crmd/34, version=0.130.12): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=nagios_monitor_0, magic=0:7;12:207:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.12) : Resource op removal Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=server02/crmd/36, version=0.130.12): ok (rc=0) Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']//lrm_resource[@id='pnp'] (origin=server02/crmd/37, version=0.130.13): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=pnp_monitor_0, magic=0:7;13:207:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.13) : Resource op removal Feb 8 13:28:33 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=server02/crmd/39, version=0.130.13): ok (rc=0) Feb 8 13:28:33 server01 crmd: [19470]: info: match_graph_event: Action apache2_monitor_0 (7) confirmed on server01 (rc=0) Feb 8 13:28:34 server01 lrmd: [19467]: info: rsc:pnp:136: probe Feb 8 13:28:34 server01 crmd: [19470]: info: process_lrm_event: LRM operation pnp_monitor_0 (call=136, rc=7, cib-update=541, confirmed=true) not running Feb 8 13:28:34 server01 crmd: [19470]: info: match_graph_event: Action pnp_monitor_0 (9) confirmed on server01 (rc=0) Feb 8 13:28:34 server01 crmd: [19470]: info: te_rsc_command: Initiating action 4: probe_complete probe_complete on server01 (local) - no waiting Feb 8 13:28:34 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:28:34 server01 crmd: [19470]: notice: run_graph: Transition 216 (Complete=6, Pending=0, Fired=0, Skipped=20, Incomplete=0, Source=/var/lib/pengine/pe-input-6641.bz2): Stopped Feb 8 13:28:34 server01 crmd: [19470]: info: te_graph_trigger: Transition 216 is now complete Feb 8 13:28:34 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:28:34 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:28:34 server01 crmd: [19470]: info: do_pe_invoke: Query 542: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:34 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=542, ref=pe_calc-dc-1328704114-410, seq=132, quorate=1 Feb 8 13:28:34 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:28:34 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:28:34 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:28:34 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:28:34 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:28:34 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:28:34 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:28:34 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:28:34 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Stopped Feb 8 13:28:34 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:28:34 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Stopped Feb 8 13:28:34 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:28:34 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:28:34 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:28:34 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 13:28:34 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:28:34 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:28:34 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:28:34 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:28:34 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:28:34 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (5s) for nagios-ip on server02 Feb 8 13:28:34 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (10s) for apache2 on server02 Feb 8 13:28:34 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for nagios on server02 Feb 8 13:28:34 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for pnp on server02 Feb 8 13:28:34 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:34 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:34 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:34 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:34 server01 pengine: [19469]: notice: LogActions: Start fs-data (server02) Feb 8 13:28:34 server01 pengine: [19469]: notice: LogActions: Start nagios-ip (server02) Feb 8 13:28:34 server01 pengine: [19469]: notice: LogActions: Start apache2 (server02) Feb 8 13:28:34 server01 pengine: [19469]: notice: LogActions: Start nagios (server02) Feb 8 13:28:34 server01 pengine: [19469]: notice: LogActions: Start pnp (server02) Feb 8 13:28:34 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 13:28:34 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 13:28:34 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:28:34 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 217: 26 actions in 26 synapses Feb 8 13:28:34 server01 crmd: [19470]: info: do_te_invoke: Processing graph 217 (ref=pe_calc-dc-1328704114-410) derived from /var/lib/pengine/pe-input-6642.bz2 Feb 8 13:28:34 server01 crmd: [19470]: info: te_rsc_command: Initiating action 6: monitor fs-data_monitor_0 on server02 Feb 8 13:28:34 server01 crmd: [19470]: info: te_rsc_command: Initiating action 7: monitor nagios-ip_monitor_0 on server02 Feb 8 13:28:34 server01 crmd: [19470]: info: te_rsc_command: Initiating action 8: monitor apache2_monitor_0 on server02 Feb 8 13:28:34 server01 crmd: [19470]: info: te_rsc_command: Initiating action 9: monitor nagios_monitor_0 on server02 Feb 8 13:28:34 server01 crmd: [19470]: info: te_rsc_command: Initiating action 10: monitor pnp_monitor_0 on server02 Feb 8 13:28:34 server01 pengine: [19469]: info: process_pe_message: Transition 217: PEngine Input stored in: /var/lib/pengine/pe-input-6642.bz2 Feb 8 13:28:34 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_monitor_0 (7) confirmed on server02 (rc=0) Feb 8 13:28:34 server01 crmd: [19470]: info: match_graph_event: Action nagios_monitor_0 (9) confirmed on server02 (rc=0) Feb 8 13:28:34 server01 crmd: [19470]: info: match_graph_event: Action fs-data_monitor_0 (6) confirmed on server02 (rc=0) Feb 8 13:28:34 server01 crmd: [19470]: WARN: status_from_rc: Action 8 (apache2_monitor_0) on server02 failed (target: 7 vs. rc: 0): Error Feb 8 13:28:34 server01 crmd: [19470]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=apache2_monitor_0, magic=0:0;8:217:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.19) : Event failed Feb 8 13:28:34 server01 crmd: [19470]: info: update_abort_priority: Abort priority upgraded from 0 to 1 Feb 8 13:28:34 server01 crmd: [19470]: info: update_abort_priority: Abort action done superceeded by restart Feb 8 13:28:34 server01 crmd: [19470]: info: match_graph_event: Action apache2_monitor_0 (8) confirmed on server02 (rc=4) Feb 8 13:28:35 server01 crmd: [19470]: info: match_graph_event: Action pnp_monitor_0 (10) confirmed on server02 (rc=0) Feb 8 13:28:35 server01 crmd: [19470]: info: te_rsc_command: Initiating action 5: probe_complete probe_complete on server02 - no waiting Feb 8 13:28:35 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:28:35 server01 crmd: [19470]: notice: run_graph: Transition 217 (Complete=6, Pending=0, Fired=0, Skipped=20, Incomplete=0, Source=/var/lib/pengine/pe-input-6642.bz2): Stopped Feb 8 13:28:35 server01 crmd: [19470]: info: te_graph_trigger: Transition 217 is now complete Feb 8 13:28:35 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:28:35 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:28:35 server01 crmd: [19470]: info: do_pe_invoke: Query 543: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:28:35 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=543, ref=pe_calc-dc-1328704115-417, seq=132, quorate=1 Feb 8 13:28:35 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:28:35 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:28:35 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:28:35 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:28:35 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:28:35 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:28:35 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:28:35 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server02 Feb 8 13:28:35 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:28:35 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Stopped Feb 8 13:28:35 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:28:35 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server02 Feb 8 13:28:35 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:28:35 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:28:35 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:28:35 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 13:28:35 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:28:35 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:28:35 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:28:35 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:28:35 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:28:35 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (5s) for nagios-ip on server02 Feb 8 13:28:35 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (10s) for apache2 on server02 Feb 8 13:28:35 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for nagios on server02 Feb 8 13:28:35 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for pnp on server02 Feb 8 13:28:35 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:35 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:35 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:35 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:28:35 server01 pengine: [19469]: notice: LogActions: Start fs-data (server02) Feb 8 13:28:35 server01 pengine: [19469]: notice: LogActions: Start nagios-ip (server02) Feb 8 13:28:35 server01 pengine: [19469]: notice: LogActions: Restart resource apache2 (Started server02) Feb 8 13:28:35 server01 pengine: [19469]: notice: LogActions: Start nagios (server02) Feb 8 13:28:35 server01 pengine: [19469]: notice: LogActions: Start pnp (server02) Feb 8 13:28:35 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 13:28:35 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 13:28:35 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:28:35 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 218: 15 actions in 15 synapses Feb 8 13:28:35 server01 crmd: [19470]: info: do_te_invoke: Processing graph 218 (ref=pe_calc-dc-1328704115-417) derived from /var/lib/pengine/pe-input-6643.bz2 Feb 8 13:28:35 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 18 fired and confirmed Feb 8 13:28:35 server01 crmd: [19470]: info: te_rsc_command: Initiating action 9: stop apache2_stop_0 on server02 Feb 8 13:28:35 server01 pengine: [19469]: info: process_pe_message: Transition 218: PEngine Input stored in: /var/lib/pengine/pe-input-6643.bz2 Feb 8 13:28:36 server01 crmd: [19470]: info: match_graph_event: Action apache2_stop_0 (9) confirmed on server02 (rc=0) Feb 8 13:28:36 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 19 fired and confirmed Feb 8 13:28:36 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 2 fired and confirmed Feb 8 13:28:36 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 16 fired and confirmed Feb 8 13:28:36 server01 crmd: [19470]: info: te_rsc_command: Initiating action 6: start fs-data_start_0 on server02 Feb 8 13:28:36 server01 crmd: [19470]: info: match_graph_event: Action fs-data_start_0 (6) confirmed on server02 (rc=0) Feb 8 13:28:36 server01 crmd: [19470]: info: te_rsc_command: Initiating action 7: start nagios-ip_start_0 on server02 Feb 8 13:28:36 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_start_0 (7) confirmed on server02 (rc=0) Feb 8 13:28:36 server01 crmd: [19470]: info: te_rsc_command: Initiating action 8: monitor nagios-ip_monitor_5000 on server02 Feb 8 13:28:36 server01 crmd: [19470]: info: te_rsc_command: Initiating action 10: start apache2_start_0 on server02 Feb 8 13:28:36 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_monitor_5000 (8) confirmed on server02 (rc=0) Feb 8 13:28:37 server01 crmd: [19470]: info: match_graph_event: Action apache2_start_0 (10) confirmed on server02 (rc=0) Feb 8 13:28:37 server01 crmd: [19470]: info: te_rsc_command: Initiating action 11: monitor apache2_monitor_10000 on server02 Feb 8 13:28:37 server01 crmd: [19470]: info: te_rsc_command: Initiating action 12: start nagios_start_0 on server02 Feb 8 13:28:37 server01 crmd: [19470]: info: match_graph_event: Action apache2_monitor_10000 (11) confirmed on server02 (rc=0) Feb 8 13:28:39 server01 crmd: [19470]: info: match_graph_event: Action nagios_start_0 (12) confirmed on server02 (rc=0) Feb 8 13:28:39 server01 crmd: [19470]: info: te_rsc_command: Initiating action 13: monitor nagios_monitor_30000 on server02 Feb 8 13:28:39 server01 crmd: [19470]: info: te_rsc_command: Initiating action 14: start pnp_start_0 on server02 Feb 8 13:28:39 server01 crmd: [19470]: info: match_graph_event: Action nagios_monitor_30000 (13) confirmed on server02 (rc=0) Feb 8 13:28:39 server01 crmd: [19470]: info: match_graph_event: Action pnp_start_0 (14) confirmed on server02 (rc=0) Feb 8 13:28:39 server01 crmd: [19470]: info: te_pseudo_action: Pseudo action 17 fired and confirmed Feb 8 13:28:39 server01 crmd: [19470]: info: te_rsc_command: Initiating action 15: monitor pnp_monitor_30000 on server02 Feb 8 13:28:39 server01 crmd: [19470]: info: match_graph_event: Action pnp_monitor_30000 (15) confirmed on server02 (rc=0) Feb 8 13:28:39 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:28:39 server01 crmd: [19470]: notice: run_graph: Transition 218 (Complete=15, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6643.bz2): Complete Feb 8 13:28:39 server01 crmd: [19470]: info: te_graph_trigger: Transition 218 is now complete Feb 8 13:28:39 server01 crmd: [19470]: info: notify_crmd: Transition 218 status: done - Feb 8 13:28:39 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:28:39 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 13:28:39 server01 ntpd[1956]: Deleting interface #13 eth0, 10.28.110.108#123, interface stats: received=0, sent=0, dropped=0, active_time=1200 secs Feb 8 13:29:07 server01 cibadmin: [19241]: info: Invoked: cibadmin -Ql -o nodes Feb 8 13:29:07 server01 cibadmin: [19243]: info: Invoked: cibadmin -Ql -o resources Feb 8 13:29:07 server01 cibadmin: [19245]: info: Invoked: cibadmin -Ql -o resources Feb 8 13:29:07 server01 crm_resource: [19247]: info: Invoked: crm_resource -C -r supervision-grp -H server01 Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource fs-data from the LRM Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'fs-data' deleted for 19247_crm_resource on server01 Feb 8 13:29:07 server01 crmd: [19470]: info: notify_deleted: Notifying 19247_crm_resource on server01 that fs-data was deleted Feb 8 13:29:07 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op fs-data_delete_60000 from 0:0:crm-resource-19247: lrm_invoke-lrmd-1328704147-429 Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']//lrm_resource[@id='fs-data'] (origin=local/crmd/544, version=0.130.31): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource nagios-ip from the LRM Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'nagios-ip' deleted for 19247_crm_resource on server01 Feb 8 13:29:07 server01 crmd: [19470]: info: notify_deleted: Notifying 19247_crm_resource on server01 that nagios-ip was deleted Feb 8 13:29:07 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op nagios-ip_delete_60000 from 0:0:crm-resource-19247: lrm_invoke-lrmd-1328704147-430 Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: - Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:29:07 server01 cib: [19466]: info: log_data_element: cib:diff: + Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/546, version=0.131.1): ok (rc=0) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']//lrm_resource[@id='nagios-ip'] (origin=local/crmd/547, version=0.131.2): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource apache2 from the LRM Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'apache2' deleted for 19247_crm_resource on server01 Feb 8 13:29:07 server01 crmd: [19470]: info: notify_deleted: Notifying 19247_crm_resource on server01 that apache2 was deleted Feb 8 13:29:07 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op apache2_delete_60000 from 0:0:crm-resource-19247: lrm_invoke-lrmd-1328704147-431 Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/549, version=0.131.2): ok (rc=0) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']//lrm_resource[@id='apache2'] (origin=local/crmd/550, version=0.131.3): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource nagios from the LRM Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'nagios' deleted for 19247_crm_resource on server01 Feb 8 13:29:07 server01 crmd: [19470]: info: notify_deleted: Notifying 19247_crm_resource on server01 that nagios was deleted Feb 8 13:29:07 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op nagios_delete_60000 from 0:0:crm-resource-19247: lrm_invoke-lrmd-1328704147-432 Feb 8 13:29:07 server01 cib: [19248]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-22.raw Feb 8 13:29:07 server01 cib: [19248]: info: write_cib_contents: Wrote version 0.131.0 of the CIB to disk (digest: 15a7682efa5c4359caf44594d40142b8) Feb 8 13:29:07 server01 cib: [19248]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.B1k3re (digest: /var/lib/heartbeat/crm/cib.ZdXLZ6) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/552, version=0.131.3): ok (rc=0) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']//lrm_resource[@id='nagios'] (origin=local/crmd/553, version=0.131.4): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_invoke: Removing resource pnp from the LRM Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_invoke: Resource 'pnp' deleted for 19247_crm_resource on server01 Feb 8 13:29:07 server01 crmd: [19470]: info: notify_deleted: Notifying 19247_crm_resource on server01 that pnp was deleted Feb 8 13:29:07 server01 crmd: [19470]: info: send_direct_ack: ACK'ing resource op pnp_delete_60000 from 0:0:crm-resource-19247: lrm_invoke-lrmd-1328704147-433 Feb 8 13:29:07 server01 cibadmin: [19250]: info: Invoked: cibadmin -Ql -o resources Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/555, version=0.131.4): ok (rc=0) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server01']//lrm_resource[@id='pnp'] (origin=local/crmd/556, version=0.131.5): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=fs-data_monitor_0, magic=0:7;5:216:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.130.31) : Resource op removal Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change Feb 8 13:29:07 server01 crmd: [19470]: info: need_abort: Aborting on change to admin_epoch Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=nagios-ip_monitor_0, magic=0:7;6:216:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.2) : Resource op removal Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=apache2_monitor_0, magic=0:7;7:216:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.3) : Resource op removal Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=nagios_monitor_0, magic=0:7;8:216:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.4) : Resource op removal Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=pnp_monitor_0, magic=0:7;9:216:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.5) : Resource op removal Feb 8 13:29:07 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] Feb 8 13:29:07 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:29:07 server01 crmd: [19470]: info: do_pe_invoke: Query 559: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:29:07 server01 crmd: [19470]: info: do_pe_invoke: Query 560: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:29:07 server01 crmd: [19470]: info: do_pe_invoke: Query 561: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:29:07 server01 crmd: [19470]: info: do_pe_invoke: Query 562: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:29:07 server01 crmd: [19470]: info: do_pe_invoke: Query 563: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:29:07 server01 crmd: [19470]: info: do_pe_invoke: Query 564: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/558, version=0.131.5): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=564, ref=pe_calc-dc-1328704147-434, seq=132, quorate=1 Feb 8 13:29:07 server01 crmd: [19470]: info: config_query_callback: Checking for expired actions every 900000ms Feb 8 13:29:07 server01 crmd: [19470]: info: config_query_callback: Sending expected-votes=2 to corosync Feb 8 13:29:07 server01 crmd: [19470]: info: ais_dispatch: Membership 132: quorum retained Feb 8 13:29:07 server01 crmd: [19470]: info: crm_ais_dispatch: Setting expected votes to 2 Feb 8 13:29:07 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:29:07 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:29:07 server01 cibadmin: [19252]: info: Invoked: cibadmin -Ql -o resources Feb 8 13:29:07 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:29:07 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:29:07 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:29:07 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:29:07 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:29:07 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server02 Feb 8 13:29:07 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:29:07 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server02 Feb 8 13:29:07 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server02 Feb 8 13:29:07 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server02 Feb 8 13:29:07 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server02 Feb 8 13:29:07 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server02 Feb 8 13:29:07 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:29:07 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 13:29:07 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:29:07 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:29:07 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:29:07 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:29:07 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:29:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:07 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:07 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server02) Feb 8 13:29:07 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server02) Feb 8 13:29:07 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server02) Feb 8 13:29:07 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server02) Feb 8 13:29:07 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server02) Feb 8 13:29:07 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 13:29:07 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 13:29:07 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:29:07 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 219: 7 actions in 7 synapses Feb 8 13:29:07 server01 crmd: [19470]: info: do_te_invoke: Processing graph 219 (ref=pe_calc-dc-1328704147-434) derived from /var/lib/pengine/pe-input-6644.bz2 Feb 8 13:29:07 server01 crmd: [19470]: info: te_rsc_command: Initiating action 9: monitor fs-data_monitor_0 on server01 (local) Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=9:219:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=fs-data_monitor_0 ) Feb 8 13:29:07 server01 lrmd: [19467]: info: rsc:fs-data:137: probe Feb 8 13:29:07 server01 crmd: [19470]: info: te_rsc_command: Initiating action 10: monitor nagios-ip_monitor_0 on server01 (local) Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=10:219:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios-ip_monitor_0 ) Feb 8 13:29:07 server01 lrmd: [19467]: info: rsc:nagios-ip:138: probe Feb 8 13:29:07 server01 crmd: [19470]: info: te_rsc_command: Initiating action 11: monitor apache2_monitor_0 on server01 (local) Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=11:219:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=apache2_monitor_0 ) Feb 8 13:29:07 server01 lrmd: [19467]: info: rsc:apache2:139: probe Feb 8 13:29:07 server01 crmd: [19470]: info: te_rsc_command: Initiating action 12: monitor nagios_monitor_0 on server01 (local) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/567, version=0.131.5): ok (rc=0) Feb 8 13:29:07 server01 lrmd: [19467]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=12:219:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=nagios_monitor_0 ) Feb 8 13:29:07 server01 lrmd: [19467]: info: rsc:nagios:140: probe Feb 8 13:29:07 server01 crmd: [19470]: info: te_rsc_command: Initiating action 13: monitor pnp_monitor_0 on server01 (local) Feb 8 13:29:07 server01 lrmd: [19467]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Feb 8 13:29:07 server01 crmd: [19470]: info: do_lrm_rsc_op: Performing key=13:219:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1 op=pnp_monitor_0 ) Feb 8 13:29:07 server01 crm_resource: [19272]: info: Invoked: crm_resource -C -r supervision-grp -H server02 Feb 8 13:29:07 server01 pengine: [19469]: info: process_pe_message: Transition 219: PEngine Input stored in: /var/lib/pengine/pe-input-6644.bz2 Feb 8 13:29:07 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios_monitor_0 (call=140, rc=7, cib-update=568, confirmed=true) not running Feb 8 13:29:07 server01 crmd: [19470]: info: match_graph_event: Action nagios_monitor_0 (12) confirmed on server01 (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: process_lrm_event: LRM operation nagios-ip_monitor_0 (call=138, rc=7, cib-update=569, confirmed=true) not running Feb 8 13:29:07 server01 crmd: [19470]: info: process_lrm_event: LRM operation fs-data_monitor_0 (call=137, rc=7, cib-update=570, confirmed=true) not running Feb 8 13:29:07 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_monitor_0 (10) confirmed on server01 (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: match_graph_event: Action fs-data_monitor_0 (9) confirmed on server01 (rc=0) Feb 8 13:29:07 server01 apache[19255]: INFO: apache not running Feb 8 13:29:07 server01 crmd: [19470]: info: process_lrm_event: LRM operation apache2_monitor_0 (call=139, rc=7, cib-update=571, confirmed=true) not running Feb 8 13:29:07 server01 crmd: [19470]: info: match_graph_event: Action apache2_monitor_0 (11) confirmed on server01 (rc=0) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']//lrm_resource[@id='fs-data'] (origin=server02/crmd/55, version=0.131.10): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=fs-data_monitor_0, magic=0:7;6:217:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.10) : Resource op removal Feb 8 13:29:07 server01 crmd: [19470]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Feb 8 13:29:07 server01 crmd: [19470]: info: update_abort_priority: Abort action done superceeded by restart Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=server02/crmd/57, version=0.131.10): ok (rc=0) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']//lrm_resource[@id='nagios-ip'] (origin=server02/crmd/58, version=0.131.11): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=nagios-ip_monitor_0, magic=0:7;7:217:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.11) : Resource op removal Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=server02/crmd/60, version=0.131.11): ok (rc=0) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']//lrm_resource[@id='apache2'] (origin=server02/crmd/61, version=0.131.12): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=apache2_monitor_0, magic=0:0;8:217:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.12) : Resource op removal Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=server02/crmd/63, version=0.131.12): ok (rc=0) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']//lrm_resource[@id='nagios'] (origin=server02/crmd/64, version=0.131.13): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=nagios_monitor_0, magic=0:7;9:217:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.13) : Resource op removal Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=server02/crmd/66, version=0.131.13): ok (rc=0) Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='server02']//lrm_resource[@id='pnp'] (origin=server02/crmd/67, version=0.131.14): ok (rc=0) Feb 8 13:29:07 server01 crmd: [19470]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=pnp_monitor_0, magic=0:7;10:217:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.14) : Resource op removal Feb 8 13:29:07 server01 cib: [19466]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=server02/crmd/69, version=0.131.14): ok (rc=0) Feb 8 13:29:08 server01 lrmd: [19467]: info: rsc:pnp:141: probe Feb 8 13:29:08 server01 crmd: [19470]: info: process_lrm_event: LRM operation pnp_monitor_0 (call=141, rc=7, cib-update=572, confirmed=true) not running Feb 8 13:29:08 server01 crmd: [19470]: info: match_graph_event: Action pnp_monitor_0 (13) confirmed on server01 (rc=0) Feb 8 13:29:08 server01 crmd: [19470]: info: te_rsc_command: Initiating action 8: probe_complete probe_complete on server01 (local) - no waiting Feb 8 13:29:08 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:29:08 server01 crmd: [19470]: notice: run_graph: Transition 219 (Complete=6, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-6644.bz2): Stopped Feb 8 13:29:08 server01 crmd: [19470]: info: te_graph_trigger: Transition 219 is now complete Feb 8 13:29:08 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:29:08 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:29:08 server01 crmd: [19470]: info: do_pe_invoke: Query 573: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:29:08 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=573, ref=pe_calc-dc-1328704148-442, seq=132, quorate=1 Feb 8 13:29:08 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:29:08 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:29:08 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:29:08 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:29:08 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:29:08 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:29:08 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:29:08 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:29:08 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Stopped Feb 8 13:29:08 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Stopped Feb 8 13:29:08 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Stopped Feb 8 13:29:08 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Stopped Feb 8 13:29:08 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Stopped Feb 8 13:29:08 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:29:08 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 13:29:08 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:29:08 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:29:08 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:29:08 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:29:08 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:29:08 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (5s) for nagios-ip on server02 Feb 8 13:29:08 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (10s) for apache2 on server02 Feb 8 13:29:08 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for nagios on server02 Feb 8 13:29:08 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for pnp on server02 Feb 8 13:29:08 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:08 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:08 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:08 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:08 server01 pengine: [19469]: notice: LogActions: Start fs-data (server02) Feb 8 13:29:08 server01 pengine: [19469]: notice: LogActions: Start nagios-ip (server02) Feb 8 13:29:08 server01 pengine: [19469]: notice: LogActions: Start apache2 (server02) Feb 8 13:29:08 server01 pengine: [19469]: notice: LogActions: Start nagios (server02) Feb 8 13:29:08 server01 pengine: [19469]: notice: LogActions: Start pnp (server02) Feb 8 13:29:08 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 13:29:08 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 13:29:08 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:29:08 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 220: 26 actions in 26 synapses Feb 8 13:29:08 server01 crmd: [19470]: info: do_te_invoke: Processing graph 220 (ref=pe_calc-dc-1328704148-442) derived from /var/lib/pengine/pe-input-6645.bz2 Feb 8 13:29:08 server01 crmd: [19470]: info: te_rsc_command: Initiating action 6: monitor fs-data_monitor_0 on server02 Feb 8 13:29:08 server01 crmd: [19470]: info: te_rsc_command: Initiating action 7: monitor nagios-ip_monitor_0 on server02 Feb 8 13:29:08 server01 crmd: [19470]: info: te_rsc_command: Initiating action 8: monitor apache2_monitor_0 on server02 Feb 8 13:29:08 server01 crmd: [19470]: info: te_rsc_command: Initiating action 9: monitor nagios_monitor_0 on server02 Feb 8 13:29:08 server01 crmd: [19470]: info: te_rsc_command: Initiating action 10: monitor pnp_monitor_0 on server02 Feb 8 13:29:08 server01 pengine: [19469]: info: process_pe_message: Transition 220: PEngine Input stored in: /var/lib/pengine/pe-input-6645.bz2 Feb 8 13:29:08 server01 crmd: [19470]: WARN: status_from_rc: Action 6 (fs-data_monitor_0) on server02 failed (target: 7 vs. rc: 0): Error Feb 8 13:29:08 server01 crmd: [19470]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=fs-data_monitor_0, magic=0:0;6:220:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.16) : Event failed Feb 8 13:29:08 server01 crmd: [19470]: info: update_abort_priority: Abort priority upgraded from 0 to 1 Feb 8 13:29:08 server01 crmd: [19470]: info: update_abort_priority: Abort action done superceeded by restart Feb 8 13:29:08 server01 crmd: [19470]: info: match_graph_event: Action fs-data_monitor_0 (6) confirmed on server02 (rc=4) Feb 8 13:29:08 server01 crmd: [19470]: WARN: status_from_rc: Action 9 (nagios_monitor_0) on server02 failed (target: 7 vs. rc: 0): Error Feb 8 13:29:08 server01 crmd: [19470]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=nagios_monitor_0, magic=0:0;9:220:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.17) : Event failed Feb 8 13:29:08 server01 crmd: [19470]: info: match_graph_event: Action nagios_monitor_0 (9) confirmed on server02 (rc=4) Feb 8 13:29:08 server01 crmd: [19470]: WARN: status_from_rc: Action 7 (nagios-ip_monitor_0) on server02 failed (target: 7 vs. rc: 0): Error Feb 8 13:29:08 server01 crmd: [19470]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=nagios-ip_monitor_0, magic=0:0;7:220:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.18) : Event failed Feb 8 13:29:08 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_monitor_0 (7) confirmed on server02 (rc=4) Feb 8 13:29:08 server01 crmd: [19470]: WARN: status_from_rc: Action 8 (apache2_monitor_0) on server02 failed (target: 7 vs. rc: 0): Error Feb 8 13:29:08 server01 crmd: [19470]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=apache2_monitor_0, magic=0:0;8:220:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.19) : Event failed Feb 8 13:29:08 server01 crmd: [19470]: info: match_graph_event: Action apache2_monitor_0 (8) confirmed on server02 (rc=4) Feb 8 13:29:09 server01 crmd: [19470]: WARN: status_from_rc: Action 10 (pnp_monitor_0) on server02 failed (target: 7 vs. rc: 0): Error Feb 8 13:29:09 server01 crmd: [19470]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=pnp_monitor_0, magic=0:0;10:220:7:ada70012-d57b-48f3-9bbb-9c35d4f020f1, cib=0.131.20) : Event failed Feb 8 13:29:09 server01 crmd: [19470]: info: match_graph_event: Action pnp_monitor_0 (10) confirmed on server02 (rc=4) Feb 8 13:29:09 server01 crmd: [19470]: info: te_rsc_command: Initiating action 5: probe_complete probe_complete on server02 - no waiting Feb 8 13:29:09 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:29:09 server01 crmd: [19470]: notice: run_graph: Transition 220 (Complete=6, Pending=0, Fired=0, Skipped=20, Incomplete=0, Source=/var/lib/pengine/pe-input-6645.bz2): Stopped Feb 8 13:29:09 server01 crmd: [19470]: info: te_graph_trigger: Transition 220 is now complete Feb 8 13:29:09 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:29:09 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:29:09 server01 crmd: [19470]: info: do_pe_invoke: Query 574: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:29:09 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=574, ref=pe_calc-dc-1328704149-449, seq=132, quorate=1 Feb 8 13:29:09 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:29:09 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:29:09 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:29:09 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:29:09 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:29:09 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation fs-data_monitor_0 found resource fs-data active on server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios_monitor_0 found resource nagios active on server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios-ip_monitor_0 found resource nagios-ip active on server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation pnp_monitor_0 found resource pnp active on server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:29:09 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:29:09 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 13:29:09 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:29:09 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:29:09 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:29:09 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:29:09 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:29:09 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (5s) for nagios-ip on server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (10s) for apache2 on server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for nagios on server02 Feb 8 13:29:09 server01 pengine: [19469]: notice: RecurringOp: Start recurring monitor (30s) for pnp on server02 Feb 8 13:29:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server02) Feb 8 13:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server02) Feb 8 13:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server02) Feb 8 13:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server02) Feb 8 13:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server02) Feb 8 13:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 13:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 13:29:09 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:29:09 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 221: 4 actions in 4 synapses Feb 8 13:29:09 server01 crmd: [19470]: info: do_te_invoke: Processing graph 221 (ref=pe_calc-dc-1328704149-449) derived from /var/lib/pengine/pe-input-6646.bz2 Feb 8 13:29:09 server01 crmd: [19470]: info: te_rsc_command: Initiating action 10: monitor nagios-ip_monitor_5000 on server02 Feb 8 13:29:09 server01 crmd: [19470]: info: te_rsc_command: Initiating action 13: monitor apache2_monitor_10000 on server02 Feb 8 13:29:09 server01 crmd: [19470]: info: te_rsc_command: Initiating action 16: monitor nagios_monitor_30000 on server02 Feb 8 13:29:09 server01 crmd: [19470]: info: te_rsc_command: Initiating action 19: monitor pnp_monitor_30000 on server02 Feb 8 13:29:09 server01 pengine: [19469]: info: process_pe_message: Transition 221: PEngine Input stored in: /var/lib/pengine/pe-input-6646.bz2 Feb 8 13:29:09 server01 crmd: [19470]: info: match_graph_event: Action pnp_monitor_30000 (19) confirmed on server02 (rc=0) Feb 8 13:29:09 server01 crmd: [19470]: info: match_graph_event: Action nagios-ip_monitor_5000 (10) confirmed on server02 (rc=0) Feb 8 13:29:09 server01 crmd: [19470]: info: match_graph_event: Action nagios_monitor_30000 (16) confirmed on server02 (rc=0) Feb 8 13:29:09 server01 crmd: [19470]: info: match_graph_event: Action apache2_monitor_10000 (13) confirmed on server02 (rc=0) Feb 8 13:29:09 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:29:09 server01 crmd: [19470]: notice: run_graph: Transition 221 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6646.bz2): Complete Feb 8 13:29:09 server01 crmd: [19470]: info: te_graph_trigger: Transition 221 is now complete Feb 8 13:29:09 server01 crmd: [19470]: info: notify_crmd: Transition 221 status: done - Feb 8 13:29:09 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:29:09 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 13:31:25 server01 cib: [19466]: info: cib_stats: Processed 301 operations (3455.00us average, 0% utilization) in the last 10min Feb 8 13:41:25 server01 cib: [19466]: info: cib_stats: Processed 40 operations (0.00us average, 0% utilization) in the last 10min Feb 8 13:44:09 server01 crmd: [19470]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Feb 8 13:44:09 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Feb 8 13:44:09 server01 crmd: [19470]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Feb 8 13:44:09 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:44:09 server01 crmd: [19470]: info: do_pe_invoke: Query 575: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:44:09 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=575, ref=pe_calc-dc-1328705049-454, seq=132, quorate=1 Feb 8 13:44:09 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:44:09 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:44:09 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:44:09 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:44:09 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:44:09 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:44:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation fs-data_monitor_0 found resource fs-data active on server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios_monitor_0 found resource nagios active on server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios-ip_monitor_0 found resource nagios-ip active on server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation pnp_monitor_0 found resource pnp active on server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:44:09 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server02 Feb 8 13:44:09 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:44:09 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 13:44:09 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:44:09 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:44:09 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:44:09 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:44:09 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:44:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:44:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:44:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:44:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:44:09 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server02) Feb 8 13:44:09 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server02) Feb 8 13:44:09 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server02) Feb 8 13:44:09 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server02) Feb 8 13:44:09 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server02) Feb 8 13:44:09 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 13:44:09 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 13:44:09 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:44:09 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 222: 0 actions in 0 synapses Feb 8 13:44:09 server01 crmd: [19470]: info: do_te_invoke: Processing graph 222 (ref=pe_calc-dc-1328705049-454) derived from /var/lib/pengine/pe-input-6647.bz2 Feb 8 13:44:09 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:44:09 server01 crmd: [19470]: notice: run_graph: Transition 222 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6647.bz2): Complete Feb 8 13:44:09 server01 crmd: [19470]: info: te_graph_trigger: Transition 222 is now complete Feb 8 13:44:09 server01 crmd: [19470]: info: notify_crmd: Transition 222 status: done - Feb 8 13:44:09 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:44:09 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 13:44:09 server01 pengine: [19469]: info: process_pe_message: Transition 222: PEngine Input stored in: /var/lib/pengine/pe-input-6647.bz2 Feb 8 13:51:25 server01 cib: [19466]: info: cib_stats: Processed 43 operations (465.00us average, 0% utilization) in the last 10min Feb 8 13:59:09 server01 crmd: [19470]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Feb 8 13:59:09 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Feb 8 13:59:09 server01 crmd: [19470]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Feb 8 13:59:09 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 13:59:09 server01 crmd: [19470]: info: do_pe_invoke: Query 576: Requesting the current CIB: S_POLICY_ENGINE Feb 8 13:59:09 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=576, ref=pe_calc-dc-1328705949-455, seq=132, quorate=1 Feb 8 13:59:09 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 13:59:09 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 13:59:09 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 13:59:09 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 13:59:09 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 13:59:09 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 13:59:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation fs-data_monitor_0 found resource fs-data active on server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios_monitor_0 found resource nagios active on server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios-ip_monitor_0 found resource nagios-ip active on server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation pnp_monitor_0 found resource pnp active on server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 13:59:09 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server02 Feb 8 13:59:09 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 13:59:09 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 13:59:09 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 13:59:09 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:59:09 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:59:09 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 13:59:09 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 13:59:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:59:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:59:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:59:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 13:59:09 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server02) Feb 8 13:59:09 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server02) Feb 8 13:59:09 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server02) Feb 8 13:59:09 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server02) Feb 8 13:59:09 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server02) Feb 8 13:59:09 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 13:59:09 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 13:59:09 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 13:59:09 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 223: 0 actions in 0 synapses Feb 8 13:59:09 server01 crmd: [19470]: info: do_te_invoke: Processing graph 223 (ref=pe_calc-dc-1328705949-455) derived from /var/lib/pengine/pe-input-6648.bz2 Feb 8 13:59:09 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 13:59:09 server01 crmd: [19470]: notice: run_graph: Transition 223 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6648.bz2): Complete Feb 8 13:59:09 server01 crmd: [19470]: info: te_graph_trigger: Transition 223 is now complete Feb 8 13:59:09 server01 crmd: [19470]: info: notify_crmd: Transition 223 status: done - Feb 8 13:59:09 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 13:59:09 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 13:59:09 server01 pengine: [19469]: info: process_pe_message: Transition 223: PEngine Input stored in: /var/lib/pengine/pe-input-6648.bz2 Feb 8 14:00:26 server01 nslcd[1978]: [c4b50e] "supervision": user not found Feb 8 14:01:25 server01 cib: [19466]: info: cib_stats: Processed 42 operations (714.00us average, 0% utilization) in the last 10min Feb 8 14:11:25 server01 cib: [19466]: info: cib_stats: Processed 39 operations (0.00us average, 0% utilization) in the last 10min Feb 8 14:14:09 server01 crmd: [19470]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Feb 8 14:14:09 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Feb 8 14:14:09 server01 crmd: [19470]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Feb 8 14:14:09 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 14:14:09 server01 crmd: [19470]: info: do_pe_invoke: Query 577: Requesting the current CIB: S_POLICY_ENGINE Feb 8 14:14:09 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=577, ref=pe_calc-dc-1328706849-456, seq=132, quorate=1 Feb 8 14:14:09 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 14:14:09 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 14:14:09 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 14:14:09 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 14:14:09 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 14:14:09 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 14:14:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation fs-data_monitor_0 found resource fs-data active on server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios_monitor_0 found resource nagios active on server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios-ip_monitor_0 found resource nagios-ip active on server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation pnp_monitor_0 found resource pnp active on server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 14:14:09 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server02 Feb 8 14:14:09 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 14:14:09 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 14:14:09 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 14:14:09 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 14:14:09 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 14:14:09 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 14:14:09 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 14:14:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:14:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:14:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:14:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:14:09 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server02) Feb 8 14:14:09 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server02) Feb 8 14:14:09 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server02) Feb 8 14:14:09 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server02) Feb 8 14:14:09 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server02) Feb 8 14:14:09 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 14:14:09 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 14:14:09 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 14:14:09 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 224: 0 actions in 0 synapses Feb 8 14:14:09 server01 crmd: [19470]: info: do_te_invoke: Processing graph 224 (ref=pe_calc-dc-1328706849-456) derived from /var/lib/pengine/pe-input-6649.bz2 Feb 8 14:14:09 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 14:14:09 server01 crmd: [19470]: notice: run_graph: Transition 224 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6649.bz2): Complete Feb 8 14:14:09 server01 crmd: [19470]: info: te_graph_trigger: Transition 224 is now complete Feb 8 14:14:09 server01 crmd: [19470]: info: notify_crmd: Transition 224 status: done - Feb 8 14:14:09 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 14:14:09 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 14:14:09 server01 pengine: [19469]: info: process_pe_message: Transition 224: PEngine Input stored in: /var/lib/pengine/pe-input-6649.bz2 Feb 8 14:16:10 server01 cibadmin: [29534]: info: Invoked: cibadmin -Ql Feb 8 14:21:25 server01 cib: [19466]: info: cib_stats: Processed 42 operations (476.00us average, 0% utilization) in the last 10min Feb 8 14:28:29 server01 lrmd: [19467]: info: rsc:drbd-nagios:1:131: monitor Feb 8 14:29:09 server01 crmd: [19470]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Feb 8 14:29:09 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Feb 8 14:29:09 server01 crmd: [19470]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Feb 8 14:29:09 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 14:29:09 server01 crmd: [19470]: info: do_pe_invoke: Query 578: Requesting the current CIB: S_POLICY_ENGINE Feb 8 14:29:09 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=578, ref=pe_calc-dc-1328707749-457, seq=132, quorate=1 Feb 8 14:29:09 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 14:29:09 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 14:29:09 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 14:29:09 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 14:29:09 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 14:29:09 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 14:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation fs-data_monitor_0 found resource fs-data active on server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios_monitor_0 found resource nagios active on server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios-ip_monitor_0 found resource nagios-ip active on server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: unpack_rsc_op: Operation pnp_monitor_0 found resource pnp active on server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 14:29:09 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server02 Feb 8 14:29:09 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 14:29:09 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 14:29:09 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 14:29:09 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 14:29:09 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 14:29:09 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 14:29:09 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 14:29:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:29:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:29:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:29:09 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server02) Feb 8 14:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server02) Feb 8 14:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server02) Feb 8 14:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server02) Feb 8 14:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server02) Feb 8 14:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 14:29:09 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 14:29:09 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 14:29:09 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 225: 0 actions in 0 synapses Feb 8 14:29:09 server01 crmd: [19470]: info: do_te_invoke: Processing graph 225 (ref=pe_calc-dc-1328707749-457) derived from /var/lib/pengine/pe-input-6650.bz2 Feb 8 14:29:09 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 14:29:09 server01 crmd: [19470]: notice: run_graph: Transition 225 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6650.bz2): Complete Feb 8 14:29:09 server01 crmd: [19470]: info: te_graph_trigger: Transition 225 is now complete Feb 8 14:29:09 server01 crmd: [19470]: info: notify_crmd: Transition 225 status: done - Feb 8 14:29:09 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 14:29:09 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 14:29:09 server01 pengine: [19469]: info: process_pe_message: Transition 225: PEngine Input stored in: /var/lib/pengine/pe-input-6650.bz2 Feb 8 14:31:25 server01 cib: [19466]: info: cib_stats: Processed 41 operations (975.00us average, 0% utilization) in the last 10min Feb 8 14:41:25 server01 cib: [19466]: info: cib_stats: Processed 40 operations (0.00us average, 0% utilization) in the last 10min Feb 8 14:44:09 server01 crmd: [19470]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Feb 8 14:44:09 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Feb 8 14:44:09 server01 crmd: [19470]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Feb 8 14:44:09 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 14:44:09 server01 crmd: [19470]: info: do_pe_invoke: Query 579: Requesting the current CIB: S_POLICY_ENGINE Feb 8 14:44:10 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=579, ref=pe_calc-dc-1328708650-458, seq=132, quorate=1 Feb 8 14:44:10 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 14:44:10 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 14:44:10 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 14:44:10 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 14:44:10 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 14:44:10 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 14:44:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation fs-data_monitor_0 found resource fs-data active on server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios_monitor_0 found resource nagios active on server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios-ip_monitor_0 found resource nagios-ip active on server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation pnp_monitor_0 found resource pnp active on server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 14:44:10 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server02 Feb 8 14:44:10 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 14:44:10 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 14:44:10 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 14:44:10 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 14:44:10 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 14:44:10 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 14:44:10 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 14:44:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:44:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:44:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:44:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:44:10 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server02) Feb 8 14:44:10 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server02) Feb 8 14:44:10 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server02) Feb 8 14:44:10 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server02) Feb 8 14:44:10 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server02) Feb 8 14:44:10 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 14:44:10 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 14:44:10 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 14:44:10 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 226: 0 actions in 0 synapses Feb 8 14:44:10 server01 crmd: [19470]: info: do_te_invoke: Processing graph 226 (ref=pe_calc-dc-1328708650-458) derived from /var/lib/pengine/pe-input-6651.bz2 Feb 8 14:44:10 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 14:44:10 server01 crmd: [19470]: notice: run_graph: Transition 226 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6651.bz2): Complete Feb 8 14:44:10 server01 crmd: [19470]: info: te_graph_trigger: Transition 226 is now complete Feb 8 14:44:10 server01 crmd: [19470]: info: notify_crmd: Transition 226 status: done - Feb 8 14:44:10 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 14:44:10 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 14:44:10 server01 pengine: [19469]: info: process_pe_message: Transition 226: PEngine Input stored in: /var/lib/pengine/pe-input-6651.bz2 Feb 8 14:59:10 server01 crmd: [19470]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Feb 8 14:59:10 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Feb 8 14:59:10 server01 crmd: [19470]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Feb 8 14:59:10 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 14:59:10 server01 crmd: [19470]: info: do_pe_invoke: Query 580: Requesting the current CIB: S_POLICY_ENGINE Feb 8 14:59:10 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=580, ref=pe_calc-dc-1328709550-459, seq=132, quorate=1 Feb 8 14:59:10 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 14:59:10 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 14:59:10 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 14:59:10 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 14:59:10 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 14:59:10 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 14:59:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation fs-data_monitor_0 found resource fs-data active on server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios_monitor_0 found resource nagios active on server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios-ip_monitor_0 found resource nagios-ip active on server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation pnp_monitor_0 found resource pnp active on server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 14:59:10 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server02 Feb 8 14:59:10 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 14:59:10 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 14:59:10 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 14:59:10 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 14:59:10 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 14:59:10 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 14:59:10 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 14:59:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:59:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:59:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:59:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 14:59:10 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server02) Feb 8 14:59:10 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server02) Feb 8 14:59:10 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server02) Feb 8 14:59:10 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server02) Feb 8 14:59:10 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server02) Feb 8 14:59:10 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 14:59:10 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 14:59:10 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 14:59:10 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 227: 0 actions in 0 synapses Feb 8 14:59:10 server01 crmd: [19470]: info: do_te_invoke: Processing graph 227 (ref=pe_calc-dc-1328709550-459) derived from /var/lib/pengine/pe-input-6652.bz2 Feb 8 14:59:10 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 14:59:10 server01 crmd: [19470]: notice: run_graph: Transition 227 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6652.bz2): Complete Feb 8 14:59:10 server01 crmd: [19470]: info: te_graph_trigger: Transition 227 is now complete Feb 8 14:59:10 server01 crmd: [19470]: info: notify_crmd: Transition 227 status: done - Feb 8 14:59:10 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 14:59:10 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 14:59:10 server01 pengine: [19469]: info: process_pe_message: Transition 227: PEngine Input stored in: /var/lib/pengine/pe-input-6652.bz2 Feb 8 15:01:25 server01 cib: [19466]: info: cib_stats: Processed 41 operations (243.00us average, 0% utilization) in the last 10min Feb 8 15:10:26 server01 nslcd[1978]: [4aef65] "supervision": user not found Feb 8 15:11:25 server01 cib: [19466]: info: cib_stats: Processed 40 operations (0.00us average, 0% utilization) in the last 10min Feb 8 15:14:10 server01 crmd: [19470]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Feb 8 15:14:10 server01 crmd: [19470]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Feb 8 15:14:10 server01 crmd: [19470]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Feb 8 15:14:10 server01 crmd: [19470]: info: do_state_transition: All 2 cluster nodes are eligible to run resources. Feb 8 15:14:10 server01 crmd: [19470]: info: do_pe_invoke: Query 581: Requesting the current CIB: S_POLICY_ENGINE Feb 8 15:14:10 server01 crmd: [19470]: info: do_pe_invoke_callback: Invoking the PE: query=581, ref=pe_calc-dc-1328710450-460, seq=132, quorate=1 Feb 8 15:14:10 server01 pengine: [19469]: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 8 15:14:10 server01 pengine: [19469]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Feb 8 15:14:10 server01 pengine: [19469]: info: determine_online_status: Node server01 is online Feb 8 15:14:10 server01 pengine: [19469]: WARN: unpack_rsc_op: Processing failed op drbd-nagios:1_demote_0 on server01: unknown exec error (-2) Feb 8 15:14:10 server01 pengine: [19469]: WARN: unpack_rsc_op: Forcing drbd-nagios:1 to stop after a failed demote action Feb 8 15:14:10 server01 pengine: [19469]: info: determine_online_status: Node server02 is online Feb 8 15:14:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation drbd-nagios:0_monitor_0 found resource drbd-nagios:0 active on server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation fs-data_monitor_0 found resource fs-data active on server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios_monitor_0 found resource nagios active on server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation nagios-ip_monitor_0 found resource nagios-ip active on server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation apache2_monitor_0 found resource apache2 active on server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: unpack_rsc_op: Operation pnp_monitor_0 found resource pnp active on server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: group_print: Resource Group: supervision-grp Feb 8 15:14:10 server01 pengine: [19469]: notice: native_print: fs-data (ocf::heartbeat:Filesystem): Started server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: native_print: nagios-ip (ocf::heartbeat:IPaddr2): Started server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: native_print: apache2 (ocf::heartbeat:apache): Started server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: native_print: nagios (lsb:nagios3): Started server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: native_print: pnp (lsb:npcd): Started server02 Feb 8 15:14:10 server01 pengine: [19469]: notice: clone_print: Master/Slave Set: ms-drbd-supervision Feb 8 15:14:10 server01 pengine: [19469]: notice: short_print: Masters: [ server02 ] Feb 8 15:14:10 server01 pengine: [19469]: notice: short_print: Slaves: [ server01 ] Feb 8 15:14:10 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 15:14:10 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 15:14:10 server01 pengine: [19469]: info: master_color: Promoting drbd-nagios:0 (Master server02) Feb 8 15:14:10 server01 pengine: [19469]: info: master_color: ms-drbd-supervision: Promoted 1 instances of a possible 1 to master Feb 8 15:14:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 15:14:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 15:14:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 15:14:10 server01 pengine: [19469]: ERROR: create_notification_boundaries: Creating boundaries for ms-drbd-supervision Feb 8 15:14:10 server01 pengine: [19469]: notice: LogActions: Leave resource fs-data (Started server02) Feb 8 15:14:10 server01 pengine: [19469]: notice: LogActions: Leave resource nagios-ip (Started server02) Feb 8 15:14:10 server01 pengine: [19469]: notice: LogActions: Leave resource apache2 (Started server02) Feb 8 15:14:10 server01 pengine: [19469]: notice: LogActions: Leave resource nagios (Started server02) Feb 8 15:14:10 server01 pengine: [19469]: notice: LogActions: Leave resource pnp (Started server02) Feb 8 15:14:10 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:0 (Master server02) Feb 8 15:14:10 server01 pengine: [19469]: notice: LogActions: Leave resource drbd-nagios:1 (Slave server01) Feb 8 15:14:10 server01 crmd: [19470]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Feb 8 15:14:10 server01 crmd: [19470]: info: unpack_graph: Unpacked transition 228: 0 actions in 0 synapses Feb 8 15:14:10 server01 crmd: [19470]: info: do_te_invoke: Processing graph 228 (ref=pe_calc-dc-1328710450-460) derived from /var/lib/pengine/pe-input-6653.bz2 Feb 8 15:14:10 server01 crmd: [19470]: info: run_graph: ==================================================== Feb 8 15:14:10 server01 crmd: [19470]: notice: run_graph: Transition 228 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-6653.bz2): Complete Feb 8 15:14:10 server01 crmd: [19470]: info: te_graph_trigger: Transition 228 is now complete Feb 8 15:14:10 server01 crmd: [19470]: info: notify_crmd: Transition 228 status: done - Feb 8 15:14:10 server01 crmd: [19470]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Feb 8 15:14:10 server01 crmd: [19470]: info: do_state_transition: Starting PEngine Recheck Timer Feb 8 15:14:10 server01 pengine: [19469]: info: process_pe_message: Transition 228: PEngine Input stored in: /var/lib/pengine/pe-input-6653.bz2