Sep 24 15:38:44 ifds2 Stateful(IFDS-Stateful)[23904]: ERROR: Stateful ifds2 main monitor
Sep 24 15:38:54 ifds2 Stateful(IFDS-Stateful)[24270]: ERROR: Stateful ifds2 main monitor
Sep 24 15:39:04 ifds2 Stateful(IFDS-Stateful)[24498]: ERROR: Stateful ifds2 main monitor
Sep 24 15:39:14 ifds2 Stateful(IFDS-Stateful)[24694]: ERROR: Stateful ifds2 main monitor
Sep 24 15:39:24 ifds2 Stateful(IFDS-Stateful)[24917]: ERROR: Stateful ifds2 main monitor
Sep 24 15:39:34 ifds2 Stateful(IFDS-Stateful)[25193]: ERROR: Stateful ifds2 main monitor
Sep 24 15:39:44 ifds2 Stateful(IFDS-Stateful)[25425]: ERROR: Stateful ifds2 main monitor
Sep 24 15:39:54 ifds2 Stateful(IFDS-Stateful)[25659]: ERROR: Stateful ifds2 main monitor
Sep 24 15:40:04 ifds2 Stateful(IFDS-Stateful)[25861]: ERROR: Stateful ifds2 main monitor
Sep 24 15:40:10 ifds2 Stateful(IFDS-Stateful)[26003]: ERROR: Stateful ifds2 main notify
Sep 24 15:40:10 ifds2 crmd[22923]:   notice: process_lrm_event: LRM operation IFDS-Stateful_notify_0 (call=57, rc=0, cib-update=0, confirmed=true) ok
Sep 24 15:40:10 ifds2 Stateful(IFDS-Stateful)[26011]: ERROR: Stateful ifds2 main notify
Sep 24 15:40:10 ifds2 crmd[22923]:   notice: process_lrm_event: LRM operation IFDS-Stateful_notify_0 (call=61, rc=0, cib-update=0, confirmed=true) ok
Sep 24 15:40:10 ifds2 corosync[19178]:  [CPG   ] got procleave message from cluster node 1
Sep 24 15:40:10 ifds2 crmd[22923]:   notice: peer_update_callback: Our peer on the DC is dead
Sep 24 15:40:10 ifds2 crmd[22923]:   notice: do_state_transition: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback ]
Sep 24 15:40:10 ifds2 crmd[22923]:   notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Sep 24 15:40:10 ifds2 corosync[19178]:  [CPG   ] got procleave message from cluster node 1
Sep 24 15:40:10 ifds2 corosync[19178]:  [CPG   ] got procleave message from cluster node 1
Sep 24 15:40:10 ifds2 corosync[19178]:  [CPG   ] got procleave message from cluster node 1
Sep 24 15:40:10 ifds2 corosync[19178]:  [CPG   ] got procleave message from cluster node 1
Sep 24 15:40:10 ifds2 attrd[22921]:   notice: attrd_local_callback: Sending full refresh (origin=crmd)
Sep 24 15:40:10 ifds2 attrd[22921]:   notice: attrd_trigger_update: Sending flush op to all hosts for: default_ping_set (100)
Sep 24 15:40:10 ifds2 attrd[22921]:   notice: attrd_trigger_update: Sending flush op to all hosts for: master-IFDS-Stateful (10)
Sep 24 15:40:10 ifds2 attrd[22921]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Sep 24 15:40:11 ifds2 corosync[19178]:  [CPG   ] got procjoin message from cluster node 1 (r(0) ip(10.40.0.77) r(1) ip(10.80.0.77) ) for pid 21783
Sep 24 15:40:11 ifds2 corosync[19178]:  [CPG   ] got procjoin message from cluster node 1 (r(0) ip(10.40.0.77) r(1) ip(10.80.0.77) ) for pid 21788
Sep 24 15:40:11 ifds2 corosync[19178]:  [CPG   ] got procjoin message from cluster node 1 (r(0) ip(10.40.0.77) r(1) ip(10.80.0.77) ) for pid 21786
Sep 24 15:40:11 ifds2 corosync[19178]:  [CPG   ] got procjoin message from cluster node 1 (r(0) ip(10.40.0.77) r(1) ip(10.80.0.77) ) for pid 21785
Sep 24 15:40:11 ifds2 pengine[22922]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Sep 24 15:40:11 ifds2 pengine[22922]:   notice: unpack_rsc_op: Operation monitor found resource IFDS-Stateful:0 active in master mode on ifds2
Sep 24 15:40:11 ifds2 pengine[22922]:   notice: process_pe_message: Calculated Transition 1: /var/lib/pacemaker/pengine/pe-input-1.bz2
Sep 24 15:40:11 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 19: monitor IFDS-Stateful_monitor_10000 on ifds2 (local)
Sep 24 15:40:11 ifds2 Stateful(IFDS-Stateful)[26048]: ERROR: Stateful ifds2 main monitor
Sep 24 15:40:11 ifds2 crmd[22923]:   notice: process_lrm_event: LRM operation IFDS-Stateful_monitor_10000 (call=64, rc=8, cib-update=87, confirmed=false) master
Sep 24 15:40:11 ifds2 crmd[22923]:   notice: run_graph: Transition 1 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1.bz2): Complete
Sep 24 15:40:11 ifds2 crmd[22923]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Sep 24 15:40:12 ifds2 corosync[19178]:  [CPG   ] got procjoin message from cluster node 1 (r(0) ip(10.40.0.77) r(1) ip(10.80.0.77) ) for pid 21790
Sep 24 15:40:12 ifds2 crmd[22923]:   notice: do_state_transition: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=peer_update_callback ]
Sep 24 15:40:14 ifds2 attrd[22921]:   notice: attrd_local_callback: Sending full refresh (origin=crmd)
Sep 24 15:40:14 ifds2 attrd[22921]:   notice: attrd_trigger_update: Sending flush op to all hosts for: default_ping_set (100)
Sep 24 15:40:14 ifds2 attrd[22921]:   notice: attrd_trigger_update: Sending flush op to all hosts for: master-IFDS-Stateful (10)
Sep 24 15:40:14 ifds2 attrd[22921]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Sep 24 15:40:15 ifds2 pengine[22922]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Sep 24 15:40:15 ifds2 pengine[22922]:   notice: unpack_rsc_op: Operation monitor found resource IFDS-Stateful:0 active in master mode on ifds2
Sep 24 15:40:15 ifds2 pengine[22922]:   notice: LogActions: Start   IFDS-VIP	(ifds1)
Sep 24 15:40:15 ifds2 pengine[22922]:   notice: LogActions: Start   IFDS-pingCheck:1	(ifds1)
Sep 24 15:40:15 ifds2 pengine[22922]:   notice: process_pe_message: Calculated Transition 2: /var/lib/pacemaker/pengine/pe-input-2.bz2
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 8: monitor IFDS-VIP_monitor_0 on ifds1
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 9: monitor IFDS-MIP1_monitor_0 on ifds1
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 10: monitor IFDS-MIP2_monitor_0 on ifds1
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 11: monitor IFDS-Stateful_monitor_0 on ifds1
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 12: monitor IFDS-pingCheck:1_monitor_0 on ifds1
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 7: probe_complete probe_complete on ifds1 - no waiting
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 14: start IFDS-VIP_start_0 on ifds1
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 54: start IFDS-pingCheck:1_start_0 on ifds1
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 55: monitor IFDS-pingCheck:1_monitor_2000 on ifds1
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 15: monitor IFDS-VIP_monitor_10000 on ifds1
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: run_graph: Transition 2 (Complete=13, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-2.bz2): Complete
Sep 24 15:40:15 ifds2 crmd[22923]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Sep 24 15:40:20 ifds2 crmd[22923]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Sep 24 15:40:20 ifds2 pengine[22922]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Sep 24 15:40:20 ifds2 pengine[22922]:   notice: unpack_rsc_op: Operation monitor found resource IFDS-Stateful:0 active in master mode on ifds2
Sep 24 15:40:20 ifds2 pengine[22922]:   notice: LogActions: Start   IFDS-Stateful:1	(ifds1)
Sep 24 15:40:20 ifds2 pengine[22922]:   notice: process_pe_message: Calculated Transition 3: /var/lib/pacemaker/pengine/pe-input-3.bz2
Sep 24 15:40:20 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 62: notify IFDS-Stateful_pre_notify_start_0 on ifds2 (local)
Sep 24 15:40:20 ifds2 Stateful(IFDS-Stateful)[26250]: ERROR: Stateful ifds2 main notify
Sep 24 15:40:20 ifds2 crmd[22923]:   notice: process_lrm_event: LRM operation IFDS-Stateful_notify_0 (call=68, rc=0, cib-update=0, confirmed=true) ok
Sep 24 15:40:20 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 25: start IFDS-Stateful_start_0 on ifds1
Sep 24 15:40:20 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 63: notify IFDS-Stateful_post_notify_start_0 on ifds2 (local)
Sep 24 15:40:20 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 64: notify IFDS-Stateful_post_notify_start_0 on ifds1
Sep 24 15:40:20 ifds2 Stateful(IFDS-Stateful)[26258]: ERROR: Stateful ifds2 main notify
Sep 24 15:40:20 ifds2 crmd[22923]:   notice: process_lrm_event: LRM operation IFDS-Stateful_notify_0 (call=72, rc=0, cib-update=0, confirmed=true) ok
Sep 24 15:40:20 ifds2 crmd[22923]:   notice: run_graph: Transition 3 (Complete=10, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-3.bz2): Stopped
Sep 24 15:40:20 ifds2 pengine[22922]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Sep 24 15:40:20 ifds2 pengine[22922]:   notice: unpack_rsc_op: Operation monitor found resource IFDS-Stateful:0 active in master mode on ifds2
Sep 24 15:40:20 ifds2 pengine[22922]:   notice: process_pe_message: Calculated Transition 4: /var/lib/pacemaker/pengine/pe-input-4.bz2
Sep 24 15:40:20 ifds2 crmd[22923]:   notice: te_rsc_command: Initiating action 27: monitor IFDS-Stateful_monitor_11000 on ifds1
Sep 24 15:40:21 ifds2 crmd[22923]:   notice: run_graph: Transition 4 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-4.bz2): Complete
Sep 24 15:40:21 ifds2 crmd[22923]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Sep 24 15:40:21 ifds2 attrd[22921]:  warning: attrd_cib_callback: Update 10 for probe_complete=true failed: Timer expired
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] IPC credentials authenticated (19178-28987-27)
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] connecting to client [28987]
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:22 ifds2 corosync[19178]:  [MAIN  ] connection created
Sep 24 15:42:22 ifds2 corosync[19178]:  [CMAP  ] lib_init_fn: conn=0x7f71071ada40
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] HUP conn (19178-28987-27)
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] qb_ipcs_disconnect(19178-28987-27) state:2
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] epoll_ctl(del): Bad file descriptor (9)
Sep 24 15:42:22 ifds2 corosync[19178]:  [MAIN  ] cs_ipcs_connection_closed() 
Sep 24 15:42:22 ifds2 corosync[19178]:  [CMAP  ] exit_fn for conn=0x7f71071ada40
Sep 24 15:42:22 ifds2 corosync[19178]:  [MAIN  ] cs_ipcs_connection_destroyed() 
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-19178-28987-27-header
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-19178-28987-27-header
Sep 24 15:42:22 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-19178-28987-27-header
Sep 24 15:42:22 ifds2 logger: Mark:pcmk:1380037342
Sep 24 15:42:26 ifds2 crm_verify[29547]:   notice: crm_log_args: Invoked: crm_verify -V -x /root/monitoring_stopped_on_ifds2/ifds2/cib.xml 
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] IPC credentials authenticated (19178-29660-27)
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] connecting to client [29660]
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:26 ifds2 corosync[19178]:  [MAIN  ] connection created
Sep 24 15:42:26 ifds2 corosync[19178]:  [CMAP  ] lib_init_fn: conn=0x7f71071ada40
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] HUP conn (19178-29660-27)
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] qb_ipcs_disconnect(19178-29660-27) state:2
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] epoll_ctl(del): Bad file descriptor (9)
Sep 24 15:42:26 ifds2 corosync[19178]:  [MAIN  ] cs_ipcs_connection_closed() 
Sep 24 15:42:26 ifds2 corosync[19178]:  [CMAP  ] exit_fn for conn=0x7f71071ada40
Sep 24 15:42:26 ifds2 corosync[19178]:  [MAIN  ] cs_ipcs_connection_destroyed() 
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-19178-29660-27-header
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-19178-29660-27-header
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-19178-29660-27-header
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] IPC credentials authenticated (19178-29662-27)
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] connecting to client [29662]
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:26 ifds2 corosync[19178]:  [MAIN  ] connection created
Sep 24 15:42:26 ifds2 corosync[19178]:  [CMAP  ] lib_init_fn: conn=0x7f71071ada40
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ]  writing total of: 8388620
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] HUP conn (19178-29662-27)
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] qb_ipcs_disconnect(19178-29662-27) state:2
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] epoll_ctl(del): Bad file descriptor (9)
Sep 24 15:42:26 ifds2 corosync[19178]:  [MAIN  ] cs_ipcs_connection_closed() 
Sep 24 15:42:26 ifds2 corosync[19178]:  [CMAP  ] exit_fn for conn=0x7f71071ada40
Sep 24 15:42:26 ifds2 corosync[19178]:  [MAIN  ] cs_ipcs_connection_destroyed() 
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-19178-29662-27-header
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-19178-29662-27-header
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-19178-29662-27-header
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] IPC credentials authenticated (19178-29667-27)
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] connecting to client [29667]
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] shm size:1048576; real_size:1048576; rb->word_size:262144
Sep 24 15:42:26 ifds2 corosync[19178]:  [MAIN  ] connection created
Sep 24 15:42:26 ifds2 corosync[19178]:  [CMAP  ] lib_init_fn: conn=0x7f71071ada40
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] HUP conn (19178-29667-27)
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] qb_ipcs_disconnect(19178-29667-27) state:2
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] epoll_ctl(del): Bad file descriptor (9)
Sep 24 15:42:26 ifds2 corosync[19178]:  [MAIN  ] cs_ipcs_connection_closed() 
Sep 24 15:42:26 ifds2 corosync[19178]:  [CMAP  ] exit_fn for conn=0x7f71071ada40
Sep 24 15:42:26 ifds2 corosync[19178]:  [MAIN  ] cs_ipcs_connection_destroyed() 
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-19178-29667-27-header
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-19178-29667-27-header
Sep 24 15:42:26 ifds2 corosync[19178]:  [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-19178-29667-27-header
