Mar 30 16:04:08 ha-d1 service_failover(service_failover)[27253]: INFO: UTM resource monitor
Mar 30 16:04:27 ha-d1 service_failover(service_failover)[12397]: INFO: UTM resource monitor
Mar 30 16:04:47 ha-d1 service_failover(service_failover)[13653]: INFO: UTM resource monitor
Mar 30 16:05:01 ha-d1 CROND[21628]: (root) CMD (/usr/sbin/atwos-heartbeat)
Mar 30 16:05:01 ha-d1 CROND[21629]: (root) CMD (run-parts /etc/cron.5min)
Mar 30 16:05:01 ha-d1 run-parts(/etc/cron.5min)[21629]: starting cfgchanges
Mar 30 16:05:01 ha-d1 run-parts(/etc/cron.5min)[21655]: finished cfgchanges
Mar 30 16:05:01 ha-d1 run-parts(/etc/cron.5min)[21629]: starting dnsmasq
Mar 30 16:05:01 ha-d1 run-parts(/etc/cron.5min)[21791]: finished dnsmasq
Mar 30 16:05:01 ha-d1 run-parts(/etc/cron.5min)[21629]: starting drbd-status
Mar 30 16:05:01 ha-d1 run-parts(/etc/cron.5min)[21810]: finished drbd-status
Mar 30 16:05:01 ha-d1 run-parts(/etc/cron.5min)[21629]: starting ha-utils
Mar 30 16:05:04 ha-d1 run-parts(/etc/cron.5min)[23851]: finished ha-utils
Mar 30 16:05:04 ha-d1 run-parts(/etc/cron.5min)[21629]: starting kswapd-check.sh
Mar 30 16:05:04 ha-d1 run-parts(/etc/cron.5min)[23876]: finished kswapd-check.sh
Mar 30 16:05:04 ha-d1 run-parts(/etc/cron.5min)[21629]: starting pipefilter
Mar 30 16:05:04 ha-d1 run-parts(/etc/cron.5min)[23886]: finished pipefilter
Mar 30 16:05:04 ha-d1 run-parts(/etc/cron.5min)[21629]: starting servmon
Mar 30 16:05:04 ha-d1 run-parts(/etc/cron.5min)[23922]: finished servmon
Mar 30 16:05:04 ha-d1 run-parts(/etc/cron.5min)[21629]: starting shaper
Mar 30 16:05:04 ha-d1 run-parts(/etc/cron.5min)[23934]: finished shaper
Mar 30 16:05:04 ha-d1 run-parts(/etc/cron.5min)[21629]: starting sys_info
Mar 30 16:05:06 ha-d1 service_failover(service_failover)[25979]: INFO: UTM resource monitor
Mar 30 16:05:07 ha-d1 sys_info[23940]: Last_Boot: Mar 28 21:21 ; CPU_Usage: 74.37%; MEM_Usage: 96.3524%; HD_Temperature: UNKNOWN; nslookup: Successful; Google_Ping: Successful; MSS_Relay: Successful
Mar 30 16:05:07 ha-d1 sys_info[23940]: coretemp-isa-0000
Mar 30 16:05:07 ha-d1 sys_info[23940]: Adapter: ISA adapter
Mar 30 16:05:07 ha-d1 sys_info[23940]: Physical id 0: +100.0 C  (high = +100.0 C, crit = +100.0 C)
Mar 30 16:05:07 ha-d1 sys_info[23940]: Core 0:        +100.0 C  (high = +100.0 C, crit = +100.0 C)
Mar 30 16:05:07 ha-d1 sys_info[23940]: 
Mar 30 16:05:07 ha-d1 run-parts(/etc/cron.5min)[27032]: finished sys_info
Mar 30 16:05:07 ha-d1 run-parts(/etc/cron.5min)[21629]: starting tw-multipath
Mar 30 16:05:07 ha-d1 run-parts(/etc/cron.5min)[27042]: finished tw-multipath
Mar 30 16:05:07 ha-d1 run-parts(/etc/cron.5min)[21629]: starting custom-logrotate
Mar 30 16:05:07 ha-d1 run-parts(/etc/cron.5min)[27054]: finished custom-logrotate
Mar 30 16:05:25 ha-d1 service_failover(service_failover)[14659]: INFO: UTM resource monitor
Mar 30 16:05:45 ha-d1 service_failover(service_failover)[3692]: INFO: UTM resource monitor
Mar 30 16:05:51 ha-d1 kernel:[153912.757767] "DROP "IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:83:48:80:08:00 SRC=11.37.10.88 DST=11.37.10.255 LEN=229 TOS=0x00 PREC=0x00 TTL=128 ID=8332 PROTO=UDP SPT=138 DPT=138 LEN=209 
Mar 30 16:06:04 ha-d1 service_failover(service_failover)[8089]: INFO: UTM resource monitor
Mar 30 16:06:23 ha-d1 service_failover(service_failover)[14683]: INFO: UTM resource monitor
Mar 30 16:06:43 ha-d1 service_failover(service_failover)[3513]: INFO: UTM resource monitor
Mar 30 16:07:02 ha-d1 service_failover(service_failover)[25395]: INFO: UTM resource monitor
Mar 30 16:07:21 ha-d1 service_failover(service_failover)[3164]: INFO: UTM resource monitor
Mar 30 16:07:41 ha-d1 service_failover(service_failover)[4481]: INFO: UTM resource monitor
Mar 30 16:08:00 ha-d1 service_failover(service_failover)[24816]: INFO: UTM resource monitor
Mar 30 16:08:15 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200071 iface 118.13.10.190 to [1 of 10]
Mar 30 16:08:15 ha-d1 conntrack-tools[1746]: device `eth0' becomes dedicated link
Mar 30 16:08:15 ha-d1 kernel:[154057.455270] e1000: eth2 NIC Link is Down
Mar 30 16:08:15 ha-d1 kernel:[154057.455451] e1000 0000:02:02.0 eth2: Reset adapter
Mar 30 16:08:16 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200073 iface 118.13.10.190 to [2 of 10]
Mar 30 16:08:17 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200075 iface 118.13.10.190 to [3 of 10]
Mar 30 16:08:17 ha-d1 corosync[1687]:   [TOTEM ] Decrementing problem counter for iface 118.13.10.190 to [2 of 10]
Mar 30 16:08:17 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200077 iface 118.13.10.190 to [3 of 10]
Mar 30 16:08:18 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200079 iface 118.13.10.190 to [4 of 10]
Mar 30 16:08:19 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200081 iface 118.13.10.190 to [5 of 10]
Mar 30 16:08:19 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation dmz1_monitor_7000: not running (node=ha-d1.tw.com, call=156, rc=7, cib-update=588, confirmed=false)
Mar 30 16:08:19 ha-d1 crmd[15384]:   notice: process_lrm_event: ha-d1.tw.com-dmz1_monitor_7000:156 [ <debug> Checking 118.13.10.192, Level 0\n<debug> 118.13.10.192 present on eth2\n<warn> Link for eth2: Not detected\n<warn> No link on eth2...\n ]
Mar 30 16:08:19 ha-d1 corosync[1687]:   [TOTEM ] Decrementing problem counter for iface 118.13.10.190 to [4 of 10]
Mar 30 16:08:19 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200083 iface 118.13.10.190 to [5 of 10]
Mar 30 16:08:19 ha-d1 crmd[15384]:  warning: update_failcount: Updating failcount for dmz1 on ha-d1.tw.com after failed monitor: rc=7 (update=value++, time=1459354099)
Mar 30 16:08:19 ha-d1 crmd[15384]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 30 16:08:19 ha-d1 attrd[15382]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-dmz1 (1)
Mar 30 16:08:19 ha-d1 attrd[15382]:   notice: attrd_perform_update: Sent update 264: fail-count-dmz1=1
Mar 30 16:08:19 ha-d1 attrd[15382]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-dmz1 (1459354099)
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: update_validation: pacemaker-1.2-style configuration is also valid for pacemaker-1.3
Mar 30 16:08:19 ha-d1 attrd[15382]:   notice: attrd_perform_update: Sent update 266: last-failure-dmz1=1459354099
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: update_validation: Upgrading pacemaker-1.3-style configuration to pacemaker-2.0 with upgrade-1.3.xsl
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: update_validation: Transformed the configuration from pacemaker-1.2 to pacemaker-2.0
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 30 16:08:19 ha-d1 pengine[15383]:  warning: unpack_rsc_op_failure: Processing failed op monitor for dmz1 on ha-d1.tw.com: not running (7)
Mar 30 16:08:19 ha-d1 pengine[15383]:    error: RecurringOp: Invalid recurring action DRBDSlave-start-interval-30s wth name: 'start'
Mar 30 16:08:19 ha-d1 pengine[15383]:    error: RecurringOp: Invalid recurring action DRBDSlave-start-interval-30s wth name: 'start'
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: LogActions: Move    inif (Started ha-d1.tw.com -> ha-d2.tw.com)
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: LogActions: Move    outif (Started ha-d1.tw.com -> ha-d2.tw.com)
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: LogActions: Recover dmz1 (Started ha-d1.tw.com -> ha-d2.tw.com)
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: LogActions: Demote  DRBDSlave:0 (Master -> Stopped ha-d1.tw.com)
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: LogActions: Promote DRBDSlave:1 (Slave -> Master ha-d2.tw.com)
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: LogActions: Move    DRBDFS (Started ha-d1.tw.com -> ha-d2.tw.com)
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: LogActions: Move    service_failover (Started ha-d1.tw.com -> ha-d2.tw.com)
Mar 30 16:08:19 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 18: stop dmz1_stop_0 on ha-d1.tw.com (local)
Mar 30 16:08:19 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 7: cancel DRBDSlave_monitor_11000 on ha-d2.tw.com
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: process_pe_message: Calculated Transition 228: /var/lib/pacemaker/pengine/pe-input-204.bz2
Mar 30 16:08:19 ha-d1 crmd[15384]:   notice: process_lrm_event: ha-d1.tw.com-dmz1_monitor_7000:156 [ <debug> Checking 118.13.10.192, Level 0\n<debug> 118.13.10.192 present on eth2\n<warn> Link for eth2: Not detected\n<warn> No link on eth2...\n ]
Mar 30 16:08:19 ha-d1 pengine[15383]:   notice: process_pe_message: Configuration ERRORs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 30 16:08:19 ha-d1 service_failover(service_failover)[12814]: INFO: UTM resource monitor
Mar 30 16:08:20 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200085 iface 118.13.10.190 to [6 of 10]
Mar 30 16:08:20 ha-d1 crmd[15384]:   notice: abort_transition_graph: Transition aborted by status-ha-d1.tw.com-fail-count-dmz1, fail-count-dmz1=1: Transient attribute change (create cib=0.77.6, source=te_update_diff:391, path=/cib/status/node_state[@id='ha-d1.tw.com']/transient_attributes[@id='ha-d1.tw.com']/instance_attributes[@id='status-ha-d1.tw.com'], 0)
Mar 30 16:08:20 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200087 iface 118.13.10.190 to [7 of 10]
Mar 30 16:08:21 ha-d1 ntpd[1794]: Deleting interface #17 eth2, 118.13.10.192#123, interface stats: received=0, sent=0, dropped=0, active_time=311 secs
Mar 30 16:08:21 ha-d1 ntpd[1794]: peers refreshed
Mar 30 16:08:21 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200089 iface 118.13.10.190 to [8 of 10]
Mar 30 16:08:21 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200091 iface 118.13.10.190 to [9 of 10]
Mar 30 16:08:21 ha-d1 corosync[1687]:   [TOTEM ] Decrementing problem counter for iface 118.13.10.190 to [8 of 10]
Mar 30 16:08:22 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200093 iface 118.13.10.190 to [9 of 10]
Mar 30 16:08:22 ha-d1 corosync[1687]:   [TOTEM ] Incrementing problem counter for seqid 200095 iface 118.13.10.190 to [10 of 10]
Mar 30 16:08:22 ha-d1 corosync[1687]:   [TOTEM ] Marking seqid 200095 ringid 1 interface 118.13.10.190 FAULTY
Mar 30 16:08:25 ha-d1 kernel:[154067.126549] drbd wwwdata: PingAck did not arrive in time.
Mar 30 16:08:25 ha-d1 kernel:[154067.126702] drbd wwwdata: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
Mar 30 16:08:25 ha-d1 kernel:[154067.134574] block drbd1: new current UUID 2BE3DB456BBBE0B7:B0DA745C79C56591:36E0631B6F022952:36DF631B6F022952
Mar 30 16:08:25 ha-d1 kernel:[154067.134595] drbd wwwdata: asender terminated
Mar 30 16:08:25 ha-d1 kernel:[154067.134597] drbd wwwdata: Terminating drbd_a_wwwdata
Mar 30 16:08:25 ha-d1 kernel:[154067.135002] drbd wwwdata: Connection closed
Mar 30 16:08:25 ha-d1 kernel:[154067.135031] drbd wwwdata: conn( NetworkFailure -> Unconnected ) 
Mar 30 16:08:25 ha-d1 kernel:[154067.135032] drbd wwwdata: receiver terminated
Mar 30 16:08:25 ha-d1 kernel:[154067.135033] drbd wwwdata: Restarting receiver thread
Mar 30 16:08:25 ha-d1 kernel:[154067.135034] drbd wwwdata: receiver (re)started
Mar 30 16:08:25 ha-d1 kernel:[154067.135040] drbd wwwdata: conn( Unconnected -> WFConnection ) 
Mar 30 16:08:29 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation dmz1_stop_0: ok (node=ha-d1.tw.com, call=168, rc=0, cib-update=590, confirmed=true)
Mar 30 16:08:29 ha-d1 crmd[15384]:   notice: run_graph: Transition 228 (Complete=6, Pending=0, Fired=0, Skipped=41, Incomplete=13, Source=/var/lib/pacemaker/pengine/pe-input-204.bz2): Stopped
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: update_validation: pacemaker-1.2-style configuration is also valid for pacemaker-1.3
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: update_validation: Upgrading pacemaker-1.3-style configuration to pacemaker-2.0 with upgrade-1.3.xsl
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: update_validation: Transformed the configuration from pacemaker-1.2 to pacemaker-2.0
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 30 16:08:29 ha-d1 pengine[15383]:  warning: unpack_rsc_op_failure: Processing failed op monitor for dmz1 on ha-d1.tw.com: not running (7)
Mar 30 16:08:29 ha-d1 pengine[15383]:    error: RecurringOp: Invalid recurring action DRBDSlave-start-interval-30s wth name: 'start'
Mar 30 16:08:29 ha-d1 pengine[15383]:    error: RecurringOp: Invalid recurring action DRBDSlave-start-interval-30s wth name: 'start'
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: LogActions: Move    inif (Started ha-d1.tw.com -> ha-d2.tw.com)
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: LogActions: Move    outif (Started ha-d1.tw.com -> ha-d2.tw.com)
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: LogActions: Start   dmz1 (ha-d2.tw.com)
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: LogActions: Demote  DRBDSlave:0 (Master -> Stopped ha-d1.tw.com)
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: LogActions: Promote DRBDSlave:1 (Slave -> Master ha-d2.tw.com)
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: LogActions: Move    DRBDFS (Started ha-d1.tw.com -> ha-d2.tw.com)
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: LogActions: Move    service_failover (Started ha-d1.tw.com -> ha-d2.tw.com)
Mar 30 16:08:29 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 13: stop outif_stop_0 on ha-d1.tw.com (local)
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: process_pe_message: Calculated Transition 229: /var/lib/pacemaker/pengine/pe-input-205.bz2
Mar 30 16:08:29 ha-d1 pengine[15383]:   notice: process_pe_message: Configuration ERRORs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 30 16:08:29 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 78: notify DRBDSlave_pre_notify_demote_0 on ha-d2.tw.com
Mar 30 16:08:31 ha-d1 ntpd[1794]: Deleting interface #16 eth1, 11.37.10.192#123, interface stats: received=0, sent=0, dropped=0, active_time=324 secs
Mar 30 16:08:31 ha-d1 ntpd[1794]: peers refreshed
Mar 30 16:08:40 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation outif_stop_0: ok (node=ha-d1.tw.com, call=170, rc=0, cib-update=592, confirmed=true)
Mar 30 16:08:40 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 10: stop inif_stop_0 on ha-d1.tw.com (local)
Mar 30 16:08:42 ha-d1 ntpd[1794]: Deleting interface #15 eth0, 182.137.10.192#123, interface stats: received=0, sent=0, dropped=0, active_time=338 secs
Mar 30 16:08:42 ha-d1 ntpd[1794]: peers refreshed
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation inif_stop_0: ok (node=ha-d1.tw.com, call=172, rc=0, cib-update=593, confirmed=true)
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 11: start inif_start_0 on ha-d2.tw.com
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 76: notify DRBDSlave_pre_notify_demote_0 on ha-d1.tw.com (local)
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation DRBDSlave_notify_0: ok (node=ha-d1.tw.com, call=173, rc=0, cib-update=0, confirmed=true)
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 52: stop DRBDFS_stop_0 on ha-d1.tw.com (local)
Mar 30 16:08:50 ha-d1 Filesystem(DRBDFS)[30831]: INFO: Running stop for /dev/drbd/by-res/wwwdata/0 on /drbd_fs
Mar 30 16:08:50 ha-d1 Filesystem(DRBDFS)[30831]: INFO: Trying to unmount /drbd_fs
Mar 30 16:08:50 ha-d1 Filesystem(DRBDFS)[30831]: INFO: unmounted /drbd_fs successfully
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation DRBDFS_stop_0: ok (node=ha-d1.tw.com, call=175, rc=0, cib-update=594, confirmed=true)
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 22: demote DRBDSlave_demote_0 on ha-d1.tw.com (local)
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: process_lrm_event: ha-d1.tw.com-DRBDSlave_monitor_13000:161 [ \n ]
Mar 30 16:08:50 ha-d1 kernel:[154091.738169] block drbd1: role( Primary -> Secondary ) 
Mar 30 16:08:50 ha-d1 kernel:[154091.738616] block drbd1: bitmap WRITE of 1 pages took 0 jiffies
Mar 30 16:08:50 ha-d1 kernel:[154091.738619] block drbd1: 4 KB (1 bits) marked out-of-sync by on disk bit-map.
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation DRBDSlave_demote_0: ok (node=ha-d1.tw.com, call=177, rc=0, cib-update=595, confirmed=true)
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 59: stop service_failover_stop_0 on ha-d1.tw.com (local)
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 79: notify DRBDSlave_post_notify_demote_0 on ha-d2.tw.com
Mar 30 16:08:50 ha-d1 service_failover(service_failover)[30929]: INFO: Stopping UTM resource agent
Mar 30 16:08:50 ha-d1 service_failover(service_failover)[30929]: INFO: Stopping dnsmasq
Mar 30 16:08:50 ha-d1 dnsmasq[18013]: exiting on receipt of SIGTERM
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: abort_transition_graph: Transition aborted by status-ha-d2.tw.com-master-DRBDSlave, master-DRBDSlave=1000: Transient attribute change (modify cib=0.77.14, source=te_update_diff:391, path=/cib/status/node_state[@id='ha-d2.tw.com']/transient_attributes[@id='ha-d2.tw.com']/instance_attributes[@id='status-ha-d2.tw.com']/nvpair[@id='status-ha-d2.tw.com-master-DRBDSlave'], 0)
Mar 30 16:08:50 ha-d1 conntrack-tools[1746]: flushing conntrack table in 60 secs
Mar 30 16:08:50 ha-d1 conntrack-tools[1746]: request resync
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation service_failover_stop_0: ok (node=ha-d1.tw.com, call=179, rc=0, cib-update=596, confirmed=true)
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 77: notify DRBDSlave_post_notify_demote_0 on ha-d1.tw.com (local)
Mar 30 16:08:50 ha-d1 attrd[15382]:   notice: attrd_trigger_update: Sending flush op to all hosts for: master-DRBDSlave (1000)
Mar 30 16:08:50 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation DRBDSlave_notify_0: ok (node=ha-d1.tw.com, call=180, rc=0, cib-update=0, confirmed=true)
Mar 30 16:08:50 ha-d1 attrd[15382]:   notice: attrd_perform_update: Sent update 270: master-DRBDSlave=1000
Mar 30 16:08:53 ha-d1 crmd[15384]:   notice: run_graph: Transition 229 (Complete=22, Pending=0, Fired=0, Skipped=28, Incomplete=8, Source=/var/lib/pacemaker/pengine/pe-input-205.bz2): Stopped
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: update_validation: pacemaker-1.2-style configuration is also valid for pacemaker-1.3
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: update_validation: Upgrading pacemaker-1.3-style configuration to pacemaker-2.0 with upgrade-1.3.xsl
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: update_validation: Transformed the configuration from pacemaker-1.2 to pacemaker-2.0
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 30 16:08:53 ha-d1 pengine[15383]:  warning: unpack_rsc_op_failure: Processing failed op monitor for dmz1 on ha-d1.tw.com: not running (7)
Mar 30 16:08:53 ha-d1 pengine[15383]:    error: RecurringOp: Invalid recurring action DRBDSlave-start-interval-30s wth name: 'start'
Mar 30 16:08:53 ha-d1 pengine[15383]:    error: RecurringOp: Invalid recurring action DRBDSlave-start-interval-30s wth name: 'start'
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: LogActions: Start   outif (ha-d2.tw.com)
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: LogActions: Start   dmz1 (ha-d2.tw.com)
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: LogActions: Stop    DRBDSlave:0 (ha-d1.tw.com)
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: LogActions: Promote DRBDSlave:1 (Slave -> Master ha-d2.tw.com)
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: LogActions: Start   DRBDFS (ha-d2.tw.com)
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: LogActions: Start   service_failover (ha-d2.tw.com)
Mar 30 16:08:53 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 7: monitor inif_monitor_3000 on ha-d2.tw.com
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: process_pe_message: Calculated Transition 230: /var/lib/pacemaker/pengine/pe-input-206.bz2
Mar 30 16:08:53 ha-d1 pengine[15383]:   notice: process_pe_message: Configuration ERRORs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 30 16:08:53 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 8: start outif_start_0 on ha-d2.tw.com
Mar 30 16:08:53 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 62: notify DRBDSlave_pre_notify_stop_0 on ha-d1.tw.com (local)
Mar 30 16:08:53 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation DRBDSlave_notify_0: ok (node=ha-d1.tw.com, call=181, rc=0, cib-update=0, confirmed=true)
Mar 30 16:08:53 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 63: notify DRBDSlave_pre_notify_stop_0 on ha-d2.tw.com
Mar 30 16:08:53 ha-d1 kernel:[154094.845637] "DROP "IN=eth1 OUT=eth1 MAC=00:50:56:83:12:84:00:50:56:83:03:eb:08:00 SRC=11.37.10.191 DST=11.37.10.192 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=30184 DF PROTO=ICMP TYPE=8 CODE=0 ID=24931 SEQ=1 
Mar 30 16:08:53 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 16: stop DRBDSlave_stop_0 on ha-d1.tw.com (local)
Mar 30 16:08:53 ha-d1 kernel:[154094.894473] drbd wwwdata: conn( WFConnection -> Disconnecting ) 
Mar 30 16:08:53 ha-d1 kernel:[154094.894485] drbd wwwdata: Discarding network configuration.
Mar 30 16:08:53 ha-d1 kernel:[154094.894516] drbd wwwdata: Connection closed
Mar 30 16:08:53 ha-d1 kernel:[154094.894524] drbd wwwdata: conn( Disconnecting -> StandAlone ) 
Mar 30 16:08:53 ha-d1 kernel:[154094.894525] drbd wwwdata: receiver terminated
Mar 30 16:08:53 ha-d1 kernel:[154094.894527] drbd wwwdata: Terminating drbd_r_wwwdata
Mar 30 16:08:53 ha-d1 kernel:[154094.894559] block drbd1: disk( UpToDate -> Failed ) 
Mar 30 16:08:53 ha-d1 kernel:[154094.894569] block drbd1: bitmap WRITE of 0 pages took 0 jiffies
Mar 30 16:08:53 ha-d1 kernel:[154094.894571] block drbd1: 4 KB (1 bits) marked out-of-sync by on disk bit-map.
Mar 30 16:08:53 ha-d1 kernel:[154094.894574] block drbd1: disk( Failed -> Diskless ) 
Mar 30 16:08:53 ha-d1 kernel:[154094.894647] block drbd1: drbd_bm_resize called with capacity == 0
Mar 30 16:08:53 ha-d1 kernel:[154094.894652] drbd wwwdata: Terminating drbd_w_wwwdata
Mar 30 16:08:53 ha-d1 attrd[15382]:   notice: attrd_trigger_update: Sending flush op to all hosts for: master-DRBDSlave (<null>)
Mar 30 16:08:53 ha-d1 crmd[15384]:   notice: process_lrm_event: Operation DRBDSlave_stop_0: ok (node=ha-d1.tw.com, call=182, rc=0, cib-update=598, confirmed=true)
Mar 30 16:08:53 ha-d1 attrd[15382]:   notice: attrd_perform_update: Sent delete 272: node=ha-d1.tw.com, attr=master-DRBDSlave, id=<n/a>, set=(null), section=status
Mar 30 16:08:53 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 64: notify DRBDSlave_post_notify_stop_0 on ha-d2.tw.com
Mar 30 16:08:53 ha-d1 attrd[15382]:   notice: attrd_perform_update: Sent delete 274: node=ha-d1.tw.com, attr=master-DRBDSlave, id=<n/a>, set=(null), section=status
Mar 30 16:08:53 ha-d1 crmd[15384]:   notice: abort_transition_graph: Transition aborted by deletion of nvpair[@id='status-ha-d1.tw.com-master-DRBDSlave']: Transient attribute change (cib=0.77.20, source=te_update_diff:391, path=/cib/status/node_state[@id='ha-d1.tw.com']/transient_attributes[@id='ha-d1.tw.com']/instance_attributes[@id='status-ha-d1.tw.com']/nvpair[@id='status-ha-d1.tw.com-master-DRBDSlave'], 0)
Mar 30 16:08:54 ha-d1 kernel:[154095.853863] "DROP "IN=eth1 OUT=eth1 MAC=00:50:56:83:12:84:00:50:56:83:03:eb:08:00 SRC=11.37.10.191 DST=11.37.10.192 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=30291 DF PROTO=ICMP TYPE=8 CODE=0 ID=24931 SEQ=2 
Mar 30 16:08:56 ha-d1 crmd[15384]:   notice: run_graph: Transition 230 (Complete=13, Pending=0, Fired=0, Skipped=19, Incomplete=4, Source=/var/lib/pacemaker/pengine/pe-input-206.bz2): Stopped
Mar 30 16:08:56 ha-d1 pengine[15383]:   notice: update_validation: pacemaker-1.2-style configuration is also valid for pacemaker-1.3
Mar 30 16:08:56 ha-d1 pengine[15383]:   notice: update_validation: Upgrading pacemaker-1.3-style configuration to pacemaker-2.0 with upgrade-1.3.xsl
Mar 30 16:08:56 ha-d1 pengine[15383]:   notice: update_validation: Transformed the configuration from pacemaker-1.2 to pacemaker-2.0
Mar 30 16:08:56 ha-d1 pengine[15383]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 30 16:08:56 ha-d1 pengine[15383]:  warning: unpack_rsc_op_failure: Processing failed op monitor for dmz1 on ha-d1.tw.com: not running (7)
Mar 30 16:08:56 ha-d1 pengine[15383]:    error: RecurringOp: Invalid recurring action DRBDSlave-start-interval-30s wth name: 'start'
Mar 30 16:08:56 ha-d1 pengine[15383]:    error: RecurringOp: Invalid recurring action DRBDSlave-start-interval-30s wth name: 'start'
Mar 30 16:08:56 ha-d1 pengine[15383]:   notice: LogActions: Start   dmz1 (ha-d2.tw.com)
Mar 30 16:08:56 ha-d1 pengine[15383]:   notice: LogActions: Promote DRBDSlave:0 (Slave -> Master ha-d2.tw.com)
Mar 30 16:08:56 ha-d1 pengine[15383]:   notice: LogActions: Start   DRBDFS (ha-d2.tw.com)
Mar 30 16:08:56 ha-d1 pengine[15383]:   notice: LogActions: Start   service_failover (ha-d2.tw.com)
Mar 30 16:08:56 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 10: monitor outif_monitor_5000 on ha-d2.tw.com
Mar 30 16:08:56 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 11: start dmz1_start_0 on ha-d2.tw.com
Mar 30 16:08:56 ha-d1 pengine[15383]:   notice: process_pe_message: Calculated Transition 231: /var/lib/pacemaker/pengine/pe-input-207.bz2
Mar 30 16:08:56 ha-d1 pengine[15383]:   notice: process_pe_message: Configuration ERRORs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 30 16:08:56 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 65: notify DRBDSlave_pre_notify_promote_0 on ha-d2.tw.com
Mar 30 16:08:59 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 12: monitor dmz1_monitor_7000 on ha-d2.tw.com
Mar 30 16:08:59 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 19: promote DRBDSlave_promote_0 on ha-d2.tw.com
Mar 30 16:08:59 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 66: notify DRBDSlave_post_notify_promote_0 on ha-d2.tw.com
Mar 30 16:08:59 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 45: start DRBDFS_start_0 on ha-d2.tw.com
Mar 30 16:08:59 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 20: monitor DRBDSlave_monitor_13000 on ha-d2.tw.com
Mar 30 16:08:59 ha-d1 crmd[15384]:   notice: abort_transition_graph: Transition aborted by status-ha-d2.tw.com-master-DRBDSlave, master-DRBDSlave=10000: Transient attribute change (modify cib=0.77.26, source=te_update_diff:391, path=/cib/status/node_state[@id='ha-d2.tw.com']/transient_attributes[@id='ha-d2.tw.com']/instance_attributes[@id='status-ha-d2.tw.com']/nvpair[@id='status-ha-d2.tw.com-master-DRBDSlave'], 0)
Mar 30 16:08:59 ha-d1 crmd[15384]:   notice: run_graph: Transition 231 (Complete=17, Pending=0, Fired=0, Skipped=6, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-207.bz2): Stopped
Mar 30 16:08:59 ha-d1 pengine[15383]:   notice: update_validation: pacemaker-1.2-style configuration is also valid for pacemaker-1.3
Mar 30 16:08:59 ha-d1 pengine[15383]:   notice: update_validation: Upgrading pacemaker-1.3-style configuration to pacemaker-2.0 with upgrade-1.3.xsl
Mar 30 16:08:59 ha-d1 pengine[15383]:   notice: update_validation: Transformed the configuration from pacemaker-1.2 to pacemaker-2.0
Mar 30 16:08:59 ha-d1 pengine[15383]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 30 16:08:59 ha-d1 pengine[15383]:  warning: unpack_rsc_op_failure: Processing failed op monitor for dmz1 on ha-d1.tw.com: not running (7)
Mar 30 16:08:59 ha-d1 pengine[15383]:    error: RecurringOp: Invalid recurring action DRBDSlave-start-interval-30s wth name: 'start'
Mar 30 16:08:59 ha-d1 pengine[15383]:    error: RecurringOp: Invalid recurring action DRBDSlave-start-interval-30s wth name: 'start'
Mar 30 16:08:59 ha-d1 pengine[15383]:   notice: LogActions: Start   service_failover (ha-d2.tw.com)
Mar 30 16:08:59 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 49: monitor DRBDFS_monitor_17000 on ha-d2.tw.com
Mar 30 16:08:59 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 54: start service_failover_start_0 on ha-d2.tw.com
Mar 30 16:08:59 ha-d1 pengine[15383]:   notice: process_pe_message: Calculated Transition 232: /var/lib/pacemaker/pengine/pe-input-208.bz2
Mar 30 16:08:59 ha-d1 pengine[15383]:   notice: process_pe_message: Configuration ERRORs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 30 16:09:00 ha-d1 crmd[15384]:   notice: te_rsc_command: Initiating action 55: monitor service_failover_monitor_19000 on ha-d2.tw.com
Mar 30 16:09:00 ha-d1 crmd[15384]:   notice: run_graph: Transition 232 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-208.bz2): Complete
Mar 30 16:09:00 ha-d1 crmd[15384]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 30 16:09:50 ha-d1 conntrack-tools[1746]: flushing kernel conntrack table (scheduled)
Mar 30 16:10:01 ha-d1 CROND[23266]: (root) CMD (/usr/sbin/atwos-heartbeat)
Mar 30 16:10:01 ha-d1 CROND[23267]: (root) CMD (/usr/sbin/configd_cleanup.sh)
Mar 30 16:10:01 ha-d1 CROND[23268]: (root) CMD (run-parts /etc/cron.5min)
Mar 30 16:10:01 ha-d1 CROND[23270]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 30 16:10:01 ha-d1 run-parts(/etc/cron.5min)[23268]: starting cfgchanges
Mar 30 16:10:01 ha-d1 run-parts(/etc/cron.5min)[23289]: finished cfgchanges
Mar 30 16:10:01 ha-d1 run-parts(/etc/cron.5min)[23268]: starting dnsmasq
Mar 30 16:10:01 configd[23277]: INFO: Read config file: /etc/sysconfig/configd/configd.conf
Mar 30 16:10:01 configd_client[23277]: INFO: Apply XML data from /tmp/configd_cleanup.xml
Mar 30 16:10:02 ha-d1 run-parts(/etc/cron.5min)[23354]: finished dnsmasq
Mar 30 16:10:02 ha-d1 run-parts(/etc/cron.5min)[23268]: starting drbd-status
Mar 30 16:10:02 ha-d1 run-parts(/etc/cron.5min)[23376]: finished drbd-status
Mar 30 16:10:02 ha-d1 run-parts(/etc/cron.5min)[23268]: starting ha-utils
Mar 30 16:10:03 ha-d1 run-parts(/etc/cron.5min)[23454]: finished ha-utils
Mar 30 16:10:03 ha-d1 run-parts(/etc/cron.5min)[23268]: starting kswapd-check.sh
Mar 30 16:10:03 ha-d1 run-parts(/etc/cron.5min)[23464]: finished kswapd-check.sh
Mar 30 16:10:03 ha-d1 run-parts(/etc/cron.5min)[23268]: starting pipefilter
Mar 30 16:10:03 ha-d1 run-parts(/etc/cron.5min)[23470]: finished pipefilter
Mar 30 16:10:03 ha-d1 run-parts(/etc/cron.5min)[23268]: starting servmon
Mar 30 16:10:03 ha-d1 run-parts(/etc/cron.5min)[23490]: finished servmon
Mar 30 16:10:03 ha-d1 run-parts(/etc/cron.5min)[23268]: starting shaper
Mar 30 16:10:03 ha-d1 run-parts(/etc/cron.5min)[23497]: finished shaper
Mar 30 16:10:03 ha-d1 run-parts(/etc/cron.5min)[23268]: starting sys_info
Mar 30 16:10:06 ha-d1 sys_info[23500]: Last_Boot: Mar 28 21:21 ; CPU_Usage: 74.3501%; MEM_Usage: 93.9042%; HD_Temperature: UNKNOWN; nslookup: Successful; Google_Ping: Successful; MSS_Relay: Successful
Mar 30 16:10:06 ha-d1 sys_info[23500]: coretemp-isa-0000
Mar 30 16:10:06 ha-d1 sys_info[23500]: Adapter: ISA adapter
Mar 30 16:10:06 ha-d1 sys_info[23500]: Physical id 0: +100.0 C  (high = +100.0 C, crit = +100.0 C)
Mar 30 16:10:06 ha-d1 sys_info[23500]: Core 0:        +100.0 C  (high = +100.0 C, crit = +100.0 C)
Mar 30 16:10:06 ha-d1 sys_info[23500]: 
Mar 30 16:10:06 ha-d1 run-parts(/etc/cron.5min)[23542]: finished sys_info
Mar 30 16:10:06 ha-d1 run-parts(/etc/cron.5min)[23268]: starting tw-multipath
Mar 30 16:10:06 ha-d1 run-parts(/etc/cron.5min)[23548]: finished tw-multipath
Mar 30 16:10:06 ha-d1 run-parts(/etc/cron.5min)[23268]: starting custom-logrotate
Mar 30 16:10:06 ha-d1 run-parts(/etc/cron.5min)[23555]: finished custom-logrotate
Mar 30 16:12:19 ha-d1 kernel:[154301.107975] "DROP "IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:83:48:80:08:00 SRC=11.37.10.88 DST=11.37.10.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=8333 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 30 16:12:20 ha-d1 kernel:[154301.842806] "DROP "IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:83:48:80:08:00 SRC=11.37.10.88 DST=11.37.10.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=8334 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 30 16:12:20 ha-d1 kernel:[154302.608930] "DROP "IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:83:48:80:08:00 SRC=11.37.10.88 DST=11.37.10.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=8335 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 30 16:12:36 ha-d1 kernel:[154318.383132] "DROP "IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:83:48:80:08:00 SRC=11.37.10.88 DST=11.37.10.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=8336 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 30 16:12:37 ha-d1 kernel:[154319.131072] "DROP "IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:83:48:80:08:00 SRC=11.37.10.88 DST=11.37.10.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=8337 PROTO=UDP SPT=137 DPT=137 LEN=58 
Mar 30 16:12:38 ha-d1 kernel:[154319.896970] "DROP "IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:83:48:80:08:00 SRC=11.37.10.88 DST=11.37.10.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=8338 PROTO=UDP SPT=137 DPT=137 LEN=58 
