Diff hostcache... OK
Diff members.txt... OK
Diff cib.xml... OK
Diff crm_mon.txt... OK
Diff ha.cf... OK
Diff logd.cf... no /tmp/.hb_report.workdir.pVbqD1/hb_report/*/logd.cf :/
Diff sysinfo.txt... --- /tmp/.hb_report.workdir.pVbqD1/hb_report/dl380g5c/sysinfo.txt	2013-01-10 18:34:53.000000000 +0900
+++ /tmp/.hb_report.workdir.pVbqD1/hb_report/dl380g5d/sysinfo.txt	2013-01-10 18:34:53.000000000 +0900
@@ -12,127 +12,7 @@
 lvm2 2.02.87-6.el6 - Red Hat x86_64
 glibc 2.12-1.47.el6 - Red Hat x86_64
 glibc 2.12-1.47.el6 - Red Hat i686
-S.5....T.    /usr/lib/ocf/resource.d/pacemaker/Dummy
-S.5....T.    /usr/lib/ocf/resource.d/pacemaker/Stateful
 S.5....T.    /usr/sbin/crm
-.......T.    /etc/ha.d/shellfuncs
-.......T.    /usr/include/heartbeat/agent_config.h
-S.5....T.    /usr/lib/ocf/lib/heartbeat/apache-conf.sh
-S.5....T.    /usr/lib/ocf/lib/heartbeat/http-mon.sh
-.......T.    /usr/lib/ocf/lib/heartbeat/ocf-binaries
-.......T.    /usr/lib/ocf/lib/heartbeat/ocf-directories
-.......T.    /usr/lib/ocf/lib/heartbeat/ocf-returncodes
-S.5....T.    /usr/lib/ocf/lib/heartbeat/ocf-shellfuncs
-.......T.    /usr/lib/ocf/lib/heartbeat/sapdb-nosha.sh
-S.5....T.    /usr/lib/ocf/lib/heartbeat/sapdb.sh
-missing     /usr/lib/ocf/resource.d/heartbeat/.ocf-directories
-.......T.    /usr/lib/ocf/resource.d/heartbeat/AoEtarget
-.......T.    /usr/lib/ocf/resource.d/heartbeat/AudibleAlarm
-.......T.    /usr/lib/ocf/resource.d/heartbeat/CTDB
-.......T.    /usr/lib/ocf/resource.d/heartbeat/ClusterMon
-.......T.    /usr/lib/ocf/resource.d/heartbeat/Delay
-.......T.    /usr/lib/ocf/resource.d/heartbeat/Dummy
-.......T.    /usr/lib/ocf/resource.d/heartbeat/EvmsSCC
-.......T.    /usr/lib/ocf/resource.d/heartbeat/Evmsd
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/Filesystem
-.......T.    /usr/lib/ocf/resource.d/heartbeat/ICP
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/IPaddr
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/IPaddr2
-.......T.    /usr/lib/ocf/resource.d/heartbeat/IPsrcaddr
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/IPv6addr
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/LVM
-.......T.    /usr/lib/ocf/resource.d/heartbeat/LinuxSCSI
-.......T.    /usr/lib/ocf/resource.d/heartbeat/MailTo
-.......T.    /usr/lib/ocf/resource.d/heartbeat/ManageRAID
-.......T.    /usr/lib/ocf/resource.d/heartbeat/ManageVE
-.......T.    /usr/lib/ocf/resource.d/heartbeat/Pure-FTPd
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/Raid1
-.......T.    /usr/lib/ocf/resource.d/heartbeat/Route
-.......T.    /usr/lib/ocf/resource.d/heartbeat/SAPDatabase
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/SAPInstance
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/SendArp
-.......T.    /usr/lib/ocf/resource.d/heartbeat/ServeRAID
-.......T.    /usr/lib/ocf/resource.d/heartbeat/SphinxSearchDaemon
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/Squid
-.......T.    /usr/lib/ocf/resource.d/heartbeat/Stateful
-.......T.    /usr/lib/ocf/resource.d/heartbeat/SysInfo
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/VIPArip
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/VirtualDomain
-.......T.    /usr/lib/ocf/resource.d/heartbeat/WAS
-.......T.    /usr/lib/ocf/resource.d/heartbeat/WAS6
-.......T.    /usr/lib/ocf/resource.d/heartbeat/WinPopup
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/Xen
-.......T.    /usr/lib/ocf/resource.d/heartbeat/Xinetd
-.......T.    /usr/lib/ocf/resource.d/heartbeat/anything
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/apache
-.......T.    /usr/lib/ocf/resource.d/heartbeat/asterisk
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/conntrackd
-.......T.    /usr/lib/ocf/resource.d/heartbeat/db2
-.......T.    /usr/lib/ocf/resource.d/heartbeat/drbd
-.......T.    /usr/lib/ocf/resource.d/heartbeat/eDir88
-.......T.    /usr/lib/ocf/resource.d/heartbeat/ethmonitor
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/exportfs
-.......T.    /usr/lib/ocf/resource.d/heartbeat/fio
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/iSCSILogicalUnit
-.......T.    /usr/lib/ocf/resource.d/heartbeat/iSCSITarget
-.......T.    /usr/lib/ocf/resource.d/heartbeat/ids
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/iscsi
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/jboss
-.......T.    /usr/lib/ocf/resource.d/heartbeat/ldirectord
-.......T.    /usr/lib/ocf/resource.d/heartbeat/lxc
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/mysql
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/mysql-proxy
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/named
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/nfsserver
-.......T.    /usr/lib/ocf/resource.d/heartbeat/nginx
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/oracle
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/oralsnr
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/pgsql
-.......T.    /usr/lib/ocf/resource.d/heartbeat/pingd
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/portblock
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/postfix
-.......T.    /usr/lib/ocf/resource.d/heartbeat/pound
-.......T.    /usr/lib/ocf/resource.d/heartbeat/proftpd
-.......T.    /usr/lib/ocf/resource.d/heartbeat/rsyncd
-.......T.    /usr/lib/ocf/resource.d/heartbeat/rsyslog
-.......T.    /usr/lib/ocf/resource.d/heartbeat/scsi2reservation
-.......T.    /usr/lib/ocf/resource.d/heartbeat/sfex
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/slapd
-.......T.    /usr/lib/ocf/resource.d/heartbeat/symlink
-.......T.    /usr/lib/ocf/resource.d/heartbeat/syslog-ng
-S.5....T.    /usr/lib/ocf/resource.d/heartbeat/tomcat
-.......T.    /usr/lib/ocf/resource.d/heartbeat/varnish
-.......T.    /usr/lib/ocf/resource.d/heartbeat/vmware
-S.5....T.    /usr/lib64/heartbeat/findif
-S.5....T.    /usr/lib64/heartbeat/send_arp
-S.5....T.    /usr/lib64/heartbeat/sfex_daemon
-S.5....T.    /usr/lib64/heartbeat/tickle_tcp
-..5....T.    /usr/sbin/ocf-tester
-S.5....T.    /usr/sbin/ocft
-S.5....T.    /usr/sbin/sfex_init
-S.5....T.    /usr/sbin/sfex_stat
-.......T.    /usr/share/resource-agents/ocft/README
-.......T.    /usr/share/resource-agents/ocft/README.zh_CN
-.......T.    /usr/share/resource-agents/ocft/caselib
-S.5....T.    /usr/share/resource-agents/ocft/configs/Filesystem
-S.5....T.    /usr/share/resource-agents/ocft/configs/IPaddr2
-.......T.    /usr/share/resource-agents/ocft/configs/IPsrcaddr
-.......T.    /usr/share/resource-agents/ocft/configs/IPv6addr
-S.5....T.    /usr/share/resource-agents/ocft/configs/LVM
-.......T.    /usr/share/resource-agents/ocft/configs/MailTo
-S.5....T.    /usr/share/resource-agents/ocft/configs/SendArp
-.......T.    /usr/share/resource-agents/ocft/configs/Xinetd
-S.5....T.    /usr/share/resource-agents/ocft/configs/apache
-.......T.    /usr/share/resource-agents/ocft/configs/db2
-.......T.    /usr/share/resource-agents/ocft/configs/drbd.linbit
-.......T.    /usr/share/resource-agents/ocft/configs/iscsi
-.......T.    /usr/share/resource-agents/ocft/configs/mysql
-.......T.    /usr/share/resource-agents/ocft/configs/named
-.......T.    /usr/share/resource-agents/ocft/configs/nfsserver
-.......T.    /usr/share/resource-agents/ocft/configs/pgsql
-.......T.    /usr/share/resource-agents/ocft/configs/portblock
-.......T.    /usr/share/resource-agents/ocft/configs/postfix
-.......T.  d /usr/share/resource-agents/ra-api-1.dtd
 S.5....T.    /etc/init.d/logd
 S.5....T.    /usr/lib64/heartbeat/ha_logd
 S.5....T.    /usr/lib64/heartbeat/lrmd
@@ -189,7 +69,6 @@
 S.5....T.  c /etc/logrotate.d/heartbeat
 .M...UG..    /var/run/heartbeat
 .M.......    /var/run/heartbeat/ccm
-.......T.    /usr/lib/ocf/resource.d/linbit/drbd
 S.5....T.  c /etc/drbd.conf
 Platform: Linux
 Kernel release: 2.6.32-220.el6.x86_64

WARN: crm_verify reported warnings at dl380g5c:
crm_verify[9302]: 2013/01/10_18:34:52 WARN: unpack_nodes: Blind faith: not fencing unseen nodes
crm_verify[9302]: 2013/01/10_18:34:52 WARN: unpack_rsc_op: Processing failed op dummy:0_demote_0 on dl380g5c: unknown error (1)
crm_verify[9302]: 2013/01/10_18:34:52 WARN: unpack_rsc_op: Forcing dummy:0 to stop after a failed demote action
WARN: crm_verify reported warnings at dl380g5d:
crm_verify[11723]: 2013/01/10_18:34:52 WARN: unpack_nodes: Blind faith: not fencing unseen nodes
crm_verify[11723]: 2013/01/10_18:34:52 WARN: unpack_rsc_op: Processing failed op dummy:0_demote_0 on dl380g5c: unknown error (1)
crm_verify[11723]: 2013/01/10_18:34:52 WARN: unpack_rsc_op: Forcing dummy:0 to stop after a failed demote action
WARN: problem with permissions/ownership at dl380g5c:
/var/lib/heartbeat/ccm wrong type or doesn't exist
WARN: problem with permissions/ownership at dl380g5d:
/var/lib/heartbeat/ccm wrong type or doesn't exist

Log patterns:
Jan 10 18:24:57 dl380g5c heartbeat: [1404]: ERROR: api_process_registration_msg: cannot add client()
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Jan 10 18:24:57 dl380g5c heartbeat: [1404]: ERROR: api_process_registration_msg: cannot add client()
Jan 10 18:24:57 dl380g5c cib: [1491]: ERROR: register_heartbeat_conn: Cannot sign on with heartbeat: 
Jan 10 18:24:57 dl380g5c cib: [1491]: CRIT: cib_init: Cannot sign in to the cluster... terminating
Jan 10 18:25:18 dl380g5d pm_logconv: [2142]: ERROR:  detect_fo_complete(): resource [vip-master] is not in this cluster.
Jan 10 18:27:04 dl380g5d pm_logconv: [2142]: ERROR:  detect_fo_complete(): resource [vip-master] is not in this cluster.
Jan 10 18:24:57 dl380g5c heartbeat: [1404]: ERROR: api_process_registration_msg: cannot add client()
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Jan 10 18:24:00 dl380g5d pengine: [4297]: ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Jan 10 18:24:57 dl380g5c heartbeat: [1404]: ERROR: api_process_registration_msg: cannot add client()
Jan 10 18:24:57 dl380g5c cib: [1491]: ERROR: register_heartbeat_conn: Cannot sign on with heartbeat: 
Jan 10 18:24:57 dl380g5c cib: [1491]: CRIT: cib_init: Cannot sign in to the cluster... terminating
Jan 10 18:25:18 dl380g5d pm_logconv: [2142]: ERROR:  detect_fo_complete(): resource [vip-master] is not in this cluster.
Jan 10 18:27:04 dl380g5d pm_logconv: [2142]: ERROR:  detect_fo_complete(): resource [vip-master] is not in this cluster.
