Diff hostcache... no /tmp/.hb_report.workdir.UKGg27/hb_report-Mon-10-Sep-2012/*/hostcache :/
Diff members.txt... OK
Diff cib.xml... OK
Diff crm_mon.txt... OK
Diff openais.conf... no /tmp/.hb_report.workdir.UKGg27/hb_report-Mon-10-Sep-2012/*/openais.conf :/
Diff logd.cf... no /tmp/.hb_report.workdir.UKGg27/hb_report-Mon-10-Sep-2012/*/logd.cf :/
Diff sysinfo.txt... OK
WARN: coredumps found at Cluster-Server-1:
	Core was generated by `/usr/libexec/pacemaker/cib'.
	Program terminated with signal 6, Aborted.
	Core was generated by `/usr/libexec/pacemaker/cib'.
	Program terminated with signal 6, Aborted.
WARN: coredumps found at Cluster-Server-2:
	Core was generated by `/usr/libexec/pacemaker/cib'.
	Program terminated with signal 6, Aborted.
	Core was generated by `/usr/libexec/pacemaker/cib'.
	Program terminated with signal 6, Aborted.
WARN: problem with permissions/ownership at Cluster-Server-1:
wrong permissions or ownership for /var/lib/heartbeat/crm:
drwxrwx--- 2 hacluster haclient 12288 Sep 10 15:27 /var/lib/heartbeat/crm
WARN: problem with permissions/ownership at Cluster-Server-2:
wrong permissions or ownership for /var/lib/heartbeat/crm:
drwxrwx--- 2 hacluster haclient 12288 Sep 10 15:27 /var/lib/heartbeat/crm

Log patterns:
Sep 10 15:24:40 Cluster-Server-1 cib: [54564]: ERROR: validate_cib_digest: Digest comparision failed: expected 83751b899e758f9b138d060ace084080 (/var/lib/heartbeat/crm/cib.ANRY1Q), calculated bd97ef3df10846e783bd64059be77e45
Sep 10 15:24:40 Cluster-Server-1 cib: [54564]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.uGGnOm failed!  Configuration contents ignored!
Sep 10 15:24:40 Cluster-Server-1 cib: [54564]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected
Sep 10 15:24:40 Cluster-Server-1 cib: [54564]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:662 : retrieveCib(tmp1, tmp2, FALSE) != NULL
Sep 10 15:24:40 Cluster-Server-1 cib: [48709]: ERROR: Managed write_cib_contents process 54564 dumped core
Sep 10 15:24:40 Cluster-Server-1 cib: [48709]: ERROR: cib_diskwrite_complete: Disk write failed: status=134, signo=6, exitcode=0
Sep 10 15:24:40 Cluster-Server-1 cib: [48709]: ERROR: cib_diskwrite_complete: Disabling disk writes after write failure
Sep 10 15:27:46 Cluster-Server-1 cib: [61587]: ERROR: validate_cib_digest: Digest comparision failed: expected 915815b4e68298e50146ce8cc380b8d9 (/var/lib/heartbeat/crm/cib.4Fs3XB), calculated 0121bf5cffe4dab4f888959f1baf7322
Sep 10 15:27:46 Cluster-Server-1 cib: [61587]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.CcXphf failed!  Configuration contents ignored!
Sep 10 15:27:46 Cluster-Server-1 cib: [61587]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected
Sep 10 15:27:46 Cluster-Server-1 cib: [61587]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:662 : retrieveCib(tmp1, tmp2, FALSE) != NULL
Sep 10 15:27:46 Cluster-Server-1 cib: [48709]: ERROR: Managed write_cib_contents process 61587 dumped core
Sep 10 15:27:46 Cluster-Server-1 cib: [48709]: ERROR: cib_diskwrite_complete: Disk write failed: status=134, signo=6, exitcode=0
Sep 10 15:27:46 Cluster-Server-1 cib: [48709]: ERROR: cib_diskwrite_complete: Disabling disk writes after write failure
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Sep 10 15:24:40 Cluster-Server-2 cib: [57193]: ERROR: validate_cib_digest: Digest comparision failed: expected 83751b899e758f9b138d060ace084080 (/var/lib/heartbeat/crm/cib.IpZiPD), calculated bd97ef3df10846e783bd64059be77e45
Sep 10 15:24:40 Cluster-Server-2 cib: [57193]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.RqPk3s failed!  Configuration contents ignored!
Sep 10 15:24:40 Cluster-Server-2 cib: [57193]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected
Sep 10 15:24:40 Cluster-Server-2 cib: [57193]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:662 : retrieveCib(tmp1, tmp2, FALSE) != NULL
Sep 10 15:24:40 Cluster-Server-2 cib: [40192]: ERROR: Managed write_cib_contents process 57193 dumped core
Sep 10 15:24:40 Cluster-Server-2 cib: [40192]: ERROR: cib_diskwrite_complete: Disk write failed: status=134, signo=6, exitcode=0
Sep 10 15:24:40 Cluster-Server-2 cib: [40192]: ERROR: cib_diskwrite_complete: Disabling disk writes after write failure
Sep 10 15:26:28 Cluster-Server-2 lrmd: [40194]: ERROR: get_resource_meta: pclose failed: Interrupted system call
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: lrm_get_rsc_type_metadata(578): got a return code HA_FAIL from a reply message of rmetadata with function get_ret_from_msg.
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: string2xml: Can't parse NULL input
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: get_rsc_restart_list: Metadata for nas::ocf:Filesystem is not valid XML
Sep 10 15:26:28 Cluster-Server-2 lrmd: [40194]: ERROR: get_resource_meta: pclose failed: Resource temporarily unavailable
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: lrm_get_rsc_type_metadata(578): got a return code HA_FAIL from a reply message of rmetadata with function get_ret_from_msg.
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: string2xml: Can't parse NULL input
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: get_rsc_restart_list: Metadata for nas::ocf:exportfs is not valid XML
Sep 10 15:27:46 Cluster-Server-2 cib: [7872]: ERROR: validate_cib_digest: Digest comparision failed: expected 915815b4e68298e50146ce8cc380b8d9 (/var/lib/heartbeat/crm/cib.L6EHMo), calculated 0121bf5cffe4dab4f888959f1baf7322
Sep 10 15:27:46 Cluster-Server-2 cib: [7872]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.nb01wl failed!  Configuration contents ignored!
Sep 10 15:27:46 Cluster-Server-2 cib: [7872]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected
Sep 10 15:27:46 Cluster-Server-2 cib: [7872]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:662 : retrieveCib(tmp1, tmp2, FALSE) != NULL
Sep 10 15:27:46 Cluster-Server-2 cib: [40192]: ERROR: Managed write_cib_contents process 7872 dumped core
Sep 10 15:27:46 Cluster-Server-2 cib: [40192]: ERROR: cib_diskwrite_complete: Disk write failed: status=134, signo=6, exitcode=0
Sep 10 15:27:46 Cluster-Server-2 cib: [40192]: ERROR: cib_diskwrite_complete: Disabling disk writes after write failure
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Sep 10 15:24:40 Cluster-Server-1 cib: [54564]: ERROR: validate_cib_digest: Digest comparision failed: expected 83751b899e758f9b138d060ace084080 (/var/lib/heartbeat/crm/cib.ANRY1Q), calculated bd97ef3df10846e783bd64059be77e45
Sep 10 15:24:40 Cluster-Server-1 cib: [54564]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.uGGnOm failed!  Configuration contents ignored!
Sep 10 15:24:40 Cluster-Server-1 cib: [54564]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected
Sep 10 15:24:40 Cluster-Server-1 cib: [54564]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:662 : retrieveCib(tmp1, tmp2, FALSE) != NULL
Sep 10 15:24:40 Cluster-Server-1 cib: [48709]: ERROR: Managed write_cib_contents process 54564 dumped core
Sep 10 15:24:40 Cluster-Server-1 cib: [48709]: ERROR: cib_diskwrite_complete: Disk write failed: status=134, signo=6, exitcode=0
Sep 10 15:24:40 Cluster-Server-1 cib: [48709]: ERROR: cib_diskwrite_complete: Disabling disk writes after write failure
Sep 10 15:27:46 Cluster-Server-1 cib: [61587]: ERROR: validate_cib_digest: Digest comparision failed: expected 915815b4e68298e50146ce8cc380b8d9 (/var/lib/heartbeat/crm/cib.4Fs3XB), calculated 0121bf5cffe4dab4f888959f1baf7322
Sep 10 15:27:46 Cluster-Server-1 cib: [61587]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.CcXphf failed!  Configuration contents ignored!
Sep 10 15:27:46 Cluster-Server-1 cib: [61587]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected
Sep 10 15:27:46 Cluster-Server-1 cib: [61587]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:662 : retrieveCib(tmp1, tmp2, FALSE) != NULL
Sep 10 15:27:46 Cluster-Server-1 cib: [48709]: ERROR: Managed write_cib_contents process 61587 dumped core
Sep 10 15:27:46 Cluster-Server-1 cib: [48709]: ERROR: cib_diskwrite_complete: Disk write failed: status=134, signo=6, exitcode=0
Sep 10 15:27:46 Cluster-Server-1 cib: [48709]: ERROR: cib_diskwrite_complete: Disabling disk writes after write failure
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Sep 10 15:24:40 Cluster-Server-2 cib: [57193]: ERROR: validate_cib_digest: Digest comparision failed: expected 83751b899e758f9b138d060ace084080 (/var/lib/heartbeat/crm/cib.IpZiPD), calculated bd97ef3df10846e783bd64059be77e45
Sep 10 15:24:40 Cluster-Server-2 cib: [57193]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.RqPk3s failed!  Configuration contents ignored!
Sep 10 15:24:40 Cluster-Server-2 cib: [57193]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected
Sep 10 15:24:40 Cluster-Server-2 cib: [57193]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:662 : retrieveCib(tmp1, tmp2, FALSE) != NULL
Sep 10 15:24:40 Cluster-Server-2 cib: [40192]: ERROR: Managed write_cib_contents process 57193 dumped core
Sep 10 15:24:40 Cluster-Server-2 cib: [40192]: ERROR: cib_diskwrite_complete: Disk write failed: status=134, signo=6, exitcode=0
Sep 10 15:24:40 Cluster-Server-2 cib: [40192]: ERROR: cib_diskwrite_complete: Disabling disk writes after write failure
Sep 10 15:26:28 Cluster-Server-2 lrmd: [40194]: ERROR: get_resource_meta: pclose failed: Interrupted system call
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: lrm_get_rsc_type_metadata(578): got a return code HA_FAIL from a reply message of rmetadata with function get_ret_from_msg.
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: string2xml: Can't parse NULL input
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: get_rsc_restart_list: Metadata for nas::ocf:Filesystem is not valid XML
Sep 10 15:26:28 Cluster-Server-2 lrmd: [40194]: ERROR: get_resource_meta: pclose failed: Resource temporarily unavailable
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: lrm_get_rsc_type_metadata(578): got a return code HA_FAIL from a reply message of rmetadata with function get_ret_from_msg.
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: string2xml: Can't parse NULL input
Sep 10 15:26:28 Cluster-Server-2 crmd: [40197]: ERROR: get_rsc_restart_list: Metadata for nas::ocf:exportfs is not valid XML
Sep 10 15:27:46 Cluster-Server-2 cib: [7872]: ERROR: validate_cib_digest: Digest comparision failed: expected 915815b4e68298e50146ce8cc380b8d9 (/var/lib/heartbeat/crm/cib.L6EHMo), calculated 0121bf5cffe4dab4f888959f1baf7322
Sep 10 15:27:46 Cluster-Server-2 cib: [7872]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.nb01wl failed!  Configuration contents ignored!
Sep 10 15:27:46 Cluster-Server-2 cib: [7872]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected
Sep 10 15:27:46 Cluster-Server-2 cib: [7872]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:662 : retrieveCib(tmp1, tmp2, FALSE) != NULL
Sep 10 15:27:46 Cluster-Server-2 cib: [40192]: ERROR: Managed write_cib_contents process 7872 dumped core
Sep 10 15:27:46 Cluster-Server-2 cib: [40192]: ERROR: cib_diskwrite_complete: Disk write failed: status=134, signo=6, exitcode=0
Sep 10 15:27:46 Cluster-Server-2 cib: [40192]: ERROR: cib_diskwrite_complete: Disabling disk writes after write failure
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Sep 10 15:21:40 Cluster-Server-2 pengine: [40196]: ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
