[Pacemaker] CTS outputs Single search timed out.

[Andrew Beekhof]

On Fri, Jan 21, 2011 at 6:03 AM, nozawat <nozawat at gmail.com> wrote:
> Hi
>
>  I ran CTS in the following environment.
>  * OS:RHEL5.5-x86_64
>  * pacemaker-1.0.9.1-1.15.el5
>  * TDN(bbs01)
>  * TNNs(cts0101 cts0102)
>
>  Probably it is a phenomenon like the following.
>  http://www.gossamer-threads.com/lists/linuxha/pacemaker/69322
>
>  SSH login without password -> OK.
>  Syslog Message transfer by syslog-ng -> OK.

You're logging to a central server?  The same server you're running CTS on?
If so, what is the contents of /share/ha/logs/ha-log-local7 on that
machine?  Because that is where CTS is looking.

>
> -------
> $ python /usr/share/pacemaker/tests/cts/CTSlab.py --nodes "cts0101 cts0102"
> --at-boot 1 --stack heartbeat --stonith no --logfile
> /share/ha/logs/ha-log-local7 --syslog-facility local7 1
> Jan 21 13:23:08 Random seed is: 1295583788
> Jan 21 13:23:08 >>>>>>>>>>>>>>>> BEGINNING 1 TESTS
> Jan 21 13:23:08 Stack:            heartbeat
> Jan 21 13:23:08 Schema:           pacemaker-1.0
> Jan 21 13:23:08 Scenario:         Random Test Execution
> Jan 21 13:23:08 Random Seed:      1295583788
> Jan 21 13:23:08 System log files: /share/ha/logs/ha-log-local7
> Jan 21 13:23:08 Cluster nodes:
> Jan 21 13:23:08 * cts0101
> Jan 21 13:23:08 * cts0102
> Jan 21 13:23:12 Testing for syslog logs
> Jan 21 13:23:12 Testing for remote logs
> Jan 21 13:24:16 Restarting logging on: ['cts0101', 'cts0102']
> Jan 21 13:25:49 Restarting logging on: ['cts0101', 'cts0102']
> Jan 21 13:28:21 Restarting logging on: ['cts0101', 'cts0102']
> Jan 21 13:31:54 Restarting logging on: ['cts0101', 'cts0102']
> Jan 21 13:35:54 ERROR: Cluster logging unrecoverable.
> Jan 21 13:35:54 Audit LogAudit FAILED.
> -----
>
>  I run it in heartbeat, but a similar error occurs in corosync.
>  I become the error in "Single search timed out" in the log and seem to
> retry.
> -----
> Jan 21 13:23:11 bbs01 CTS: debug: Audit DiskspaceAudit passed.
> Jan 21 13:23:12 bbs01 CTS: Testing for syslog logs
> Jan 21 13:23:12 bbs01 CTS: Testing for remote logs
> Jan 21 13:23:12 bbs01 CTS: debug: lw: cts0101:/share/ha/logs/ha-log-local7:
> Installing /tmp/cts_log_watcher.py on cts0101
> Jan 21 13:23:12 bbs01 CTS: debug: lw: cts0102:/share/ha/logs/ha-log-local7:
> Installing /tmp/cts_log_watcher.py on cts0102
> Jan 21 13:23:13 cts0102 logger: Test message from cts0102
> Jan 21 13:23:13 cts0101 logger: Test message from cts0101
> Jan 21 13:23:44 bbs01 CTS: debug: lw: LogAudit: Single search timed out:
> timeout=30, start=1295583793, limit=1295583824, now=1295583824
> Jan 21 13:24:16 bbs01 CTS: debug: lw: LogAudit: Single search timed out:
> timeout=30, start=1295583824, limit=1295583855, now=1295583856
> Jan 21 13:24:16 bbs01 CTS: Restarting logging on: ['cts0101', 'cts0102']
> Jan 21 13:24:16 bbs01 CTS: debug: cmd: async: target=cts0101, rc=22203:
> /etc/init.d/syslog-ng restart 2>&1 > /dev/null
> Jan 21 13:24:16 bbs01 CTS: debug: cmd: async: target=cts0102, rc=22204:
> /etc/init.d/syslog-ng restart 2>&1 > /dev/null
> Jan 21 13:25:17 cts0102 logger: Test message from cts0102
> Jan 21 13:25:17 cts0101 logger: Test message from cts0101
> -----
>
>  The test case seems to be carried out after this error.
>  However, the script is finished by an error. It is because "Audit LogAudit
> FAILED" occurs.
>  Is it right that how becomes the result of the CTS?
>
> Regards,
> Tomo
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>
>