<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7652.24">
<TITLE>Cluster Test Suite: Bad News</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<P><FONT SIZE=2 FACE="Arial">Hi list,</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">after manually (positive) testing of my new cluster i've initiated one round of the cluster test suite with the following command:</FONT></P>
<P><FONT SIZE=2 FACE="Arial">./CTSlab.py --at-boot 1 --nodes 'mysqlha1 mysqlha2' --stack ais --syslog-facility local6 --schema pacemaker-1.0 --logfile /var/log/messages --stonith 1 --standby 1 --fencing 1 --stonith-type ssh --once 1</FONT></P>
<P><FONT SIZE=2 FACE="Arial">it seems there's a problem with openais but i'm not sure about the significance of these cts results; </FONT>
<BR><FONT SIZE=2 FACE="Arial">after cts finished, my crm on both nodes was not reachable (but openais was running)</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">any hints or comments are very much appreciated.</FONT>
</P>
<BR>
<P><FONT SIZE=2 FACE="Arial">here my CTS:BadNews filtered for one node only:</FONT>
<BR><FONT SIZE=2 FACE="Arial">===============================================================</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 attrd: ERROR: ais_dispatch: Receiving message body failed: (-1) unknown: Resource temporarily unavailable (11)</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 attrd: ERROR: ais_dispatch: AIS connection failed</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 attrd: CRIT: attrd_ais_destroy: Lost connection to OpenAIS service!</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 attrd: ERROR: attrd_cib_connection_destroy: Connection to the CIB terminated...</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: attrd_connection_destroy: Lost connection to attrd</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 cib: ERROR: ais_dispatch: Receiving message body failed: (-1) unknown: Resource temporarily unavailable (11)</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 cib: ERROR: ais_dispatch: AIS connection failed</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 cib: ERROR: cib_ais_destroy: AIS connection terminated</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: CRIT: cib_native_dispatch: Lost connection to the CIB service [1655/callback].</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: CRIT: cib_native_dispatch: Lost connection to the CIB service [1655/command].</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 stonithd: ERROR: ais_dispatch: Receiving message body failed: (-1) unknown: Resource temporarily unavailable (11)</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: crmd_cib_connection_destroy: Connection to the CIB terminated...</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 stonithd: ERROR: ais_dispatch: AIS connection failed</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: do_log: FSA: Input I_ERROR from crmd_cib_connection_destroy() received in state S_IDLE</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 stonithd: ERROR: AIS connection terminated</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: info: do_state_transition: State transition S_IDLE -> S_RECOVERY [ input=I_ERROR cause=C_FSA_INTERNAL origin=crmd_cib_connection_destroy ]</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: do_recover: Action A_RECOVER (0000000001000000) not supported</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: do_log: FSA: Input I_TERMINATE from do_recover() received in state S_RECOVERY</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: verify_stopped: Resource res.stonith.ssh:1 was active at shutdown. You may ignore this error if it is unmanaged.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: verify_stopped: Resource res.drbd.mysqldb:1 was active at shutdown. You may ignore this error if it is unmanaged.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: do_exit: Could not recover from internal error</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 drbd ERROR: mysqldb: Called drbdadm -c /etc/drbd.conf --peer mysqlha2 up mysqldb</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 drbd ERROR: mysqldb: Exit code 1</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 lrmd: info: RA output: (res.drbd.mysqldb:1:start:stderr) 0: Failure: (124) Device is attached to a disk (use detach first) Command 'drbdsetup 0 disk /dev/sdb /dev/sdb internal --set-defaults --create-device --on-io-error=detach' terminated with exit code 10 2009/10/27_16:57:43 ERROR: mysqldb: Called drbdadm -c /etc/drbd.conf --peer mysqlha2 up mysqldb</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 drbd ERROR: mysqldb: Command output:</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 lrmd: info: RA output: (res.drbd.mysqldb:1:start:stderr) 2009/10/27_16:57:43 ERROR: mysqldb: Exit code 1</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 lrmd: info: RA output: (res.drbd.mysqldb:1:start:stderr) 2009/10/27_16:57:43 ERROR: mysqldb: Command output:</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 IPaddr2 ERROR: Could not send gratuitous arps</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 lrmd: info: RA output: (res.ip.stacked:start:stderr) 2009/10/27_16:57:50 ERROR: Could not send gratuitous arps</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: info: te_fence_node: Executing reboot fencing operation (83) on mysqlha2 (timeout=60000)</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 drbd ERROR: mysqlstack: Called drbdadm -c /etc/drbd.conf -S up mysqlstack</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 drbd ERROR: mysqlstack: Exit code 1</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 lrmd: info: RA output: (res.drbd.mysqlstack:0:start:stderr) 2009/10/27_17:02:00 ERROR: mysqlstack: Called drbdadm -c /etc/drbd.conf -S up mysqlstack</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 drbd ERROR: mysqlstack: Command output:</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 lrmd: info: RA output: (res.drbd.mysqlstack:0:start:stderr) 2009/10/27_17:02:00 ERROR: mysqlstack: Exit code 1</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 lrmd: info: RA output: (res.drbd.mysqlstack:0:start:stderr) 2009/10/27_17:02:00 ERROR: mysqlstack: Command output:</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 lrmd: info: RA output: (res.ip.stacked:start:stderr) 2009/10/27_17:02:04 ERROR: Could not send gratuitous arps</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 lrmd: info: RA output: (res.ip.mysql:start:stderr) 2009/10/27_17:02:08 ERROR: Could not send gratuitous arps</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 openais [crm ] ERROR: pcmk_wait_dispatch: Child process mgmtd terminated with signal 15 (pid=1684, core=false)</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: CRIT: cib_native_dispatch: Lost connection to the CIB service [1679/callback].</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: CRIT: cib_native_dispatch: Lost connection to the CIB service [1679/command].</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: verify_stopped: Resource res.ip.stacked was active at shutdown. You may ignore this error if it is unmanaged.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: verify_stopped: Resource res.ocf.mysql was active at shutdown. You may ignore this error if it is unmanaged.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: verify_stopped: Resource res.drbd.mysqlstack:0 was active at shutdown. You may ignore this error if it is unmanaged.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: verify_stopped: Resource res.fs.mysql was active at shutdown. You may ignore this error if it is unmanaged.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: verify_stopped: Resource res.ip.mysql was active at shutdown. You may ignore this error if it is unmanaged.</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 lrmd: info: RA output: (res.ip.mysql:start:stderr) 2009/10/27_17:30:21 ERROR: Could not send gratuitous arps</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: ais_dispatch: Receiving message body failed: (-1) unknown: Resource temporarily unavailable (11)</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: ais_dispatch: AIS connection failed</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: crm_ais_destroy: AIS connection terminated</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 openais [crm ] ERROR: pcmk_wait_dispatch: Child process mgmtd terminated with signal 15 (pid=29800, core=false)</FONT></P>
<P><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: ERROR: stonithd_op_result_ready: not signed on</FONT>
<BR><FONT SIZE=2 FACE="Arial">mysqlha1 crmd: CRIT: tengine_stonith_connection_destroy: Fencing daemon connection failed</FONT>
</P>
<P><FONT SIZE=2 FACE="Arial">CTS: Overall Results:{'auditfail': 2, 'failure': 8, 'skipped': 0, 'success': 7, 'BadNews': 593}</FONT>
<BR><FONT SIZE=2 FACE="Arial">===============================================================</FONT>
</P>
<BR>
<P><FONT SIZE=2 FACE="Arial">environment + versions:</FONT><FONT FACE="Times New Roman"><BR>
</FONT><FONT SIZE=2 FACE="Arial">===============================================================</FONT>
<BR><FONT SIZE=2 FACE="Arial">OS: RHEL 5.4 x86_64</FONT><FONT FACE="Times New Roman"> </FONT>
<BR><FONT SIZE=2 FACE="Arial">drbd 8.3.4 compiled from source</FONT><BR>
<FONT SIZE=2 FACE="Arial">openais.x86_64 0.80.6-8.el5_4.1</FONT><FONT FACE="Times New Roman"><BR>
</FONT><FONT SIZE=2 FACE="Arial">heartbeat.x86_64 3.0.0-33.2</FONT><FONT FACE="Times New Roman"><BR>
</FONT><FONT SIZE=2 FACE="Arial">resource-agents.x86_64 1.0-31.4</FONT><FONT FACE="Times New Roman"><BR>
</FONT><FONT SIZE=2 FACE="Arial">pacemaker.x86_64 1.0.5-4.1</FONT><FONT FACE="Times New Roman"><BR>
</FONT><FONT SIZE=2 FACE="Arial">pacemaker-libs.x86_64 1.0.5-4.1</FONT><FONT FACE="Times New Roman"> </FONT>
<BR><FONT SIZE=2 FACE="Arial">===============================================================</FONT>
</P>
<BR>
<P><FONT SIZE=2 FACE="Arial">/etc/ais/openais.conf:</FONT>
<BR><FONT SIZE=2 FACE="Arial">===============================================================</FONT>
<BR><FONT SIZE=2 FACE="Arial">totem {</FONT>
<BR><FONT SIZE=2 FACE="Arial"> version: 2</FONT>
<BR><FONT SIZE=2 FACE="Arial"> secauth: on</FONT>
<BR><FONT SIZE=2 FACE="Arial"> threads: 0</FONT>
<BR><FONT SIZE=2 FACE="Arial"> rrp_mode: active</FONT>
<BR><FONT SIZE=2 FACE="Arial"> clear_node_high_bit: yes</FONT>
<BR><FONT SIZE=2 FACE="Arial"> vsftype: none</FONT>
<BR><FONT SIZE=2 FACE="Arial"> token: 3000</FONT>
<BR><FONT SIZE=2 FACE="Arial"> join: 60</FONT>
<BR><FONT SIZE=2 FACE="Arial"> consensus: 1500</FONT>
<BR><FONT SIZE=2 FACE="Arial"> max_messages: 20</FONT>
<BR><FONT SIZE=2 FACE="Arial"> interface {</FONT>
<BR><FONT SIZE=2 FACE="Arial"> ringnumber: 0</FONT>
<BR><FONT SIZE=2 FACE="Arial"> bindnetaddr: 172.30.0.0</FONT>
<BR><FONT SIZE=2 FACE="Arial"> mcastaddr: 226.94.1.1</FONT>
<BR><FONT SIZE=2 FACE="Arial"> mcastport: 5405</FONT>
<BR><FONT SIZE=2 FACE="Arial"> }</FONT>
<BR><FONT SIZE=2 FACE="Arial"> interface {</FONT>
<BR><FONT SIZE=2 FACE="Arial"> ringnumber: 1</FONT>
<BR><FONT SIZE=2 FACE="Arial"> bindnetaddr: 10.6.0.0</FONT>
<BR><FONT SIZE=2 FACE="Arial"> mcastaddr: 226.94.1.2</FONT>
<BR><FONT SIZE=2 FACE="Arial"> mcastport: 5405</FONT>
<BR><FONT SIZE=2 FACE="Arial"> }</FONT>
<BR><FONT SIZE=2 FACE="Arial">}</FONT>
<BR><FONT SIZE=2 FACE="Arial">logging {</FONT>
<BR><FONT SIZE=2 FACE="Arial"> debug: off</FONT>
<BR><FONT SIZE=2 FACE="Arial"> timestamp: on</FONT>
<BR><FONT SIZE=2 FACE="Arial"> to_stderr: yes</FONT>
<BR><FONT SIZE=2 FACE="Arial"> to_syslog: yes</FONT>
<BR><FONT SIZE=2 FACE="Arial"> syslog_facility: local7</FONT>
<BR><FONT SIZE=2 FACE="Arial"> to_file: no</FONT>
<BR><FONT SIZE=2 FACE="Arial">}</FONT>
<BR><FONT SIZE=2 FACE="Arial">amf {</FONT>
<BR><FONT SIZE=2 FACE="Arial"> mode: disabled</FONT>
<BR><FONT SIZE=2 FACE="Arial">}</FONT>
<BR><FONT SIZE=2 FACE="Arial">service {</FONT>
<BR><FONT SIZE=2 FACE="Arial"> ver: 0</FONT>
<BR><FONT SIZE=2 FACE="Arial"> name: pacemaker</FONT>
<BR><FONT SIZE=2 FACE="Arial"> use_mgmtd: yes</FONT>
<BR><FONT SIZE=2 FACE="Arial">}</FONT>
<BR><FONT SIZE=2 FACE="Arial">aisexec {</FONT>
<BR><FONT SIZE=2 FACE="Arial"> user: root</FONT>
<BR><FONT SIZE=2 FACE="Arial"> group: root</FONT>
<BR><FONT SIZE=2 FACE="Arial">}</FONT>
<BR><FONT SIZE=2 FACE="Arial">===============================================================</FONT>
</P>
<BR>
<P><FONT SIZE=2 FACE="Arial">crm configure show:</FONT>
<BR><FONT SIZE=2 FACE="Arial">===============================================================</FONT>
<BR><FONT SIZE=2 FACE="Arial">node mysqlha1 \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">attributes standby="off"</FONT>
<BR><FONT SIZE=2 FACE="Arial">node mysqlha2 \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">attributes standby="off"</FONT>
<BR><FONT SIZE=2 FACE="Arial">primitive res.drbd.mysqldb ocf:linbit:drbd \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">params drbd_resource="mysqldb" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">op monitor interval="59s" role="Master" timeout="30s" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">op monitor interval="60s" role="Slave" timeout="30s" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">meta target-role="Started" is-managed="true"</FONT>
<BR><FONT SIZE=2 FACE="Arial">primitive res.drbd.mysqlstack ocf:linbit:drbd \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">params drbd_resource="mysqlstack" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">op monitor interval="58s" role="Master" timeout="29s" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">op monitor interval="59s" role="Slave" timeout="28s" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">meta is-managed="true"</FONT>
<BR><FONT SIZE=2 FACE="Arial">primitive res.fs.mysql ocf:heartbeat:Filesystem \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">params device="/dev/drbd10" directory="/opt/data" fstype="ext3" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">op monitor interval="20" timeout="10s" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">op start interval="0" timeout="5s"</FONT>
<BR><FONT SIZE=2 FACE="Arial">primitive res.ip.mysql ocf:heartbeat:IPaddr2 \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">params ip="172.30.2.10" nic="eth0" cidr_netmask="22" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">op monitor interval="2s" timeout="1s"</FONT>
<BR><FONT SIZE=2 FACE="Arial">primitive res.ip.stacked ocf:heartbeat:IPaddr2 \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">params ip="10.6.0.132" nic="eth1" cidr_netmask="24" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">op monitor interval="3s" timeout="2s" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">meta is-managed="true"</FONT>
<BR><FONT SIZE=2 FACE="Arial">primitive res.ocf.mysql ocf:heartbeat:mysql \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">params binary="/usr/bin/mysqld_safe" config="/etc/my.cnf" user="mysql" group="mysql" log="/var/log/mysqld.log" pid="/var/run/mysqld/mysqld.pid" datadir="/opt/data" \</FONT></P>
<P> <FONT SIZE=2 FACE="Arial">op monitor interval="30s" timeout="10s"</FONT>
<BR><FONT SIZE=2 FACE="Arial">primitive res.stonith.ssh stonith:ssh \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">params hostlist="mysqlHA1 mysqlHA2"</FONT>
<BR><FONT SIZE=2 FACE="Arial">group group.mysql res.fs.mysql res.ip.mysql res.ocf.mysql \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">meta is-managed="true"</FONT>
<BR><FONT SIZE=2 FACE="Arial">ms ms.drbd.mysqldb res.drbd.mysqldb \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" globally_unique="false"</FONT>
<BR><FONT SIZE=2 FACE="Arial">ms ms.drbd.mysqlstack res.drbd.mysqlstack \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">meta master-max="1" master-node-max="1" clone-max="1" clone-node-max="1" notify="true" globally_unique="false"</FONT>
<BR><FONT SIZE=2 FACE="Arial">clone clone.res.stonith.ssh res.stonith.ssh \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">params globally-unique="false"</FONT>
<BR><FONT SIZE=2 FACE="Arial">colocation co.ms.drbd.mysqlstack_on_ms.drbd.mysqldb inf: ms.drbd.mysqlstack ms.drbd.mysqldb:Master</FONT>
<BR><FONT SIZE=2 FACE="Arial">colocation co.ms.drbd.mysqlstack_on_res.ip.stacked inf: ms.drbd.mysqlstack res.ip.stacked</FONT>
<BR><FONT SIZE=2 FACE="Arial">colocation co.res.fs.mysql_on_ms.drbd.mysqlstack_master inf: group.mysql ms.drbd.mysqlstack:Master</FONT>
<BR><FONT SIZE=2 FACE="Arial">colocation co.res.ip.stacked_on_ms.drbd.mysqldb_master inf: res.ip.stacked ms.drbd.mysqldb:Master</FONT>
<BR><FONT SIZE=2 FACE="Arial">order o.ms.drbd.mysqldb_before_ms.drbd.mysqlstack inf: ms.drbd.mysqldb:promote ms.drbd.mysqlstack:start</FONT>
<BR><FONT SIZE=2 FACE="Arial">order o.ms.drbd.mysqlstack_before_res.fs.mysql inf: ms.drbd.mysqlstack:promote res.fs.mysql:start</FONT>
<BR><FONT SIZE=2 FACE="Arial">property $id="cib-bootstrap-options" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">stonith-enabled="true" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">expected-quorum-votes="2" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">no-quorum-policy="ignore" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">last-lrm-refresh="1256717460" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">dc-version="1.0.5-462f1569a43740667daf7b0f6b521742e9eb8fa7" \</FONT>
<BR> <FONT SIZE=2 FACE="Arial">cluster-infrastructure="openais"</FONT>
<BR><FONT SIZE=2 FACE="Arial">===============================================================</FONT>
</P>
<BR>
<P><FONT COLOR="#000000" SIZE=2 FACE="Arial">Mit freundlichen Grüßen / with kind regards</FONT>
</P>
<P><B><FONT COLOR="#000000" FACE="Arial">Torsten Schmidt</FONT></B>
</P>
</BODY>
</HTML>