No subject


Mon Mar 29 03:57:18 EDT 2010


(exit 6),
which means &quot;program is not configured&quot;. <br>
<br>
<o:p></o:p></p>

<pre>Jul&nbsp; 7 11:47:58 qad01 pengine: [4359]: ERROR: unpack_rsc_op: =
Hard error<o:p></o:p></pre><pre>- mysql-server_start_0 failed with =
rc=3D6: Preventing mysql-server from<o:p></o:p></pre><pre>re-starting =
anywhere in the cluster<o:p></o:p></pre>

<p class=3DMsoNormal>Because the mysql-server primitive and/or the mysql =
RA were
not properly configured at the time the error message was written, it =
led to:<br>
<br>
<br>
<o:p></o:p></p>

<pre>Jul&nbsp; 7 11:47:58 qad01 pengine: [4359]: WARN: =
common_apply_stickiness:<o:p></o:p></pre><pre>Forcing mysql-server away =
from qad01 after 1000000 =
failures<o:p></o:p></pre><pre>(max=3D1000000)<o:p></o:p></pre>

<p class=3DMsoNormal><br>
Therefore, export the variables manually, check the exit codes on each
operation of the script, they should match normal operation as described =
in the
RA specification draft, if everything is OK, move on the the crm shell, =
cleanup
the mysql-server resource, cleanup the max failures (the cluster =
resource
manager keeps track of these, I've had to manually remove everything =
from
/var/lib/heartbeat/crm/*, haven't found a way to clear this option, yet, =
and
loaded the saved config with crm load config.crm to have the cluster =
functional
again). <br>
<br>
Also keep in mind that the mysql RA script needs to be the same (as well =
as /etc/my.cnf)
on all cluster nodes, and even if Pacemaker propagates the changes to =
all nodes
via multicast, /var/lib/heartbeat/crm/* files are placed on all cluster =
nodes
as well. And the mysql RA script needs to work the same way on all =
nodes.<br>
<br>
And I'll stop, I've said enough already.<br>
<br>
Cheers.<br>
<br>
<br>
Jake Bogie wrote: <o:p></o:p></p>

<pre>So I took Dan's advice this time and cleaned up my =
resource<o:p></o:p></pre><pre>configuration, updated the script, and =
verified...however I'm still not<o:p></o:p></pre><pre>getting the =
resource =
online...<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>[root at qad01 =
heartbeat]# crm resource start =
mysql-server<o:p></o:p></pre><pre>[root at qad01 heartbeat]# crm =
status<o:p></o:p></pre><pre>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<o:p></o:=
p></pre><pre>Last updated: Wed Jul&nbsp; 7 11:49:20 =
2010<o:p></o:p></pre><pre>Stack: openais<o:p></o:p></pre><pre>Current =
DC: qad01 - partition with quorum<o:p></o:p></pre><pre>Version: =
1.0.9-89bd754939df5150de7cd76835f98fe90851b677<o:p></o:p></pre><pre>2 =
Nodes configured, 2 expected votes<o:p></o:p></pre><pre>3 Resources =
configured.<o:p></o:p></pre><pre>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<o:p=
></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Online: [ qad02 qad01 =
]<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre> Resource Group: =
mysql<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; fs_mysql&nbsp;&nbsp; =
(ocf::heartbeat:Filesystem):&nbsp;&nbsp;&nbsp; Started =
qad01<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; ip_mysql&nbsp;&nbsp; =
(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Started =
qad01<o:p></o:p></pre><pre> Master/Slave Set: =
ms_drbd_mysql<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; Masters: [ =
qad01 ]<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; Slaves: [ qad02 =
]<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Failed =
actions:<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp; mysql-server_start_0 =
(node=3Dqad01, call=3D6, rc=3D6, =
status=3Dcomplete):<o:p></o:p></pre><pre>not =
configured<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>[root at qad01 =
heartbeat]# ./mysql start<o:p></o:p></pre><pre>mysql[5750]: DEBUG: MySQL =
is not running<o:p></o:p></pre><pre>mysql[5750]: DEBUG: MySQL is not =
running<o:p></o:p></pre><pre>100707 11:49:55 [Warning] option =
'group_concat_max_len': unsigned value<o:p></o:p></pre><pre>0 adjusted =
to 4<o:p></o:p></pre><pre>100707 11:49:55 [Note] Plugin 'FEDERATED' is =
disabled.<o:p></o:p></pre><pre>InnoDB: The InnoDB memory heap is =
disabled<o:p></o:p></pre><pre>InnoDB: Mutexes and rw_locks use GCC =
atomic builtins<o:p></o:p></pre><pre>InnoDB: Compressed tables use zlib =
1.2.3<o:p></o:p></pre><pre>100707 11:49:55&nbsp; InnoDB: highest =
supported file format is Barracuda.<o:p></o:p></pre><pre>100707 =
11:49:55&nbsp; InnoDB: Warning: allocated tablespace 1, old =
maximum<o:p></o:p></pre><pre>was 0<o:p></o:p></pre><pre>100707 11:49:55 =
InnoDB Plugin 1.0.9 started; log sequence =
number<o:p></o:p></pre><pre>28732335894<o:p></o:p></pre><pre>100707 =
11:49:55 [Note] Event Scheduler: Loaded 0 =
events<o:p></o:p></pre><pre>100707 11:49:55 [Note] /usr/sbin/mysqld: =
ready for connections.<o:p></o:p></pre><pre>Version: =
'5.1.48-community'&nbsp; socket: '/var/lib/mysql/mysql.sock'&nbsp; =
port:<o:p></o:p></pre><pre>3306&nbsp; MySQL Community Server =
(GPL)<o:p></o:p></pre><pre>mysql[5750]: INFO: MySQL =
started<o:p></o:p></pre><pre>[root at qad01 heartbeat]# ./mysql =
status<o:p></o:p></pre><pre>[root at qad01 heartbeat]# ./mysql =
monitor<o:p></o:p></pre><pre>[root at qad01 heartbeat]# ./mysql =
validate-all<o:p></o:p></pre><pre>[root at qad01 heartbeat]# crm =
status<o:p></o:p></pre><pre>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<o:p></o:=
p></pre><pre>Last updated: Wed Jul&nbsp; 7 11:50:23 =
2010<o:p></o:p></pre><pre>Stack: openais<o:p></o:p></pre><pre>Current =
DC: qad01 - partition with quorum<o:p></o:p></pre><pre>Version: =
1.0.9-89bd754939df5150de7cd76835f98fe90851b677<o:p></o:p></pre><pre>2 =
Nodes configured, 2 expected votes<o:p></o:p></pre><pre>3 Resources =
configured.<o:p></o:p></pre><pre>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<o:p=
></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Online: [ qad02 qad01 =
]<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre> Resource Group: =
mysql<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; fs_mysql&nbsp;&nbsp; =
(ocf::heartbeat:Filesystem):&nbsp;&nbsp;&nbsp; Started =
qad01<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; ip_mysql&nbsp;&nbsp; =
(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Started =
qad01<o:p></o:p></pre><pre> Master/Slave Set: =
ms_drbd_mysql<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; Masters: [ =
qad01 ]<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; Slaves: [ qad02 =
]<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Failed =
actions:<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp; mysql-server_start_0 =
(node=3Dqad01, call=3D6, rc=3D6, =
status=3Dcomplete):<o:p></o:p></pre><pre>not =
configured<o:p></o:p></pre><pre>[root at qad01 heartbeat]# ./mysql =
stop<o:p></o:p></pre><pre>100707 11:50:31 [Note] /usr/sbin/mysqld: =
Normal =
shutdown<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>./mysql: line =
426: (/1000)-5: syntax error: operand expected =
(error<o:p></o:p></pre><pre>token is =
&quot;/1000)-5&quot;)<o:p></o:p></pre><pre>100707 11:50:31 [Note] Event =
Scheduler: Purging the queue. 0 events<o:p></o:p></pre><pre>100707 =
11:50:31&nbsp; InnoDB: Starting =
shutdown...<o:p></o:p></pre><pre>[root at qad01 heartbeat]# 100707 =
11:50:36&nbsp; InnoDB: Shutdown completed; =
log<o:p></o:p></pre><pre>sequence number =
28732335904<o:p></o:p></pre><pre>100707 11:50:36 [Note] =
/usr/sbin/mysqld: Shutdown =
complete<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>[root at qad01 =
heartbeat]#<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>[root at qad01 =
heartbeat]# crm configure show =
mysql-server<o:p></o:p></pre><pre>primitive mysql-server =
ocf:heartbeat:mysql =
\<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; op =
monitor interval=3D&quot;30s&quot; timeout=3D&quot;30s&quot; =
\<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; op =
start interval=3D&quot;0&quot; timeout=3D&quot;120&quot; =
\<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; op =
stop interval=3D&quot;0&quot; timeout=3D&quot;120&quot; =
\<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; params =
binary=3D&quot;/usr/sbin/mysqld&quot; =
config=3D&quot;/etc/my.cnf&quot;<o:p></o:p></pre><pre>datadir=3D&quot;/dr=
bd/mysql/data/&quot; user=3D&quot;mysql&quot; =
group=3D&quot;mysql&quot;<o:p></o:p></pre><pre>log=3D&quot;/var/log/mysql=
d.log&quot; =
pid=3D&quot;/drbd/mysql/data/qadb.pid&quot;<o:p></o:p></pre><pre>socket=3D=
&quot;/var/lib/mysql/mysql.sock&quot; =
test_user=3D&quot;qaclus&quot;<o:p></o:p></pre><pre>test_passwd=3D&quot;i=
situp&quot; test_table=3D&quot;cluster_check.connectioncheck&quot; =
\<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; meta =
target-role=3D&quot;Started&quot;<o:p></o:p></pre><pre><o:p>&nbsp;</o:p><=
/pre><pre>[root at qad01 heartbeat]# cat =
mysql<o:p></o:p></pre><pre>#!/bin/sh<o:p></o:p></pre><pre>#<o:p></o:p></p=
re><pre>#<o:p></o:p></pre><pre># =
MySQL<o:p></o:p></pre><pre>#<o:p></o:p></pre><pre># Description:&nbsp; =
Manages a MySQL database as Linux-HA =
resource<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>#<o:p></o:p></p=
re><pre># Author:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Alan =
Robertson&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : DB2 =
Script<o:p></o:p></pre><pre># =
Author:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Jakub =
Janczak&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : =
Rewrite as MySQL<o:p></o:p></pre><pre># =
Author:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Andrew =
Beekhof&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Cleanup =
and import<o:p></o:p></pre><pre># =
Author:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Sebastian =
Reitenbach&nbsp;&nbsp;&nbsp; : add OpenBSD defaults, =
more<o:p></o:p></pre><pre>cleanup<o:p></o:p></pre><pre># Author:&nbsp; =
Narayan Newton&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : Add Gentoo/Debian =
defaults<o:p></o:p></pre><pre>#<o:p></o:p></pre><pre># =
Support:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <a
href=3D"mailto:linux-ha at lists.linux-ha.org">linux-ha at lists.linux-ha.org</=
a><o:p></o:p></pre><pre># License:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GNU =
General Public License (GPL)<o:p></o:p></pre><pre># =
Copyright:&nbsp;&nbsp;&nbsp; (C) 2002 - 2005 International Business =
Machines, Inc.<o:p></o:p></pre><pre>#<o:p></o:p></pre><pre># An example =
usage in =
/etc/ha.d/haresources:<o:p></o:p></pre><pre>#&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp; node1&nbsp; 10.0.0.170 =
mysql<o:p></o:p></pre><pre>#<o:p></o:p></pre><pre># See usage() function =
below for more details...<o:p></o:p></pre><pre>#<o:p></o:p></pre><pre># =
OCF instance parameters:<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_binary<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_config<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_datadir<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_user<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_group<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_test_table<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_test_user<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_test_passwd<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_enable_creation<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_additional_parameters<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_log<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_pid<o:p></o:p></pre><pre>#&nbsp;&nbsp; =
OCF_RESKEY_socket<o:p></o:p></pre><pre>##################################=
#####################################<o:p></o:p></pre><pre># =
Initialization:<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>: =
${OCF_FUNCTIONS_DIR=3D${OCF_ROOT}/resource.d/heartbeat}<o:p></o:p></pre><=
pre>. =
${OCF_FUNCTIONS_DIR}/.ocf-shellfuncs<o:p></o:p></pre><pre><o:p>&nbsp;</o:=
p></pre><pre>############################################################=
###########<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre># Added =
exports<o:p></o:p></pre><pre>export =
OCF_ROOT=3D/usr/lib/ocf/<o:p></o:p></pre><pre>export =
OCF_RESKEY_binary_default=3D&quot;/usr/sbin/mysqld&quot;<o:p></o:p></pre>=
<pre>export =
OCF_RESKEY_config_default=3D&quot;/etc/my.cnf&quot;<o:p></o:p></pre><pre>=
export =
OCF_RESKEY_datadir_default=3D&quot;/drbd/mysql/data&quot;<o:p></o:p></pre=
><pre>export =
OCF_RESKEY_user_default=3D&quot;mysql&quot;<o:p></o:p></pre><pre>export =
OCF_RESKEY_group_default=3D&quot;mysql&quot;<o:p></o:p></pre><pre>export =
OCF_RESKEY_log_default=3D&quot;/var/log/mysqld.log&quot;<o:p></o:p></pre>=
<pre>export =
OCF_RESKEY_pid_default=3D&quot;/drbd/mysql/data/qadb.pid&quot;<o:p></o:p>=
</pre><pre>export =
OCF_RESKEY_socket_default=3D&quot;/var/lib/mysql/mysql.sock&quot;<o:p></o=
:p></pre><pre>export =
OCF_RESKEY_test_user_default=3D&quot;qaclus&quot;<o:p></o:p></pre><pre>ex=
port =
OCF_RESKEY_test_table_default=3D&quot;cluster_check.connectioncheck&quot;=
<o:p></o:p></pre><pre>export =
OCF_RESKEY_test_passwd_default=3D&quot;isitup&quot;<o:p></o:p></pre><pre>=
<o:p>&nbsp;</o:p></pre><pre># Fill in some defaults if no values are =
specified<o:p></o:p></pre><pre>HOSTOS=3D`uname`<o:p></o:p></pre><pre>if =
[ &quot;X${HOSTOS}&quot; =3D &quot;XOpenBSD&quot; =
];then<o:p></o:p></pre><pre>OCF_RESKEY_binary_default=3D&quot;/usr/local/=
bin/mysqld_safe&quot;<o:p></o:p></pre><pre>OCF_RESKEY_config_default=3D&q=
uot;/etc/my.cnf&quot;<o:p></o:p></pre><pre>OCF_RESKEY_datadir_default=3D&=
quot;/var/mysql&quot;<o:p></o:p></pre><pre>OCF_RESKEY_user_default=3D&quo=
t;_mysql&quot;<o:p></o:p></pre><pre>OCF_RESKEY_group_default=3D&quot;_mys=
ql&quot;<o:p></o:p></pre><pre>OCF_RESKEY_log_default=3D&quot;/var/log/mys=
qld.log&quot;<o:p></o:p></pre><pre>OCF_RESKEY_pid_default=3D&quot;/var/my=
sql/mysqld.pid&quot;<o:p></o:p></pre><pre>OCF_RESKEY_socket_default=3D&qu=
ot;/var/run/mysql/mysql.sock&quot;<o:p></o:p></pre><pre>OCF_RESKEY_test_u=
ser_default=3D&quot;root&quot;<o:p></o:p></pre><pre>OCF_RESKEY_test_table=
_default=3D&quot;mysql.user&quot;<o:p></o:p></pre><pre>OCF_RESKEY_test_pa=
sswd_default=3D&quot;&quot;<o:p></o:p></pre><pre>OCF_RESKEY_enable_creati=
on_default=3D0<o:p></o:p></pre><pre>OCF_RESKEY_additional_parameters_defa=
ult=3D&quot;&quot;<o:p></o:p></pre><pre>else<o:p></o:p></pre><pre>OCF_RES=
KEY_binary_default=3D&quot;/usr/sbin/mysqld&quot;<o:p></o:p></pre><pre>OC=
F_RESKEY_config_default=3D&quot;/etc/my.cnf&quot;<o:p></o:p></pre><pre>OC=
F_RESKEY_datadir_default=3D&quot;/drbd/mysql/data&quot;<o:p></o:p></pre><=
pre>OCF_RESKEY_user_default=3D&quot;mysql&quot;<o:p></o:p></pre><pre>OCF_=
RESKEY_group_default=3D&quot;mysql&quot;<o:p></o:p></pre><pre>OCF_RESKEY_=
log_default=3D&quot;/var/log/mysqld.log&quot;<o:p></o:p></pre><pre>OCF_RE=
SKEY_pid_default=3D&quot;/drbd/mysql/data/qadb.pid&quot;<o:p></o:p></pre>=
<pre>OCF_RESKEY_socket_default=3D&quot;/var/lib/mysql/mysql.sock&quot;<o:=
p></o:p></pre><pre>OCF_RESKEY_test_user_default=3D&quot;qaclus&quot;<o:p>=
</o:p></pre><pre>OCF_RESKEY_test_table_default=3D&quot;cluster_check.conn=
ectioncheck&quot;<o:p></o:p></pre><pre>OCF_RESKEY_test_passwd_default=3D&=
quot;isitup&quot;<o:p></o:p></pre><pre>OCF_RESKEY_enable_creation_default=
=3D0<o:p></o:p></pre><pre>OCF_RESKEY_additional_parameters_default=3D&quo=
t;&quot;<o:p></o:p></pre><pre>Fi<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></=
pre><pre>[root at qad01 heartbeat]# cat /var/log/messages | grep =
mysql-server<o:p></o:p></pre><pre>Jul&nbsp; 7 11:43:38 qad01 pengine: =
[4359]: ERROR: unpack_rsc_op: Hard error<o:p></o:p></pre><pre>- =
mysql-server_start_0 failed with rc=3D6: Preventing mysql-server =
from<o:p></o:p></pre><pre>re-starting anywhere in the =
cluster<o:p></o:p></pre><pre>Jul&nbsp; 7 11:43:38 qad01 pengine: [4359]: =
WARN: unpack_rsc_op: Processing<o:p></o:p></pre><pre>failed op =
mysql-server_start_0 on qad01: not configured =
(6)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:43:38 qad01 pengine: [4359]: =
notice: =
native_print:<o:p></o:p></pre><pre>mysql-server&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; (ocf::heartbeat:mysql): =
Stopped<o:p></o:p></pre><pre>Jul&nbsp; 7 11:43:38 qad01 pengine: [4359]: =
info: get_failcount: mysql-server<o:p></o:p></pre><pre>has failed =
INFINITY times on qad01<o:p></o:p></pre><pre>Jul&nbsp; 7 11:43:38 qad01 =
pengine: [4359]: WARN: =
common_apply_stickiness:<o:p></o:p></pre><pre>Forcing mysql-server away =
from qad01 after 1000000 =
failures<o:p></o:p></pre><pre>(max=3D1000000)<o:p></o:p></pre><pre>Jul&nb=
sp; 7 11:43:38 qad01 pengine: [4359]: info: native_color: =
Resource<o:p></o:p></pre><pre>mysql-server cannot run =
anywhere<o:p></o:p></pre><pre>Jul&nbsp; 7 11:43:38 qad01 pengine: =
[4359]: notice: LogActions: Leave<o:p></o:p></pre><pre>resource =
mysql-server&nbsp; (Stopped)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 =
qad01 crmd: [4360]: info: =
abort_transition_graph:<o:p></o:p></pre><pre>te_update_diff:267 - =
Triggered transition abort =
(complete=3D1,<o:p></o:p></pre><pre>tag=3Dlrm_rsc_op, =
id=3Dmysql-server_monitor_0,<o:p></o:p></pre><pre>magic=3D0:7;7:0:7:e87a7=
3c4-97b8-4f63-9e69-89ec59fce708, cib=3D0.287.3) =
:<o:p></o:p></pre><pre>Resource op =
removal<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 pengine: [4359]: =
ERROR: unpack_rsc_op: Hard error<o:p></o:p></pre><pre>- =
mysql-server_start_0 failed with rc=3D6: Preventing mysql-server =
from<o:p></o:p></pre><pre>re-starting anywhere in the =
cluster<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 pengine: [4359]: =
WARN: unpack_rsc_op: Processing<o:p></o:p></pre><pre>failed op =
mysql-server_start_0 on qad01: not configured =
(6)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 pengine: [4359]: =
notice: =
native_print:<o:p></o:p></pre><pre>mysql-server&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; (ocf::heartbeat:mysql): =
Stopped<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 pengine: [4359]: =
info: get_failcount: mysql-server<o:p></o:p></pre><pre>has failed =
INFINITY times on qad01<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 =
pengine: [4359]: WARN: =
common_apply_stickiness:<o:p></o:p></pre><pre>Forcing mysql-server away =
from qad01 after 1000000 =
failures<o:p></o:p></pre><pre>(max=3D1000000)<o:p></o:p></pre><pre>Jul&nb=
sp; 7 11:47:58 qad01 pengine: [4359]: info: native_color: =
Resource<o:p></o:p></pre><pre>mysql-server cannot run =
anywhere<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 attrd: [4358]: =
info: attrd_trigger_update: Sending<o:p></o:p></pre><pre>flush op to all =
hosts for: fail-count-mysql-server =
(INFINITY)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 attrd: =
[4358]: info: attrd_trigger_update: Sending<o:p></o:p></pre><pre>flush =
op to all hosts for: last-failure-mysql-server =
(1278516515)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 pengine: =
[4359]: notice: LogActions: Leave<o:p></o:p></pre><pre>resource =
mysql-server&nbsp; (Stopped)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 =
qad01 crmd: [4360]: info: te_rsc_command: =
Initiating<o:p></o:p></pre><pre>action 7: monitor mysql-server_monitor_0 =
on qad02<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 crmd: [4360]: =
info: match_graph_event: =
Action<o:p></o:p></pre><pre>mysql-server_monitor_0 (7) confirmed on =
qad02 (rc=3D0)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 pengine: =
[4359]: ERROR: unpack_rsc_op: Hard error<o:p></o:p></pre><pre>- =
mysql-server_start_0 failed with rc=3D6: Preventing mysql-server =
from<o:p></o:p></pre><pre>re-starting anywhere in the =
cluster<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 pengine: [4359]: =
WARN: unpack_rsc_op: Processing<o:p></o:p></pre><pre>failed op =
mysql-server_start_0 on qad01: not configured =
(6)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 pengine: [4359]: =
notice: =
native_print:<o:p></o:p></pre><pre>mysql-server&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; (ocf::heartbeat:mysql): =
Stopped<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 pengine: [4359]: =
info: get_failcount: mysql-server<o:p></o:p></pre><pre>has failed =
INFINITY times on qad01<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 =
pengine: [4359]: WARN: =
common_apply_stickiness:<o:p></o:p></pre><pre>Forcing mysql-server away =
from qad01 after 1000000 =
failures<o:p></o:p></pre><pre>(max=3D1000000)<o:p></o:p></pre><pre>Jul&nb=
sp; 7 11:47:58 qad01 pengine: [4359]: info: native_color: =
Resource<o:p></o:p></pre><pre>mysql-server cannot run =
anywhere<o:p></o:p></pre><pre>Jul&nbsp; 7 11:47:58 qad01 pengine: =
[4359]: notice: LogActions: Leave<o:p></o:p></pre><pre>resource =
mysql-server&nbsp; (Stopped)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:01 =
qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard =
error<o:p></o:p></pre><pre>- mysql-server_start_0 failed with rc=3D6: =
Preventing mysql-server from<o:p></o:p></pre><pre>re-starting anywhere =
in the cluster<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:01 qad01 pengine: =
[4359]: WARN: unpack_rsc_op: Processing<o:p></o:p></pre><pre>failed op =
mysql-server_start_0 on qad01: not configured =
(6)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:01 qad01 pengine: [4359]: =
notice: =
native_print:<o:p></o:p></pre><pre>mysql-server&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; (ocf::heartbeat:mysql): =
Stopped<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:01 qad01 pengine: [4359]: =
info: get_failcount: mysql-server<o:p></o:p></pre><pre>has failed =
INFINITY times on qad01<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:01 qad01 =
pengine: [4359]: WARN: =
common_apply_stickiness:<o:p></o:p></pre><pre>Forcing mysql-server away =
from qad01 after 1000000 =
failures<o:p></o:p></pre><pre>(max=3D1000000)<o:p></o:p></pre><pre>Jul&nb=
sp; 7 11:48:01 qad01 pengine: [4359]: info: native_color: =
Resource<o:p></o:p></pre><pre>mysql-server cannot run =
anywhere<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:01 qad01 pengine: =
[4359]: notice: LogActions: Leave<o:p></o:p></pre><pre>resource =
mysql-server&nbsp; (Stopped)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:10 =
qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard =
error<o:p></o:p></pre><pre>- mysql-server_start_0 failed with rc=3D6: =
Preventing mysql-server from<o:p></o:p></pre><pre>re-starting anywhere =
in the cluster<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:10 qad01 pengine: =
[4359]: WARN: unpack_rsc_op: Processing<o:p></o:p></pre><pre>failed op =
mysql-server_start_0 on qad01: not configured =
(6)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:10 qad01 pengine: [4359]: =
notice: =
native_print:<o:p></o:p></pre><pre>mysql-server&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; (ocf::heartbeat:mysql): =
Stopped<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:10 qad01 pengine: [4359]: =
info: get_failcount: mysql-server<o:p></o:p></pre><pre>has failed =
INFINITY times on qad01<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:10 qad01 =
pengine: [4359]: WARN: =
common_apply_stickiness:<o:p></o:p></pre><pre>Forcing mysql-server away =
from qad01 after 1000000 =
failures<o:p></o:p></pre><pre>(max=3D1000000)<o:p></o:p></pre><pre>Jul&nb=
sp; 7 11:48:10 qad01 pengine: [4359]: info: native_color: =
Resource<o:p></o:p></pre><pre>mysql-server cannot run =
anywhere<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:10 qad01 pengine: =
[4359]: notice: LogActions: Leave<o:p></o:p></pre><pre>resource =
mysql-server&nbsp; (Stopped)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:11 =
qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard =
error<o:p></o:p></pre><pre>- mysql-server_start_0 failed with rc=3D6: =
Preventing mysql-server from<o:p></o:p></pre><pre>re-starting anywhere =
in the cluster<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:11 qad01 pengine: =
[4359]: WARN: unpack_rsc_op: Processing<o:p></o:p></pre><pre>failed op =
mysql-server_start_0 on qad01: not configured =
(6)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:11 qad01 pengine: [4359]: =
notice: =
native_print:<o:p></o:p></pre><pre>mysql-server&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; (ocf::heartbeat:mysql): =
Stopped<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:11 qad01 pengine: [4359]: =
info: get_failcount: mysql-server<o:p></o:p></pre><pre>has failed =
INFINITY times on qad01<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:11 qad01 =
pengine: [4359]: WARN: =
common_apply_stickiness:<o:p></o:p></pre><pre>Forcing mysql-server away =
from qad01 after 1000000 =
failures<o:p></o:p></pre><pre>(max=3D1000000)<o:p></o:p></pre><pre>Jul&nb=
sp; 7 11:48:11 qad01 pengine: [4359]: info: native_color: =
Resource<o:p></o:p></pre><pre>mysql-server cannot run =
anywhere<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:11 qad01 pengine: =
[4359]: notice: LogActions: Leave<o:p></o:p></pre><pre>resource =
mysql-server&nbsp; (Stopped)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:26 =
qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard =
error<o:p></o:p></pre><pre>- mysql-server_start_0 failed with rc=3D6: =
Preventing mysql-server from<o:p></o:p></pre><pre>re-starting anywhere =
in the cluster<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:26 qad01 pengine: =
[4359]: WARN: unpack_rsc_op: Processing<o:p></o:p></pre><pre>failed op =
mysql-server_start_0 on qad01: not configured =
(6)<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:26 qad01 pengine: [4359]: =
notice: =
native_print:<o:p></o:p></pre><pre>mysql-server&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; (ocf::heartbeat:mysql): =
Stopped<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:26 qad01 pengine: [4359]: =
info: get_failcount: mysql-server<o:p></o:p></pre><pre>has failed =
INFINITY times on qad01<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:26 qad01 =
pengine: [4359]: WARN: =
common_apply_stickiness:<o:p></o:p></pre><pre>Forcing mysql-server away =
from qad01 after 1000000 =
failures<o:p></o:p></pre><pre>(max=3D1000000)<o:p></o:p></pre><pre>Jul&nb=
sp; 7 11:48:26 qad01 pengine: [4359]: info: native_color: =
Resource<o:p></o:p></pre><pre>mysql-server cannot run =
anywhere<o:p></o:p></pre><pre>Jul&nbsp; 7 11:48:26 qad01 pengine: =
[4359]: notice: LogActions: Leave<o:p></o:p></pre><pre>resource =
mysql-server&nbsp; =
(Stopped)<o:p></o:p></pre><pre>___<o:p></o:p></pre><pre><o:p>&nbsp;</o:p>=
</pre><pre>Message: 7<o:p></o:p></pre><pre>Date: Wed, 07 Jul 2010 =
12:55:51 +0300<o:p></o:p></pre><pre>From: Dan Frincu <a
href=3D"mailto:dfrincu at streamwide.ro">&lt;dfrincu at streamwide.ro&gt;</a><o=
:p></o:p></pre><pre>To: The Pacemaker cluster resource =
manager<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
<a
href=3D"mailto:pacemaker at oss.clusterlabs.org">&lt;pacemaker at oss.clusterla=
bs.org&gt;</a><o:p></o:p></pre><pre>Subject: Re: [Pacemaker] Upgraded =
mysql from 5.0 to 5.1<o:p></o:p></pre><pre>Message-ID: <a
href=3D"mailto:4C344F27.1060707 at streamwide.ro">&lt;4C344F27.1060707 at strea=
mwide.ro&gt;</a><o:p></o:p></pre><pre>Content-Type: text/plain; =
charset=3DISO-8859-1; =
format=3Dflowed<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Have =
you copied twice this =
line?<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>socket=3D&quot;/va=
r/lib/mysql/mysql.sock&quot; =
binary=3D&quot;/usr/sbin/mysqld&quot;<o:p></o:p></pre><pre>socket=3D&quot=
;/var/lib/mysql/mysql.sock&quot; =
binary=3D&quot;/usr/sbin/mysqld&quot;<o:p></o:p></pre><pre><o:p>&nbsp;</o=
:p></pre><pre>I think so. Regardless, to test a resource agent manually =
requires that <o:p></o:p></pre><pre>you define some variables and then =
call the script by hand. Also, check <o:p></o:p></pre><pre>all the =
actions (start,stop,restart,promote,etc) and their exit codes, =
<o:p></o:p></pre><pre>to see if they match the OCF RA specification. =
Most of the problems =
that<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>you will have with =
a resource agent and it's resource can be found if =
<o:p></o:p></pre><pre>you're manually testing the RA =
script.<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Go to =
/usr/lib/ocf/resource.d/heartbeat/<o:p></o:p></pre><pre>Open the mysql =
RA script. Go to line 63 and starting from that line =
<o:p></o:p></pre><pre>update the values in the script to match the =
contents of /etc/my.cnf. <o:p></o:p></pre><pre>Then update the crm =
configure for the primitive mysql-server to match =
as<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>well.<o:p></o:p></pre=
><pre> From what I remember, the values in =
<o:p></o:p></pre><pre>OCF_RESKEY_{binary_default,pid_default,socket_defau=
lt} are wrong in the <o:p></o:p></pre><pre>RA script vs what's actually =
installed.<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Then =
&quot;export OCF_ROOT=3D/usr/lib/ocf/&quot; and all OCF_RESKEY_* with =
their <o:p></o:p></pre><pre>defined values, then call the script with no =
parameters. It should <o:p></o:p></pre><pre>provide the usage of the =
script. Then take step by step each action and =
<o:p></o:p></pre><pre>check it's exit code, see if it matches the OCF RA =
specification, and <o:p></o:p></pre><pre>also check to see if it =
actually starts the resource or not. The thing <o:p></o:p></pre><pre>is, =
once the script works as it should, all the issues have been =
<o:p></o:p></pre><pre>resolved, the cluster will work with the =
mysql-server =
resource.<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Regards,<o:p><=
/o:p></pre><pre>Dan<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Jake=
 Bogie wrote:<o:p></o:p></pre><pre>&nbsp; <o:p></o:p></pre>

<blockquote style=3D'margin-top:5.0pt;margin-bottom:5.0pt'><pre>So I =
took Raoul's advice and ditched the lsb:mysql check and went =
for<o:p></o:p></pre><pre>the ocf:heartbeat version =
however...<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>I'm getting =
this now...<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>What am I =
missing? I'm having a hard time finding a document on how =
to<o:p></o:p></pre><pre>setup this resource =
agent.<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D<o:p></o:p></pre><pre>Last updated: Tue Jul&nbsp; 6 =
12:44:07 2010<o:p></o:p></pre><pre>Stack: =
openais<o:p></o:p></pre><pre>Current DC: qad02 - partition with =
quorum<o:p></o:p></pre><pre>Version: =
1.0.9-89bd754939df5150de7cd76835f98fe90851b677<o:p></o:p></pre><pre>2 =
Nodes configured, 2 expected votes<o:p></o:p></pre><pre>3 Resources =
configured.<o:p></o:p></pre><pre>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<o:p=
></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Online: [ qad02 qad01 =
]<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre> Resource Group: =
mysql<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; fs_mysql&nbsp;&nbsp; =
(ocf::heartbeat:Filesystem):&nbsp;&nbsp;&nbsp; Started =
qad02<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; ip_mysql&nbsp;&nbsp; =
(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Started =
qad02<o:p></o:p></pre><pre> Master/Slave Set: =
ms_drbd_mysql<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; Masters: [ =
qad02 ]<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp; Slaves: [ qad01 =
]<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>Failed =
actions:<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp; mysql-server_start_0 =
(node=3Dqad01, call=3D6, rc=3D6, =
status=3Dcomplete):<o:p></o:p></pre><pre>not =
configured<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp; mysql-server_start_0 =
(node=3Dqad02, call=3D33, rc=3D5, =
status=3Dcomplete):<o:p></o:p></pre><pre>not =
installed<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>###<o:p></o:p>=
</pre><pre><o:p>&nbsp;</o:p></pre><pre>primitive mysql-server =
ocf:heartbeat:mysql =
\<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; op =
monitor interval=3D&quot;30s&quot; timeout=3D&quot;30s&quot; =
\<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; op =
start interval=3D&quot;0&quot; timeout=3D&quot;120&quot; =
\<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; op =
stop interval=3D&quot;0&quot; timeout=3D&quot;120&quot; =
\<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; params =
config=3D&quot;/etc/my.cnf&quot; =
datadir=3D&quot;/drbd/mysql/data/&quot;<o:p></o:p></pre><pre>socket=3D&qu=
ot;/var/lib/mysql/mysql.sock&quot; =
binary=3D&quot;/usr/sbin/mysqld&quot;<o:p></o:p></pre><pre>socket=3D&quot=
;/var/lib/mysql/mysql.sock&quot; =
binary=3D&quot;/usr/sbin/mysqld&quot;<o:p></o:p></pre><pre>pid=3D&quot;/d=
rbd/mysql/data/mysql.pid&quot; =
test_passwd=3D&quot;isitup&quot;<o:p></o:p></pre><pre>test_table=3D&quot;=
cluster_check.connectioncheck&quot; test_user=3D&quot;qaclus&quot; =
\<o:p></o:p></pre><pre><o:p>&nbsp;</o:p></pre><pre>______________________=
_________________________<o:p></o:p></pre><pre>Pacemaker mailing list: =
<a
href=3D"mailto:Pacemaker at oss.clusterlabs.org">Pacemaker at oss.clusterlabs.o=
rg</a><o:p></o:p></pre><pre><a
href=3D"http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss=
.clusterlabs.org/mailman/listinfo/pacemaker</a><o:p></o:p></pre><pre><o:p=
>&nbsp;</o:p></pre><pre>Project Home: <a
href=3D"http://www.clusterlabs.org">http://www.clusterlabs.org</a><o:p></=
o:p></pre><pre>Getting started:<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp; =
<o:p></o:p></pre></blockquote>

<pre><a =
href=3D"http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://w=
ww.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><o:p></o:p></pre><pre>=
&nbsp; <o:p></o:p></pre>

<blockquote =
style=3D'margin-top:5.0pt;margin-bottom:5.0pt'><pre>Bugs:<o:p></o:p></pre=
><pre>&nbsp;&nbsp;&nbsp; <o:p></o:p></pre></blockquote>

<pre><a
href=3D"http://developerbugs.linux-foundation.org/enter_bug.cgi?product=3D=
Pacemake">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=
=3DPacemake</a><o:p></o:p></pre><pre>r<o:p></o:p></pre><pre>&nbsp; =
<o:p></o:p></pre>

<blockquote style=3D'margin-top:5.0pt;margin-bottom:5.0pt'><pre>&nbsp; =
<o:p></o:p></pre><pre>&nbsp;&nbsp;&nbsp;&nbsp;<o:p></o:p></pre></blockquo=
te>

<pre><o:p>&nbsp;</o:p></pre><pre>&nbsp; <o:p></o:p></pre>

<p class=3DMsoNormal><br>
<br>
<o:p></o:p></p>

<pre>-- <o:p></o:p></pre><pre>Dan FRINCU<o:p></o:p></pre><pre>Systems =
Engineer<o:p></o:p></pre><pre>CCNA, RHCE<o:p></o:p></pre><pre>Streamwide =
Romania<o:p></o:p></pre><pre>E-mail: <a
href=3D"mailto:dfrincu at streamwide.ro">dfrincu at streamwide.ro</a><o:p></o:p=
></pre><pre>Phone: +40 (0) 21 320 41 24<o:p></o:p></pre></div>

</body>

</html>

------_=_NextPart_001_01CB1E9C.96FA31BE--



More information about the Pacemaker mailing list