[Pacemaker] Master/Slave resource cannot start
Diego Remolina
diego.remolina at physics.gatech.edu
Fri Aug 21 15:11:41 UTC 2009
Here is what I am seeing now right after stopping openais, updating
heartbeat and pacemaker and trying to start openais again:
[root at phys-file02 ~]# /etc/init.d/openais status
Stopped
[root at phys-file02 ~]# /etc/init.d/openais start
Starting OpenAIS daemon (aisexec): starting... rc=0: OK
[root at phys-file02 ~]# crm status
Connection to cluster failed: connection failed
[root at phys-file02 ~]# crm status
Connection to cluster failed: connection failed
[root at phys-file02 ~]# crm status
Connection to cluster failed: connection failed
[root at phys-file02 ~]# yum -y install gdb
At this point, I installed gdb and here is what I get:
[root at phys-file02 ~]# ps -ef | grep aisexec
root 19423 1 0 11:01 pts/1 00:00:00 aisexec
root 19520 19241 0 11:02 pts/1 00:00:00 grep aisexec
[root at phys-file02 ~]# gdb aisexec 19423
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(no debugging symbols found)
Attaching to program: /usr/sbin/aisexec, process 19423
Reading symbols from /lib64/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x2ae946b8fec0 (LWP 19423)]
[New Thread 0x40638fe0 (LWP 19425)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/libexec/lcrso/objdb.lcrso...done.
Loaded symbols for /usr/libexec/lcrso/objdb.lcrso
Reading symbols from /usr/libexec/lcrso/aisparser.lcrso...done.
Loaded symbols for /usr/libexec/lcrso/aisparser.lcrso
Reading symbols from /usr/libexec/lcrso/pacemaker.lcrso...done.
Loaded symbols for /usr/libexec/lcrso/pacemaker.lcrso
Reading symbols from /usr/lib64/libplumb.so.2...done.
Loaded symbols for /usr/lib64/libplumb.so.2
Reading symbols from /usr/lib64/libpils.so.2...done.
Loaded symbols for /usr/lib64/libpils.so.2
Reading symbols from /usr/lib64/libbz2.so.1...done.
Loaded symbols for /usr/lib64/libbz2.so.1
Reading symbols from /usr/lib64/libxslt.so.1...done.
Loaded symbols for /usr/lib64/libxslt.so.1
Reading symbols from /usr/lib64/libxml2.so.2...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/libpam.so.0...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libglib-2.0.so.0...done.
Loaded symbols for /lib64/libglib-2.0.so.0
Reading symbols from /usr/lib64/libltdl.so.3...done.
Loaded symbols for /usr/lib64/libltdl.so.3
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libaudit.so.0...done.
Loaded symbols for /lib64/libaudit.so.0
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
0x0000003be08dee5e in __lll_lock_wait_private () from /lib64/libc.so.6
(gdb) where
#0 0x0000003be08dee5e in __lll_lock_wait_private () from /lib64/libc.so.6
#1 0x0000003be088c74d in _L_lock_1685 () from /lib64/libc.so.6
#2 0x0000003be088c497 in __tz_convert () from /lib64/libc.so.6
#3 0x0000000000418a16 in _log_printf ()
#4 0x0000000000418cb1 in internal_log_printf2 ()
#5 0x00002aaaab0b8819 in pcmk_plugin_init () from
/usr/libexec/lcrso/pacemaker.lcrso
#6 0x00002aaaab0b946a in pcmk_startup () from
/usr/libexec/lcrso/pacemaker.lcrso
#7 0x000000000041a422 in openais_service_link_and_init ()
#8 0x000000000041a5c8 in openais_service_defaults_link_and_init ()
#9 0x0000000000418117 in main ()
(gdb) thread 0
Thread ID 0 not known.
(gdb) thread 1
[Switching to thread 1 (Thread 0x2ae946b8fec0 (LWP 19423))]#0
0x0000003be08dee5e in __lll_lock_wait_private () from /lib64/libc.so.6
(gdb) where
#0 0x0000003be08dee5e in __lll_lock_wait_private () from /lib64/libc.so.6
#1 0x0000003be088c74d in _L_lock_1685 () from /lib64/libc.so.6
#2 0x0000003be088c497 in __tz_convert () from /lib64/libc.so.6
#3 0x0000000000418a16 in _log_printf ()
#4 0x0000000000418cb1 in internal_log_printf2 ()
#5 0x00002aaaab0b8819 in pcmk_plugin_init () from
/usr/libexec/lcrso/pacemaker.lcrso
#6 0x00002aaaab0b946a in pcmk_startup () from
/usr/libexec/lcrso/pacemaker.lcrso
#7 0x000000000041a422 in openais_service_link_and_init ()
#8 0x000000000041a5c8 in openais_service_defaults_link_and_init ()
#9 0x0000000000418117 in main ()
(gdb) thread 3
Thread ID 3 not known.
(gdb) thread 4
Thread ID 4 not known.
(gdb) thread 5
Thread ID 5 not known.
(gdb) thread 6
Thread ID 6 not known.
Like I said, I have not used gdb before, so if I am doing something
wrong, let me know what I should do or where can I read some docs to try
and understand what I am supposed to do with it to give you useful output.
Here is the log file where I do not see any valuable crm info up until
the point where I installed gdb on the system
Aug 21 11:01:31 phys-file02 openais[19423]: [MAIN ] AIS Executive
Service RELEASE 'subrev 1152 version 0.80'
Aug 21 11:01:31 phys-file02 openais[19423]: [MAIN ] Copyright (C)
2002-2006 MontaVista Software, Inc and contributors.
Aug 21 11:01:31 phys-file02 openais[19423]: [MAIN ] Copyright (C) 2006
Red Hat, Inc.
Aug 21 11:01:31 phys-file02 openais[19423]: [MAIN ] AIS Executive
Service: started and ready to provide service.
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] Token Timeout (3000
ms) retransmit timeout (294 ms)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] token hold (225 ms)
retransmits before loss (10 retrans)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] join (60 ms)
send_join (0 ms) consensus (1500 ms) merge (200 ms)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] downcheck (1000 ms)
fail to recv const (50 msgs)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] seqno unchanged
const (30 rotations) Maximum network MTU 1500
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] window size per
rotation (50 messages) maximum messages per rotation (20 messages)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] send threads (0 threads)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] RRP token expired
timeout (294 ms)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] RRP token problem
counter (2000 ms)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] RRP threshold (10
problem count)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] RRP mode set to passive.
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM]
heartbeat_failures_allowed (0)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] max_network_delay
(50 ms)
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] HeartBeat is
Disabled. To enable set heartbeat_failures_allowed > 0
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] Receive multicast
socket recv buffer size (262142 bytes).
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] Transmit multicast
socket send buffer size (262142 bytes).
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] The network
interface [10.0.0.22] is now up.
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] Created or loaded
sequence id 184.10.0.0.22 for this ring.
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] Receive multicast
socket recv buffer size (262142 bytes).
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] Transmit multicast
socket send buffer size (262142 bytes).
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] The network
interface [10.0.1.22] is now up.
Aug 21 11:01:32 phys-file02 openais[19423]: [TOTEM] entering GATHER
state from 15.
Aug 21 11:01:32 phys-file02 openais[19423]: [crm ] info:
process_ais_conf: Reading configure
Aug 21 11:01:32 phys-file02 openais[19423]: [MAIN ] info:
config_find_next: Processing additional logging options...
Aug 21 11:01:32 phys-file02 openais[19423]: [MAIN ] info:
get_config_opt: Found 'on' for option: debug
Aug 21 11:01:32 phys-file02 openais[19423]: [MAIN ] info:
get_config_opt: Defaulting to 'off' for option: to_file
Aug 21 11:01:32 phys-file02 openais[19423]: [MAIN ] info:
get_config_opt: Found 'daemon' for option: syslog_facility
Aug 21 11:01:32 phys-file02 openais[19423]: [MAIN ] info:
config_find_next: Processing additional service options...
Aug 21 11:01:32 phys-file02 openais[19423]: [MAIN ] info:
get_config_opt: Defaulting to 'no' for option: use_logd
Aug 21 11:01:58 phys-file02 crm_shadow: [19439]: info: Invoked: crm_shadow
Aug 21 11:01:58 phys-file02 crm_shadow: [19453]: info: Invoked: crm_shadow
Aug 21 11:01:58 phys-file02 crm_shadow: [19455]: info: Invoked: crm_shadow
Aug 21 11:02:01 phys-file02 crm_shadow: [19467]: info: Invoked: crm_shadow
Aug 21 11:02:01 phys-file02 crm_shadow: [19481]: info: Invoked: crm_shadow
Aug 21 11:02:01 phys-file02 crm_shadow: [19483]: info: Invoked: crm_shadow
Aug 21 11:02:03 phys-file02 crm_shadow: [19495]: info: Invoked: crm_shadow
Aug 21 11:02:03 phys-file02 crm_shadow: [19509]: info: Invoked: crm_shadow
Aug 21 11:02:03 phys-file02 crm_shadow: [19511]: info: Invoked: crm_shadow
Aug 21 11:02:16 phys-file02 yum: Installed: gdb-6.8-27.el5.x86_64
Again, killin aisexec and restarting openais seems to work.
[root at phys-file02 ~]# /etc/init.d/openais stop
Stopping OpenAIS daemon (aisexec):
......................................................................................................................................
[root at phys-file02 ~]# pkill -9 aisexec
[root at phys-file02 ~]# ps -ef | grep aise
root 19546 19241 0 11:10 pts/1 00:00:00 grep aise
[root at phys-file02 ~]# /etc/init.d/openais start
Starting OpenAIS daemon (aisexec): starting... rc=0: OK
[root at phys-file02 ~]# crm status
============
Last updated: Fri Aug 21 11:10:51 2009
Stack: openais
Current DC: phys-file01.physics.gatech.edu - partition with quorum
Version: 1.0.5-462f1569a43740667daf7b0f6b521742e9eb8fa7
2 Nodes configured, 2 expected votes
4 Resources configured.
============
Online: [ phys-file01.physics.gatech.edu phys-file02.physics.gatech.edu ]
Master/Slave Set: ms-drbd_export
Masters: [ phys-file01.physics.gatech.edu ]
Slaves: [ phys-file02.physics.gatech.edu ]
Master/Slave Set: ms-drbd_scratch
Masters: [ phys-file01.physics.gatech.edu ]
Slaves: [ phys-file02.physics.gatech.edu ]
Resource Group: fileserver
fs_export (ocf::heartbeat:Filesystem): Started
phys-file01.physics.gatech.edu
fs_scratch (ocf::heartbeat:Filesystem): Started
phys-file01.physics.gatech.edu
virtual-ip-1 (ocf::heartbeat:IPaddr2): Started
phys-file01.physics.gatech.edu
nfs (lsb:nfs): Started phys-file01.physics.gatech.edu
samba (lsb:smb): Started phys-file01.physics.gatech.edu
Clone Set: pingd-clone
Started: [ phys-file01.physics.gatech.edu
phys-file02.physics.gatech.edu ]
[root at phys-file02 ~]#
Diego
Andrew Beekhof wrote:
> On Wed, Aug 12, 2009 at 3:35 PM, Diego
> Remolina<diego.remolina at physics.gatech.edu> wrote:
>>> could you instead attach to it with gdb and see what it was doing?
>> I will try, but cannot promise it will be soon, beginning of the semester is
>> very busy and I am not familiar with gdb...
>
> gdb aisexec $PID_OF_AISEXEC
> # where
>
> then, for every thread it has:
>
> # thread 0
> # where
> # thread 1
> # where
> ...
>
> I think you get the idea :-)
>
>> RedHat.... one is x86_64, the other is the 32 bit one....
>>
>> [root at phys-file01 windows7]# rpm -qa --qf
>> "%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n" | grep openais
>> openais-0.80.5-13.1.x86_64
>> libopenais2-0.80.5-13.1.i386
>> libopenais2-0.80.5-13.1.x86_64
>
> how about trying with just one?
> maybe something is confused.
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
More information about the Pacemaker
mailing list