[Pacemaker] alignment issues on arm (debian armel arch)

[Greg Walton]

I've been slowly working at getting pacemaker(stable-1.0) with 
corosync(flatiron) to run on arm little endian arch.

I'm having alignment issues that break everything, crm status never 
reports any nodes joining. I've simplified things down to starting 
corosync on a single node, and expecting it to contact crmd and join the 
cluster via ipc(which it's not doing).


I have isolated one alignment case I've been trying to fix:

The AIS_Message struct pointed to by 'msg" in in ais_dispatch() (from 
lib/common/ais.c) shows up with odd alignment sometimes  which causes 
access to msg->is_compressed to throw a bus error.

What I've been doing is switching the kernel to signal on unaligned 
access "echo 5 > /proc/cpu/alignment" which gets me a core file.. 99 
times out of 100 crmd is the first to core out, within 2 or 3 seconds at 
most.

I should add that I can switch on kernel alignment fixups via "echo 3 > 
/proc/cpu/alignment" and the single node will sign on with crmd... at 
least initially, i haven't actually tested beyond "crm status"

With some debugging lines in ais.c, I can see that crmd does NOT core 
when the AIS_Message struct starts in an address that is a multiple of 
4. I've fiddled with various __attrib__ align statements in the 
AIS_Message struct in include/crm/ais.h and regardless of what align 
attributes I apply to struct members or the struct, the entire struct 
shows up starting at different alignments, 4 first then some odd/non 
alignment and cores out.

So I'm trying to track down where the memory for the AIS_Message 
struct/buffer is getting allocated at. I've backed into corosync with 
coroipcc_dispatch_get(), then socket_recv(), and finally to recvmsg(). I 
only see 1 char at a time being read from the socket, and returned back 
to ais_dispatch()... but I got getting fairly confused by all the 
dispatches and handles being passed around and I'm not sure how the 
dispatch scheme is supposed to work anyway.

So, the big question I have is: In ais_dispatch(), the memory that "msg" 
points to get's allocated by what/where... and/or is it in corosync or 
pacemaker?


Here's an example of what I'm seeing in gdb( that source line #455 is 
something I added for debugging, a few lines later the code normally 
uses msg->is_compressed and you'd get the bus error there instead..

Core was generated by `/usr/lib/heartbeat/crmd'.
Program terminated with signal 7, Bus error.
#0  0x40105048 in ais_dispatch (sender=7, user_data=0xfebc) at ais.c:455
455         crm_debug("msg->is_compressed:%d", msg->is_compressed);
(gdb) print msg
$1 = (AIS_Message *) 0x40b6832b
(gdb) print &msg->is_compressed
$2 = (int *) 0x40b6834b
(gdb) print *msg
$3 = {header = {size = 778, id = 0, error = CS_OK}, id = 1, 
is_compressed = 0,
   host = {id = 0, pid = 0, local = 1, type = crm_msg_crmd, size = 0,
     uname = '\000' <repeats 255 times>}, sender = {id = 855754762,
     pid = 24720, local = 0, type = crm_msg_crmd, size = 5,
     uname = "serva", '\000' <repeats 250 times>}, size = 178,
   compressed_size = 0,
   data = 0x40b6857f "<create_request_adv origin=\"post_cache_update\" 
t=\"crmd\" version=\"3.0.1\" subt=\"request\" 
reference=\"noop-crmd-1292462365-0\" crm_task=\"noop\" 
crm_sys_to=\"crmd\" crm_sys_from=\"crmd\"/>"}
(gdb) bt
#0  0x40105048 in ais_dispatch (sender=7, user_data=0xfebc) at ais.c:455
#1  0x401ced6c in G_fd_dispatch (source=0x5a4f0, callback=0, user_data=0x0)
     at GSource.c:303
#2  0x405308b4 in g_main_context_dispatch () from /lib/libglib-2.0.so.0
#3  0x405346b4 in ?? () from /lib/libglib-2.0.so.0
Cannot access memory at address 0x0
#4  0x405346b4 in ?? () from /lib/libglib-2.0.so.0
Cannot access memory at address 0x0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)


Note that msg=0x40b6832b is not 4 byte aligned by my math. In some cases 
msg does point to 4 byte aligned memory. I added another debug line to 
print out %p for msg each time ais_dispatch() is called.

Here is part of a corosync.log with my debug prints for msg, you can see 
crmd cores and re-registers with lrmd after each bus error, then it's 
aligned good 1 time, then cores again after that, over and over.... 4 
byte aligned 1 time then the next it's not.

root at serva:/var/lib/heartbeat/cores/hacluster# grep msg /tmp/corosync.log
Dec 15 20:19:13 corosync [TOTEM ] downcheck (1000 ms) fail to recv const 
(50 msgs)
Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: msg %p:0x40b26000
Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:16 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24696] registered
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: msg %p:0x40b6832b
Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: (char*)msg:
Dec 15 20:19:17 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24702] registered
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: msg %p:0x40b6832b
Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: (char*)msg:
Dec 15 20:19:18 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24704] registered
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: msg %p:0x40b6832b
Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: (char*)msg:
Dec 15 20:19:19 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24706] registered
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: msg %p:0x40b6832b
Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: (char*)msg:
Dec 15 20:19:20 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24708] registered
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: msg %p:0x40b6832b
Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: (char*)msg:
Dec 15 20:19:21 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24710] registered
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: msg %p:0x40b6832b
Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: (char*)msg:
Dec 15 20:19:22 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24712] registered
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: msg %p:0x40b6832b
Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: (char*)msg:
Dec 15 20:19:23 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24715] registered
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: msg %p:0x40b6832b
Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: (char*)msg:
Dec 15 20:19:24 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24717] registered
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: msg %p:0x40b6832b
Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: (char*)msg:
Dec 15 20:19:25 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24720] registered
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: (char*)msg:+
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: 
msg->is_compressed:0
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment 
of msg->is_compressed:0
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment 
of ais_data_len(msg):211
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment 
of msg->size():211
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment 
of msg->compressed_size():0
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: msg %p:0x40b6832b
Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: (char*)msg:
Dec 15 20:19:26 serva lrmd: [24693]: debug: on_msg_register:client crmd 
[24722] registered
Dec 15 20:19:26 serva crmd: [24722]: debug: ais_dispatch: msg %p:0x40b68000
Dec 15 20:19:26 serva crmd: [24722]: debug: ais_dispatch: (char*)msg:+
root at serva:/var/lib/heartbeat/cores/hacluster#


here's a short tail of dmesg (which is the "warn" that you can configure 
/proc/cpu/alignment for) that shows the various PIDs of crmd whenthey 
bus error.
Notice the address 0c40b6834b is the same as my gdb ouptut above shows 
for print &msg->is_compressed.

[728065.861236] Alignment trap: crmd (18435) PC=0x40105008 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[728066.902288] Alignment trap: crmd (18438) PC=0x40105008 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[728067.944698] Alignment trap: crmd (18440) PC=0x40105008 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[735468.169460] Alignment trap: crmd (24696) PC=0x40105048 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[735469.296659] Alignment trap: crmd (24702) PC=0x40105048 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[735470.337662] Alignment trap: crmd (24704) PC=0x40105048 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[735471.403065] Alignment trap: crmd (24706) PC=0x40105048 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[735472.395324] Alignment trap: crmd (24708) PC=0x40105048 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[735473.426591] Alignment trap: crmd (24710) PC=0x40105048 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[735474.462234] Alignment trap: crmd (24712) PC=0x40105048 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[735475.505742] Alignment trap: crmd (24715) PC=0x40105048 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[735476.542277] Alignment trap: crmd (24717) PC=0x40105048 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
[735477.587241] Alignment trap: crmd (24720) PC=0x40105048 
Instr=0xe5933020 Address=0x40b6834b FSR 0x001
root at serva:/var/lib/heartbeat/cores/hacluster#


Sorry for the massive cut and paste in my first email to the list, I 
just wanted to throw in all the details I've got and hope that someone 
can enlighten me further and/or tell me what I missed.

Thanks,
Greg

oh yeah this should cover details on the arch/build some on the arch/build:

root at serva:/var/lib/heartbeat/cores/hacluster# uname -a
Linux serva 2.6.32-5-kirkwood #1 Fri Nov 26 07:01:06 UTC 2010 armv5tel 
GNU/Linux


root at serva:/var/lib/heartbeat/cores/hacluster# readelf  -h 
/usr/lib/heartbeat/crmd
ELF Header:
   Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
   Class:                             ELF32
   Data:                              2's complement, little endian
   Version:                           1 (current)
   OS/ABI:                            UNIX - System V
   ABI Version:                       0
   Type:                              EXEC (Executable file)
   Machine:                           ARM
   Version:                           0x1
   Entry point address:               0xc1b4
   Start of program headers:          52 (bytes into file)
   Start of section headers:          4217016 (bytes into file)
   Flags:                             0x5000002, has entry point, 
Version5 EABI
   Size of this header:               52 (bytes)
   Size of program headers:           32 (bytes)
   Number of program headers:         8
   Size of section headers:           40 (bytes)
   Number of section headers:         41
   Section header string table index: 38
root at serva:/var/lib/heartbeat/cores/hacluster#