<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body text="#000000" bgcolor="#ffffff">
Hello,<br>
I'll explain what’s happened after a network black-out<br>
I've a cluster with pacemaker on Opensuse 11.2 64bit <br>
<small>============<br>
Last updated: Wed Aug 18 18:13:33 2010<br>
Current DC: nodo1 (nodo1)<br>
Version: 1.0.2-ec6b0bbee1f3aa72c4c2559997e675db6ab39160<br>
3 Nodes configured.<br>
11 Resources configured.<br>
============<br>
<br>
Node: nodo1 (nodo1): online<br>
Node: nodo3 (nodo3): online<br>
Node: nodo4 (nodo4): online<br>
<br>
Clone Set: dlm-clone<br>
dlm:0 (ocf::pacemaker:controld): Started nodo3<br>
dlm:1 (ocf::pacemaker:controld): Started nodo1<br>
dlm:2 (ocf::pacemaker:controld): Started nodo4<br>
Clone Set: o2cb-clone<br>
o2cb:0 (ocf::ocfs2:o2cb): Started nodo3<br>
o2cb:1 (ocf::ocfs2:o2cb): Started nodo1<br>
o2cb:2 (ocf::ocfs2:o2cb): Started nodo4<br>
Clone Set: XencfgFS-Clone<br>
XencfgFS:0 (ocf::heartbeat:Filesystem): Started nodo3<br>
XencfgFS:1 (ocf::heartbeat:Filesystem): Started nodo1<br>
XencfgFS:2 (ocf::heartbeat:Filesystem): Started nodo4<br>
Clone Set: XenimageFS-Clone<br>
XenimageFS:0 (ocf::heartbeat:Filesystem): Started nodo3<br>
XenimageFS:1 (ocf::heartbeat:Filesystem): Started nodo1<br>
XenimageFS:2 (ocf::heartbeat:Filesystem): Started nodo4<br>
rsa1-fencing (stonith:external/ibmrsa-telnet): Started nodo4<br>
rsa2-fencing (stonith:external/ibmrsa-telnet): Started nodo3<br>
rsa3-fencing (stonith:external/ibmrsa-telnet): Started nodo4<br>
rsa4-fencing (stonith:external/ibmrsa-telnet): Started nodo3<br>
mailsrv-rm (ocf::heartbeat:Xen): Started nodo3<br>
dbsrv-rm (ocf::heartbeat:Xen): Started nodo4<br>
websrv-rm (ocf::heartbeat:Xen): Started nodo4<br>
</small><br>
After a switch failure all the nodes and the rsa stonith devices was
unreachable.<br>
<br>
On the cluster happen the following error on one node<br>
<br>
<meta http-equiv="CONTENT-TYPE" content="text/html; charset=ISO-8859-1">
<title></title>
<meta name="GENERATOR" content="OpenOffice.org 3.2 (Linux)">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 cluster-dlm: receive_plocks_stored:
receive_plocks_stored 1778493632:2 need_plocks 0#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272025] ------------[ cut here
]------------
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:11:38 nodo1 kernel: [ 4154.272036] kernel BUG at
/usr/src/packages/BUILD/kernel-xen-2.6.31.12/linux-2.6.31/fs/inode.c:1323!
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:11:38 nodo1 kernel: [ 4154.272042] invalid opcode: 0000 [#1]
SMP </font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:11:38 nodo1 kernel: [ 4154.272046] last sysfs file:
/sys/kernel/dlm/0BB443F896254AD3BA8FB960C425B666/control
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:11:38 nodo1 kernel: [ 4154.272050] CPU 1 </font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272053] Modules linked in:
nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_physdev
iptable_filter ip_tables x_tables ocfs2 ocfs2_nodemanager quota_tree
ocfs2_stack_user ocfs2_stackglue dlm configfs netbk coretemp blkbk
blkback_pagemap blktap xenbus_be ipmi_si edd dm_round_robin
scsi_dh_rdac dm_multipath scsi_dh bridge stp llc bonding ipv6 fuse
ext4 jbd2 crc16 loop dm_mod sr_mod ide_pci_generic ide_core iTCO_wdt
ata_generic ibmpex i5k_amb ibmaem iTCO_vendor_support ipmi_msghandler
bnx2 i5000_edac 8250_pnp shpchp ata_piix pcspkr ics932s401 joydev
edac_core i2c_i801 ses pci_hotplug 8250 i2c_core serio_raw enclosure
serial_core button sg reiserfs usbhid hid uhci_hcd ehci_hcd xenblk
cdrom xennet fan processor pata_acpi lpfc thermal thermal_sys hwmon
aacraid [last unloaded: ocfs2_stackglue]
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272111] Pid: 8889, comm: dlm_send
Not tainted 2.6.31.12-0.2-xen #1 IBM System x3650 -[7979AC1]-
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272113] RIP:
e030:[<ffffffff801331c2>] [<ffffffff801331c2>]
iput+0x82/0x90
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272121] RSP: e02b:ffff88014ec03c30
EFLAGS: 00010246
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272122] RAX: 0000000000000000 RBX:
ffff880148a703c8 RCX: 0000000000000000
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272123] RDX: ffffc90000010000 RSI:
ffff880148a70380 RDI: ffff880148a703c8
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272125] RBP: ffff88014ec03c50 R08:
b038000000000000 R09: fe99594c51a57607
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272126] R10: ffff880040410270 R11:
0000000000000000 R12: ffff8801713e6e08
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272128] R13: ffff88014ec03d20 R14:
0000000000000000 R15: ffffc9000331d108
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272133] FS: 00007ff4cb11a730(0000)
GS:ffffc90000010000(0000) knlGS:0000000000000000
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272135] CS: e033 DS: 0000 ES: 0000
CR0: 000000008005003b
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272136] CR2: 00007ff4c5c45000 CR3:
0000000135b2a000 CR4: 0000000000002660
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272138] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272140] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272142] Process dlm_send (pid: 8889,
threadinfo ffff88014ec02000, task ffff8801381e45c0)</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272143] Stack:
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272144] 0000000000000000
00000000072f0874 ffff880148a70380 ffff880148a70380
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272146] <0> ffff88014ec03c80
ffffffff803add09 ffff88014ec03c80 00000000072f0874
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272147] <0> ffff8801713e6df8
ffff8801713e6e08 ffff88014ec03de0 ffffffffa05661e1
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272150] Call Trace:
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272164] [<ffffffff803add09>]
sock_release+0x89/0xa0
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272177] [<ffffffffa05661e1>]
tcp_connect_to_sock+0x161/0x2b0 [dlm]
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272206] [<ffffffffa0568764>]
process_send_sockets+0x34/0x60 [dlm]
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272222] [<ffffffff800693f3>]
run_workqueue+0x83/0x230
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272227] [<ffffffff80069654>]
worker_thread+0xb4/0x140
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272231] [<ffffffff8006fac6>]
kthread+0xb6/0xc0
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272236] [<ffffffff8000d38a>]
child_rip+0xa/0x20
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272240] Code: 42 20 48 c7 c2 b0 4c
13 80 48 85 c0 48 0f 44 c2 48 89 df ff d0 48 8b 45 e8 65 48 33 04 25
28 00 00 00 75 0b 48 83 c4 18 5b c9 c3 <0f> 0b eb fe e8 35 c6
f1 ff 0f 1f 44 00 00 55 48 8d 97 10 02 00 </font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272256] RIP [<ffffffff801331c2>]
iput+0x82/0x90
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272259] RSP <ffff88014ec03c30>
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272264] ---[ end trace
7707d0d92a7f5415 ]---
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1">Aug
18
13:11:38 nodo1 kernel: [ 4154.272495] dlm: connect from non
cluster node
</font></p>
<br>
and after few log lines the following line repeated until the node was
killed by me<br>
<meta http-equiv="CONTENT-TYPE" content="text/html; charset=ISO-8859-1">
<title></title>
<meta name="GENERATOR" content="OpenOffice.org 3.2 (Linux)">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<p style="margin-bottom: 0in;">
<meta http-equiv="CONTENT-TYPE" content="text/html; charset=ISO-8859-1">
<title></title>
<meta name="GENERATOR" content="OpenOffice.org 3.2 (Linux)">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: start_kernel: start_kernel cg 3
member_count 1#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: update_dir_members: dir_member
1812048064#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: update_dir_members: dir_member
1778493632#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: set_configfs_members: set_members
rmdir
"/sys/kernel/config/dlm/cluster/spaces/0BB443F896254AD3BA8FB960C425B666/nodes/1812048064"#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/0BB443F896254AD3BA8FB960C425B666/control"#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: set_fs_notified: set_fs_notified no
nodeid 1812048064#012</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: set_fs_notified: set_fs_notified no
nodeid 1812048064#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: set_fs_notified: set_fs_notified no
nodeid 1812048064#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: set_fs_notified: set_fs_notified no
nodeid 1812048064#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: set_fs_notified: set_fs_notified no
nodeid 1812048064#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: set_fs_notified: set_fs_notified no
nodeid 1812048064#012
</font></p>
<p style="margin-bottom: 0in;"><font style="font-size: 8pt;" size="1"
color="#ff0000">Aug
18 13:12:31 nodo1 cluster-dlm: set_fs_notified: set_fs_notified no
nodeid 1812048064#012</font></p>
</p>
<br>
Attached the log file<br>
<br>
Someone can explain what is the reason?<br>
<br>
Regards,<br>
Roberto.<br>
<br>
<br>
</body>
</html>