[ClusterLabs] gfs2 crashes when i, e.g., dd to a lvm volume

[Lars Ellenberg]

On Thu, Oct 08, 2015 at 01:50:50PM +0200, J. Echter wrote:
> Hi,
> 
> i have a strange issue on CentOS 6.5
> 
> If i install a new vm on node1 it works well.
> 
> If i install a new vm on node2 it gets stuck.
> 
> Same if i do a dd if=/dev/zero of=/dev/DATEN/vm-test (on node2)
> 
> On node1 it works:
> 
> dd if=/dev/zero of=vm-test
> Schreiben in „vm-test“: Auf dem Gerät ist kein Speicherplatz mehr verfügbar
> 83886081+0 Datensätze ein
> 83886080+0 Datensätze aus
> 42949672960 Bytes (43 GB) kopiert, 2338,15 s, 18,4 MB/s
> 
> 
> dmesg shows the following (while dd'ing on node2):
> 
> INFO: task flush-253:18:9820 blocked for more than 120 seconds.
>       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> flush-253:18  D 0000000000000000     0  9820      2 0x00000080
>  ffff8805f9bc3490 0000000000000046 ffff8805f9bc3458 ffff8805f9bc3454
>  ffff880620768c00 ffff88062fc23080 0000006fc80f5a1e ffff8800282959c0
>  00000000000002aa 000000010002be11 ffff8806241d65f8 ffff8805f9bc3fd8
> Call Trace:
>  [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd]

What DRBD version is this?
What does the IO stack look like?

DRBD seems to block and wait for something in its make request function.
So maybe it's backend is blocked for some reason?

You'd see this for example on a thin provisioned backend that is
configured to block when out of physical space...

>  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
>  [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40
>  [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd]
>  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
>  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
>  [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod]
>  [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd]
>  [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60
>  [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0
>  [<ffffffff81276544>] ? blk_queue_bio+0x494/0x610
>  [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod]
>  [<ffffffff81274910>] generic_make_request+0x240/0x5a0
>  [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20
>  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
>  [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20
>  [<ffffffff81274ce0>] submit_bio+0x70/0x120

-- 
: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.