[ClusterLabs] gfs2 crashes when i, e.g., dd to a lvm volume

Thu Oct 8 11:50:50 UTC 2015

Hi,

i have a strange issue on CentOS 6.5

If i install a new vm on node1 it works well.

If i install a new vm on node2 it gets stuck.

Same if i do a dd if=/dev/zero of=/dev/DATEN/vm-test (on node2)

On node1 it works:

dd if=/dev/zero of=vm-test
Schreiben in „vm-test“: Auf dem Gerät ist kein Speicherplatz mehr verfügbar
83886081+0 Datensätze ein
83886080+0 Datensätze aus
42949672960 Bytes (43 GB) kopiert, 2338,15 s, 18,4 MB/s

dmesg shows the following (while dd'ing on node2):

INFO: task flush-253:18:9820 blocked for more than 120 seconds.
       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-253:18  D 0000000000000000     0  9820      2 0x00000080
  ffff8805f9bc3490 0000000000000046 ffff8805f9bc3458 ffff8805f9bc3454
  ffff880620768c00 ffff88062fc23080 0000006fc80f5a1e ffff8800282959c0
  00000000000002aa 000000010002be11 ffff8806241d65f8 ffff8805f9bc3fd8
Call Trace:
  [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd]
  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
  [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40
  [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd]
  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
  [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod]
  [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd]
  [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60
  [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0
  [<ffffffff81276544>] ? blk_queue_bio+0x494/0x610
  [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod]
  [<ffffffff81274910>] generic_make_request+0x240/0x5a0
  [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20
  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
  [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20
  [<ffffffff81274ce0>] submit_bio+0x70/0x120
  [<ffffffff811c78ed>] submit_bh+0x11d/0x1f0
  [<ffffffff811ca0a8>] __block_write_full_page+0x1c8/0x330
  [<ffffffff811c9070>] ? end_buffer_async_write+0x0/0x190
  [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20
  [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20
  [<ffffffff811ca2f0>] block_write_full_page_endio+0xe0/0x120
  [<ffffffff81126ff0>] ? find_get_pages_tag+0x40/0x130
  [<ffffffff811ca345>] block_write_full_page+0x15/0x20
  [<ffffffff811cf108>] blkdev_writepage+0x18/0x20
  [<ffffffff8113b387>] __writepage+0x17/0x40
  [<ffffffff8113c64d>] write_cache_pages+0x1fd/0x4c0
  [<ffffffff8113b370>] ? __writepage+0x0/0x40
  [<ffffffff8113c934>] generic_writepages+0x24/0x30
  [<ffffffff8113c961>] do_writepages+0x21/0x40
  [<ffffffff811beb3d>] writeback_single_inode+0xdd/0x290
  [<ffffffff811bef3d>] writeback_sb_inodes+0xbd/0x170
  [<ffffffff811bf09b>] writeback_inodes_wb+0xab/0x1b0
  [<ffffffff811bf493>] wb_writeback+0x2f3/0x410
  [<ffffffff8108a752>] ? del_timer_sync+0x22/0x30
  [<ffffffff811bf755>] wb_do_writeback+0x1a5/0x240
  [<ffffffff811bf853>] bdi_writeback_task+0x63/0x1b0
  [<ffffffff810a12e7>] ? bit_waitqueue+0x17/0xd0
  [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100
  [<ffffffff8114b7c6>] bdi_start_fn+0x86/0x100
  [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100
  [<ffffffff810a0fce>] kthread+0x9e/0xc0
  [<ffffffff8100c28a>] child_rip+0xa/0x20
  [<ffffffff810a0f30>] ? kthread+0x0/0xc0
  [<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task flush-253:18:9820 blocked for more than 120 seconds.
       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-253:18  D 0000000000000000     0  9820      2 0x00000080
  ffff8805f9bc3490 0000000000000046 ffff8805f9bc3458 ffff8805f9bc3454
  ffff880620768c00 ffff88062fc23080 0000006fc80f5a1e ffff8800282959c0
  00000000000002aa 000000010002be11 ffff8806241d65f8 ffff8805f9bc3fd8
Call Trace:
  [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd]
  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
  [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40
  [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd]
  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
  [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod]
  [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd]
  [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60
  [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0
  [<ffffffff81276544>] ? blk_queue_bio+0x494/0x610
  [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod]
  [<ffffffff81274910>] generic_make_request+0x240/0x5a0
  [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20
  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
  [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20
  [<ffffffff81274ce0>] submit_bio+0x70/0x120
  [<ffffffff811c78ed>] submit_bh+0x11d/0x1f0
  [<ffffffff811ca0a8>] __block_write_full_page+0x1c8/0x330
  [<ffffffff811c9070>] ? end_buffer_async_write+0x0/0x190
  [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20
  [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20
  [<ffffffff811ca2f0>] block_write_full_page_endio+0xe0/0x120
  [<ffffffff81126ff0>] ? find_get_pages_tag+0x40/0x130
  [<ffffffff811ca345>] block_write_full_page+0x15/0x20
  [<ffffffff811cf108>] blkdev_writepage+0x18/0x20
  [<ffffffff8113b387>] __writepage+0x17/0x40
  [<ffffffff8113c64d>] write_cache_pages+0x1fd/0x4c0
  [<ffffffff8113b370>] ? __writepage+0x0/0x40
  [<ffffffff8113c934>] generic_writepages+0x24/0x30
  [<ffffffff8113c961>] do_writepages+0x21/0x40
  [<ffffffff811beb3d>] writeback_single_inode+0xdd/0x290
  [<ffffffff811bef3d>] writeback_sb_inodes+0xbd/0x170
  [<ffffffff811bf09b>] writeback_inodes_wb+0xab/0x1b0
  [<ffffffff811bf493>] wb_writeback+0x2f3/0x410
  [<ffffffff8108a752>] ? del_timer_sync+0x22/0x30
  [<ffffffff811bf755>] wb_do_writeback+0x1a5/0x240
  [<ffffffff811bf853>] bdi_writeback_task+0x63/0x1b0
  [<ffffffff810a12e7>] ? bit_waitqueue+0x17/0xd0
  [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100
  [<ffffffff8114b7c6>] bdi_start_fn+0x86/0x100
  [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100
  [<ffffffff810a0fce>] kthread+0x9e/0xc0
  [<ffffffff8100c28a>] child_rip+0xa/0x20
  [<ffffffff810a0f30>] ? kthread+0x0/0xc0
  [<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task glock_workqueue:7144 blocked for more than 120 seconds.
       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glock_workque D 0000000000000001     0  7144      2 0x00000080
  ffff880602d13740 0000000000000046 0000000000000000 0000000000000000
  0000000000000000 ffff880622eb3080 0000009d020d96ca 0000000000000246
  ffff880602d13720 000000010005b5aa ffff880622cbbad8 ffff880602d13fd8
Call Trace:
  [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd]
  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
  [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40
  [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd]
  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
  [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod]
  [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd]
  [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60
  [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0
  [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod]
  [<ffffffff81274910>] generic_make_request+0x240/0x5a0
  [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20
  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
  [<ffffffff81274ce0>] submit_bio+0x70/0x120
  [<ffffffff811c78ed>] submit_bh+0x11d/0x1f0
  [<ffffffffa04f0ddf>] buf_lo_before_commit+0x10f/0x1c0 [gfs2]
  [<ffffffffa04ef6c8>] gfs2_log_flush+0x1d8/0x600 [gfs2]
  [<ffffffff8100969d>] ? __switch_to+0x7d/0x340
  [<ffffffffa04ec4bd>] inode_go_sync+0x7d/0x160 [gfs2]
  [<ffffffffa04eafd6>] do_xmote+0x156/0x280 [gfs2]
  [<ffffffffa04eb1f1>] run_queue+0xf1/0x1d0 [gfs2]
  [<ffffffffa04eb9fa>] glock_work_func+0x7a/0x1d0 [gfs2]
  [<ffffffffa04eb980>] ? glock_work_func+0x0/0x1d0 [gfs2]
  [<ffffffff8109a780>] worker_thread+0x170/0x2a0
  [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40
  [<ffffffff8109a610>] ? worker_thread+0x0/0x2a0
  [<ffffffff810a0fce>] kthread+0x9e/0xc0
  [<ffffffff8100c28a>] child_rip+0xa/0x20
  [<ffffffff810a0f30>] ? kthread+0x0/0xc0
  [<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task gfs2_logd:7162 blocked for more than 120 seconds.
       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
gfs2_logd     D 0000000000000003     0  7162      2 0x00000080
  ffff880600d13d00 0000000000000046 0000000000000000 0000000000000000
  0000000000000000 0000000000000000 000000a727fdf37d 0000000000000000
  0000000000000000 0000000100066027 ffff88062257a5f8 ffff880600d13fd8
Call Trace:
  [<ffffffff8153b665>] rwsem_down_failed_common+0x95/0x1d0
  [<ffffffff8153b7c3>] rwsem_down_write_failed+0x23/0x30
  [<ffffffff8129d6f3>] call_rwsem_down_write_failed+0x13/0x20
  [<ffffffff8153acc2>] ? down_write+0x32/0x40
  [<ffffffffa04ef51f>] gfs2_log_flush+0x2f/0x600 [gfs2]
  [<ffffffffa04ee57f>] ? gfs2_ail1_empty+0x2f/0x1b0 [gfs2]
  [<ffffffffa04efbc9>] gfs2_logd+0xd9/0x140 [gfs2]
  [<ffffffffa04efaf0>] ? gfs2_logd+0x0/0x140 [gfs2]
  [<ffffffff810a0fce>] kthread+0x9e/0xc0
  [<ffffffff8100c28a>] child_rip+0xa/0x20
  [<ffffffff810a0f30>] ? kthread+0x0/0xc0
  [<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task gfs2_quotad:7163 blocked for more than 120 seconds.
       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
gfs2_quotad   D 0000000000000001     0  7163      2 0x00000080
  ffff880600d17c20 0000000000000046 0000000000000000 0000000400000018
  aa838d160000001c ffffffffa0508110 000000a03da58698 ffffffffa05080c0
  ffff880600000005 000000010005eb43 ffff880624105068 ffff880600d17fd8
Call Trace:
  [<ffffffffa0508110>] ? gdlm_ast+0x0/0x210 [gfs2]
  [<ffffffffa05080c0>] ? gdlm_bast+0x0/0x50 [gfs2]
  [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
  [<ffffffffa04e710e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
  [<ffffffff815397ff>] __wait_on_bit+0x5f/0x90
  [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
  [<ffffffff815398a8>] out_of_line_wait_on_bit+0x78/0x90
  [<ffffffff810a14e0>] ? wake_bit_function+0x0/0x50
  [<ffffffffa04e80e5>] gfs2_glock_wait+0x45/0x90 [gfs2]
  [<ffffffffa04eb619>] gfs2_glock_nq+0x2c9/0x410 [gfs2]
  [<ffffffff8108a6cb>] ? try_to_del_timer_sync+0x7b/0xe0
  [<ffffffffa0504549>] gfs2_statfs_sync+0x59/0x1c0 [gfs2]
  [<ffffffff815393ca>] ? schedule_timeout+0x19a/0x2e0
  [<ffffffffa0504541>] ? gfs2_statfs_sync+0x51/0x1c0 [gfs2]
  [<ffffffffa04fbce7>] quotad_check_timeo+0x57/0xb0 [gfs2]
  [<ffffffffa04fbf74>] gfs2_quotad+0x234/0x2b0 [gfs2]
  [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40
  [<ffffffffa04fbd40>] ? gfs2_quotad+0x0/0x2b0 [gfs2]
  [<ffffffff810a0fce>] kthread+0x9e/0xc0
  [<ffffffff8100c28a>] child_rip+0xa/0x20
  [<ffffffff810a0f30>] ? kthread+0x0/0xc0
  [<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task flush-253:18:9820 blocked for more than 120 seconds.
       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-253:18  D 0000000000000000     0  9820      2 0x00000080
  ffff8805f9bc3490 0000000000000046 ffff8805f9bc3458 ffff8805f9bc3454
  ffff880620768c00 ffff88062fc23080 0000006fc80f5a1e ffff8800282959c0
  00000000000002aa 000000010002be11 ffff8806241d65f8 ffff8805f9bc3fd8
Call Trace:
  [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd]
  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
  [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40
  [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd]
  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
  [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod]
  [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd]
  [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60
  [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0
  [<ffffffff81276544>] ? blk_queue_bio+0x494/0x610
  [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod]
  [<ffffffff81274910>] generic_make_request+0x240/0x5a0
  [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20
  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
  [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20
  [<ffffffff81274ce0>] submit_bio+0x70/0x120
  [<ffffffff811c78ed>] submit_bh+0x11d/0x1f0
  [<ffffffff811ca0a8>] __block_write_full_page+0x1c8/0x330
  [<ffffffff811c9070>] ? end_buffer_async_write+0x0/0x190
  [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20
  [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20
  [<ffffffff811ca2f0>] block_write_full_page_endio+0xe0/0x120
  [<ffffffff81126ff0>] ? find_get_pages_tag+0x40/0x130
  [<ffffffff811ca345>] block_write_full_page+0x15/0x20
  [<ffffffff811cf108>] blkdev_writepage+0x18/0x20
  [<ffffffff8113b387>] __writepage+0x17/0x40
  [<ffffffff8113c64d>] write_cache_pages+0x1fd/0x4c0
  [<ffffffff8113b370>] ? __writepage+0x0/0x40
  [<ffffffff8113c934>] generic_writepages+0x24/0x30
  [<ffffffff8113c961>] do_writepages+0x21/0x40
  [<ffffffff811beb3d>] writeback_single_inode+0xdd/0x290
  [<ffffffff811bef3d>] writeback_sb_inodes+0xbd/0x170
  [<ffffffff811bf09b>] writeback_inodes_wb+0xab/0x1b0
  [<ffffffff811bf493>] wb_writeback+0x2f3/0x410
  [<ffffffff8108a752>] ? del_timer_sync+0x22/0x30
  [<ffffffff811bf755>] wb_do_writeback+0x1a5/0x240
  [<ffffffff811bf853>] bdi_writeback_task+0x63/0x1b0
  [<ffffffff810a12e7>] ? bit_waitqueue+0x17/0xd0
  [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100
  [<ffffffff8114b7c6>] bdi_start_fn+0x86/0x100
  [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100
  [<ffffffff810a0fce>] kthread+0x9e/0xc0
  [<ffffffff8100c28a>] child_rip+0xa/0x20
  [<ffffffff810a0f30>] ? kthread+0x0/0xc0
  [<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task clusterfs.sh:11652 blocked for more than 120 seconds.
       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
clusterfs.sh  D 0000000000000001     0 11652  11588 0x00000080
  ffff8804f9363c58 0000000000000082 ffff88061f81a6c0 0000000600000000
  0000000000000000 ffff880400000000 0000000000000001 ffff8804f9363db8
  ffff8804f9363ce8 ffff8804f9363cd8 ffff880624ead068 ffff8804f9363fd8
Call Trace:
  [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
  [<ffffffffa04e710e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
  [<ffffffff815397ff>] __wait_on_bit+0x5f/0x90
  [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
  [<ffffffff815398a8>] out_of_line_wait_on_bit+0x78/0x90
  [<ffffffff810a14e0>] ? wake_bit_function+0x0/0x50
  [<ffffffffa04e80e5>] gfs2_glock_wait+0x45/0x90 [gfs2]
  [<ffffffffa04eb619>] gfs2_glock_nq+0x2c9/0x410 [gfs2]
  [<ffffffffa04f9f97>] gfs2_getattr+0xd7/0x1d0 [gfs2]
  [<ffffffffa04f9f8f>] ? gfs2_getattr+0xcf/0x1d0 [gfs2]
  [<ffffffff81197691>] vfs_getattr+0x51/0x80
  [<ffffffff811974d4>] ? cp_new_stat+0xe4/0x100
  [<ffffffff81197724>] vfs_fstatat+0x64/0xa0
  [<ffffffff811977ce>] vfs_lstat+0x1e/0x20
  [<ffffffff811977f4>] sys_newlstat+0x24/0x50
  [<ffffffff810e8a57>] ? audit_syscall_entry+0x1d7/0x200
  [<ffffffff810e884e>] ? __audit_syscall_exit+0x25e/0x290
  [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task nrpe:11727 blocked for more than 120 seconds.
       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nrpe          D 0000000000000000     0 11727  11696 0x00000080
  ffff8804f91d7988 0000000000000082 ffffea0011683018 0000000114570cc0
  ffff880626a297c0 00000002000022c0 000000000000004b 00000040ffffffff
  0000000000000000 ffff880000049b08 ffff880624a9b068 ffff8804f91d7fd8
Call Trace:
  [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
  [<ffffffffa04e710e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]
  [<ffffffff815397ff>] __wait_on_bit+0x5f/0x90
  [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
  [<ffffffff815398a8>] out_of_line_wait_on_bit+0x78/0x90
  [<ffffffff810a14e0>] ? wake_bit_function+0x0/0x50
  [<ffffffffa04e80e5>] gfs2_glock_wait+0x45/0x90 [gfs2]
  [<ffffffffa04eb619>] gfs2_glock_nq+0x2c9/0x410 [gfs2]
  [<ffffffff811ad4a2>] ? dput+0x32/0x160
  [<ffffffffa04f922f>] gfs2_permission+0x17f/0x210 [gfs2]
  [<ffffffffa04f9226>] ? gfs2_permission+0x176/0x210 [gfs2]
  [<ffffffff811a1da3>] __link_path_walk+0xb3/0x1000
  [<ffffffff811a21eb>] __link_path_walk+0x4fb/0x1000
  [<ffffffff811a2faa>] path_walk+0x6a/0xe0
  [<ffffffff811a31bb>] filename_lookup+0x6b/0xc0
  [<ffffffff811a42e7>] user_path_at+0x57/0xa0
  [<ffffffff8119fd1b>] ? putname+0x2b/0x40
  [<ffffffff811a42f2>] ? user_path_at+0x62/0xa0
  [<ffffffff811974d4>] ? cp_new_stat+0xe4/0x100
  [<ffffffff81197710>] vfs_fstatat+0x50/0xa0
  [<ffffffff8119788b>] vfs_stat+0x1b/0x20
  [<ffffffff811978b4>] sys_newstat+0x24/0x50
  [<ffffffff810e8a57>] ? audit_syscall_entry+0x1d7/0x200
  [<ffffffff810e884e>] ? __audit_syscall_exit+0x25e/0x290
  [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
INFO: task glock_workqueue:7144 blocked for more than 120 seconds.
       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glock_workque D 0000000000000001     0  7144      2 0x00000080
  ffff880602d13740 0000000000000046 0000000000000000 0000000000000000
  0000000000000000 ffff880622eb3080 0000009d020d96ca 0000000000000246
  ffff880602d13720 000000010005b5aa ffff880622cbbad8 ffff880602d13fd8
Call Trace:
  [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd]
  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
  [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40
  [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd]
  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
  [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0
  [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod]
  [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd]
  [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60
  [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0
  [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod]
  [<ffffffff81274910>] generic_make_request+0x240/0x5a0
  [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20
  [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140
  [<ffffffff81274ce0>] submit_bio+0x70/0x120
  [<ffffffff811c78ed>] submit_bh+0x11d/0x1f0
  [<ffffffffa04f0ddf>] buf_lo_before_commit+0x10f/0x1c0 [gfs2]
  [<ffffffffa04ef6c8>] gfs2_log_flush+0x1d8/0x600 [gfs2]
  [<ffffffff8100969d>] ? __switch_to+0x7d/0x340
  [<ffffffffa04ec4bd>] inode_go_sync+0x7d/0x160 [gfs2]
  [<ffffffffa04eafd6>] do_xmote+0x156/0x280 [gfs2]
  [<ffffffffa04eb1f1>] run_queue+0xf1/0x1d0 [gfs2]
  [<ffffffffa04eb9fa>] glock_work_func+0x7a/0x1d0 [gfs2]
  [<ffffffffa04eb980>] ? glock_work_func+0x0/0x1d0 [gfs2]
  [<ffffffff8109a780>] worker_thread+0x170/0x2a0
  [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40
  [<ffffffff8109a610>] ? worker_thread+0x0/0x2a0
  [<ffffffff810a0fce>] kthread+0x9e/0xc0
  [<ffffffff8100c28a>] child_rip+0xa/0x20
  [<ffffffff810a0f30>] ? kthread+0x0/0xc0
  [<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task gfs2_logd:7162 blocked for more than 120 seconds.
       Not tainted 2.6.32-573.7.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
gfs2_logd     D 0000000000000003     0  7162      2 0x00000080
  ffff880600d13d00 0000000000000046 0000000000000000 0000000000000000
  0000000000000000 0000000000000000 000000a727fdf37d 0000000000000000
  0000000000000000 0000000100066027 ffff88062257a5f8 ffff880600d13fd8
Call Trace:
  [<ffffffff8153b665>] rwsem_down_failed_common+0x95/0x1d0
  [<ffffffff8153b7c3>] rwsem_down_write_failed+0x23/0x30
  [<ffffffff8129d6f3>] call_rwsem_down_write_failed+0x13/0x20
  [<ffffffff8153acc2>] ? down_write+0x32/0x40
  [<ffffffffa04ef51f>] gfs2_log_flush+0x2f/0x600 [gfs2]
  [<ffffffffa04ee57f>] ? gfs2_ail1_empty+0x2f/0x1b0 [gfs2]
  [<ffffffffa04efbc9>] gfs2_logd+0xd9/0x140 [gfs2]
  [<ffffffffa04efaf0>] ? gfs2_logd+0x0/0x140 [gfs2]
  [<ffffffff810a0fce>] kthread+0x9e/0xc0
  [<ffffffff8100c28a>] child_rip+0xa/0x20
  [<ffffffff810a0f30>] ? kthread+0x0/0xc0
  [<ffffffff8100c280>] ? child_rip+0x0/0x20

any hint on fixing that?