[ClusterLabs] Antw: [EXT] Re: Antw: Hanging OCFS2 Filesystem any one else?

[Ulrich Windl]

Hi Guys!

Just to keep you informed on the issue:
I was informed that I'm not the only one seeing this problem, and there seems
to be some "negative interference" between BtrFS reorganizing its extents
periodically and OCFS2 making reflink snapshots (a local cron job here) in
current SUSE SLES kernels. It seems that happens almost exactly at 0:00 o'
clock.

The only thing that BtrFS and OCFS2 have in common here is that BtrFS provides
the mount point for OCFS2.

Regards,
Ulrich

>>> Ulrich Windl schrieb am 02.06.2021 um 11:00 in Nachricht <60B748A4.E0C :
161 :
60728>:
>>>> Gang He <GHe at suse.com> schrieb am 02.06.2021 um 08:34 in Nachricht
>
<AM6PR04MB6488DE7D2DA906BAD73FA3A1CF3D9 at AM6PR04MB6488.eurprd04.prod.outlook.c

> om>
> 
> > Hi Ulrich,
> > 
> > The hang problem looks like a fix
(90bd070aae6c4fb5d302f9c4b9c88be60c8197ec 
> > ocfs2: fix deadlock between setattr and dio_end_io_write), but it is not 
> 100% 
> > sure.
> > If possible, could you help to report a bug to SUSE, then we can work on 
> > that further.
> 
> Hi!
> 
> Actually a service request for the issue is open at SUSE. However I don't 
> know which L3 engineer is working on it.
> I have some "funny" effects, like these:
> On one node "ls" hangs, but can be interrupted with ^C; on another node "ls"

> also hangs, but cannot be stopped with ^C or ^Z
> (Most processes cannot even be killed with "kill -9")
> "ls" on the directory also hangs, just as an "rm" for a non-existent file
> 
> What I really wonder is what triggered the effect, and more importantly  how

> to recover from it.
> Initially I had suspected a rather full (95%) flesystem, but that means 
> there are still 24GB available.
> The other suspect was concurrent creation of reflink snapshots while the 
> file being snapshot did change (e.g. allocate a hole in a sparse file)
> 
> Regards,
> Ulrich
> 
> > 
> > Thanks
> > Gang
> > 
> > ________________________________________
> > From: Users <users‑bounces at clusterlabs.org> on behalf of Ulrich Windl 
> > <Ulrich.Windl at rz.uni‑regensburg.de>
> > Sent: Tuesday, June 1, 2021 15:14
> > To: users at clusterlabs.org 
> > Subject: [ClusterLabs] Antw: Hanging OCFS2 Filesystem any one else?
> > 
> >>>> Ulrich Windl schrieb am 31.05.2021 um 12:11 in Nachricht <60B4B65A.A8F
: 161 
> 
> > :
> > 60728>:
> >> Hi!
> >>
> >> We have an OCFS2 filesystem shared between three cluster nodes (SLES 15
SP2,
> >> Kernel 5.3.18‑24.64‑default). The filesystem is filled up to about 95%,
and
> >> we have an odd effect:
> >> A stat() systemcall to some of the files hangs indefinitely (state "D").
> >> ("ls ‑l" and "rm" also hang, but I suspect those are calling state()
> >> internally, too).
> >> My only suspect is that the effect might be related to the 95% being
used.
> >> The other suspect is that concurrent reflink calls may trigger the
effect.
> >>
> >> Did anyone else experience something similar?
> > 
> > Hi!
> > 
> > I have some details:
> > It seems there is a reader/writer deadlock trying to allocate additional 
> > blocks for a file.
> > The stacktrace looks like this:
> > Jun 01 07:56:31 h16 kernel:  rwsem_down_write_slowpath+0x251/0x620
> > Jun 01 07:56:31 h16 kernel:  ? __ocfs2_change_file_space+0xb3/0x620
[ocfs2]
> > Jun 01 07:56:31 h16 kernel:  __ocfs2_change_file_space+0xb3/0x620 [ocfs2]
> > Jun 01 07:56:31 h16 kernel:  ocfs2_fallocate+0x82/0xa0 [ocfs2]
> > Jun 01 07:56:31 h16 kernel:  vfs_fallocate+0x13f/0x2a0
> > Jun 01 07:56:31 h16 kernel:  ksys_fallocate+0x3c/0x70
> > Jun 01 07:56:31 h16 kernel:  __x64_sys_fallocate+0x1a/0x20
> > Jun 01 07:56:31 h16 kernel:  do_syscall_64+0x5b/0x1e0
> > 
> > That is the only writer (on that host), bit there are multiple readers
like 
> > this:
> > Jun 01 07:56:31 h16 kernel:  rwsem_down_read_slowpath+0x172/0x300
> > Jun 01 07:56:31 h16 kernel:  ? dput+0x2c/0x2f0
> > Jun 01 07:56:31 h16 kernel:  ? lookup_slow+0x27/0x50
> > Jun 01 07:56:31 h16 kernel:  lookup_slow+0x27/0x50
> > Jun 01 07:56:31 h16 kernel:  walk_component+0x1c4/0x300
> > Jun 01 07:56:31 h16 kernel:  ? path_init+0x192/0x320
> > Jun 01 07:56:31 h16 kernel:  path_lookupat+0x6e/0x210
> > Jun 01 07:56:31 h16 kernel:  ? __put_lkb+0x45/0xd0 [dlm]
> > Jun 01 07:56:31 h16 kernel:  filename_lookup+0xb6/0x190
> > Jun 01 07:56:31 h16 kernel:  ? kmem_cache_alloc+0x3d/0x250
> > Jun 01 07:56:31 h16 kernel:  ? getname_flags+0x66/0x1d0
> > Jun 01 07:56:31 h16 kernel:  ? vfs_statx+0x73/0xe0
> > Jun 01 07:56:31 h16 kernel:  vfs_statx+0x73/0xe0
> > Jun 01 07:56:31 h16 kernel:  ? fsnotify_grab_connector+0x46/0x80
> > Jun 01 07:56:31 h16 kernel:  __do_sys_newstat+0x39/0x70
> > Jun 01 07:56:31 h16 kernel:  ? do_unlinkat+0x92/0x320
> > Jun 01 07:56:31 h16 kernel:  do_syscall_64+0x5b/0x1e0
> > 
> > So that will match the hanging stat() quite nicely!
> > 
> > However the PID displayed as holding the writer does not exist in the
system 
> 
> > (on that node).
> > 
> > Regards,
> > Ulrich
> > 
> > 
> >>
> >> Regards,
> >> Ulrich
> >>
> >>
> >>
> >>
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users 
> > 
> > ClusterLabs home: https://www.clusterlabs.org/ 
> > 
> > 
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users 
> > 
> > ClusterLabs home: https://www.clusterlabs.org/ 
> 
> 
> 
>