[ClusterLabs] Antw: Re: Antw: [EXT] Re: Antw: Hanging OCFS2 Filesystem any one else?

Wed Jun 16 03:01:32 EDT 2021

>>> Gang He <ghe at suse.com> schrieb am 16.06.2021 um 07:48 in Nachricht
<7b2df128-4701-704c-869f-e08b0e37ee79 at suse.com>:
> Hi Ulrich,
> 
> On 2021/6/15 17:01, Ulrich Windl wrote:
>> Hi Guys!
>> 
>> Just to keep you informed on the issue:
>> I was informed that I'm not the only one seeing this problem, and there 
> seems
>> to be some "negative interference" between BtrFS reorganizing its extents
>> periodically and OCFS2 making reflink snapshots (a local cron job here) in
>> current SUSE SLES kernels. It seems that happens almost exactly at 0:00 o'
>> clock.
> We encountered the same hang in local environment, the problem looks 
> like caused by btrfs btrfs-balance job run, but I need to crash the 
> kernel for the further analysis.
> Hi Ulrich, do you know how to reproduce this hang stably? e.g. run 
> reflink snapshot script and trigger the btrfs-balance job

Hi!

My guess is (as reflink snapshot is very (or at least "quite") fast while
BtrFS balance takes some time) that you'll have to start BtrFS balance, and
while it's running add one or more reflink snapshots.
In out original case BtrFS balance was run at midnight, and the reflink
snapshots were created hourly. The midnight snapshots did not complete.
The really interesting thing is that the only common point of OCFS2 and BtrFS
is the mountpoint provided by BtrFS.
My guess is that the issue won't happen if the OCFS2 is not mounted on the
BtrFS being balanced; otherwise it would be a disastrous bug (I'm not saying
that the bug in its present form is less severe).

Regards,
Ulrich

> 
> 
> Thanks
> Gang
> 
>> 
>> The only thing that BtrFS and OCFS2 have in common here is that BtrFS 
> provides
>> the mount point for OCFS2.
>> 
>> Regards,
>> Ulrich
>> 
>>>>> Ulrich Windl schrieb am 02.06.2021 um 11:00 in Nachricht <60B748A4.E0C
:
>> 161 :
>> 60728>:
>>>>>> Gang He <GHe at suse.com> schrieb am 02.06.2021 um 08:34 in Nachricht
>>>
>>
<AM6PR04MB6488DE7D2DA906BAD73FA3A1CF3D9 at AM6PR04MB6488.eurprd04.prod.outlook.c

>> 
>>> om>
>>>
>>>> Hi Ulrich,
>>>>
>>>> The hang problem looks like a fix
>> (90bd070aae6c4fb5d302f9c4b9c88be60c8197ec
>>>> ocfs2: fix deadlock between setattr and dio_end_io_write), but it is not
>>> 100%
>>>> sure.
>>>> If possible, could you help to report a bug to SUSE, then we can work on
>>>> that further.
>>>
>>> Hi!
>>>
>>> Actually a service request for the issue is open at SUSE. However I don't
>>> know which L3 engineer is working on it.
>>> I have some "funny" effects, like these:
>>> On one node "ls" hangs, but can be interrupted with ^C; on another node
"ls"
>> 
>>> also hangs, but cannot be stopped with ^C or ^Z
>>> (Most processes cannot even be killed with "kill -9")
>>> "ls" on the directory also hangs, just as an "rm" for a non-existent file
>>>
>>> What I really wonder is what triggered the effect, and more importantly 
how
>> 
>>> to recover from it.
>>> Initially I had suspected a rather full (95%) flesystem, but that means
>>> there are still 24GB available.
>>> The other suspect was concurrent creation of reflink snapshots while the
>>> file being snapshot did change (e.g. allocate a hole in a sparse file)
>>>
>>> Regards,
>>> Ulrich
>>>
>>>>
>>>> Thanks
>>>> Gang
>>>>
>>>> ________________________________________
>>>> From: Users <users‑bounces at clusterlabs.org> on behalf of Ulrich Windl
>>>> <Ulrich.Windl at rz.uni‑regensburg.de>
>>>> Sent: Tuesday, June 1, 2021 15:14
>>>> To: users at clusterlabs.org 
>>>> Subject: [ClusterLabs] Antw: Hanging OCFS2 Filesystem any one else?
>>>>
>>>>>>> Ulrich Windl schrieb am 31.05.2021 um 12:11 in Nachricht
<60B4B65A.A8F
>> : 161
>>>
>>>> :
>>>> 60728>:
>>>>> Hi!
>>>>>
>>>>> We have an OCFS2 filesystem shared between three cluster nodes (SLES 15
>> SP2,
>>>>> Kernel 5.3.18‑24.64‑default). The filesystem is filled up to about 95%,
>> and
>>>>> we have an odd effect:
>>>>> A stat() systemcall to some of the files hangs indefinitely (state
"D").
>>>>> ("ls ‑l" and "rm" also hang, but I suspect those are calling state()
>>>>> internally, too).
>>>>> My only suspect is that the effect might be related to the 95% being
>> used.
>>>>> The other suspect is that concurrent reflink calls may trigger the
>> effect.
>>>>>
>>>>> Did anyone else experience something similar?
>>>>
>>>> Hi!
>>>>
>>>> I have some details:
>>>> It seems there is a reader/writer deadlock trying to allocate additional
>>>> blocks for a file.
>>>> The stacktrace looks like this:
>>>> Jun 01 07:56:31 h16 kernel:  rwsem_down_write_slowpath+0x251/0x620
>>>> Jun 01 07:56:31 h16 kernel:  ? __ocfs2_change_file_space+0xb3/0x620
>> [ocfs2]
>>>> Jun 01 07:56:31 h16 kernel:  __ocfs2_change_file_space+0xb3/0x620
[ocfs2]
>>>> Jun 01 07:56:31 h16 kernel:  ocfs2_fallocate+0x82/0xa0 [ocfs2]
>>>> Jun 01 07:56:31 h16 kernel:  vfs_fallocate+0x13f/0x2a0
>>>> Jun 01 07:56:31 h16 kernel:  ksys_fallocate+0x3c/0x70
>>>> Jun 01 07:56:31 h16 kernel:  __x64_sys_fallocate+0x1a/0x20
>>>> Jun 01 07:56:31 h16 kernel:  do_syscall_64+0x5b/0x1e0
>>>>
>>>> That is the only writer (on that host), bit there are multiple readers
>> like
>>>> this:
>>>> Jun 01 07:56:31 h16 kernel:  rwsem_down_read_slowpath+0x172/0x300
>>>> Jun 01 07:56:31 h16 kernel:  ? dput+0x2c/0x2f0
>>>> Jun 01 07:56:31 h16 kernel:  ? lookup_slow+0x27/0x50
>>>> Jun 01 07:56:31 h16 kernel:  lookup_slow+0x27/0x50
>>>> Jun 01 07:56:31 h16 kernel:  walk_component+0x1c4/0x300
>>>> Jun 01 07:56:31 h16 kernel:  ? path_init+0x192/0x320
>>>> Jun 01 07:56:31 h16 kernel:  path_lookupat+0x6e/0x210
>>>> Jun 01 07:56:31 h16 kernel:  ? __put_lkb+0x45/0xd0 [dlm]
>>>> Jun 01 07:56:31 h16 kernel:  filename_lookup+0xb6/0x190
>>>> Jun 01 07:56:31 h16 kernel:  ? kmem_cache_alloc+0x3d/0x250
>>>> Jun 01 07:56:31 h16 kernel:  ? getname_flags+0x66/0x1d0
>>>> Jun 01 07:56:31 h16 kernel:  ? vfs_statx+0x73/0xe0
>>>> Jun 01 07:56:31 h16 kernel:  vfs_statx+0x73/0xe0
>>>> Jun 01 07:56:31 h16 kernel:  ? fsnotify_grab_connector+0x46/0x80
>>>> Jun 01 07:56:31 h16 kernel:  __do_sys_newstat+0x39/0x70
>>>> Jun 01 07:56:31 h16 kernel:  ? do_unlinkat+0x92/0x320
>>>> Jun 01 07:56:31 h16 kernel:  do_syscall_64+0x5b/0x1e0
>>>>
>>>> So that will match the hanging stat() quite nicely!
>>>>
>>>> However the PID displayed as holding the writer does not exist in the
>> system
>>>
>>>> (on that node).
>>>>
>>>> Regards,
>>>> Ulrich
>>>>
>>>>
>>>>>
>>>>> Regards,
>>>>> Ulrich
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Manage your subscription:
>>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>
>>>> ClusterLabs home: https://www.clusterlabs.org/ 
>>>>
>>>>
>>>> _______________________________________________
>>>> Manage your subscription:
>>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>
>>>> ClusterLabs home: https://www.clusterlabs.org/ 
>>>
>>>
>>>
>>>
>> 
>> 
>> 
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> ClusterLabs home: https://www.clusterlabs.org/ 
>> 
> 
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> ClusterLabs home: https://www.clusterlabs.org/ 