Rich Freeman via plug on 13 Sep 2019 06:01:00 -0700


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] large_dir support on ext4


On Fri, Sep 13, 2019 at 8:42 AM Walt Mankowski via plug
<plug@lists.phillylinux.org> wrote:
>
> I used tune2fs to add large_dir support to the filesystem in hopes of
> getting around this problem. It returned very quickly, which makes me
> suspect it will only be applied to new directories, not existing
> ones. I'm asking here on the off chance anyone might happen to know
> for sure.
>

I don't know the answer to that question, but you might want to
consider also enabling dir_index.

I'm not sure if the feature has any downsides (I'm guessing it would
negatively impact file creation/linking/unlinking slightly), but it is
supposed to greatly speed name look-ups in large directories.

Now, if you don't actually do name lookups but do something like using
readdir to iterate through all the inodes in a directory then it won't
provide benefit.  Anything that actually uses a filename to open a
file directly would benefit, however, as otherwise the kernel
basically has to scan the entire directory to find a file which I'm
guessing is O(N).  I don't know enough about ext4 to know if the
metadata can get fragmented, but I'm guessing it can, and thus that
cost would apply to any seeks needed to read the entire directory
listing.

I suspect that it doesn't work retroactively, so you will want to
enable this before creating all your files.

Offtopic, but I know from reading about btrfs for my presentation that
it tries to index the directory metadata both for name access and
iterating over directories, and I think the latter is designed to
attempt to do it in on-disk order to reduce seeking without any
guarantees.

-- 
Rich
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug