Walt Mankowski on 2 Sep 2009 20:45:28 -0700

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] File System Directory Limits

On Wed, Sep 02, 2009 at 09:10:51PM -0400, JP Vossen wrote:
> > Date: Wed, 2 Sep 2009 16:17:37 -0400
> > From: Tim Allen <flipper@peregrinesalon.com>
> > 
> > We're trying to replicate what the SEC does (don't ask).
> > Unfortunately, this somehow involved creating upwards of 100,000
> > subdirectories in a root directory.
> Which "SEC?"
> I kind of doubt they are doing this on ZFS, so...  How *are they* doing 
> it.  If you know what they are doing do you know someone you can ask? 
> What "root" dir on what file system on what OS on what hardware, 
> starting when?
> I'd be interested to know the answer just to see what setup allowed them 
> to be that boneheaded long enough ago that everyone didn't immediately 
> realize what a bad idea it was.

Yes, I'd also be interested in why replicating what the SEC does
requires you putting 100,000 subdirectories in the root.  Is there
really no way to break things up logically into subdirectories?

> > UFS limits us at 32k, NetApp at 64k. If we were designing this, yes,
> > we'd want to come up with a better scheme or hash things out, but it
> > really is essential we replicate their set up for the closest
> > simulation.
> > 
> > Does anyone know of a file system that can handle upwards of 100,000
> > sub-directories in root?
> Shot in the dark?  Can you do the 100K in a sub-dir and chroot into 
> that?  (This may be a wildly wrong-headed idea, I've never looked into 
> those kinds of limits.)

I should also point out that it's not just a question of whether or
not the file system can support that many subdirs at all.  You also
need to consider performance.  Many file systems do a linear search
through the directory table to find subdirectories.  My experience
with ext3 is that performance slows down considerably long before you
reach whatever the hard limit is.

While I've never used it, I hear that reiserfs is supposed to be very
good at storing large numbers of subdirs efficiently.


Attachment: signature.asc
Description: Digital signature

Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug