|
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
|
Re: [PLUG] Is there a limit to the amount of files in a directory before it starts causing trouble?
|
On 2/9/06, Marc Zucchelli <marcz908@yahoo.com> wrote:
> I just finished a craigslist type website for one of
> my clients, its for college students to list their
> stuff, books, etc.
>
> When someone uploads a picture, the pictures ALL get
> saved in the same directory. It just occured to me,
> what if theres 10's of millions of images sitting in
> one directory, or even 10's of thousands.
>
> Can ext2 or ext3 handle something like that?
>
> Will this cause memory problems?
>
> The directory will never be read, like ls does. File
> paths are stored in the db, so pictures will be
> accessed and deleted directly.
If you are using a newish (2.6 based) distro, you are probably in good
shape out of the box. Except for Debian Sarge, because they also
support a 2.4 kernel.
Newer versions of the ext3 driver have a feature called dir_index that
makes this use pattern fairly effecient. This causes the directory
entries to be stored in a b-tree, so that lookups are O(log(n)).
Note that the filesystem on disk must also be marked as supporting the
feature. It can be enabled for an existing filesystem with tune2fs
(and a pass over the fs with 'e2fsck -D' to index existing
directories). I played around a bit and couldn't find a way to test if
the dir_index feature was enabled on a particluar filesystem.
Reiserfs, XFS, HFS (and probbably JFS) always store directories as b-trees.
Ext2 will always do a linear read on directories (bleh).
If I were in your shoes, I'd reformat as reiserfs (assuming the app
does not run on a shared server).
--
Will Dyson
___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
|
|