Limit on number of files in a directory ?

Limit on number of files in a directory ?

Post by Bruce G Wils » Tue, 04 Mar 1997 04:00:00



Is there a limit in Unix on the number of files that a directory can hold ?
I'm working with a server based application that will save a large number
of files (some large, some small) on behalf of users of the system.  Each
user will have a subdirectory under a common directory; there will eventually
be a few thousand users.  Will I begin to run into trouble if I create a
few thousand directories under one subdirectory ?  How about if I have
a few tens of thousands of files in an archive directory ?

This is on Solaris 2.5 .

TIA,
        - Bruce

--
------------------------------------------------------------------------
Bruce Wilson, Contract Software Developer
Arlington, Massachusetts, USA
"If it ain't baroque, don't fix it."

 
 
 

Limit on number of files in a directory ?

Post by Bryan O'Sulliva » Tue, 04 Mar 1997 04:00:00


b> Will I begin to run into trouble if I create a few thousand
b> directories under one subdirectory ?

Yes.  Most Unix filesystems store directory entries in an array or
list that may or may not be sorted.  This means that searching through
the directory takes time proportional to the number of the files in
the directory.  Once you get above a small few thousand entries in a
single directory, things start to get very slow.

Solaris 2.5, at least, imposes no fixed limit on the number of entries
you can have in a directory (I have had to handle directories with
upwards of 125,000 files in them after episodes of sendmail going
berserk).  This doesn't mean you should do that, though.

        <b

--
Let us pray:
What a Great System.
Please Do Not Crash.


 
 
 

Limit on number of files in a directory ?

Post by Casper H.S. Dik - Network Security Engine » Tue, 04 Mar 1997 04:00:00



>Solaris 2.5, at least, imposes no fixed limit on the number of entries
>you can have in a directory (I have had to handle directories with
>upwards of 125,000 files in them after episodes of sendmail going
>berserk).  This doesn't mean you should do that, though.

There is, however, a limit of the number of subdirectories and that's
about 64K (the MAXLINK value comes into play: each subdirectory is linked
to the parent and the parent's link count can not be highe rthan 32K
or 64K or some such limit)

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

 
 
 

Limit on number of files in a directory ?

Post by Ken Pizzin » Wed, 05 Mar 1997 04:00:00




Quote:>Is there a limit in Unix on the number of files that a directory can hold ?
>I'm working with a server based application that will save a large number
>of files (some large, some small) on behalf of users of the system.  Each
>user will have a subdirectory under a common directory; there will eventually
>be a few thousand users.  Will I begin to run into trouble if I create a
>few thousand directories under one subdirectory ?  How about if I have
>a few tens of thousands of files in an archive directory ?

I remember back in the dark ages (v7 and/or SysIII) we had
to deal with a limit of a maximum of 1000 links for an inode.
Since each subdirectory has a link to its parent, this meant a
limit of 998 subdirectories.  I haven't had a need to keep track
of what the modern limits on number of links is, but this is one
basic limitation you might run into.

As far as number of files in a directory go, you can go hog wild.
But keep in mind that many (most?) Unix implementations do a
linear search through the directory each time that a name lookup
in that directory happens.  This can be a rather painful thing
to experience with tens of thousands of files.

                --Ken Pizzini

 
 
 

Limit on number of files in a directory ?

Post by Andrew Giert » Wed, 05 Mar 1997 04:00:00


 Bruce> Is there a limit in Unix on the number of files that a
 Bruce> directory can hold ?

Not much of a theoretical limit, but there are practical limits.

(There *is* a theoretical limit on the number of *subdirectories* that
any particular directory can hold: LINK_MAX-2.)

 Bruce> I'm working with a server based application that will save a
 Bruce> large number of files (some large, some small) on behalf of
 Bruce> users of the system.  Each user will have a subdirectory under
 Bruce> a common directory; there will eventually be a few thousand
 Bruce> users.  Will I begin to run into trouble if I create a few
 Bruce> thousand directories under one subdirectory ?

Some systems have LINK_MAX set arbitrarily at around 1000; more have it
fixed at 32767. I don't know what Solaris has for this.

But performance considerations usually kick in well before that. Most
filesystems still do linear searches of directories, so as the number of
files goes up, the efficiency goes down. A good rule of thumb is that
the size of the directory file itself should be kept to a few disk blocks.

A usual trick is to organize directories like this:

/users/a/andrew

or

/users/a/n/andrew

or

/users/a/n/d/andrew

 Bruce> How about if I have a few tens of thousands of files in an
 Bruce> archive directory ?

Same problem. It'll work, but be prepared for anything accessing those
files to take a performance hit.

I've had cases where an out-of-control print program has deposited
thousands of files in a print queue. Cleaning them up took so long that
I had to submit the rm command to 'batch'.

 Bruce> This is on Solaris 2.5 .

Never used it, myself.

--
Andrew.

 
 
 

Limit on number of files in a directory ?

Post by Patrick Leu » Wed, 05 Mar 1997 04:00:00



: filesystems still do linear searches of directories, so as the number of
: files goes up, the efficiency goes down. A good rule of thumb is that
: the size of the directory file itself should be kept to a few disk blocks.

why would you want to put lots of files in the same directory?
is there really a good reason to?
if it becomes too difficult for you to find a file, it probably is a good
time to be creating sub directories

: A usual trick is to organize directories like this:
: /users/a/andrew
: or
: /users/a/n/andrew
: or
: /users/a/n/d/andrew

good point!!!!
remember that UN*X does caching on files specified in env var $PATH
it's best to keep your filenames short.
guess why all the main directories are only a few chars long? ;-))

read Optimizing UNIX for Performance by Amir Majidimehr for more details

Patrick

 
 
 

Limit on number of files in a directory ?

Post by David Binet » Fri, 07 Mar 1997 04:00:00



Quote:>Is there a limit in Unix on the number of files that a directory can hold ?
>I'm working with a server based application that will save a large number
>of files (some large, some small) on behalf of users of the system.  Each
>user will have a subdirectory under a common directory; there will eventually
>be a few thousand users.  Will I begin to run into trouble if I create a
>few thousand directories under one subdirectory ?  How about if I have
>a few tens of thousands of files in an archive directory ?

>This is on Solaris 2.5 .

Aside from the limits  (or lack thereof) imposed by the kernel
you shoul?d consider the implications of searching this dir. with
command line tools

for i in *

or

rm *

or

ls -al *.ext

or
find .

may all fail because of their own limitations on
command line length, (also the shell has limits)

condifer a case with thousands of numeric files in the form
16256343

with 10*thousands of files you might only be able
to process with them (via ksh for example) with
for i in 0 1 2 3 4 5 6 7 8 9
do
        whatever *$i
done

instead of the more obvious

whatever *

if you plan to use shell scripts to manipulate them...
you will find more limitations than if you are doing your
programming via C and direct directory acceses
 (with the portability problems)

as all of the previous posters suggested you might have an easier life
if you can find some way to distribute the files across several
subdirectories.

AND NOW MY RANT,  why must this be!!
why can an OS as advanced as UNIX not manage a truly useful
functionality such as maintaining a database on the filesystem
these things (my opinion) could be VERY useful.

i mean (sarcastically) its ONLY a file system
you wouldn#t want to do anything USEFUL with it.

I think the 'pick' OS or OS9 (i dont remember which) had this
functionality.

ok, thats my 2 cents worth,  hope its worth something to you.

-db-

--
MY DNA and genetic structure is copyright 1957-1997 David J. Binette
ALL RIGHTS RESERVED
unauthorised use, duplication, storage or retransmission is strictly prohibited.

http://www.sce.de/~dbin
*/ unmatched closing comment

 
 
 

Limit on number of files in a directory ?

Post by Ian Crozie » Sat, 08 Mar 1997 04:00:00



> Is there a limit in Unix on the number of files that a directory can hold ?
> I'm working with a server based application that will save a large number
> of files (some large, some small) on behalf of users of the system.  Each
> user will have a subdirectory under a common directory; there will eventually
> be a few thousand users.  Will I begin to run into trouble if I create a
> few thousand directories under one subdirectory ?  How about if I have
> a few tens of thousands of files in an archive directory ?

> This is on Solaris 2.5 .

> TIA,
>         - Bruce

> --
> ------------------------------------------------------------------------
> Bruce Wilson, Contract Software Developer
> Arlington, Massachusetts, USA
> "If it ain't baroque, don't fix it."

I don't know the answer to the limit per directory, but there's a limit
per filesystem based on the number of inodes created at newfs time.

#/usr/ucb/df -i

will show you how many inodes per file system are available.

x

 
 
 

Limit on number of files in a directory ?

Post by Ken Pizzin » Sun, 09 Mar 1997 04:00:00




>AND NOW MY RANT,  why must this be!!
>why can an OS as advanced as UNIX not manage a truly useful
>functionality such as maintaining a database on the filesystem
>these things (my opinion) could be VERY useful.

Yes it could be useful, but it usually better to use
a tool expressly designed for that purpose: a database
program with it associated data files.  Not that it
wouldn't be nice for such functionality to be integrated
well with the filesystem, but they do try and solve
somewhat different problems.

Quote:>i mean (sarcastically) its ONLY a file system
>you wouldn#t want to do anything USEFUL with it.

<sarcasm>
Damn, you're right.  I can't do a single damn useful thing
with this braindead excuse of a system for storing files.
</sarcasm>

Quote:>I think the 'pick' OS or OS9 (i dont remember which) had this
>functionality.

Pick.

                --Ken Pizzini