> The particular limit that worries me is this one from sysctl():
> #define KERN_MAXFILES 7 /* int: max open files */
> On my system, this has a value of 12288.
> According to the man page for the open() command, it can return the
> following error:
> [ENFILE] The system file table is full.
Those pertain to the global file table, not the per-process file descriptor
table.
Quote:> I infer that this means that the limit specified in KERN_MAXFILES has
> been met, and so if I use up the file table in my process, then any
> process running might get this error, and that many processes may not be
> ready for it since the operation they might be engaged in would
> ordinarily seem safe and innocuous. If it was gotten in a really bad
> place, it could even potentially cause a kernel panic, for all I know.
Any process that executes a system call which requires the allocation of a
file table entry will fail with ENFILE if all entries are in use. This
normally includes the open(2) and accept(2) system calls among others. It
does not include dup(2) or dup(2) since those calls only affect the process
file descriptor table.
Exhausting the file table shouldn't directly cause a kernel panic. However,
the system could fail because a kernel related process was unable to open a
file (e.g., to dynamically load a kernel module).
Quote:> The development platfrom is a MacOS X box. In a technical q&a on their
> developer site, they write:
> "By default, there is a global limit for the total number of open files:
> this limit is set to the value 12288. Also, each unique open file
> consumes a vnode in the Virtual-Filesystem Interface layer. The number
> of available vnodes is a function of installed RAM (on a 128M system,
> there will be approximately 2864 vnodes available). During normal system
> operation, the system will maintain between 300 and 400 open files. ...
> Advanced applications can override global open file limit and the global
> vnode quota at runtime using the sysctl routine."
> The note was none too helpful as to exactly what counted against the open
> file limit or with regard to any adverse consequences of raising it: why
> does it exist at all? What might happen if I increased it to 2^31?
If you are truly interested in the*details I would recommend a good
book on UNIX kernel internals (e.g., "The Magic Garden Explained"). The
short answer is that classic UNIX allocated the "file" structures it needed
at boot time as a fixed size array. This allowed the algorithms which
manipulate those structures to be simpler and faster. Something extremely
important given the capabilities of the systems available 20+ years ago and
still fairly important for a broad class of systems today.
Since the structures are allocated at boot time you can't arbitrarily
increase the limit. The system will either exhaust its physical memory or,
more likely, its kernel virtual address space. That concern does not apply
to all limits. Some, such as the maximum size of a shared memory segment
(traditionally controlled by the SHMMAX parameter) impose limits on what
processes can do but do not directly influence how much memory or KVA the
kernel consumes.
Quote:>> Even on systems where the "file table" is allowed to grow dynamically
>> (i.e., it isn't a table in the traditional sense) you might still need
>> to be concerned with the memory consumed and performance as the number
>> of entries increases.
> Memory is more manageable - I can readily detect my app's own footprint,
> the amount of logical and physical memory, free disk space, and how much
> page swapping my app is doing. Given this, I should be able to manage
> memory with some reasonable effort.
That is not the memory I was talking about. I was referring to the memory,
and kernel virtual address space, consumed by the structures associated
with such things as the network endpoints (i.e., sockets) owned by your
process.
Quote:> If performance becomes a problem, well, it becomes a problem. The user
> will need to upgrade his hardware in some appropriate way if he can't get
> the performance he wants with the hardware he has. Performance problems
> also happily tend to manifest themselves in an obvious way - the system
> gets slower as the load increases, and the user can peek in to figure out
> what's wrong.
One of my areas of expertise is tuning UNIX kernels. And after doing that
for twelve years, including being flown halfway around the world many times
to assist in performance problem determination, I can assure you that
performance problems rarely "manifest themselves in an obvious way." My
biggest headaches as a software support engineer come from customers whose
programming staff think machine resources are infinite. Typically because
they've learned to program on single-user PCs with amounts of memory and
CPU processing power that was unheard of even on a mainframe twenty years
ago.
With respect to having a large number of network endpoints open one concern
is how efficiently the kernel can map incoming packets to the appropriate
network endpoint so the correct process can read the data. You may find
that the amount of kernel time spent processing inbound TCP/IP packets
increases non-linearly with the number of open connections.
Quote:> On the other hand, exhausting the file table could cause system death
> without warning. Everything might be just peachy one second, dead as a
> doornail the next. What really complicates matters is that I know of no
> way to find out how many entries at any given time are in use, which
> makes managing that number difficult.
Actually, you should be able to determine how many file table entries are
in use at any given instant. On many UNIX implementations "sar -v" will
tell you. Obviously you would want to obtain that information without the
cost of spawning a sar(1) command. I've never worked with Mac OS X so I
have no idea how to go about it on that OS. Also, obviously even if the
file table has x free entries at some point in time doesn't mean that a
microsecond later those entries will not have been allocated for some
purpose.
Quote:>> Also, since there are other resources associated with a TCP/IP endpoint
>> (i.e., a "socket") focusing solely on the number of file table entries
>> consumed is unlikely to keep the system out of trouble.
> Umm, apart from obvious system resources like memory, cpu and swap space,
> and cpu bandwidth, like what?
Does your program use SysV IPC semaphores? Then you have to worry about
that resource. Does your program open any files? Then you have to worry
about the number of inodes the kernel can cache. Does your program use
advisory locks (e.g., flock() or fcntl(F_SETLK))? Then that might be a
finite resource you have to worry about. Do you use shmat() or mmap()?
There might be limits on the number of such objects. Etcetera.
Basically, anything your program does which involves interaction with the
outside world (including interacting with other processes) will require
kernel data structures to manage that interaction. Those structures consume
both physical memory as well as kernel virtual address space. So you need
to consider whether that is a significant factor in the resource
requirements of your application.
Also, if the structures are statically allocated or there is a hard limit
on the number of instances of a given structure that can be instantiated
then you have to worry about that limit. The good news is that modern UNIX
implemenations have moved away from using statically allocated arrays of
structures. So there shouldn't be too many you have to worry about.
by