4.8-STABLE "Too many open files"

4.8-STABLE "Too many open files"

Post by Nicholas Acc » Sat, 05 Jul 2003 00:47:57



Hello,

I have a box running 4.8-STABLE with mysqld and an in-house developed app.
The problem is as follows:-

My dev people noticed that both mysqld and our application run out of
file descriptors after a few hours, when checking with sysctl, I noticed
that the number of open files reaches the 32K limit and that none of these
files gets closed.

We traced it back to kqueue/libc_r, I checked the newsgroups and freebsd
bugs DB, but nothing like that appears or seems relevant.

We have an older box with 4.6.2-RELEASE-p7, where this problem does not
happen. Was there any modifications to libc between 4.6.2 and 4.8?

Right now what I did is write a script that checks KQUEUE (from lsof output)
and restarts mysqld when the number reaches 16000, but I would really like
to find a more "scientific" solution for this.

Any comments?

Thanks

 
 
 

4.8-STABLE "Too many open files"

Post by Lowell Gilber » Sat, 05 Jul 2003 22:41:27


comp.unix.bsd.freebsd is not widely propagated, so you won't
necessarily get much help here (in comp.unix.bsd.freebsd).
See http://www.freebsd.org/support.html for better ideas...
[followups redirected accordingly]


> I have a box running 4.8-STABLE with mysqld and an in-house developed app.
> The problem is as follows:-

> My dev people noticed that both mysqld and our application run out of
> file descriptors after a few hours, when checking with sysctl, I noticed
> that the number of open files reaches the 32K limit and that none of these
> files gets closed.

> We traced it back to kqueue/libc_r, I checked the newsgroups and freebsd
> bugs DB, but nothing like that appears or seems relevant.

> We have an older box with 4.6.2-RELEASE-p7, where this problem does not
> happen. Was there any modifications to libc between 4.6.2 and 4.8?

Many, of course.

Quote:> Right now what I did is write a script that checks KQUEUE (from lsof output)
> and restarts mysqld when the number reaches 16000, but I would really like
> to find a more "scientific" solution for this.

This is an application problem; if the daemon really opens tens of
thousands of file handles without closing them, there's not much the
OS can do other than contain the problem.  Run the daemon under an
account with a reasonable "openfiles" limit, and it will do more or
less what you're doing by hand.  You might want to restart it out of
init, or even just send it a signal periodically, but I really can't
tell you what your situation requires.

You should try straightening this out mysqld rather than trying to get
the OS to work around the application problem, though.

 
 
 

4.8-STABLE "Too many open files"

Post by Nicholas Acc » Sun, 06 Jul 2003 02:26:53


Lowell,
Thanks for the reply.

Of course I figured out that there were many mods between 4.6 and 4.8, but I
thought someone would have some insight regarding this, anyway...

Maybe the problem is in mysqld as you say, but since I noticed this behaviour
not only in mysqld, but also in our own app, it really got me thinking that
maybe it was the library/kernel/freebsd-something.

I will try to look at it from the mysqld side, and will report anything
that comes up.

Thanks :)

-nick


> comp.unix.bsd.freebsd is not widely propagated, so you won't
> necessarily get much help here (in comp.unix.bsd.freebsd).
> See http://www.freebsd.org/support.html for better ideas...
> [followups redirected accordingly]


[snip]
> > happen. Was there any modifications to libc between 4.6.2 and 4.8?

> Many, of course.

> > Right now what I did is write a script that checks KQUEUE (from lsof output)
> > and restarts mysqld when the number reaches 16000, but I would really like
> > to find a more "scientific" solution for this.

> This is an application problem; if the daemon really opens tens of
> thousands of file handles without closing them, there's not much the
> OS can do other than contain the problem.  Run the daemon under an
> account with a reasonable "openfiles" limit, and it will do more or
> less what you're doing by hand.  You might want to restart it out of
> init, or even just send it a signal periodically, but I really can't
> tell you what your situation requires.

> You should try straightening this out mysqld rather than trying to get
> the OS to work around the application problem, though.

 
 
 

4.8-STABLE "Too many open files"

Post by Nicholas Acc » Fri, 11 Jul 2003 22:33:26


OK, my dev ppl did something interesting, I'm not sure what this means,
so maybe someone out there could clarify,

we copied libc_r.so from 4.6.2-REL-p7 and using LD_LIBRARY_PATH forced mysqld
to use that version instead of the once that came with 4.8

this has been up and running for exactly 1 week now, everything is working
perfectly.

I'm not sure what to think.

tc
-nick


> Lowell,
> Thanks for the reply.

> Of course I figured out that there were many mods between 4.6 and 4.8, but I
> thought someone would have some insight regarding this, anyway...

> Maybe the problem is in mysqld as you say, but since I noticed this behaviour
> not only in mysqld, but also in our own app, it really got me thinking that
> maybe it was the library/kernel/freebsd-something.

> I will try to look at it from the mysqld side, and will report anything
> that comes up.

> Thanks :)

> -nick


> > comp.unix.bsd.freebsd is not widely propagated, so you won't
> > necessarily get much help here (in comp.unix.bsd.freebsd).
> > See http://www.freebsd.org/support.html for better ideas...
> > [followups redirected accordingly]


>  [snip]
> > > happen. Was there any modifications to libc between 4.6.2 and 4.8?

> > Many, of course.

> > > Right now what I did is write a script that checks KQUEUE (from lsof output)
> > > and restarts mysqld when the number reaches 16000, but I would really like
> > > to find a more "scientific" solution for this.

> > This is an application problem; if the daemon really opens tens of
> > thousands of file handles without closing them, there's not much the
> > OS can do other than contain the problem.  Run the daemon under an
> > account with a reasonable "openfiles" limit, and it will do more or
> > less what you're doing by hand.  You might want to restart it out of
> > init, or even just send it a signal periodically, but I really can't
> > tell you what your situation requires.

> > You should try straightening this out mysqld rather than trying to get
> > the OS to work around the application problem, though.

 
 
 

4.8-STABLE "Too many open files"

Post by Donn Mille » Fri, 11 Jul 2003 22:57:32



> OK, my dev ppl did something interesting, I'm not sure what this means,
> so maybe someone out there could clarify,

> we copied libc_r.so from 4.6.2-REL-p7 and using LD_LIBRARY_PATH forced mysqld
> to use that version instead of the once that came with 4.8

> this has been up and running for exactly 1 week now, everything is working
> perfectly.

You can monitor the number of open files (and also set limits on open
files) by using 'sysctl'.  For example, to see how many files are open:

sysctl kern.openfiles

To see what the limit is:

sysctl kern.maxfiles

You can also set the limit by modifying kern.maxfiles with sysctl.  See
man 8 sysctl.

-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----==  Over 80,000 Newsgroups - 16 Different Servers! =-----