Weblogic - Too many open files - FAILURE

Weblogic - Too many open files - FAILURE

Post by Jude Farm » Wed, 20 Feb 2002 22:37:18



Hi everyone

Right does anyone know what would cause WebLogic 5.1 running on
Solaris 8 to do this across several different instances running on the
same server even though this has been upped in /etc/services -

* Set the file descriptors higher for weblogic
set rlim_fd_max=4096 /* hard limit */

and the startup scripts have the SP10 entries in them -

# Grab some file descriptors.
maxfiles=`ulimit -H -n`
if [ $? -eq 0 -a "$maxfiles" != 1024 ]; then
  if [ "$maxfiles" = "unlimited" ]; then
    maxfiles=1025
  fi
  if [ "$maxfiles" -lt 1024 ]; then
    ulimit -n $maxfiles
  else
    ulimit -n 1024
  fi
fi

I have to say that I'm pretty new to Weblogic, but I've pretty much
tried everything that's I've seen raised as solutions, but I'm still
seeing the error below occuring over and over again.

Any help is much apprciated as I'm stumped

Thanks in advance

Jude

ue Feb 19 11:22:21 GMT 2002:<E> <ListenThread> Listen failed, failure
count: '2147'
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:421)
        at java.net.ServerSocket.implAccept(ServerSocket.java:243)
        at java.net.ServerSocket.accept(ServerSocket.java:222)
        at weblogic.t3.srvr.ListenThread.run(ListenThread.java:286)

Tue Feb 19 11:22:22 GMT 2002:<E> <ListenThread> Listen failed, failure
count: '2148'
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:421)
        at java.net.ServerSocket.implAccept(ServerSocket.java:243)
        at java.net.ServerSocket.accept(ServerSocket.java:222)
        at weblogic.t3.srvr.ListenThread.run(ListenThread.java:286)

 
 
 

Weblogic - Too many open files - FAILURE

Post by Mike Delane » Thu, 21 Feb 2002 07:42:29



:  
:  Right does anyone know what would cause WebLogic 5.1 running on
:  Solaris 8 to do this across several different instances running on the
:  same server even though this has been upped in /etc/services -
:  
:  * Set the file descriptors higher for weblogic
:  set rlim_fd_max=4096 /* hard limit */
:  
:  and the startup scripts have the SP10 entries in them -
:  
:  # Grab some file descriptors.
:  maxfiles=`ulimit -H -n`
:  if [ $? -eq 0 -a "$maxfiles" != 1024 ]; then
:    if [ "$maxfiles" = "unlimited" ]; then
:      maxfiles=1025
:    fi
:    if [ "$maxfiles" -lt 1024 ]; then
:      ulimit -n $maxfiles
:    else
:      ulimit -n 1024
:    fi
:  fi

Umm, OK.  That seems a bit daft.  That little routine above is
limiting the maximum number of file descriptors to the *lower* of
1023 or the hard limit.  You'd think that routine should just
be setting the soft limit to the same value as the hard limit:

# Grab some file descriptors.
ulimit -n `ulimit -H -n`

You might want to ask WebLogic's support folks about that, as the only
reason I can think of for doing what you've shown above is to work around
some type of bug relating to larger numbers of file descriptors.

--

"...Microsoft follows standards.  In much the same manner that fish follow
migrating caribou." "Now I have this image in my mind of a fish embracing and
extending a caribou." -- Paul Tomblin and Christian Bauernfeind in the SDM

 
 
 

Weblogic - Too many open files - FAILURE

Post by Jude Farm » Thu, 21 Feb 2002 19:18:04




> :  
> :  Right does anyone know what would cause WebLogic 5.1 running on
> :  Solaris 8 to do this across several different instances running on the
> :  same server even though this has been upped in /etc/services -
> :  
> :  * Set the file descriptors higher for weblogic
> :  set rlim_fd_max=4096 /* hard limit */
> :  
> :  and the startup scripts have the SP10 entries in them -
> :  
> :  # Grab some file descriptors.
> :  maxfiles=`ulimit -H -n`
> :  if [ $? -eq 0 -a "$maxfiles" != 1024 ]; then
> :    if [ "$maxfiles" = "unlimited" ]; then
> :      maxfiles=1025
> :    fi
> :    if [ "$maxfiles" -lt 1024 ]; then
> :      ulimit -n $maxfiles
> :    else
> :      ulimit -n 1024
> :    fi
> :  fi

> Umm, OK.  That seems a bit daft.  That little routine above is
> limiting the maximum number of file descriptors to the *lower* of
> 1023 or the hard limit.  You'd think that routine should just
> be setting the soft limit to the same value as the hard limit:

> # Grab some file descriptors.
> ulimit -n `ulimit -H -n`

> You might want to ask WebLogic's support folks about that, as the only
> reason I can think of for doing what you've shown above is to work around
> some type of bug relating to larger numbers of file descriptors.

Thanks. Yeah I agree with you, it is a bit daft and I did actaully
notice that before so I've have in fact hacked it to set it to the max
of 4096. However it's made no difference. I've also written a script
to monitor the number of open files the application is using and the
majority of the time it's less that 100. I'm now starting to wonder if
it's not actually a problem with WebLogic, but in fact a problem with
how WebLogic is talking to the Oracle Database. It seem as though the
two applications may get themselves into a deadlock situation where
Oracle isn't allowing anymore connections yet WebLogic is still trying
to open connections and hence the used file handles starts to sky
rocket until it hits the limit and crashes. I think that the amount of
file handles allowed is a red herring.

I'd be interested to hear if this sounds like a plausible notion.

ps. I'd love to give the Weblogic support folks a call but all they'll
say is "go to SP10", which we can't do at the moment - not very
helpful. And also they're fix for this, as you pointed out, doesn't
even work.

Anyway thanks in advance again

Jude

 
 
 

Weblogic - Too many open files - FAILURE

Post by Jeremiah DeWitt Weine » Fri, 22 Feb 2002 04:00:57



> However it's made no difference. I've also written a script
> to monitor the number of open files the application is using and the
> majority of the time it's less that 100.

        How are you monitoring it?  You said you're on Solaris 8, so
you're in luck - plimit and pfiles are very useful for this sort of
thing.  It sounds like you may already be doing this, but check with
plimit to see what the process thinks its limits are - sometimes a
limit gets set somewhere other than where you're looking.

(snip theory on filehandles)

Quote:> I'd be interested to hear if this sounds like a plausible notion.

        I can't speak as to the exact nature of the problem, but I agree
that it's probably not something as simple as it sounds.  We had a
problem at work a while back that was similar; it was Dynamo, not Weblogic,
but it's also written in java, and the error message was similar, and
similarly misleading.

        <RANT>  You know, problems like this /could/ be reasonably easy
to figure out - you'd truss the process, or write a library interposer
or use ldd to figure out a dependency problem or something - and then
Sun had to go*IT ALL UP by creating a new language/environment for
which the old tools don't work.  Write once, run anywhere, my tuches.
Ever truss'ed a java process?  HelloWorld in C: 26 lines.  HelloWorld
in java: 1118 lines.  16 lines of mutex activity and 24 of semaphore for
freakin' "Hello World".  And people wonder why java's slow. </rant>

--
If mail to me bounces, try removing the "+STRING" part of the address.

 
 
 

Weblogic - Too many open files - FAILURE

Post by Mike Delane » Sat, 23 Feb 2002 11:02:57



:  
:  Thanks. Yeah I agree with you, it is a bit daft and I did actaully
:  notice that before so I've have in fact hacked it to set it to the max
:  of 4096. However it's made no difference. I've also written a script
:  to monitor the number of open files the application is using and the
:  majority of the time it's less that 100.

Is that including the periods where the exceptions are being generated,
or is it mainly* around 100 and working, then suddenly spiking up
and throwing exceptions?

:  I'm now starting to wonder if
:  it's not actually a problem with WebLogic, but in fact a problem with
:  how WebLogic is talking to the Oracle Database. It seem as though the
:  two applications may get themselves into a deadlock situation where
:  Oracle isn't allowing anymore connections yet WebLogic is still trying
:  to open connections and hence the used file handles starts to sky
:  rocket until it hits the limit and crashes. I think that the amount of
:  file handles allowed is a red herring.

I don't think so.  At least with 4.x, when WebLogic would get into a
situation where all avalible connections to the database were busy, it
would start complaining about being unable to increase the connection
pool size.

From the error message you included with your origional post:

Quote:> java.net.SocketException: Too many open files
>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:421)
>         at java.net.ServerSocket.implAccept(ServerSocket.java:243)
>         at java.net.ServerSocket.accept(ServerSocket.java:222)
>         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:286)

It looks like WL is running into problems accepting new incomming
connections from the network, not opening additional connections to
the database.  It's been awhile since I've dealt with WebLogic, but
IIRC, it was reasonably good about returning the results of strerr()
in it's traces for exceptions resulting from errors in system and libc
calls.  It certainly does *look* as if the underlying call to accept()
is failing with EMFILE.

At this point, I'd see if I could find some evidence for or against
the server being "slashdotted" from elsewhere in the network.

Is there some other component in your environment that would be connecting
to WL, or are the client connections limited to actual users connecting
via their browsers?

--

"...Microsoft follows standards.  In much the same manner that fish follow
migrating caribou." "Now I have this image in my mind of a fish embracing and
extending a caribou." -- Paul Tomblin and Christian Bauernfeind in the SDM

 
 
 

1. OpenWin Filemgr: 'cannot open file: too many files open' ?

Here's a strange problem with the FileManager in Solaris 2.3.  The
only way to fix it right now is to restart the file manager over.

 When double-clicking on a text file, file manager is set-up to
 start textedit on that file.  I quit textedit.  I double-click
 on a text file again.  And I quit again textedit.  I can do that
 approximately 54 times.  After that, you can't open any file or
 change directory within FileManager.

Does anybody know if this is a configuration problem or a bug ?
And if it's a bug, is there any patch for it ?

Thanks

Hugo Frappier
Montreal

2. lex/yacc

3. Help: What controls number of files a user can open (file open command)?

4. background screens

5. Open File Manager (open file backup) for Linux?

6. setuid programs on NFS-mounted disks

7. File open or not open ???

8. #9 Vision 330

9. map a process open file to a open socket ?

10. Can't open a broadcast socket because there are too many open files?

11. File open or not open???

12. cannot open /dev/mem: too many open files??

13. Open files vs. open inodes