New connections stall with 20k+ open sockets

New connections stall with 20k+ open sockets

Post by Karthik Arumugha » Tue, 30 Jul 2002 04:40:09



Hello,

I am running a large IRC server that at times has held over 40,000
simultaneous connections. Normally the server will hover around 15k - 20k
connections, but at times it goes well past that.

I've been having an issue where when the server goes past 20k connections or
so, it'll start ignoring syn packets on the most heavily used ports. I've
experienced this under 2.4.18 and older 2.4 kernels, and I'm currently
running 2.5.29. Distribution is Debian unstable (not that that should matter
here). I'm using a Netgear GA620 gig-e card, x86 architecture.

One thing I've done is to bind multiple IPs to the box and add additional
ports. The lesser used ports generally accept connections fine. Almost half
of all connections come in to one ip+port pair though, and that's where the
biggest problem is. At 30k users or so, about 80+% of syn packets to that
port are ignored. I can see the incoming syn packets fine in tcpdump,
there's just no syn-ack reply. This makes new users sit there for a long
time timing out rather than successfully connecting. The ignored connections
do not show up as SYN_RECV.

Currently, with 30199 connected users all working properly, I have a total
of 243 sockets in SYN_RECV state. At times (such as if the server is
restarted), there are several hundred connection requests per second, which
generally get handled fine if there are not many users connected. The
problem only occurs when many sockets are open, whether or not the
connection rate is high. tcp_max_syn_backlog is currently set to 4096, and
I've reduced tcp_synack_retries to 2.

Also, there's plenty of memory:

             total       used       free     shared    buffers     cached
Mem:        517452     284576     232876          0          0      13712
-/+ buffers/cache:     270864     246588
Swap:            0          0          0

Any other relevant details I'm missing? I've tried screwing around with
various parameters in /proc/sys/net/ipv4 without luck; unfortunately I don't
understand the internals of how a received syn packet is handled, and what
would cause it to be silently discarded.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

New connections stall with 20k+ open sockets

Post by Willy Tarrea » Tue, 30 Jul 2002 17:50:10



> I've been having an issue where when the server goes past 20k connections or
> so, it'll start ignoring syn packets on the most heavily used ports. I've
> experienced this under 2.4.18 and older 2.4 kernels, and I'm currently
> running 2.5.29. Distribution is Debian unstable (not that that should matter
> here). I'm using a Netgear GA620 gig-e card, x86 architecture.

I've had such a behaviour with an HTTP reverse proxy I wrote, until I
realized that when you have thousands of connections, the select() call
slows down a bit, and the accept() was not called often enough to catch
all the new connections. I simply solved the problem by calling as many
accept() as possible each time the listen socket wakes up. I'm pretty
sure you are in such a situation.

Cheers,
Willy

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

1. Opening a socket connection while in a socket connection.

Hello,

I am new to using sockets and am working on a project that requires
me to audit services accessed under a client/server program (TCP/IP)
socket connection.

What I want to do is send a datapacket to a collector machine each
time a client accesses a service on the server.  It seems that
I am unable to do this from the server.  Basically, in short form
I need to be able to do the following:

1. Client connects (TCP/IP socket connection) to server
2. Client requests a service from the server
3. At that point I want to send a datapacket to a audit server.

Any Ideas?  I am familiar with most of the concepts involved in
using sockets.  Maybe I have been working on this to long and have
gotten tunnel vision.

Thanks

***********************************************************
Brent Curtiss


Academic Computing Services
The Ohio State University
***********************************************************

2. RFD: New group c.o.l.experts

3. Linux sockets problem (closed connections staying open?)

4. cancel <1087683729.741415@smtp.tvcom.ru>

5. Open Socket Connections

6. favorite apps and games?

7. open socket connection

8. CD-ROM

9. Socket connection to port fails despite port open / listening !

10. socket problem: new port for each connection needed?

11. Socket problem: select() not returning when a new connection is available

12. map a process open file to a open socket ?

13. Can't open a broadcast socket because there are too many open files?