50 thousand TCP connections on Solaris 2.0/SVR4 - is it possible?

50 thousand TCP connections on Solaris 2.0/SVR4 - is it possible?

Post by Jim Robins » Sat, 20 Jun 1992 05:29:08



We have an application that potentially wishes to manage an * number
of TCP connections, about 10,000 to 50,000.  The target system will be
Solaris 2.0 running on a SPARC.

Prior to Solaris 2.0 (and SVR4 in general) there would have been a number
of problems associated with attempting to manage that many TCP connections
with only a few (say 3) processes.

1) Maximum number of per-process open file descriptors was much too small.
   (able to be bumped up to 256 on SunOS 4.1).
2) System open file table would normally be too small requiring kernel
   reconfiguration.
3) Kernel memory devoted to networking code (mbufs) was fixed at a "small"
   amount (1 or 2 Meg) and there was no way to bump this up.
4) It was questionable whether various kernel algorithms would scale
   gracefully to handle 10k to 50k TCP connections.
5) Other issues that have not been considered.

We do not not have access to a SVR4 system but it is my understanding that
problem 1 is solved in that SVR4 allows a process to have an "unlimited"
(in quotes for obvious reasons) amount of open file descriptors. [see next
paragraph]

Problem 2 (which really is just a bit of a pain) is alleviated in that
many/most/all kernel tables now dynamically expand.

Problem 3 has also been solved in that at worst it will be necessary to
bump a number in a system configuration file and reboot.

Problem 4 could be a show stopper. It has been a long time since I have
looked at the BSD networking code, but I seem to remember a *linear* search
being performed when a lookup is done to determine the appropriate socket
for a TCP segment. Am I misremembering? As well, there are probably other
linear searches that I am unaware of that could be a problem.

I would greatly appreciate any comments and/or corrections re the above
with particular attention paid to problems 4 and 5. The thought of managing
that many TCP connections on a single SPARC with just a few processes is
somewhat mind-boggling and I am sure that there would be, perhaps very
serious, problems.

Note: the above presupposes that we are willing to throw in as much memory
as required to do the job.

Thanks.
--
Jim Robinson

{ubc-cs!van-bc,uunet}!mdivax1!robinson

 
 
 

50 thousand TCP connections on Solaris 2.0/SVR4 - is it possible?

Post by Frank Ch. Eigle » Sat, 20 Jun 1992 06:12:41



>We have an application that potentially wishes to manage an * number
>of TCP connections, about 10,000 to 50,000.  The target system will be
>Solaris 2.0 running on a SPARC.
>[...]

Sorry for the silly questions, but are you sure you need that many *TCP*
connections?  Have you considered something built on top of UDP?

--



 
 
 

50 thousand TCP connections on Solaris 2.0/SVR4 - is it possible?

Post by Peter Desnoye » Sat, 20 Jun 1992 14:04:35



>Problem 4 could be a show stopper. It has been a long time since I have
>looked at the BSD networking code, but I seem to remember a *linear* search
>being performed when a lookup is done to determine the appropriate socket
>for a TCP segment. Am I misremembering? As well, there are probably other
>linear searches that I am unaware of that could be a problem.

It sounds like you might have a serious enough application to warrant
getting a source license and modifying the lookup function yourself.
Another thing to try would be finding systems that advertise phenomenal
TPC numbers - from previous discussions in this newsgroup it would seem
that maintaining large numbers of TCP connections efficiently is
necessary for good TPC (transaction processing consortium benchmark)
numbers.

                                Peter Desnoyers
--

 
 
 

50 thousand TCP connections on Solaris 2.0/SVR4 - is it possible?

Post by Adnan C. Yaq » Sun, 21 Jun 1992 03:27:23


Jim> We have an application that potentially wishes to manage an * number
Jim> of TCP connections, about 10,000 to 50,000.  The target system will be
Jim> Solaris 2.0 running on a SPARC.

Jim> 3) Kernel memory devoted to networking code (mbufs) was fixed at a "small"
Jim>    amount (1 or 2 Meg) and there was no way to bump this up.

Jim> Problem 3 has also been solved in that at worst it will be necessary to
Jim> bump a number in a system configuration file and reboot.

In TCP, the protocol must allocate enough memory to hold "window size"
bytes of data.  For common implementations, "window size" is 4Kbytes.
Therefore, for 50,000 connections, this is about 200Mbytes of mbufs
(actually mblks in SVR4), and that is only the data for the windows,
not to mention the TCP control blocks, IP memory needs, ARP tables...

I hope we know what we are doing here :-)
--

Allen-Bradley Company, Inc., 747 Alpha Drive, Highland Hts., OH 44143, USA
Phone: +1 216 646 4670 FAX: +1 216 646 4484

 
 
 

50 thousand TCP connections on Solaris 2.0/SVR4 - is it possible?

Post by Jim Robins » Sun, 21 Jun 1992 05:25:35





>Jim> We have an application that potentially wishes to manage an * number
>Jim> of TCP connections, about 10,000 to 50,000.  The target system will be
>Jim> Solaris 2.0 running on a SPARC.

>Jim> 3) Kernel memory devoted to networking code (mbufs) was fixed at a "small"
>Jim>    amount (1 or 2 Meg) and there was no way to bump this up.

>Jim> Problem 3 has also been solved in that at worst it will be necessary to
>Jim> bump a number in a system configuration file and reboot.

>In TCP, the protocol must allocate enough memory to hold "window size"
>bytes of data.  For common implementations, "window size" is 4Kbytes.
>Therefore, for 50,000 connections, this is about 200Mbytes of mbufs
>(actually mblks in SVR4), and that is only the data for the windows,
>not to mention the TCP control blocks, IP memory needs, ARP tables...

We had considered this and one possibility is to reduce the window size
(which is realized as the socket buffer sizes) using setsockopt(). That
would normally result in a performance hit due to reduced pipelining of
messages.  However, since our metric of interest is *total* message
thruput, and since our message size averages 100 to 200 bytes, we should be
able to get away with a fairly small window.

I suppose if we really wanted to get fancy, socket buffer size
could be dynamically managed by the application depending on the number of
current TCP connections; but, like Scarlet O'Hara, I would rather think
about that tomorrow.

Quote:>I hope we know what we are doing here :-)

That remains to be seen :-).
--
Jim Robinson

{ubc-cs!van-bc,uunet}!mdivax1!robinson
 
 
 

50 thousand TCP connections on Solaris 2.0/SVR4 - is it possible?

Post by Boyd Rober » Sun, 21 Jun 1992 06:02:16


50000 TCP/IP connections.  Just how many packets a second will that
be?  And just how many cycles will you need to service them all?


``When the going gets wierd, the weird turn pro...''

 
 
 

50 thousand TCP connections on Solaris 2.0/SVR4 - is it possible?

Post by Dan Bernste » Tue, 23 Jun 1992 08:57:59


Consider: From one Sun 4 with no kernel modifications, using just a few
hundred TCP connections at once, I brought down the PSINet-NSFNet
gateway for hours one day last December. (It was an accident, I swear.)
The idiots designing networks keep talking about how many *bytes* per
second the wires can handle---and don't realize how few *packets* per
second the routers can handle.

Now what the hell do you want to do with 50,000 connections at once?

---Dan

 
 
 

50 thousand TCP connections on Solaris 2.0/SVR4 - is it possible?

Post by Ian Heave » Tue, 23 Jun 1992 22:18:16





>Jim> We have an application that potentially wishes to manage an * number
>Jim> of TCP connections, about 10,000 to 50,000.  The target system will be
>Jim> Solaris 2.0 running on a SPARC.

>Jim> 3) Kernel memory devoted to networking code (mbufs) was fixed at a "small"
>Jim>    amount (1 or 2 Meg) and there was no way to bump this up.

>Jim> Problem 3 has also been solved in that at worst it will be necessary to
>Jim> bump a number in a system configuration file and reboot.

>In TCP, the protocol must allocate enough memory to hold "window size"
>bytes of data.  For common implementations, "window size" is 4Kbytes.
>Therefore, for 50,000 connections, this is about 200Mbytes of mbufs
>(actually mblks in SVR4), and that is only the data for the windows,
>not to mention the TCP control blocks, IP memory needs, ARP tables...

I don't believe it is necessary to allocate a full window size of memory
unless you are actually filling that window, which presumably is only
true for a small fraction of the TCP connections.  Clearly TCP control
blocks scale linearly with the number of connections, but IP and ARP
memory requirements are related to the number of gateways and hosts
you are dealing with, so they shouldn't change.

Caveat: my comments refer to our own TCP/IP implementation.  Solaris
TCP/IP may do something a bit different, but should still be sensible.

ian

---

Spider Software
Spider Park, Stanwell Street            
Edinburgh, EH6 5NG, Scotland            +44 31 554 9424 (Ext 4346)
--

 
 
 

50 thousand TCP connections on Solaris 2.0/SVR4 - is it possible?

Post by Barry Margol » Wed, 24 Jun 1992 01:21:11


[I've redirected followups to comp.protocols.tcp-ip.]


>I don't believe it is necessary to allocate a full window size of memory
>unless you are actually filling that window, which presumably is only
>true for a small fraction of the TCP connections.

The window is supposed to indicate how much data the TCP level is willing
to buffer up.  So, if 1000 connections each advertise an 8KB window, you're
claiming that you're prepared to buffer up 8MB.  I suppose you can be
optimistic that most senders won't actually get that far ahead of the
receiving processes, so the total of all the window sizes will be less than
the amount of space reserved.

Actually, I suspect many TCP implementations do this.  They use dynamic
allocation of buffers (e.g. Unix mbufs), rather than allocating a separate
buffer for each, and have a global constant that is the per-connection
window size.  This brings up the question of how meaningful the window
really is in such implementations?  If most of the receiver processes stop
reading their input (perhaps they're all waiting for some resource to
become available), the buffer pool might be exhausted before many of the
windows are filled.  At that point I guess the system can simply close all
of the windows.  One problem with an optimistic scheme like this is that it
can result in deadlock if the common resource most of the receivers are
waiting for is held by a process that's waiting on another TCP connection
(this is similar to overallocation of disk quota: if the disk fills
completely, you can't compress files to make room because you temporarily
need space for both the compressed and uncompressed versions).
--
Barry Margolin
System Manager, Thinking Machines Corp.


 
 
 

1. Linux server to hold thousands of tcp connections?

I am building an IM server for LINUX that will accept connections and
hold them open for asyn communication (very much the same as AOL).  I
need some ideas about how to go about this.

Should I use BSD sockets or go to a lower layer?  What are the max # of
BSD sockets that could probably be held open at the same time on LINUX?
How do I go to a lower layer if I wanted too?  Resources?  Any ideas
would be great?

joe r

2. Adding ibcs to kernels 1.3x

3. pci ultra scsi cable to 50 pin scsi ii (vhdc ro 50 pin)

4. RPC over NAT with iptables

5. ?? How is it possible to compress gzipped file > 50% ??

6. OT: defekte RAM module unter linux verwenden!

7. Is it possible to set 50 line mode ?

8. shell for startup scripts

9. possible SYN flood on <my IP address>:80 from 130.244.157.50

10. Make Your H/50 & F/50 haul ASS!

11. RH9: ADSL connection drops every ~50 secs if inactive.

12. wyse-50'able modem connection

13. passing of TCP-connection to a child process possible?