Maximum outbound connections limit?

Maximum outbound connections limit?

Post by Benjamin Kaufma » Sun, 08 Jul 2001 00:36:17



I seem to have hit an outbound connection limit of 32K in some TCP based test
programs.
I can get file descriptors (each process only opens up 1000 connections) but the
connects start to fail with EADDRNOTAVAIL and then even OTHER USERS cannot get
out with telnet or ftp. They get an erro similar to mine of "cannot assign
requested address."  However, one can still log into the machine.
I'm still investigating but I couldn't find anything in the documentation which
limits this number.

BTW, this is a 4500 with 6 CPUs and at least 1 gig of RAM.  It runs at almost no
cpu until the limit is hit. Then each connect takes a large amount of CPU.
The clients bind to INADDR_ANY.

Ben

 
 
 

Maximum outbound connections limit?

Post by Barry Margoli » Sun, 08 Jul 2001 00:58:09




Quote:>I seem to have hit an outbound connection limit of 32K in some TCP based test
>programs.
>I can get file descriptors (each process only opens up 1000 connections) but the
>connects start to fail with EADDRNOTAVAIL and then even OTHER USERS cannot get
>out with telnet or ftp. They get an erro similar to mine of "cannot assign
>requested address."  However, one can still log into the machine.
>I'm still investigating but I couldn't find anything in the documentation which
>limits this number.

You've run out of local port numbers.  TCP uses ports 32K-64K as local port
numbers in outbound connections.  I expect you can change the boundaries of
this range using "ndd", but I don't know the variable name offhand.

--

Genuity, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

 
 
 

Maximum outbound connections limit?

Post by Benjamin Kaufma » Sun, 08 Jul 2001 01:12:00


Barry,

Thanks.  I was hoping for something "obvious"

Ben




>>I seem to have hit an outbound connection limit of 32K in some TCP based test
>>programs.
>>I can get file descriptors (each process only opens up 1000 connections) but the
>>connects start to fail with EADDRNOTAVAIL and then even OTHER USERS cannot get
>>out with telnet or ftp. They get an erro similar to mine of "cannot assign
>>requested address."  However, one can still log into the machine.
>>I'm still investigating but I couldn't find anything in the documentation which
>>limits this number.

>You've run out of local port numbers.  TCP uses ports 32K-64K as local port
>numbers in outbound connections.  I expect you can change the boundaries of
>this range using "ndd", but I don't know the variable name offhand.

 
 
 

Maximum outbound connections limit?

Post by Benjamin Kaufma » Sun, 08 Jul 2001 01:19:56


Barry,

Wouldn't that error out on the bind? I'm getting the error from connect.

Ben




>>I seem to have hit an outbound connection limit of 32K in some TCP based test
>>programs.
>>I can get file descriptors (each process only opens up 1000 connections) but the
>>connects start to fail with EADDRNOTAVAIL and then even OTHER USERS cannot get
>>out with telnet or ftp. They get an erro similar to mine of "cannot assign
>>requested address."  However, one can still log into the machine.
>>I'm still investigating but I couldn't find anything in the documentation which
>>limits this number.

>You've run out of local port numbers.  TCP uses ports 32K-64K as local port
>numbers in outbound connections.  I expect you can change the boundaries of
>this range using "ndd", but I don't know the variable name offhand.

 
 
 

Maximum outbound connections limit?

Post by Barry Margoli » Sun, 08 Jul 2001 02:00:22




Quote:>Wouldn't that error out on the bind? I'm getting the error from connect.

Are you sure the program is calling bind()?  It's not necessary for
outgoing connections unless you need to use a specific source port or
address.

--

Genuity, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

 
 
 

Maximum outbound connections limit?

Post by Darren Dunha » Sun, 08 Jul 2001 02:20:57





>>I seem to have hit an outbound connection limit of 32K in some TCP based test
>>programs.
>>I can get file descriptors (each process only opens up 1000 connections) but the
>>connects start to fail with EADDRNOTAVAIL and then even OTHER USERS cannot get
>>out with telnet or ftp. They get an erro similar to mine of "cannot assign
>>requested address."  However, one can still log into the machine.
>>I'm still investigating but I couldn't find anything in the documentation which
>>limits this number.
> You've run out of local port numbers.  TCP uses ports 32K-64K as local port
> numbers in outbound connections.  I expect you can change the boundaries of
> this range using "ndd", but I don't know the variable name offhand.

It's probably these.  I have no idea what effects changing them can
have..

# /usr/sbin/ndd /dev/tcp tcp_smallest_anon_port
32768
# /usr/sbin/ndd /dev/tcp tcp_largest_anon_port
65535

--

Unix System Administrator                    Taos - The SysAdmin Company
Got some Dr Pepper?                           San Francisco, CA bay area
          < How are you gentlemen!! Take off every '.SIG'!! >

 
 
 

Maximum outbound connections limit?

Post by Casper H.S. Dik - Network Security Engine » Sun, 08 Jul 2001 02:49:50


[[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]


>Wouldn't that error out on the bind? I'm getting the error from connect.

No, when you connect out (without doing a bind) you are in effect doing
a bind & connect; the bind fails and so will the connect.

You're running into a limitation in TCP; you would need to reuse the
addresses (which only works if you're connecting to different servers
or ports)

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

 
 
 

Maximum outbound connections limit?

Post by George William Herbe » Sun, 08 Jul 2001 03:28:34





>>>I seem to have hit an outbound connection limit of 32K in some TCP based test
>>>programs. [...]

>> You've run out of local port numbers.  TCP uses ports 32K-64K as local port
>> numbers in outbound connections.  I expect you can change the boundaries of
>> this range using "ndd", but I don't know the variable name offhand.

>It's probably these.  I have no idea what effects changing them can
>have..
># /usr/sbin/ndd /dev/tcp tcp_smallest_anon_port
>32768
># /usr/sbin/ndd /dev/tcp tcp_largest_anon_port
>65535

Well, I wouldn't recommend an increase in tcp_largest_anon_port .
From /usr/include/inet/tcp.h (copyright sun, yadda yadda):
        typedef struct tcphdr_s {
                uint8_t         th_lport[2];    /* Source port */
                uint8_t         th_fport[2];    /* Destination port */
        ...

You might experiment with decreasing tcp_smallest_anon_port, however.
You could get something like 64,000 ports max, perhaps, lowering the
smallest anon port...

-george william herbert

 
 
 

Maximum outbound connections limit?

Post by Barry Margoli » Sun, 08 Jul 2001 03:20:37




Quote:>You're running into a limitation in TCP; you would need to reuse the
>addresses (which only works if you're connecting to different servers
>or ports)

Strictly speaking, he's running into a limitation of this implementation of
TCP, although it's a limitation that most implementations share.  When
making an outbound connection, TCP could compare all the other fields of
the tuple to find when scanning existing connections.  But to keep the
search for a local port simple, most implementations just look for one that
isn't in use at all (this can be done using a simple bit vector, rather
than having to search all the open connections or using a more complex data
structure to track port numbers).

Limiting a system to 32K concurrent outbound connections is rarely a
problem.  It sounds like the OP only ran into it because he was running an
application designed to stress the system, not a normal use.  Systems with
lots of connections are usually servers, and most of the connections are
inbound, not outbound.

It should be noted also that some versions of Unix have a fixed-size file
table in the kernel, which holds all open streams.  This puts a limit on
the total number of files, network connections, devices, etc. that can be
open at once.  I think SysVR4 has eliminated this, but some other flavors
of Unix may still have it.

--

Genuity, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

 
 
 

Maximum outbound connections limit?

Post by Casper H.S. Dik - Network Security Engine » Sun, 08 Jul 2001 03:27:04


[[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]


Quote:>># /usr/sbin/ndd /dev/tcp tcp_smallest_anon_port
>>32768
>># /usr/sbin/ndd /dev/tcp tcp_largest_anon_port
>>65535
>Well, I wouldn't recommend an increase in tcp_largest_anon_port .

Obviously, you can't.

You can, however, decrease tcp_smallest_anon_port.

Quote:>From /usr/include/inet/tcp.h (copyright sun, yadda yadda):
>    typedef struct tcphdr_s {
>            uint8_t         th_lport[2];    /* Source port */
>            uint8_t         th_fport[2];    /* Destination port */
>    ...
>You might experiment with decreasing tcp_smallest_anon_port, however.
>You could get something like 64,000 ports max, perhaps, lowering the
>smallest anon port...

It might start to conflict with some other things if you do that
(X11 for one)

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

 
 
 

Maximum outbound connections limit?

Post by George William Herbe » Sun, 08 Jul 2001 05:57:25



>[[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]

>>># /usr/sbin/ndd /dev/tcp tcp_smallest_anon_port
>>>32768
>>># /usr/sbin/ndd /dev/tcp tcp_largest_anon_port
>>>65535

>>Well, I wouldn't recommend an increase in tcp_largest_anon_port .

>Obviously, you can't.

Well, it would at least be undefined ;-)

Quote:>You can, however, decrease tcp_smallest_anon_port.

>>From /usr/include/inet/tcp.h (copyright sun, yadda yadda):
>>        typedef struct tcphdr_s {
>>                uint8_t         th_lport[2];    /* Source port */
>>                uint8_t         th_fport[2];    /* Destination port */
>>        ...

>>You might experiment with decreasing tcp_smallest_anon_port, however.
>>You could get something like 64,000 ports max, perhaps, lowering the
>>smallest anon port...

>It might start to conflict with some other things if you do that
>(X11 for one)

The problem description sounded like a headless server, but yes,
X11's usually got a bunch of ports in the 32k range which any decrease
would stomp on.

Is that X11 port stuff locally reconfigurable?  Been too long since
I hacked X configs.

-george william herbert

 
 
 

Maximum outbound connections limit?

Post by Casper H.S. Dik - Network Security Engine » Sun, 08 Jul 2001 07:35:18


[[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]


Quote:>The problem description sounded like a headless server, but yes,
>X11's usually got a bunch of ports in the 32k range which any decrease
>would stomp on.
>Is that X11 port stuff locally reconfigurable?  Been too long since
>I hacked X configs.

Well, on many configs these days, the X server is always started at boot,
so that may not be much of a problem.

Besides, starting at 6500 still gives you around 25000 ports more.

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

 
 
 

Maximum outbound connections limit?

Post by Benjamin Kaufma » Sun, 08 Jul 2001 09:37:10


Barry,

Planning to run possibly more than 64K connections so I multi-honed the system.
Got my (today's goal) of 45,000 connections.

Ben




>>You're running into a limitation in TCP; you would need to reuse the
>>addresses (which only works if you're connecting to different servers
>>or ports)

>Strictly speaking, he's running into a limitation of this implementation of
>TCP, although it's a limitation that most implementations share.  When
>making an outbound connection, TCP could compare all the other fields of
>the tuple to find when scanning existing connections.  But to keep the
>search for a local port simple, most implementations just look for one that
>isn't in use at all (this can be done using a simple bit vector, rather
>than having to search all the open connections or using a more complex data
>structure to track port numbers).

>Limiting a system to 32K concurrent outbound connections is rarely a
>problem.  It sounds like the OP only ran into it because he was running an
>application designed to stress the system, not a normal use.  Systems with
>lots of connections are usually servers, and most of the connections are
>inbound, not outbound.

>It should be noted also that some versions of Unix have a fixed-size file
>table in the kernel, which holds all open streams.  This puts a limit on
>the total number of files, network connections, devices, etc. that can be
>open at once.  I think SysVR4 has eliminated this, but some other flavors
>of Unix may still have it.

 
 
 

Maximum outbound connections limit?

Post by Alan Coopersmit » Sun, 08 Jul 2001 11:22:22



|Is that X11 port stuff locally reconfigurable?  Been too long since
|I hacked X configs.

The X protocol defines the TCP port as 6000 + displaynumber, so all you
can do is adjust the display (and you can't go negative).  The font
server often runs on port 7000 or 7100, but that's adjustable (or you
can just turn it off if you don't need it for Xterminals or remote
sessions that need access to the fonts).

--
________________________________________________________________________


  Working for, but definitely not speaking for, Sun Microsystems, Inc.

 
 
 

Maximum outbound connections limit?

Post by Benjamin Kaufma » Sun, 08 Jul 2001 20:36:49


My mistake.   Bind isn't called unless a local address is specified.

Ben




>>Wouldn't that error out on the bind? I'm getting the error from connect.

>Are you sure the program is calling bind()?  It's not necessary for
>outgoing connections unless you need to use a specific source port or
>address.