Very Computer

I am facing a very puzzling problem.

Scenario:
on the server I have a program listening on port 9999
a client tries to connect (via telnet) to the server on port 9999

Sometimes, establishing the connection takes a long time (one minute
or more).
What happens is that the client sends a SYN packet, the syn packet is
received by the server, but the server does not reply to the SYN
packet with a SYN/ACK (the program listening on the server does not
see the incoming connection).

After 4/5 SYN sents by the client, the servers finally answers with a
SYN/ACK, the connection is established and the program on the server
sees the connection).

I ran a packet sniffer on both client and server, and the result is
the same.

19:53:45.038667 IP (tos 0x0, ttl  58, id 27948, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8571162
0,nop,wscale 7>

19:53:57.044368 IP (tos 0x0, ttl  58, id 27949, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8574162
0,nop,wscale 7>

19:54:21.045412 IP (tos 0x0, ttl  58, id 27950, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8580162
0,nop,wscale 7>

19:55:09.040188 IP (tos 0x0, ttl  58, id 27951, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8592162
0,nop,wscale 7>

19:57:08.584133 IP (tos 0x0, ttl  58, id 64380, offset 0, flags [DF],
proto 6, length: 60) client.43425 > server.9999: S [tcp sum ok]
2090300578:2090300578(0) win 5840 <mss 1460,sackOK,timestamp 8622046
0,nop,wscale 7>

19:57:08.595922 IP (tos 0x0, ttl  58, id 64381, offset 0, flags [DF],
proto 6, length: 52) client.43425 > server.9999: . [tcp sum ok]
2090300579:2090300579(0) ack 1191201502 win 46 <nop,nop,timestamp
8622049 2164260539>

19:57:14.223948 IP (tos 0x0, ttl  58, id 64382, offset 0, flags [DF],
proto 6, length: 52) client.43425 > server.9999: F [tcp sum ok] 0:0(0)
ack 1 win 46 <nop,nop,timestamp 8623457 2164260539>

19:57:14.236287 IP (tos 0x0, ttl  58, id 64383, offset 0, flags [DF],
proto 6, length: 52) client.43425 > server.9999: . [tcp sum ok] 1:1(0)
ack 2 win 46 <nop,nop,timestamp 8623459 2164266179>

As you can see, the server ignores the first SYNs but at some point,
for no apparent reason, finally replies with a SYN/ACK.

Does anyone have any clue?

Thank you.

Informations on the server:
CentOS release 4.6 (Final)

Linux ebrick.net 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 EDT
2008 i686 i686 i386 GNU/Linux

dmesg (partial):
divert: allocating divert_blk for eth0
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
ACPI: PCI Interrupt 0000:00:1b.0[A] -> GSI 21 (level, low) -> IRQ 193
PCI: Setting latency timer of device 0000:00:1b.0 to 64
e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex,
Flow Control: RX
e1000: eth0: e1000_watchdog_task: 10/100 speed: disabling TSO

Quote:> Hello all.

> I am facing a very puzzling problem.

> Scenario:
> on the server I have a program listening on port 9999
> a client tries to connect (via telnet) to the server on port 9999

What is the program doing while it's not seeing the connection? Is it
blocked in 'accept' or doing something else entirely?

DS

Still, this problem surfaces with any kind of program listening on the
server (it also shows on Apache HTTPD and SSH).

Bye

$ netstat -anp
tcp        0      1 client:42776     server:443        SYN_SENT

The listening server may be trying to do a reverse lookup on the
client's IP.  If the resolver can't reach the DNS servers, it will wait
for them to time out.  After the time out, the server process abandons
the reverse lookup and accepts the connection (if it is configured to
allow unresolvable IP's).  However, while the server is waiting for DNS
the client will very likely send more SYN packets assuming the packet
wasn't received as it wasn't responded to in a timely fashion.

Check your DNS/resolver config :)

Cheers,

James

rick jones
--
web2.0 n, the dot.com reunion tour...
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Quote:> I've seen this before and based on your later follow-up in this thread
> that this affects SSH and HTTP connections too it leads me to suggest a
> DNS problem.

Quote:> The listening server may be trying to do a reverse lookup on the
> client's IP. ?If the resolver can't reach the DNS servers, it will wait
> for them to time out. ?After the time out, the server process abandons
> the reverse lookup and accepts the connection (if it is configured to
> allow unresolvable IP's). ?However, while the server is waiting for DNS
> the client will very likely send moreSYNpackets assuming the packet
> wasn't received as it wasn't responded to in a timely fashion.
> Check your DNS/resolver config :)

But it seems it is not the case now.

During the tests, I run also a:

 tcpdump -s 0 -vv -i eth0 host ns1 or host ns2 or host ns3

(where ns1-3 are my name servers).
During the tests, nor the listening program nor the server does any
DNS lookup.

Bye

Quote:> DoesLinuxactually have a mode to allow connection accept in that
> fashion? ?I always thought it waited at least until theSYN|ACK was
> sent in response to theSYNbefore the listening application was
> notified there was anything to consider accepting. ?IIRC there is no
> means in the standard sockets interface to do such a split accept() in
> the first place?

Bye