Linux ignoring SYN packets

Linux ignoring SYN packets

Post by ErMej » Thu, 22 May 2008 02:20:33



Hello all.

I am facing a very puzzling problem.

Scenario:
on the server I have a program listening on port 9999
a client tries to connect (via telnet) to the server on port 9999

Sometimes, establishing the connection takes a long time (one minute
or more).
What happens is that the client sends a SYN packet, the syn packet is
received by the server, but the server does not reply to the SYN
packet with a SYN/ACK (the program listening on the server does not
see the incoming connection).

After 4/5 SYN sents by the client, the servers finally answers with a
SYN/ACK, the connection is established and the program on the server
sees the connection).

I ran a packet sniffer on both client and server, and the result is
the same.

19:53:45.038667 IP (tos 0x0, ttl  58, id 27948, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8571162
0,nop,wscale 7>

19:53:57.044368 IP (tos 0x0, ttl  58, id 27949, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8574162
0,nop,wscale 7>

19:54:21.045412 IP (tos 0x0, ttl  58, id 27950, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8580162
0,nop,wscale 7>

19:55:09.040188 IP (tos 0x0, ttl  58, id 27951, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8592162
0,nop,wscale 7>

19:57:08.584133 IP (tos 0x0, ttl  58, id 64380, offset 0, flags [DF],
proto 6, length: 60) client.43425 > server.9999: S [tcp sum ok]
2090300578:2090300578(0) win 5840 <mss 1460,sackOK,timestamp 8622046
0,nop,wscale 7>

19:57:08.595922 IP (tos 0x0, ttl  58, id 64381, offset 0, flags [DF],
proto 6, length: 52) client.43425 > server.9999: . [tcp sum ok]
2090300579:2090300579(0) ack 1191201502 win 46 <nop,nop,timestamp
8622049 2164260539>

19:57:14.223948 IP (tos 0x0, ttl  58, id 64382, offset 0, flags [DF],
proto 6, length: 52) client.43425 > server.9999: F [tcp sum ok] 0:0(0)
ack 1 win 46 <nop,nop,timestamp 8623457 2164260539>

19:57:14.236287 IP (tos 0x0, ttl  58, id 64383, offset 0, flags [DF],
proto 6, length: 52) client.43425 > server.9999: . [tcp sum ok] 1:1(0)
ack 2 win 46 <nop,nop,timestamp 8623459 2164266179>

As you can see, the server ignores the first SYNs but at some point,
for no apparent reason, finally replies with a SYN/ACK.

Does anyone have any clue?

Thank you.

Informations on the server:
CentOS release 4.6 (Final)


Linux ebrick.net 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 EDT
2008 i686 i686 i386 GNU/Linux

dmesg (partial):
divert: allocating divert_blk for eth0
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
ACPI: PCI Interrupt 0000:00:1b.0[A] -> GSI 21 (level, low) -> IRQ 193
PCI: Setting latency timer of device 0000:00:1b.0 to 64
e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex,
Flow Control: RX
e1000: eth0: e1000_watchdog_task: 10/100 speed: disabling TSO

 
 
 

Linux ignoring SYN packets

Post by David Schwart » Thu, 22 May 2008 07:13:24



Quote:> Hello all.

> I am facing a very puzzling problem.

> Scenario:
> on the server I have a program listening on port 9999
> a client tries to connect (via telnet) to the server on port 9999

What's the listen backlog set to? (When you called 'listen', the
second parameter.)

What is the program doing while it's not seeing the connection? Is it
blocked in 'accept' or doing something else entirely?

DS

 
 
 

Linux ignoring SYN packets

Post by ErMej » Thu, 22 May 2008 17:17:50




> > Hello all.

> > I am facing a very puzzling problem.

> > Scenario:
> > on the server I have a program listening on port 9999
> > a client tries to connect (via telnet) to the server on port 9999
> What is the program doing while it's not seeing the connection? Is it
> blocked in 'accept' or doing something else entirely?

It is blocked in the 'accept' statement.

Still, this problem surfaces with any kind of program listening on the
server (it also shows on Apache HTTPD and SSH).

Bye

 
 
 

Linux ignoring SYN packets

Post by ErMej » Thu, 22 May 2008 18:12:22



> Hello all.

> I am facing a very puzzling problem.

> Scenario:
> on the server I have a program listening on port 9999
> a client tries to connect (via telnet) to the server on port 9999

> Sometimes, establishing the connection takes a long time (one minute
> or more).
> What happens is that the client sends aSYNpacket, thesynpacket is
> received by the server, but the server does not reply to theSYN
> packet with aSYN/ACK (the program listening on the server does not
> see the incoming connection).

> After 4/5SYNsents by the client, the servers finally answers with aSYN/ACK, the connection is established and the program on the server
> sees the connection).

> I ran a packet sniffer on both client and server, and the result is
> the same.

> 19:53:45.038667 IP (tos 0x0, ttl ?58, id 27948, offset 0, flags [DF],
> proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
> 3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8571162
> 0,nop,wscale 7>

> 19:53:57.044368 IP (tos 0x0, ttl ?58, id 27949, offset 0, flags [DF],
> proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
> 3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8574162
> 0,nop,wscale 7>

> 19:54:21.045412 IP (tos 0x0, ttl ?58, id 27950, offset 0, flags [DF],
> proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
> 3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8580162
> 0,nop,wscale 7>

> 19:55:09.040188 IP (tos 0x0, ttl ?58, id 27951, offset 0, flags [DF],
> proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
> 3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8592162
> 0,nop,wscale 7>

> 19:57:08.584133 IP (tos 0x0, ttl ?58, id 64380, offset 0, flags [DF],
> proto 6, length: 60) client.43425 > server.9999: S [tcp sum ok]
> 2090300578:2090300578(0) win 5840 <mss 1460,sackOK,timestamp 8622046
> 0,nop,wscale 7>

> 19:57:08.595922 IP (tos 0x0, ttl ?58, id 64381, offset 0, flags [DF],
> proto 6, length: 52) client.43425 > server.9999: . [tcp sum ok]
> 2090300579:2090300579(0) ack 1191201502 win 46 <nop,nop,timestamp
> 8622049 2164260539>

> 19:57:14.223948 IP (tos 0x0, ttl ?58, id 64382, offset 0, flags [DF],
> proto 6, length: 52) client.43425 > server.9999: F [tcp sum ok] 0:0(0)
> ack 1 win 46 <nop,nop,timestamp 8623457 2164260539>

> 19:57:14.236287 IP (tos 0x0, ttl ?58, id 64383, offset 0, flags [DF],
> proto 6, length: 52) client.43425 > server.9999: . [tcp sum ok] 1:1(0)
> ack 2 win 46 <nop,nop,timestamp 8623459 2164266179>

> As you can see, the server ignores the first SYNs but at some point,
> for no apparent reason, finally replies with aSYN/ACK.

> Does anyone have any clue?

> Thank you.

> Informations on the server:
> CentOS release 4.6 (Final)


> 2008 i686 i686 i386 GNU/Linux

> dmesg (partial):
> divert: allocating divert_blk for eth0
> e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
> ACPI: PCI Interrupt 0000:00:1b.0[A] -> GSI 21 (level, low) -> IRQ 193
> PCI: Setting latency timer of device 0000:00:1b.0 to 64
> e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex,
> Flow Control: RX
> e1000: eth0: e1000_watchdog_task: 10/100 speed: disabling TSO

Additionally, on the client:

$ netstat -anp
tcp        0      1 client:42776     server:443        SYN_SENT

 
 
 

Linux ignoring SYN packets

Post by Centurio » Fri, 23 May 2008 07:21:22



> Hello all.

> I am facing a very puzzling problem.

> Scenario:
> on the server I have a program listening on port 9999
> a client tries to connect (via telnet) to the server on port 9999

> Sometimes, establishing the connection takes a long time (one minute
> or more).
> What happens is that the client sends a SYN packet, the syn packet is
> received by the server, but the server does not reply to the SYN
> packet with a SYN/ACK (the program listening on the server does not
> see the incoming connection).

> After 4/5 SYN sents by the client, the servers finally answers with a
> SYN/ACK, the connection is established and the program on the server
> sees the connection).

I've seen this before and based on your later follow-up in this thread
that this affects SSH and HTTP connections too it leads me to suggest a
DNS problem.

The listening server may be trying to do a reverse lookup on the
client's IP.  If the resolver can't reach the DNS servers, it will wait
for them to time out.  After the time out, the server process abandons
the reverse lookup and accepts the connection (if it is configured to
allow unresolvable IP's).  However, while the server is waiting for DNS
the client will very likely send more SYN packets assuming the packet
wasn't received as it wasn't responded to in a timely fashion.

Check your DNS/resolver config :)

Cheers,

James

 
 
 

Linux ignoring SYN packets

Post by Rick Jone » Fri, 23 May 2008 08:46:26



> I've seen this before and based on your later follow-up in this
> thread that this affects SSH and HTTP connections too it leads me to
> suggest a DNS problem.
> The listening server may be trying to do a reverse lookup on the
> client's IP.  If the resolver can't reach the DNS servers, it will
> wait for them to time out.  After the time out, the server process
> abandons the reverse lookup and accepts the connection (if it is
> configured to allow unresolvable IP's).  However, while the server
> is waiting for DNS the client will very likely send more SYN packets
> assuming the packet wasn't received as it wasn't responded to in a
> timely fashion.

Does Linux actually have a mode to allow connection accept in that
fashion?  I always thought it waited at least until the SYN|ACK was
sent in response to the SYN before the listening application was
notified there was anything to consider accepting.  IIRC there is no
means in the standard sockets interface to do such a split accept() in
the first place?

rick jones
--
web2.0 n, the dot.com reunion tour...
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

 
 
 

Linux ignoring SYN packets

Post by ErMej » Fri, 23 May 2008 17:44:06


Quote:> I've seen this before and based on your later follow-up in this thread
> that this affects SSH and HTTP connections too it leads me to suggest a
> DNS problem.

Basically, as far as I can tell, it affects any kind of program
listening on a socket on the server.

Quote:> The listening server may be trying to do a reverse lookup on the
> client's IP. ?If the resolver can't reach the DNS servers, it will wait
> for them to time out. ?After the time out, the server process abandons
> the reverse lookup and accepts the connection (if it is configured to
> allow unresolvable IP's). ?However, while the server is waiting for DNS
> the client will very likely send moreSYNpackets assuming the packet
> wasn't received as it wasn't responded to in a timely fashion.
> Check your DNS/resolver config :)

Good guess, we also experienced this problem.

But it seems it is not the case now.

During the tests, I run also a:

 tcpdump -s 0 -vv -i eth0 host ns1 or host ns2 or host ns3

(where ns1-3 are my name servers).
During the tests, nor the listening program nor the server does any
DNS lookup.

Bye

 
 
 

Linux ignoring SYN packets

Post by ErMej » Fri, 23 May 2008 17:45:39


Quote:> DoesLinuxactually have a mode to allow connection accept in that
> fashion? ?I always thought it waited at least until theSYN|ACK was
> sent in response to theSYNbefore the listening application was
> notified there was anything to consider accepting. ?IIRC there is no
> means in the standard sockets interface to do such a split accept() in
> the first place?

I do not know if such a 'split accept' exists, anyway my test server
program uses a normal 'accept', i.e. it does not receive anything
untile the 3-way handshake is complete.

Bye

 
 
 

1. Create TCP syn packet with given seq num and few other TCP parameters

Hi All,
Can somebody please tell me how can i create a TCP syn packet  with
given seq number  and few other para meters set  with value provided by
my code ??
This is to be done from my kernel module which inserts its hooks using
Netfilter.
I looked at tcp_output.c and few other files but there doesn't seem to
be a very clean interface
which i can directly use from my module .

Can somebody point me the to way in which this should be done without
much efforts ??
I mean somebody must have done similar kind of thing . Is there
something i can use directly ??

Regards,
Rohit

2. Time for kent.general?

3. SYN packet problem

4. dos partition accidentally killed

5. SYN packets dropping into a black hole

6. Problems with the BusLogic driver(s)

7. My machine doesnt respond to any SYN or ICMP packets!

8. Upstream system node to be eating packets

9. removing old SYN packets

10. repeated SYN packets to port 80

11. 2.2.19 pre13 doesn't like retransmitted SYN ACK packets

12. DRDoS detection (was: repeated SYN packets to port 80)

13. SYN packet filtering