Routing issues - ping works one way but not the other

Routing issues - ping works one way but not the other

Post by David Brow » Wed, 13 Oct 2010 18:37:30



I've got a routing issue that I can't quite figure out.  My (very
simplified) setup is this:

Box A is on 192.168.0.1 and is the router, dns server, etc. for the
192.168.0.x network.  It has other interfaces as well, for access
further into the network and out onto the internet.  iptables are set to
allow all traffic in, out and forwarded.  There is a "route -net
192.168.1.0/24 gw 192.168.0.2" in the route table.

Box B is on 192.168.0.2 with 192.168.0.1 as the default gateway.  It's a
client machine on the 192.168.0.x network.

Box C is a router with two ports.  One is at 192.168.0.3, the other is
192.168.1.1.  iptables are set to allow all traffic in, out and
forwarded.  The default route is set to 192.168.0.1 (box A).

Box D is on 192.168.1.2 with 192.168.1.1 as the default gateway.  It's a
client machine on the 192.168.1.x network.

If I log into box B, and type "ping D" I get a response.  The route
flow is B to A (the default gateway), then to C (due to the specific
route command), then to D.  I've confirmed this path with traceroute.

If I log into box D and type "ping B", I get no response.  Traceroute
shows the flow from D to C as expected, but nothing beyond that.  I know
that the packet is going directly from C to B (see below for how I
know), as expected.  But somehow the reply from box B is not getting
back to D.

(If I ping from box D to something outside these networks, accessed
through router A, it works fine.)

I can't see why I can ping one way, and not the other.

I can see that the paths from B to D and from D to B are asymmetrical -
when box B wants to send to D, the path goes through A and then C due to
the default route, while on the path from D to B, the route goes
directly from C to D and misses out A, since the network segments match.

But since pings work fine from B to D, this must mean that both the
outgoing and return paths are working as expected.  Why then does it not
work when the orders are reversed?

If I run "route -net 192.168.1.0/24 gw 192.168.0.2" on B, then pings
work properly both ways.  The routes are then symmetrical.  But putting
extra routes on B does not scale well if there are many B's and many C's
in the network.

Thanks for any ideas here - this is greatly annoying me.

 
 
 

Routing issues - ping works one way but not the other

Post by Pascal Hambour » Wed, 13 Oct 2010 20:06:28


Hello,

David Brown a crit :

Quote:> I've got a routing issue that I can't quite figure out.  My (very
> simplified) setup is this:

> Box A is on 192.168.0.1 and is the router, dns server, etc. for the
> 192.168.0.x network.  It has other interfaces as well, for access
> further into the network and out onto the internet.  iptables are set to
> allow all traffic in, out and forwarded.  There is a "route -net
> 192.168.1.0/24 gw 192.168.0.2" in the route table.

Don't you mean "192.168.1.0/24 gw 192.168.0.3" ?

Quote:> Box B is on 192.168.0.2 with 192.168.0.1 as the default gateway.  It's a
> client machine on the 192.168.0.x network.

> Box C is a router with two ports.  One is at 192.168.0.3, the other is
> 192.168.1.1.  iptables are set to allow all traffic in, out and
> forwarded.  The default route is set to 192.168.0.1 (box A).

> Box D is on 192.168.1.2 with 192.168.1.1 as the default gateway.  It's a
> client machine on the 192.168.1.x network.

> If I log into box B, and type "ping D" I get a response.  The route
> flow is B to A (the default gateway), then to C (due to the specific
> route command), then to D.  I've confirmed this path with traceroute.

> If I log into box D and type "ping B", I get no response.  Traceroute
> shows the flow from D to C as expected, but nothing beyond that.  I know
> that the packet is going directly from C to B (see below for how I
> know), as expected.  But somehow the reply from box B is not getting
> back to D.

> (If I ping from box D to something outside these networks, accessed
> through router A, it works fine.)

What about ping A from D ? I guess it works fine ?

Quote:> I can't see why I can ping one way, and not the other.

Is there some NAT or stateful filtering on the box A and box C ? These
don't work well with asymmetric routing.
Can you run tcpdump on the boxes and see what's going on ?

Quote:> If I run "route -net 192.168.1.0/24 gw 192.168.0.2" on B, then pings
> work properly both ways.

Then I guess it rules out box B not replying to ping at all.

 
 
 

Routing issues - ping works one way but not the other

Post by David Brow » Wed, 13 Oct 2010 21:11:48



Quote:> Hello,

> David Brown a crit :
>> I've got a routing issue that I can't quite figure out.  My (very
>> simplified) setup is this:

>> Box A is on 192.168.0.1 and is the router, dns server, etc. for the
>> 192.168.0.x network.  It has other interfaces as well, for access
>> further into the network and out onto the internet.  iptables are set to
>> allow all traffic in, out and forwarded.  There is a "route -net
>> 192.168.1.0/24 gw 192.168.0.2" in the route table.

> Don't you mean "192.168.1.0/24 gw 192.168.0.3" ?

Yes.  That was just a typo in my translation from the real addresses to
simpler example address (I've changed the names and addresses to
simplify matters, and to protect the guilty parties!)

Quote:

>> Box B is on 192.168.0.2 with 192.168.0.1 as the default gateway.  It's a
>> client machine on the 192.168.0.x network.

>> Box C is a router with two ports.  One is at 192.168.0.3, the other is
>> 192.168.1.1.  iptables are set to allow all traffic in, out and
>> forwarded.  The default route is set to 192.168.0.1 (box A).

>> Box D is on 192.168.1.2 with 192.168.1.1 as the default gateway.  It's a
>> client machine on the 192.168.1.x network.

>> If I log into box B, and type "ping D" I get a response.  The route
>> flow is B to A (the default gateway), then to C (due to the specific
>> route command), then to D.  I've confirmed this path with traceroute.

>> If I log into box D and type "ping B", I get no response.  Traceroute
>> shows the flow from D to C as expected, but nothing beyond that.  I know
>> that the packet is going directly from C to B (see below for how I
>> know), as expected.  But somehow the reply from box B is not getting
>> back to D.

>> (If I ping from box D to something outside these networks, accessed
>> through router A, it works fine.)

> What about ping A from D ? I guess it works fine ?

Yes - all other combinations of pinging works find - only D to B fails.

Quote:>> I can't see why I can ping one way, and not the other.

> Is there some NAT or stateful filtering on the box A and box C ? These
> don't work well with asymmetric routing.

There is no NAT or any kind of filtering on box C - everything passing
through is forwarded directly.  Box A does have filtering and NAT, but
not on the interfaces in question (though see below for an update).

Quote:> Can you run tcpdump on the boxes and see what's going on ?

Not easily - the actual boxes in question are somewhat old (such as
Debian Etch and an ancient Red Hat - if it ain't broke, don't fix it)
and others are small OpenWRT routers.  But if it comes down to that,
I'll put in other machines or test out the same arrangement with
VirtualBox machines where I can mess around without fear of disrupting
the working systems.

Quote:>> If I run "route -net 192.168.1.0/24 gw 192.168.0.2" on B, then pings
>> work properly both ways.

> Then I guess it rules out box B not replying to ping at all.

That's my conclusion too.

I've been doing some more testing while writing this reply, and I now
know where the reply from B is being stopped.

A is refusing to forward it from B to C because of the iptables rule
"iptables -A FORWARD -m state --state INVALID -j DROP".  I have always
used this rule (and the same for INPUT and OUTPUT chains) at the start
of iptables firewalls.

Assuming that is the case (and I'll do some more tests to make sure),
the question then is why is this reply packet being judged as invalid?

And if I am correct in thinking that dropping INVALID packets is
considered best practice, is there any risk in skipping that rule?  The
scope here is only for packets arriving and leaving on the same internal
LAN interface - anything on other interfaces or originating from outside
will still be dropped if it is INVALID.

Thanks,

David

 
 
 

Routing issues - ping works one way but not the other

Post by Pascal Hambour » Wed, 13 Oct 2010 22:07:45


David Brown a crit :


>> Is there some NAT or stateful filtering on the box A and box C ? These
>> don't work well with asymmetric routing.

> There is no NAT or any kind of filtering on box C - everything passing
> through is forwarded directly.  Box A does have filtering and NAT, but
> not on the interfaces in question (though see below for an update).
[...]
> A is refusing to forward it from B to C because of the iptables rule
> "iptables -A FORWARD -m state --state INVALID -j DROP".  I have always
> used this rule (and the same for INPUT and OUTPUT chains) at the start
> of iptables firewalls.

> Assuming that is the case (and I'll do some more tests to make sure),
> the question then is why is this reply packet being judged as invalid?

Because box A's connection tracking state machine did not see the echo
request it replies to, due to the asymmetric routing. In the other way,
box A sees the echo request which has state NEW, and does not see the
echo reply, but that does not matter.

Quote:> And if I am correct in thinking that dropping INVALID packets is
> considered best practice, is there any risk in skipping that rule?  The
> scope here is only for packets arriving and leaving on the same internal
> LAN interface - anything on other interfaces or originating from outside
> will still be dropped if it is INVALID.

You can safely ACCEPT any packet arriving and leaving on the same
internal LAN interface, regardless of its state.
 
 
 

Routing issues - ping works one way but not the other

Post by David Brow » Wed, 13 Oct 2010 23:01:25



> David Brown a crit :

>>> Is there some NAT or stateful filtering on the box A and box C ? These
>>> don't work well with asymmetric routing.

>> There is no NAT or any kind of filtering on box C - everything passing
>> through is forwarded directly.  Box A does have filtering and NAT, but
>> not on the interfaces in question (though see below for an update).
> [...]
>> A is refusing to forward it from B to C because of the iptables rule
>> "iptables -A FORWARD -m state --state INVALID -j DROP".  I have always
>> used this rule (and the same for INPUT and OUTPUT chains) at the start
>> of iptables firewalls.

>> Assuming that is the case (and I'll do some more tests to make sure),
>> the question then is why is this reply packet being judged as invalid?

> Because box A's connection tracking state machine did not see the echo
> request it replies to, due to the asymmetric routing. In the other way,
> box A sees the echo request which has state NEW, and does not see the
> echo reply, but that does not matter.

That makes sense.  The "INVALID" check is doing a bit more than I
thought - I thought it was just format checking for the basic telegram.
  But if it is part of the connection tracking mechanism, then I can now
see why the reply is marked INVALID.

Quote:>> And if I am correct in thinking that dropping INVALID packets is
>> considered best practice, is there any risk in skipping that rule?  The
>> scope here is only for packets arriving and leaving on the same internal
>> LAN interface - anything on other interfaces or originating from outside
>> will still be dropped if it is INVALID.

> You can safely ACCEPT any packet arriving and leaving on the same
> internal LAN interface, regardless of its state.

Okay.

Thank you for your explanation here - I now have the routing working as
I want, and I know why it didn't work before.

mvh.,

David

 
 
 

Routing issues - ping works one way but not the other

Post by Andrew Gideo » Wed, 13 Oct 2010 23:35:05



> Because box A's connection tracking state machine did not see the echo
> request it replies to, due to the asymmetric routing. In the other way,
> box A sees the echo request which has state NEW, and does not see the
> echo reply, but that does not matter.

Is there any way to "fix" this by sharing connection state amongst
multiple routers?  I'm imaging that this would be tough at best given the
speeds involved.  Packets used to share state between routers would have
to be quicker than the reply packets to newly established connections.

On the other hand, this sounds like a fairly common problem.

        - Andrew

 
 
 

Routing issues - ping works one way but not the other

Post by Pascal Hambour » Thu, 14 Oct 2010 00:16:36


Andrew Gideon a crit :


>> Because box A's connection tracking state machine did not see the echo
>> request it replies to, due to the asymmetric routing. In the other way,
>> box A sees the echo request which has state NEW, and does not see the
>> echo reply, but that does not matter.

> Is there any way to "fix" this by sharing connection state amongst
> multiple routers?

Check conntrackd from conntrack-tools.
<http://conntrack-tools.netfilter.org/>
 
 
 

Routing issues - ping works one way but not the other

Post by David Schwart » Thu, 14 Oct 2010 13:49:03



Quote:> Is there any way to "fix" this by sharing connection state amongst
> multiple routers? ?I'm imaging that this would be tough at best given the
> speeds involved. ?Packets used to share state between routers would have
> to be quicker than the reply packets to newly established connections.

Yes, but no.

There is way to share connection tracking state. But for precisely the
reasons you explained, it is useless to cure the asymmetric routing
problem. It is, however, useful to solve the 'long-lived connections
break if a firewall fails over to a backup' problem.

IMO, connection tracking should *never* be performed by any device
that isn't the sole path (so long as it doesn't fail) both into and
out of a stub network.

DS

 
 
 

Routing issues - ping works one way but not the other

Post by David Brow » Thu, 14 Oct 2010 19:20:49




>> Is there any way to "fix" this by sharing connection state amongst
>> multiple routers?  I'm imaging that this would be tough at best given the
>> speeds involved.  Packets used to share state between routers would have
>> to be quicker than the reply packets to newly established connections.

> Yes, but no.

> There is way to share connection tracking state. But for precisely the
> reasons you explained, it is useless to cure the asymmetric routing
> problem. It is, however, useful to solve the 'long-lived connections
> break if a firewall fails over to a backup' problem.

> IMO, connection tracking should *never* be performed by any device
> that isn't the sole path (so long as it doesn't fail) both into and
> out of a stub network.

That's what I've got now - the router does connection tracking for paths
through it (and there it is the sole path), while I've short-circuited
the connection tracking for packets that go in and out again on the
local LAN interface.
 
 
 

1. Ethernet Routing: ping works one way but not back!

Folks,

B can ping A or B. C can ping B using either of B's IP addresses. When C tries
to ping A, A's lights flash just the same way as when B pings A; but C just
gets a timeout message.

A                                       B               C
ISDN PBX                Redhat Linux 5.0 (2.0.32)       Win95
                        eth0 ne2000     eth1 ne1000
192.168.42.1            192.168.42.100  10.0.2.100      10.0.2.166
10baseT                 10baseT         10base2         10base2

(I hope the diagram works for you. It does for me, 78-wide with a fixed-pitch
font. But then, I can understand my children and you can't).

Since A is clearly receiving C's pings, it seems to me that the cause is
likely to be that B is not forwarding A's responses on to C. Why?

B has static routes set up:


Kernel IP routing table

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface

10.0.2.0        0.0.0.0         255.255.255.0   U      1500 0          0 eth0

10.0.2.0        10.0.2.100      255.255.255.0   UG     1500 0          0 eth0

192.168.42.0    0.0.0.0         255.255.255.0   U      1500 0          0 eth1

192.168.42.0    192.168.42.100  255.255.255.0   UG     1500 0          0 eth1

127.0.0.0       0.0.0.0         255.0.0.0       U      3584 0          0 lo

0.0.0.0         10.0.2.100      0.0.0.0         UG     1500 0          0 eth0

To me these look as if B is set up to route both ways properly.

IPv4 Forwarding is on. Dual cards are enabled via /etc/conf.modules. The LAN
cards are on seperate (unique) IRQs and their io ranges don't overlap with
each other or anything else. One, of course, is 16-bit and the other 8-bit but
the ne driver seems to identify each correctly.

Bridging is not enabled in the Kernel (I've not yet been able to rebuild the
kernel because of make "No rule to make target 'config'" errors- that's
another story). Do I need it?

I told C that B is a gateway. There's nothing I can change at A until I can
talk to it with C! A's only user interface is through Win95 software and the
10baseT port!

Any ideas?
--
Graham Harris
Equinus- Today's Business & Technology Consultancy for the Travel industry
http://www.equinus.com

2. w2k + mdk8 hell

3. Can I prevent pinging from others and still ping others?

4. How to print to the printer connected with the terminal?

5. alarm() does not work/ping only sends one packet on SMP machine (2.2.16)

6. Problems with #9 GXE64Pro resolved?

7. Ping work one way but not the other...

8. Problems getting printer working

9. ping beyond one hop is not working

10. Setting up SLIP, Ping only works one one of the machines

11. Can't ping local host but can ping others

12. D-link DE220 can PING his own IP, but can't PING others'

13. Can ping linux itself, but can't ping others