Very Computer

I've got a routing issue that I can't quite figure out.  My (very
simplified) setup is this:

Box A is on 192.168.0.1 and is the router, dns server, etc. for the
192.168.0.x network.  It has other interfaces as well, for access
further into the network and out onto the internet.  iptables are set to
allow all traffic in, out and forwarded.  There is a "route -net
192.168.1.0/24 gw 192.168.0.2" in the route table.

Box B is on 192.168.0.2 with 192.168.0.1 as the default gateway.  It's a
client machine on the 192.168.0.x network.

Box C is a router with two ports.  One is at 192.168.0.3, the other is
192.168.1.1.  iptables are set to allow all traffic in, out and
forwarded.  The default route is set to 192.168.0.1 (box A).

Box D is on 192.168.1.2 with 192.168.1.1 as the default gateway.  It's a
client machine on the 192.168.1.x network.

If I log into box B, and type "ping D" I get a response.  The route
flow is B to A (the default gateway), then to C (due to the specific
route command), then to D.  I've confirmed this path with traceroute.

If I log into box D and type "ping B", I get no response.  Traceroute
shows the flow from D to C as expected, but nothing beyond that.  I know
that the packet is going directly from C to B (see below for how I
know), as expected.  But somehow the reply from box B is not getting
back to D.

(If I ping from box D to something outside these networks, accessed
through router A, it works fine.)

I can't see why I can ping one way, and not the other.

I can see that the paths from B to D and from D to B are asymmetrical -
when box B wants to send to D, the path goes through A and then C due to
the default route, while on the path from D to B, the route goes
directly from C to D and misses out A, since the network segments match.

But since pings work fine from B to D, this must mean that both the
outgoing and return paths are working as expected.  Why then does it not
work when the orders are reversed?

If I run "route -net 192.168.1.0/24 gw 192.168.0.2" on B, then pings
work properly both ways.  The routes are then symmetrical.  But putting
extra routes on B does not scale well if there are many B's and many C's
in the network.

Thanks for any ideas here - this is greatly annoying me.

Hello,

David Brown a crit :

Quote:> I've got a routing issue that I can't quite figure out.  My (very
> simplified) setup is this:

> Box A is on 192.168.0.1 and is the router, dns server, etc. for the
> 192.168.0.x network.  It has other interfaces as well, for access
> further into the network and out onto the internet.  iptables are set to
> allow all traffic in, out and forwarded.  There is a "route -net
> 192.168.1.0/24 gw 192.168.0.2" in the route table.

Don't you mean "192.168.1.0/24 gw 192.168.0.3" ?

Quote:> Box B is on 192.168.0.2 with 192.168.0.1 as the default gateway.  It's a
> client machine on the 192.168.0.x network.

> Box C is a router with two ports.  One is at 192.168.0.3, the other is
> 192.168.1.1.  iptables are set to allow all traffic in, out and
> forwarded.  The default route is set to 192.168.0.1 (box A).

> Box D is on 192.168.1.2 with 192.168.1.1 as the default gateway.  It's a
> client machine on the 192.168.1.x network.

> If I log into box B, and type "ping D" I get a response.  The route
> flow is B to A (the default gateway), then to C (due to the specific
> route command), then to D.  I've confirmed this path with traceroute.

> If I log into box D and type "ping B", I get no response.  Traceroute
> shows the flow from D to C as expected, but nothing beyond that.  I know
> that the packet is going directly from C to B (see below for how I
> know), as expected.  But somehow the reply from box B is not getting
> back to D.

> (If I ping from box D to something outside these networks, accessed
> through router A, it works fine.)

What about ping A from D ? I guess it works fine ?

Quote:> I can't see why I can ping one way, and not the other.

Is there some NAT or stateful filtering on the box A and box C ? These
don't work well with asymmetric routing.
Can you run tcpdump on the boxes and see what's going on ?

Quote:> If I run "route -net 192.168.1.0/24 gw 192.168.0.2" on B, then pings
> work properly both ways.

Then I guess it rules out box B not replying to ping at all.

Quote:> Hello,

> David Brown a crit :
>> I've got a routing issue that I can't quite figure out.  My (very
>> simplified) setup is this:

>> Box A is on 192.168.0.1 and is the router, dns server, etc. for the
>> 192.168.0.x network.  It has other interfaces as well, for access
>> further into the network and out onto the internet.  iptables are set to
>> allow all traffic in, out and forwarded.  There is a "route -net
>> 192.168.1.0/24 gw 192.168.0.2" in the route table.

> Don't you mean "192.168.1.0/24 gw 192.168.0.3" ?

Yes.  That was just a typo in my translation from the real addresses to
simpler example address (I've changed the names and addresses to
simplify matters, and to protect the guilty parties!)

Quote:

>> Box B is on 192.168.0.2 with 192.168.0.1 as the default gateway.  It's a
>> client machine on the 192.168.0.x network.

>> Box C is a router with two ports.  One is at 192.168.0.3, the other is
>> 192.168.1.1.  iptables are set to allow all traffic in, out and
>> forwarded.  The default route is set to 192.168.0.1 (box A).

>> Box D is on 192.168.1.2 with 192.168.1.1 as the default gateway.  It's a
>> client machine on the 192.168.1.x network.

>> If I log into box B, and type "ping D" I get a response.  The route
>> flow is B to A (the default gateway), then to C (due to the specific
>> route command), then to D.  I've confirmed this path with traceroute.

>> If I log into box D and type "ping B", I get no response.  Traceroute
>> shows the flow from D to C as expected, but nothing beyond that.  I know
>> that the packet is going directly from C to B (see below for how I
>> know), as expected.  But somehow the reply from box B is not getting
>> back to D.

>> (If I ping from box D to something outside these networks, accessed
>> through router A, it works fine.)

> What about ping A from D ? I guess it works fine ?

Yes - all other combinations of pinging works find - only D to B fails.

Quote:>> I can't see why I can ping one way, and not the other.

> Is there some NAT or stateful filtering on the box A and box C ? These
> don't work well with asymmetric routing.

There is no NAT or any kind of filtering on box C - everything passing
through is forwarded directly.  Box A does have filtering and NAT, but
not on the interfaces in question (though see below for an update).

Quote:> Can you run tcpdump on the boxes and see what's going on ?

Not easily - the actual boxes in question are somewhat old (such as
Debian Etch and an ancient Red Hat - if it ain't broke, don't fix it)
and others are small OpenWRT routers.  But if it comes down to that,
I'll put in other machines or test out the same arrangement with
VirtualBox machines where I can mess around without fear of disrupting
the working systems.

Quote:>> If I run "route -net 192.168.1.0/24 gw 192.168.0.2" on B, then pings
>> work properly both ways.

> Then I guess it rules out box B not replying to ping at all.

That's my conclusion too.

I've been doing some more testing while writing this reply, and I now
know where the reply from B is being stopped.

A is refusing to forward it from B to C because of the iptables rule
"iptables -A FORWARD -m state --state INVALID -j DROP".  I have always
used this rule (and the same for INPUT and OUTPUT chains) at the start
of iptables firewalls.

Assuming that is the case (and I'll do some more tests to make sure),
the question then is why is this reply packet being judged as invalid?

And if I am correct in thinking that dropping INVALID packets is
considered best practice, is there any risk in skipping that rule?  The
scope here is only for packets arriving and leaving on the same internal
LAN interface - anything on other interfaces or originating from outside
will still be dropped if it is INVALID.

Thanks,

David

David Brown a crit :

Because box A's connection tracking state machine did not see the echo
request it replies to, due to the asymmetric routing. In the other way,
box A sees the echo request which has state NEW, and does not see the
echo reply, but that does not matter.

Quote:> And if I am correct in thinking that dropping INVALID packets is
> considered best practice, is there any risk in skipping that rule?  The
> scope here is only for packets arriving and leaving on the same internal
> LAN interface - anything on other interfaces or originating from outside
> will still be dropped if it is INVALID.

You can safely ACCEPT any packet arriving and leaving on the same
internal LAN interface, regardless of its state.

That makes sense.  The "INVALID" check is doing a bit more than I
thought - I thought it was just format checking for the basic telegram.
  But if it is part of the connection tracking mechanism, then I can now
see why the reply is marked INVALID.

Quote:>> And if I am correct in thinking that dropping INVALID packets is
>> considered best practice, is there any risk in skipping that rule?  The
>> scope here is only for packets arriving and leaving on the same internal
>> LAN interface - anything on other interfaces or originating from outside
>> will still be dropped if it is INVALID.

> You can safely ACCEPT any packet arriving and leaving on the same
> internal LAN interface, regardless of its state.

Okay.

Thank you for your explanation here - I now have the routing working as
I want, and I know why it didn't work before.

mvh.,

David

Is there any way to "fix" this by sharing connection state amongst
multiple routers?  I'm imaging that this would be tough at best given the
speeds involved.  Packets used to share state between routers would have
to be quicker than the reply packets to newly established connections.

On the other hand, this sounds like a fairly common problem.

        - Andrew

Andrew Gideon a crit :

Check conntrackd from conntrack-tools.
<http://conntrack-tools.netfilter.org/>

Quote:> Is there any way to "fix" this by sharing connection state amongst
> multiple routers? ?I'm imaging that this would be tough at best given the
> speeds involved. ?Packets used to share state between routers would have
> to be quicker than the reply packets to newly established connections.

Yes, but no.

There is way to share connection tracking state. But for precisely the
reasons you explained, it is useless to cure the asymmetric routing
problem. It is, however, useful to solve the 'long-lived connections
break if a firewall fails over to a backup' problem.

IMO, connection tracking should *never* be performed by any device
that isn't the sole path (so long as it doesn't fail) both into and
out of a stub network.

DS

That's what I've got now - the router does connection tracking for paths
through it (and there it is the sole path), while I've short-circuited
the connection tracking for packets that go in and out again on the
local LAN interface.