Packet "Mixing" Between Multiple NICs on Host

Packet "Mixing" Between Multiple NICs on Host

Post by Jonathan B. Hore » Wed, 31 Mar 2004 19:34:35



Shalom!

In, perhaps, a misguided desire for elegance, I moved our DNS server from an
aged and infirm host onto our existing file/mail server (Dell PowerEdge/2450
w/dual PIII/866 CPUs and 2GB RAM, running Fedora Core 1 w/all updates).  Since
the DNS server sat on a different subnet, I added a second NIC to the
file/mail server and created the appropriate files in
/etc/sysconfig/network-scripts (ifcfg-eth1 and route-eth{0,1}).

"route -n" shows:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
128.139.197.0   0.0.0.0         255.255.255.0   U     0      0        0 eth1
128.139.206.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         128.139.197.16  0.0.0.0         UG    0      0        0 eth1
0.0.0.0         128.139.206.1   0.0.0.0         UG    1      0        0 eth0

The problem is that although packets received from the two subnets arrive
through the corresponding device, *packets sent to a host on a subnet other
than 128.139.197.0 exit through eth1.*

Running "ping 128.139.206.12" from a host on the 128.139.206.0 subnet shows
that packets exit via eth1, rather than via eth0:


tcpdump: listening on eth1
11:54:38.192269 efes.iucc.ac.il > horen.tau.ac.il: icmp: echo reply
11:54:39.202538 efes.iucc.ac.il > horen.tau.ac.il: icmp: echo reply
11:54:40.212855 efes.iucc.ac.il > horen.tau.ac.il: icmp: echo reply

I understand that this is because the metric for eth1 is "0", while the metric
for eth1 is "1".  If I understand correctly, changing the metric for eth0 to
"0" would mean that every packet would be sent to *both* interfaces, giving me
a 50% packet loss.

Is there a way to configure routing on this server so that a packet's
source-address is "honored" by the system when responding?

worst-case, I'll cobble together a separate DNS server from an unused PIII/500...

TIA!

--
JONATHAN B. HOREN                            UNIX SYSTEMS ADMINISTRATOR

T: +972-(0)3-640-5203                               Tel-Aviv University
F: +972-(0)3-640-9118                           Ramat-Aviv 69978 Israel

 
 
 

Packet "Mixing" Between Multiple NICs on Host

Post by P Gent » Thu, 01 Apr 2004 01:53:04



> Shalom!

> In, perhaps, a misguided desire for elegance, I moved our DNS server from an
> aged and infirm host onto our existing file/mail server (Dell PowerEdge/2450
> w/dual PIII/866 CPUs and 2GB RAM, running Fedora Core 1 w/all updates).  Since
> the DNS server sat on a different subnet, I added a second NIC to the
> file/mail server and created the appropriate files in
> /etc/sysconfig/network-scripts (ifcfg-eth1 and route-eth{0,1}).

> "route -n" shows:

> Kernel IP routing table
> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
> 128.139.197.0   0.0.0.0         255.255.255.0   U     0      0        0 eth1
> 128.139.206.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
> 169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1
> 127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
> 0.0.0.0         128.139.197.16  0.0.0.0         UG    0      0        0 eth1
> 0.0.0.0         128.139.206.1   0.0.0.0         UG    1      0        0 eth0

> The problem is that although packets received from the two subnets arrive
> through the corresponding device, *packets sent to a host on a subnet other
> than 128.139.197.0 exit through eth1.*

> Running "ping 128.139.206.12" from a host on the 128.139.206.0 subnet shows
> that packets exit via eth1, rather than via eth0:


> tcpdump: listening on eth1
> 11:54:38.192269 efes.iucc.ac.il > horen.tau.ac.il: icmp: echo reply
> 11:54:39.202538 efes.iucc.ac.il > horen.tau.ac.il: icmp: echo reply
> 11:54:40.212855 efes.iucc.ac.il > horen.tau.ac.il: icmp: echo reply

> I understand that this is because the metric for eth1 is "0", while the metric
> for eth1 is "1".  If I understand correctly, changing the metric for eth0 to
> "0" would mean that every packet would be sent to *both* interfaces, giving me
> a 50% packet loss.

> Is there a way to configure routing on this server so that a packet's
> source-address is "honored" by the system when responding?

> worst-case, I'll cobble together a separate DNS server from an unused PIII/500...

> TIA!

Since your nics are on different segments, getting "correct" behavior
is do-able if a bit tedious the first time you use the necessary
features (ie., multiple routing tables).  As is, your basic problem is
that you're trying to do all your routing via the single "main"
routing table (252 more can be added) and use 2 default
route/gateways.

BTW, "tricks" for using the one main table are unreliable as the
algorithm used for route table processing is indeterminent re: "equal
prefix length" entries -- ie., first match is not assured and the
unequal metric trick is a failover hobble -> best to do it "right" as
below docs suggest.

Also a consideration re: a solution is that you're running (what could
be) 3 heavily used services on one machine -- might it prove better
and easier to run DNS on that aged PIII-500?  (BTW, my experience is
that such a box with good IO can be quite adequate and reliable --
sure runs much cooler than today's multi-GHz sizzlers!)

Anyway, you need to look at the following to understand what's going
on and how to "fix" it -- especially since you'll be the one having to
maintain it over time.

The venerable Advanced Routing HowTo:
http://lartc.org/howto/
or here for downloadable "All HowTos" which have other pertinent docs:
http://tldp.org/docs.html#howto

The best background/reference of iproute2 features and utils (still
incomplete):
http://linux-ip.net/

The "ip" command reference should be part of your Linux docs ->
ip-cref.ps or ip-cref.txt.  A handier (?) single html version here:
http://bookshelf.sleepnet.net/files/ip-cref.html

Once you get the hang of it, it's not hard to fix your problem and a
look at the above will reveal a wealth of Linux solutions for routing.

While you're at it, you may want to check for Netfilter/iptables
plug-ins (quite a range of pre-assembled solutions for some problems):
http://www.netfilter.org/patch-o-matic/
http://www.linuxguruz.com/iptables/

hth,
prg
email above disabled