Multiple NICs on one PCI Card & Linux Support

Multiple NICs on one PCI Card & Linux Support

Post by David Lee Lamber » Sat, 18 Apr 1998 04:00:00





> >I need to route between 5 ethernet segments.

> >1 is 10 mb and the other 4 are 100mb.

> >We're currently using an SGI for this, but we need more of it's
> >cycles elsewhere, and would like to replace it.

> >I've got a budgetary constraint, so a dedicated router is out of the
> >question.  I would like to use a PC, but I need to get enough
> >interfaces in it.  I will be buying 5 cards if I have to, but I was
> >hoping that there is a PCI-based 10 or 100 mb quad-card available,
> >and if so, if there are Linux or Solaris x86 drivers for it?
> I doubt if Linux can route or bridge 4 100Mb/s ports. Certainly not if you
> are using an older PC. That's why switches use hardware instead of software.

> Do you really need to route? I think a 4 port 100Mb/s hub and
> routing/bridging the 10Mb/s would be more effective.

Well,  an older,  ISA-based PC probably won't be able to do it.
Consider a 16-MHz 386sx:

16 MHz
16 bit data bus

= 256 Mb/s

I'd advise against using ISA cards in an i586 machine at 100Mbps.  The bus
bandwidth there is roughly as follows:

8.3 MHz (for AT compatibility)
16 bit data bus

= 132 Mb/s

Now,  a single NE2000 clone is OK,  generally.  However,  it would be bad
to try using several of them.

However,  a PCI-based system should be able to do it:




Now,  an i586 system with 60-ns memory and 64-bit FPM DRAM has a memory
bandwidth of 1066 Gbps.  Thus,  I don't see that there would be a problem
using a Linux box as a router between several Ethernet cards,  even Fast
Ethernet.  Now,  we may need a new bus architecture for Gigabit
Ethernet... mabey.

--
m/lamber45\100(egr|pilot)\.msu\.edu/ and print <<MHM16x20
David Lee Lambert -- Just another perl hacker
webstuph at http://web.egr.msu.edu/~lamber45
MHM16x20

 
 
 

Multiple NICs on one PCI Card & Linux Support

Post by Eric Gisi » Sat, 18 Apr 1998 04:00:00




>> I doubt if Linux can route or bridge 4 100Mb/s ports. Certainly not if
you
>> are using an older PC. That's why switches use hardware instead of
software.

>> Do you really need to route? I think a 4 port 100Mb/s hub and
>> routing/bridging the 10Mb/s would be more effective.

>However,  a PCI-based system should be able to do it:





The limiting factor is not bus/memory bandwidth, it's per-frame processing.
A single 100Mb/s port is typically 20K frames/s. If you can handle a frame
in 1000 instructions, then it trivial. More likely its 10000 instructions,
and software-based bridges and routers don't work.

 
 
 

Multiple NICs on one PCI Card & Linux Support

Post by David Lee Lamber » Tue, 21 Apr 1998 04:00:00





> >> I doubt if Linux can route or bridge 4 100Mb/s ports. Certainly not if
> you
> >> are using an older PC. That's why switches use hardware instead of
> software.

> >> Do you really need to route? I think a 4 port 100Mb/s hub and
> >> routing/bridging the 10Mb/s would be more effective.

> >However,  a PCI-based system should be able to do it:




> The limiting factor is not bus/memory bandwidth, it's per-frame processing.
> A single 100Mb/s port is typically 20K frames/s. If you can handle a frame
> in 1000 instructions, then it trivial. More likely its 10000 instructions,
> and software-based bridges and routers don't work.

With a proper bus-mastering network card (not an NE2000,  of course),  I
imagine that typical frame overhead,  even for a complicated firewall,
would be about 100 instructions plus any I/O from the processor.  An
Ethernet packet is 500 to 1500 bytes;  say the processor has to do a CRC
on it,  so it prefetches and puts 4 bytes at a time to memory or the card.
250 to 750 adds or xors or whatever.  The limit is still the cache misses
involved in getting the data off the bus,  or the DMA speed from one card
to another,  or something like that -- not the processor speed.  It might
be the processor speed on a 33MHz 486,  or on a 386SX,  but one can't use
PCI on an SX anyway;  one would use 16-bit NE2000 cards for that.  

Now if the processor is encrypting the packets, or doing application-level
translation, there may be a limit.  Also, disk I/O can't possibly keep up
with multiple 100base-TX cards.  However, I still maintain that a
non-swapping Linux system makes a great router.

By the way,  are there any Linux FC-AL drivers/boards?

--
m/lamber45\100(egr|pilot)\.msu\.edu/ and print <<MHM16x20
David Lee Lambert -- Just another perl hacker
webstuph at http://web.egr.msu.edu/~lamber45
MHM16x20

 
 
 

Multiple NICs on one PCI Card & Linux Support

Post by Wolfgang Rupprech » Tue, 21 Apr 1998 04:00:00



Quote:> With a proper bus-mastering network card (not an NE2000,  of course),  I
> imagine that typical frame overhead,  even for a complicated firewall,
> would be about 100 instructions plus any I/O from the processor.  

Van Jacobson had a paper (*) that claimed 37 sparc instructions, 11
loads, 2 stores for routing a packet.  So I guess that should be the
goal of any good router.  And yes, I've personally seen routers miss
that mark by 2 or 3 orders of magnitude.

As you said, it should be absolutely no problem for even an anemic CPU
to fill several 100BaseT pipes.

-wolfgang

* Jacobson, V., Design Changes to the Kernel Network Architecture for
4.4BSD (viewgraphs). 4.4BSD Class, Berkeley, CA, May 5 & 7, 1992.
http://ee.lbl.gov/nrg-talks.html

 
 
 

Multiple NICs on one PCI Card & Linux Support

Post by Eric Gisi » Tue, 21 Apr 1998 04:00:00






>> The limiting factor is not bus/memory bandwidth, it's per-frame
processing.
>> A single 100Mb/s port is typically 20K frames/s. If you can handle a
frame
>> in 1000 instructions, then it trivial. More likely its 10000
instructions,
>> and software-based bridges and routers don't work.

>With a proper bus-mastering network card (not an NE2000,  of course),
I
>imagine that typical frame overhead,  even for a complicated firewall,
>would be about 100 instructions plus any I/O from the processor.  An
>Ethernet packet is 500 to 1500 bytes;  say the processor has to do a
CRC
>on it,  so it prefetches and puts 4 bytes at a time to memory or the
card.
>250 to 750 adds or xors or whatever.  The limit is still the cache

misses

I seriously suggest you read the Linux (or any Unix) kernel source!
There is no way the MAC driver and IP routing or ethernet bridging code
are 100 instructions per frame, and 1000 is also unlikely.

I recently read the docs for a (old) DOS based ethernet bridge for
286/386 PCs. The author didn't use device drivers, he coded for one
specific device. He also coded the main loop in assembler, with macro
inlining instead of function calls. Bridging is not trivial, that's why
today's switches use hardware.

 
 
 

Multiple NICs on one PCI Card & Linux Support

Post by David Lee Lamber » Tue, 21 Apr 1998 04:00:00







> >> The limiting factor is not bus/memory bandwidth, it's per-frame
> processing.
> >> A single 100Mb/s port is typically 20K frames/s. If you can handle a
> frame
> >> in 1000 instructions, then it trivial. More likely its 10000
> instructions,
> >> and software-based bridges and routers don't work.

> >With a proper bus-mastering network card (not an NE2000,  of course),
> I
> >imagine that typical frame overhead,  even for a complicated firewall,
> >would be about 100 instructions plus any I/O from the processor.  An
> >Ethernet packet is 500 to 1500 bytes;  say the processor has to do a
> CRC
> >on it,  so it prefetches and puts 4 bytes at a time to memory or the
> card.
> >250 to 750 adds or xors or whatever.  The limit is still the cache
> misses

> I seriously suggest you read the Linux (or any Unix) kernel source!
> There is no way the MAC driver and IP routing or ethernet bridging code
> are 100 instructions per frame, and 1000 is also unlikely.

> I recently read the docs for a (old) DOS based ethernet bridge for
> 286/386 PCs. The author didn't use device drivers, he coded for one
> specific device. He also coded the main loop in assembler, with macro
> inlining instead of function calls. Bridging is not trivial, that's why
> today's switches use hardware.

In the 286,  DRAM and I/O ports were locked to the processor's bus cycle.
In a modern Intelish system,  the CPU is a multiple-pipeline device with a
multi-level cache,  often with dynamic branch prediction and instruction
rewriting.  In addition,  we now have optimizing compilers,  although an
assembly-recoding might be the best optimisation path.  Packets need not
even be transferred off the card,  in a multi-port board.  I've looked at
the source,  but most of it is concerned with card initialization and
debugging.  The ne.c driver has to generate wait states,  but the tulip
driver seems to just change pointers around -- perhaps five actual bus
cycles per packet,  for that part.  Right?

Anyway,  why do switches use hardware if "bridging is not trivial"?  I
though that hardware was used for trivial but speed-intensive tasks,
while software was used for more complex but slower tasks.

--
m/lamber45\100(egr|pilot)\.msu\.edu/ and print <<MHM16x20
David Lee Lambert -- Just another perl hacker
webstuph at http://web.egr.msu.edu/~lamber45
MHM16x20

 
 
 

Multiple NICs on one PCI Card & Linux Support

Post by Eric Gisi » Wed, 22 Apr 1998 04:00:00



>Van Jacobson had a paper (*) that claimed 37 sparc instructions, 11
>loads, 2 stores for routing a packet.  So I guess that should be the
>goal of any good router.  And yes, I've personally seen routers miss
>that mark by 2 or 3 orders of magnitude.

Thanks for some facts. I'll revise my original estimate of 1000
instructions.

We have 40 for routing, 20 for ARP, total 60. Bridging requires two hash
table lookups, and should also be around 60. Devices drivers are tricky,
even with bus mastering or shared memory it could be anywhere from 20 to
100. In the best case, that totals 100 instructions for routing or
bridging a packet.

10,000 incoming packets/s is 1M instructions/s.

Quote:>As you said, it should be absolutely no problem for even an anemic CPU
>to fill several 100BaseT pipes.

Agreed. So why does anyone buy an 8 port switch? Throw a few dual PCI
ethernet cards in an old Pentium with a floppy drive.
 
 
 

Multiple NICs on one PCI Card & Linux Support

Post by Peter E. Fr » Wed, 22 Apr 1998 04:00:00



> Agreed. So why does anyone buy an 8 port switch? Throw a few dual PCI
> ethernet cards in an old Pentium with a floppy drive.

  Awwww...  I just got an 8x10Mb+2x100Mb (unmanaged) switch for $250
(auction).  I was planning to use (and actually had set up) a
Linux-based box as a router (with two DEC 21140s and two PCNet-PCIs) --
but the switch is a tad easier to deal with.  It also collapses nicely
into a flat addressing scheme -- you should have seen my 4+3-subnet
plan... (I like routing...)
  I suppose I could have looked into bridging with Linux -- my
experience with bridging devices has not been pleasant, so the word
"bridge" pretty much turns me off.  If it acts as a neat MAC-level
bridge, though, it should work approximately as well as the switch.
  Don't mention dual or quad ethernet cards, though -- your switch'll be
cheaper these days.
  Incidentally, I still have one 21140 and one PCNet-PCI in the machine,
for firewalling.  Something Linux does a tad better than the switch.

Peter E. Fry

 
 
 

Multiple NICs on one PCI Card & Linux Support

Post by Wolfgang Rupprech » Wed, 22 Apr 1998 04:00:00



> We have 40 for routing, 20 for ARP, total 60.

Arp in the steady state is virtually free.  It's only the first ARP
that takes many instructions.  In a well thought out system one would
get the MAC address from the same forwarding table that got one the
outgoing interface.  Thats one (or two) indexed read instructions and
6bytes of data fetch and two write instructions (a 4-byte and a
2-byte).  Careful alignments and 8-byte long-long instructions can
recover an extra instruction in the read phase.

In a really speed-optimized setup one would keep the first 12 bytes
(perhaps rounded up to 16) of the ethernet packet in the forwarding
table.  One would just slap this over the old ether header and
re-queue this onto the output buffer ring.

For IP routing one also needs to decrement the TTL and change the
header checksum by a constant, and test the TTL for zero.  One also
needs to check that the ip version is 0x45.  If not the packet
requires special handling.

Quote:> Bridging requires two hash table lookups, and should also be around
> 60.

It should be possible to redo the hash code to be *much* better than
that.

#define hash(x) (((x) ^ ((x) >> HASHBITS)) & HASHMASK)

This should be 3 instructions and 2 immediate operands.

Quote:> Devices drivers are tricky, even with bus mastering or shared
> memory it could be anywhere from 20 to 100.

The device drivers in a well thought out hardware are simply adding a
pointer to the next free slot of the ethernet chip's output buffer
descriptor ring.

Of course we are talking about good chips like the TULIP 10/100 chip
(or the LANCE in vanj's case).

Quote:> In the best case, that totals 100 instructions for routing or
> bridging a packet.

Vanj said he did routing in 37 instructions.  I believe this is
ether-to-ether but I could be misunderstanding.

        "The total cost for IP to forward a packet is 37 instructions
        and 13 memory references (11 loads and 2 stores)"

I believe there is a bit of fluff in there.  The initial kernel dive
was 600 instructions (25 uS on the Sparc).  Once there any number of
packets could be serviced though.

Quote:> Agreed. So why does anyone buy an 8 port switch? Throw a few dual PCI
> ethernet cards in an old Pentium with a floppy drive.

convenience, support, marketing, stupidity.

Same reason why folks buy soda at $0.75 / can.  One can make a quart
of it in a seltzer bottle for 1 cent for the sugar, flavorings and
CO2.

Also, remember there is quite a bit of profit-taking in the
quad-ethernet card market.  Single port TULIP cards are now < $30, but
the few quad cards that exist are $600-$800.  There just isn't enough
competition in that sector of the market to deflate those prices.
Switches are now going for $100/port, so its no longer advantageous to
get a quad ether card at 1.5x-2x that per-port cost unless one really
needs to route.

-wolfgang

 
 
 

Multiple NICs on one PCI Card & Linux Support

Post by Wolfgang Rupprech » Wed, 22 Apr 1998 04:00:00



Quote:> I was planning to use (and actually had set up) a
> Linux-based box as a router (with two DEC 21140s and two PCNet-PCIs) --
> but the switch is a tad easier to deal with.  It also collapses nicely
> into a flat addressing scheme -- you should have seen my 4+3-subnet
> plan... (I like routing...)

Depending on how much effort you wanted to put into it, you could
setup host routes on the private ethernets and treat those ethernets
as a point-to-point medium.  You don't *really* need to waste any IP
space for host 0 and host (netsize-1).

You could then use gated or proxy arp to advertise the "straggler"
hosts that appear to be on the main ether but are really off of
private ethernet ports on the main linux box.

-wolfgang

 
 
 

1. Multiple NICs on one PCI Card & Linux Support

With a proper bus-mastering network card (not an NE2000,  of course),  I
imagine that typical frame overhead,  even for a complicated firewall,
would be about 100 instructions plus any I/O from the processor.  An
Ethernet packet is 500 to 1500 bytes;  say the processor has to do a CRC
on it,  so it prefetches and puts 4 bytes at a time to memory or the card.
250 to 750 adds or xors or whatever.  The limit is still the cache misses
involved in getting the data off the bus,  or the DMA speed from one card
to another,  or something like that -- not the processor speed.  It might
be the processor speed on a 33MHz 486,  or on a 386SX,  but one can't use
PCI on an SX anyway;  one would use 16-bit NE2000 cards for that.  

Now if the processor is encrypting the packets, or doing application-level
translation, there may be a limit.  Also, disk I/O can't possibly keep up
with multiple 100base-TX cards.  However, I still maintain that a
non-swapping Linux system makes a great router.

By the way,  are there any Linux FC-AL drivers/boards?

--
m/lamber45\100(egr|pilot)\.msu\.edu/ and print <<MHM16x20
David Lee Lambert -- Just another perl hacker
webstuph at http://web.egr.msu.edu/~lamber45
MHM16x20

2. Thoughts: 24.1" vs 2 x 21" monitors

3. PCI NIC card in same box as ISA NIC card ?

4. Routing via DSL

5. PCI NIC? Which one is supported

6. RH 5.1 and Popmail

7. Linux can not see more than one NIC card in one PC??

8. pkgadd and installpatch with -R argument

9. 4.2 nic aliasing, multiple IPs, one nic does not work???

10. Jumpstart multiple subnets with one server and quad NIC card

11. Multiple IP's/Gateways on one NIC card with IPChains Firewall

12. Multiple IP's on one NIC Card???

13. multiple IP address in one NIC card.