Bridging two media

Bridging two media

Post by Jim Mac » Tue, 01 Jul 2003 00:21:29



Hi -
  This isn't a pure Ethernet question, but I suspect there are folks
here who have faced similar issues.  In any case, if you can recommend a
better group, please do.

  I'm bridging a serial HiPPI network that runs over fiber (Essential /
ODS / SBS) to a standard GigE network.  Both are IP-based networks.  I'm
using a spare Win2K SP3 WS machine to do this, dual 866 w/256MB.

  The setup works, but is in serious need of tuning.  There's no routing
software in use, just whatever 2K will do by itself.

  I have a HiPPI NIC from SBS, and a quality GigE NIC, a Syskonnect 9821
V2.0, with HW checksumming and interrupt moderation.

  The fiber network is running on Irix machines connected to an
Essential HiPPI switch, and by itself works very well, giving a
sustained transfer rate of 50MB/sec between any two Irix machines.  It
uses an MTU of 65260, which I can't change (and wouldn't want to) and
nominally runs at 800Mb/sec.

  The GigE network uses an MTU of 1500, primarily due to the need to
maintain compatibility with 100BT machines.  It's a busy network, but in
general I can expect transfers of 20-30MB/s, given fast enough disks,
etc.

  To further complicate things, the 2K driver for the fiber NIC allows
an MTU from 1500 to 16242, but NOT as high as the 65260 used by the rest
of the fiber network.  When a connection is made from an Irix box to the
2K box, the MTU is negotiated to the lower value.

  Unfortunately, the best transfer rate I have ever seen through this 2K
router is about 10 MB/s end-to-end, fiber-to-copper or the reverse.  In
fact, on average it's about the same as using a 100BT all-Ethernet
connection from the Irix box.  The difference is that Ethernet transfers
are steady but slow, while bridged fiber transfers are bursty, with
decided pauses that bring down the average.

 With all that background (and thanks for sticking with me), is there
anything that could be done to speed this up?  Am I missing something
critical?  BTW, the HiPPI card is not supported under Linux, or really
anything except NT/2K/XP.

--
        Jim

 
 
 

Bridging two media

Post by Rich Seifer » Mon, 07 Jul 2003 08:52:02




Quote:

>   I'm bridging a serial HiPPI network that runs over fiber (Essential /
> ODS / SBS) to a standard GigE network.  Both are IP-based networks.  I'm
> using a spare Win2K SP3 WS machine to do this, dual 866 w/256MB.

Just as an aside (I discuss the real question below), don't you find the
ideal of Serial HiPPI (SERIAL hi-performance PARALLEL interface)
strange? (Yes, I know how it came about, but it just comes across as
really odd.)

Quote:>   The setup works, but is in serious need of tuning.  There's no routing
> software in use, just whatever 2K will do by itself.

>   I have a HiPPI NIC from SBS, and a quality GigE NIC, a Syskonnect 9821
> V2.0, with HW checksumming and interrupt moderation.

>   The fiber network is running on Irix machines connected to an
> Essential HiPPI switch, and by itself works very well, giving a
> sustained transfer rate of 50MB/sec between any two Irix machines.  It
> uses an MTU of 65260, which I can't change (and wouldn't want to) and
> nominally runs at 800Mb/sec.

>   The GigE network uses an MTU of 1500, primarily due to the need to
> maintain compatibility with 100BT machines.  It's a busy network, but in
> general I can expect transfers of 20-30MB/s, given fast enough disks,
> etc.

>   To further complicate things, the 2K driver for the fiber NIC allows
> an MTU from 1500 to 16242, but NOT as high as the 65260 used by the rest
> of the fiber network.  When a connection is made from an Irix box to the
> 2K box, the MTU is negotiated to the lower value.

I suspect that the 2K driver does not want to deal with the HUGE buffer
memories required to maintain queues of frames, each of which is over
65,000 bytes. I know of folks who make Ethernet equipment capable of
supporting longer frames than the 1500 standard, but none that can go to
HiPPI levels.

Quote:>   Unfortunately, the best transfer rate I have ever seen through this 2K
> router is about 10 MB/s end-to-end, fiber-to-copper or the reverse.  In
> fact, on average it's about the same as using a 100BT all-Ethernet
> connection from the Irix box.  The difference is that Ethernet transfers
> are steady but slow, while bridged fiber transfers are bursty, with
> decided pauses that bring down the average.

>  With all that background (and thanks for sticking with me), is there
> anything that could be done to speed this up?  Am I missing something
> critical?  BTW, the HiPPI card is not supported under Linux, or really
> anything except NT/2K/XP.

HiPPI is simply a unique beast. It was never intended as a
general-purpose network, in the style of Ethernet. Bridging the two is
problematic, as you have noted. The cleanest solution is to route
between the two, since pure bridging will not work given the MTU
difference. Of course, performance will suffer; either the HiPPI devices
will have to use the lower MTU, or the router will have to fragment
every HiPPI packet (and the Ethernet device reassemble them), which is
highly performance-intensive, and generally implemented outside of the
fast-path of the code.

Perhaps someone has built a "transparent fragmenting bridge", in the
manner that used to be common for FDDI, but this will still suffer the
performance degradation. The market for HiPPI equipment is so small and
specialized that I doubt anyone has tried to solve this problem
commercially.

Why not just use Ethernet throughout?

--
Rich Seifert              Networks and Communications Consulting

(408) 395-5700            Los Gatos, CA 95033
(408) 395-1966 FAX

 
 
 

Bridging two media

Post by Jim Mac » Mon, 07 Jul 2003 10:47:43


Thanks for responding, Rich.

Quote:> Just as an aside (I discuss the real question below), don't you find
> the ideal of Serial HiPPI (SERIAL hi-performance PARALLEL interface)
> strange? (Yes, I know how it came about, but it just comes across as
> really odd.)

:-)  Isn't evolution interesting?  It's even more interesting that this
decade-old technology is still being extended.  HiPPI-800 has been
superseded by HiPPI-6400, running at 800 MB/sec.  Yikes.

Quote:> I suspect that the 2K driver does not want to deal with the HUGE
> buffer memories required to maintain queues of frames, each of which
> is over 65,000 bytes.

Sure, but since this is a dedicated machine, I'm willing to throw all
its resources at this task if it would help.  Perhaps there's some
memory limit dictated by the 2K driver model, but that seems unlikely.
More likely is that the driver writers didn't envision this application.

Quote:> I know of folks who make Ethernet equipment
> capable of supporting longer frames than the 1500 standard, but none
> that can go to HiPPI levels.

The typical jumbo-enabled Ether NIC tops out at 16K.  Still, with newer
NICs offering interrupt moderation, even 1500 isn't that bad.

The GigE side of this setup is easily capable of 50MB/sec in and out,
and the HiPPI side can equal or surpass that.  It's the interface that's
falling down.  I have managed to tweak this a bit, and we can see peaks
of 25MB/sec, though not on a long-term average.  Typical transfers
involve thousands of 1MB files at a time (SD video frames).

Quote:>>  With all that background (and thanks for sticking with me), is there
>> anything that could be done to speed this up?  Am I missing something
>> critical?  BTW, the HiPPI card is not supported under Linux, or
>> really anything except NT/2K/XP.

> HiPPI is simply a unique beast. It was never intended as a
> general-purpose network, in the style of Ethernet. Bridging the two is
> problematic, as you have noted. The cleanest solution is to route
> between the two, since pure bridging will not work given the MTU
> difference. Of course, performance will suffer; either the HiPPI
> devices will have to use the lower MTU, or the router will have to
> fragment every HiPPI packet (and the Ethernet device reassemble
> them), which is highly performance-intensive, and generally
> implemented outside of the fast-path of the code.

And there's the crux.  Would it be better for performance sake to set
the 2K HiPPI MTU at the maximum that both Irix and 2K can handle (16K),
and let fragmentation occur at the route to the GigE side, or to set
both NICs on the router to 1500 (dictated by GigE) and let the HiPPI
side negotiate to that with the Irix box?  The 2K box is not being
stressed, it seems to peak at 20-25% CPU, and the stats for the
Syskonnect seem to show very few 'avoided interrupts'.

Quote:> Perhaps someone has built a "transparent fragmenting bridge", in the
> manner that used to be common for FDDI, but this will still suffer the
> performance degradation. The market for HiPPI equipment is so small
> and specialized that I doubt anyone has tried to solve this problem
> commercially.

As far as I can tell, you're right.  Are we, as I suspect, driver bound
here, so that no software solution could do any better?  Or is there
dedicated routing software that could help?  As you say, it's ideally a
bridge -- we don't need to manage thousands of IP addresses... just two.

Quote:> Why not just use Ethernet throughout?

As the wise man once said, the answer to 'Why?' is always 'Money'.  :-)

It would take roughly $25K, plus many hours, to convert the Irix side to
GigE, and would involve a wholesale, one-time replacement of all HiPPI
gear, since there is no room in the Irix boxes for another adapter.
Tough to justify when it works very well in its own little world.

But in the end, we may just do it.  The HiPPI switch is no longer
manufactured, so failed parts become a downtime nightmare while we
search the refurb sites.  We're looking at the Extreme Summit 7i, which
claims to do wire-speed routing and can mix MTUs.

Thanks again for your insights.

--
        Jim

 
 
 

Bridging two media

Post by Rich Seifer » Tue, 08 Jul 2003 07:26:35




Quote:> Sure, but since this is a dedicated machine, I'm willing to throw all
> its resources at this task if it would help.  Perhaps there's some
> memory limit dictated by the 2K driver model, but that seems unlikely.
> More likely is that the driver writers didn't envision this application.

Quite likely the case. You really can't blame them for not thinking of
HiPPI these days.

Quote:

> The typical jumbo-enabled Ether NIC tops out at 16K.  Still, with newer
> NICs offering interrupt moderation, even 1500 isn't that bad.

With a properly designed operating system, the MTU is almost completely
irrelevant. If the NIC itself is allowed access to the system page
tables, and can move data directly into application data space (i.e.,
the NIC does the virtual-to-physical memory mapping), there is NO
interrupt overhead, no O/S overhead, and NO CPU performance implication
for any chosen MTU. (We did exactly this in the design of the VAX CI
(VAXcluster) technology, as far back as 1978. Dave Cutler (who was the
chief architect of VMS, the VAX O/S, and later Windows NT/XP) should
know better.

Quote:

> And there's the crux.  Would it be better for performance sake to set
> the 2K HiPPI MTU at the maximum that both Irix and 2K can handle (16K),
> and let fragmentation occur at the route to the GigE side, or to set
> both NICs on the router to 1500 (dictated by GigE) and let the HiPPI
> side negotiate to that with the Irix box?  The 2K box is not being
> stressed, it seems to peak at 20-25% CPU, and the stats for the
> Syskonnect seem to show very few 'avoided interrupts'.

You would likely do better by setting the MTU to the lowest common
denominator and avoiding fragmentation at all costs. Any degradation due
to lower MTU will surely be less than the HUGE hit that fragmentation
implies.

Quote:> Are we, as I suspect, driver bound
> here, so that no software solution could do any better?  Or is there
> dedicated routing software that could help?  As you say, it's ideally a
> bridge -- we don't need to manage thousands of IP addresses... just two.

In theory, if you had a router that performed fragmentation in the
fast-path of the code (rather than as an "exception condition"), or even
better, fragmentation in hardware, things would be better. You would
still have the problem of reassembly at the Ethernet end, which would
still limit total throughput.

--
Rich Seifert              Networks and Communications Consulting

(408) 395-5700            Los Gatos, CA 95033
(408) 395-1966 FAX

 
 
 

1. media converter vs. WLAN-bridge problem

Hello all,

we have a strange problem involving a WLAN-bridge (Lucent) and
a couple of AT-MC100-Series media converter. The setup is as
follows:

                      100TX       100FX       100TX
Radio link <--> WLAN- <---> media <---> media <---> router
               bridge       conv.       conv.

Having connected everything as shown above and upon rebooting
the bridge, its ethernet interface hangs and thus its radio
link isn't initialized. If, in this "hang"-state, the 100TX
connection between the bridge and the first media converter
is unplugged, the bridge interface comes up, the radio link
is initialized and after replugging the 100TX everything works
just fine.
Has anyone experienced something similar or can anyone explain
this behaviour?
The devices itself aren't broken. We reproduced the problem
with another (Lucent) brigde, another set of AT-MC100 and
even with a pair of AT-10TX media converters.
If we introduce a Cisco2950-switch between the bridge and the
media converter everything works fine, the problem doesn't
occur. If we introduce a hub or a mini-switch instead of the
C2950 the problem occurs again!

Any suggestions are welcome!

Regards,

        Frank

--
"On the fourth day of xmas i stole from that lady:"
"Four family fotos."
"Three jars of pennys."
"Two former husbands."
"And a slipper on a shoe-tree."

2. SourceSafe

3. bridging between different media

4. PB100 keeps wanting to rebuild desktop, why?

5. Media One Two Way Service in Minneapolis/St. Paul?

6. Multi-user debugging (URGENT)

7. Media Conspiracies (was Re: Technology Companies; Online Media)

8. System 7 BUG

9. Bridging two different subnets over IRB (urgent)

10. Two separate subnets over Cisco 350 Wireless Bridges

11. Two secure bridged connections over one ADSL line.

12. Bridging two Peer-to-Peer Networks

13. Bridge and route two different IP subnets