Linksys LAN -- network fails at a switch

Linksys LAN -- network fails at a switch

Post by Sally Shear » Tue, 01 Feb 2005 06:41:58



I have an all-Linksys network with WRT54g router/AP and four Linksys
switches. I need switches to connect 20 wired locations in all, but
only a half dozen hosts on the network. WRT54g provides NAT and DHCP
for the whole network as well as a wireless access point. The gateway
router is FlowPoint connecting to ADSL. The WRT54g is NOT the gateway
router.  Works fine.

Topology:  Gateway -- switch -- switch -- WRT54g -- 2 switches

Problem: Intermittantly, the entire network fails. Specifically, none
of the hosts on the wired LAN (and on WLAN) can see each other and
cannot see the outside world. The failure is over the whole LAN, not
just the portion below one switch. Outside users can see the gateway
router, but not the hosts on the LAN.

I've only had three instances of this and I don't know what's going on,
but each time I can isolate the problem (by unpluging CAT5 cables) to
one of the switches. Cycling the power on that switch restores the
network. The first time, I replaced the suspicious switch. Now I think
it's not a failed unit but something more general.

I did see the problem after a power failure. When the building power
came back on, the network was in the failed state described above.
Solved it by cycling the power on the Linksys switches. The WRT54g
seems to recover from power outage just fine.

Anyone else seeing this? Know how to set up to avoid the problem?

  -- Sally

--
Sally Shears (a.k.a. "Molly")

http://theWorld.com/~sshears

 
 
 

Linksys LAN -- network fails at a switch

Post by prg » Tue, 01 Feb 2005 13:00:39


Not sure there is any "more specific" info you could provide that would
tie this down -- not for me anyway.

Could it be that the switches and hosts are coming back up after a
power outage in such a manner/sequence that the problem converges on
the suspect location and craps the spanning tree -- if you're running
spanning tree, that is.

Also could the learning mode be corrupted or the MAC/IP tables be
getting boinked by way of a "sneaky" hidden pathway in the physical
connections that causes one (or more) of the MACs to appear as
originating on different ports or on different sides of the switch.
Perhaps you need spanning tree running?

Can you monitor or at least dump the switch MAC tables?  Nearby host
arp cache might help also.  Any possibility that someone has manually
changed/set the MAC address of a host nic that duplicates MAC elswhere?
Does anyone move from one switch to another, thus causing their MAC to
appear in different parts of the switch fabric?

I've had times when MACs began appearing on different sides of a switch
for no apparent reason.  This can bring the switch(es) to their knees.
Hard to "see" it without sniffing the wire and the problem source can
be difficult to track down.

Resetting the switches clears the tables and you start clean again.
After some interval X, the problem accumulates/reappears and brings the
switches down again.

Double check, then double check again, that there are no loops in the
physical pathways.  That must be confirmed (as well as you can) before
getting to the ugly stuff.

Confirm that each switch is properly configured for the role it plays.
Think.  Should I really be using a router here instead of a switch?
Think again.  Confirm to your own satisfaction that a switch is the
correct appliance.

You've already played with cable replacement.  Tried different ports?
Tried swapping out the "bad" switch with one upstream/downstream or
sibbling to see if you can make the problem "follow" the switch.

If it does not follow the switch, begin to suspect a loop or
malfunction that originates elsewhere but converges at this point.
What about your setup/usage would cause the problem to converge here
and not elsewhere?

If you cannot narrow it down at this point, you'll have to get out the
sniffer and monitor packet flows.  If you have several Linux boxes
spread around you can use them to monitor traffic with something like
MTRG or something, anything, that will reveal a spike as switches start
bouncing packets around.

If you're real lucky, a sniffer will reveal that a MAC is showing up on
two sides of one or more switches but does not cause serious problems
till after interval X.  This situation sometimes shows up as eratic
behavior with one or two particular hosts before it shows elsewhere.
Keep an eye out for sporadic host behavior or complaints about
clunkiness.  Don't rule out a misbehaving nic.
good luck -- you likely need it ;)
prg
email above disabled

 
 
 

Linksys LAN -- network fails at a switch

Post by Sally Shear » Wed, 02 Feb 2005 00:50:06




Quote:> Could it be that the switches and hosts are coming back up after a
> power outage in such a manner/sequence that the problem converges on
> the suspect location ...

This is exactly what I'm worried about.  

Quote:> ...and craps the spanning tree -- if you're running
> spanning tree, that is.

If "spanning tree" is running, I don't know it. I have SuSE 8.1 and
9.1, pretty standard setup. Linksys consumer switches (the $50 at Best
Buy type) and a WRT54g. These don't provide the spanning tree protocol,
do they?  (Googling, I can see that Linksys WET54GS5 does do spanning
tree.)

Quote:> I've had times when MACs began appearing on different sides of a switch
> for no apparent reason.  This can bring the switch(es) to their knees.
> Hard to "see" it without sniffing the wire and the problem source can
> be difficult to track down.

No loops in the physical wiring. I'm absolutely sure of this.

But... what if a host (and its MAC address) are moved from one part of
the LAN to another?

I move a laptop (Apple OS X) all the time from wireless (with a NAT'd
non-routable IP) to wired (with fixed IP) at the same OR different part
of the network to outside the building (i.e. yet another IP somewhere
else on the internet). Could moving this Mac's MACs be causing the
problem?

FWIW, the Mac laptop has two nics, one wired and one wireless.
Different MAC addresses. OS X in most configurations automatically
changes from one nic to the other as needed. Recently someone opined in
alt.os.linux.suse "a host with two nics on the same LAN is trouble." I
wonder if I should start being extremely careful that no laptop ever
has ethernet plugged in which it's wireless card is operating. Anyone
have comments on two-nics-same-LAN?

I've seen the problem three times
 - Twice with the moving laptop outside the building
 - Once after a power failure with the laptop in it's usual place on
the LAN.

Quote:> Resetting the switches clears the tables and you start clean again.
> After some interval X, the problem accumulates/reappears and brings the
> switches down again.

Yup, that sounds like what I've seen.

Quote:> Double check, then double check again, that there are no loops in the
> physical pathways.  

No loops in physical wiring. Is it conceivable that I created a loop
involving wireless via the Apple OS X auto-network-config when I
plugged into ethernet with wireless still running? I'll watch out for
that.

Quote:> Tried swapping out the "bad" switch with one upstream/downstream or
> sibbling to see if you can make the problem "follow" the switch.

A little hard to tell, but I do have one suspect switch which I'll
watch.

Quote:> good luck -- you likely need it ;)
> prg
> email above disabled

PRG, I really appreciate the suggestions and investigation protocol.
Thank you.

  -- Sally

 
 
 

Linksys LAN -- network fails at a switch

Post by prg » Wed, 02 Feb 2005 14:00:43




prg

> > Could it be that the switches and hosts are coming back up after a
> > power outage in such a manner/sequence that the problem converges
on
> > the suspect location ...

> This is exactly what I'm worried about.

This might be a one time thing related to computer boot ups and switch
learning rates.  But it may be useful as a locational clue to a problem
source.  Or maybe not :(

Quote:> > ...and craps the spanning tree -- if you're running
> > spanning tree, that is.

> If "spanning tree" is running, I don't know it. I have SuSE 8.1 and
> 9.1, pretty standard setup. Linksys consumer switches (the $50 at
Best
> Buy type) and a WRT54g. These don't provide the spanning tree
protocol,
> do they?  (Googling, I can see that Linksys WET54GS5 does do spanning
> tree.)

This was more just a check -- have noted in the past year that a number
of commodity switches are offering spanning tree.  Doubt that it will
even engage without another spanning tree switch in the fabric.  Guides
don't speack much about configuring it -- just on or off.

Quote:> > I've had times when MACs began appearing on different sides of a
switch
> > for no apparent reason.  This can bring the switch(es) to their
knees.
> > Hard to "see" it without sniffing the wire and the problem source
can
> > be difficult to track down.

> No loops in the physical wiring. I'm absolutely sure of this.

> But... what if a host (and its MAC address) are moved from one part
of
> the LAN to another?

> I move a laptop (Apple OS X) all the time from wireless (with a NAT'd
> non-routable IP) to wired (with fixed IP) at the same OR different
part
> of the network to outside the building (i.e. yet another IP somewhere
> else on the internet). Could moving this Mac's MACs be causing the
> problem?

Well, at the switch level, only MACs and ports are used in the lookup
tables.  This is strictly level 2.  If you are moving within the same
switch group/lan and the ageing timers are long enough, your old
MAC/port will still be present when you hook up elsewhere -- you
"announce" your presence to the switch with the very first frame you
send out.

Quote:> FWIW, the Mac laptop has two nics, one wired and one wireless.
> Different MAC addresses. OS X in most configurations automatically
> changes from one nic to the other as needed. Recently someone opined
in
> alt.os.linux.suse "a host with two nics on the same LAN is trouble."

This is true at the IP level, especially re: arp cache/table of
MACs/IPs as gateways can get confused when they have MAC/IP pairs
moving around.  It's similar to MACs moving about in a switched LAN.
Any host/nic setup that produces "gratuitous" arp (announcing arp) can
exacerbate this.  But these will (or at least should if not turned
off/filtered) return ICMP errors.

If only one nic is active at a time, no problem.  Both at the same time
on a Linux box is usually trouble because Linux stack is designed to
respond for any host IP out any nic (a sort of failover "feature").
Don't think Mac stack behaves this way -- it's BSD based, afaik.

Quote:> I
> wonder if I should start being extremely careful that no laptop ever
> has ethernet plugged in which it's wireless card is operating. Anyone
> have comments on two-nics-same-LAN?

Do you gain anything with both nics up simultaneously?  Failover and
bonding are the usual reasons for wanting both up.  The route path to
the nics is likely different -- true in your case?  Normal Linux
operation does not use metrics and with its quirky responding behavior
it's quite possible that packets come in on one nic but out the other
-- routers and GWs upstream don't like this bouncing MAC address in the
arp traffic and the ethernet frames could, I imagine, make switches
bark in the right circumstances.

- Show quoted text -

Quote:> I've seen the problem three times
>  - Twice with the moving laptop outside the building
>  - Once after a power failure with the laptop in it's usual place on
> the LAN.

> > Resetting the switches clears the tables and you start clean again.
> > After some interval X, the problem accumulates/reappears and brings
the
> > switches down again.

> Yup, that sounds like what I've seen.

> > Double check, then double check again, that there are no loops in
the
> > physical pathways.

> No loops in physical wiring. Is it conceivable that I created a loop
> involving wireless via the Apple OS X auto-network-config when I
> plugged into ethernet with wireless still running? I'll watch out for
> that.

> > Tried swapping out the "bad" switch with one upstream/downstream or
> > sibbling to see if you can make the problem "follow" the switch.

> A little hard to tell, but I do have one suspect switch which I'll
> watch.

I was sorta hoping that you would find someone complaining about eratic
networking, suspect misbehaving nic, and locate your problem source.
Alas, it's never that easy.

I would keep an eye on the moving laptop.  Switches should have a
timeout for table entries with no response for XXX seconds (Ciscos are
300) so you might want to check what the timers are set at and change
them or wait before plugging in at a different switch.  We've had
trouble with students at school when moving about the right classes --
like several rooms away but the same switch or sibling.

Checking switch configs and stats can help but is usually time
consuming and tedious.  You might want to try a sniffer -- I like
ethereal -- and watch for "unusual" ethernet frames, especially what
appear to be multiple, identical frames.  Also keep an eye out for a
high volume of broadcasts or anything that looks like an errant "keep
alive" packet.

good luck,
prg
email above disabled

 
 
 

Linksys LAN -- network fails at a switch

Post by Sally Shear » Wed, 02 Feb 2005 23:15:32






> prg


...snip...

Quote:> > > ...and craps the spanning tree -- if you're running
> > > spanning tree, that is.

> > If "spanning tree" is running, I don't know it. I have SuSE 8.1 and
> > 9.1, pretty standard setup. Linksys consumer switches (the $50 at Best
> > Buy type) and a WRT54g. These don't provide the spanning tree protocol,
> > do they?  (Googling, I can see that Linksys WET54GS5 does do spanning
> > tree.)

> This was more just a check -- have noted in the past year that a number
> of commodity switches are offering spanning tree.  Doubt that it will
> even engage without another spanning tree switch in the fabric.  Guides
> don't speack much about configuring it -- just on or off.

I think my switches are more basic than this. Linksys EZXS55W ver 3 for
example. I don't think there are ANY options.

(Aside... This is so much just an appliance that the model number,
although printed on the label, is SO SMALL you need a magnifying glass
to find it.)

Quote:> > > I've had times when MACs began appearing on different sides of a switch
> > > for no apparent reason.  This can bring the switch(es) to their knees.
> > > Hard to "see" it without sniffing the wire and the problem source can
> > > be difficult to track down.

> > No loops in the physical wiring. I'm absolutely sure of this.

> > But... what if a host (and its MAC address) are moved from one part of
> > the LAN to another?

> > I move a laptop (Apple OS X) all the time from wireless (with a NAT'd
> > non-routable IP) to wired (with fixed IP) at the same OR different part
> > of the network to outside the building (i.e. yet another IP somewhere
> > else on the internet). Could moving this Mac's MACs be causing the
> > problem?

> Well, at the switch level, only MACs and ports are used in the lookup
> tables.  This is strictly level 2.  If you are moving within the same
> switch group/lan and the ageing timers are long enough, your old
> MAC/port will still be present when you hook up elsewhere -- you
> "announce" your presence to the switch with the very first frame you
> send out.

I can move hosts around on the LAN; they work and no obvious problems.
The "announce" and new routing must be working OK.

By the way, when the failure occurs, it stays failed. Does not recover
after a timeout. Stayed broken for 24 hours once.

- Show quoted text -

Quote:> > FWIW, the Mac laptop has two nics, one wired and one wireless.
> > Different MAC addresses. OS X in most configurations automatically
> > changes from one nic to the other as needed. Recently someone opined in
> > alt.os.linux.suse "a host with two nics on the same LAN is trouble."

> This is true at the IP level, especially re: arp cache/table of
> MACs/IPs as gateways can get confused when they have MAC/IP pairs
> moving around.  It's similar to MACs moving about in a switched LAN.
> Any host/nic setup that produces "gratuitous" arp (announcing arp) can
> exacerbate this.  But these will (or at least should if not turned
> off/filtered) return ICMP errors.

> If only one nic is active at a time, no problem.  Both at the same time
> on a Linux box is usually trouble because Linux stack is designed to
> respond for any host IP out any nic (a sort of failover "feature").
> Don't think Mac stack behaves this way -- it's BSD based, afaik.

Yes, Apple OS X is BSD-based.

Quote:> > I
> > wonder if I should start being extremely careful that no laptop ever
> > has ethernet plugged in which it's wireless card is operating. Anyone
> > have comments on two-nics-same-LAN?

> Do you gain anything with both nics up simultaneously?  Failover and
> bonding are the usual reasons for wanting both up.  

If both nics are up, it's not intentional.

Quote:> ... The route path to
> the nics is likely different -- true in your case?  

Not sure I know. Wired and wireless would be on different parts of the
same LAN, under different switches.

...snip...

Quote:> I was sorta hoping that you would find someone complaining about eratic
> networking, suspect misbehaving nic, and locate your problem source.
> Alas, it's never that easy.

Nope. Not sure anybody would notice a hiccup or slowdown.

...snip...

Quote:> good luck,
> prg
> email above disabled

Thanks again, prg... I have some things to watch for.

My conclusions:
 1. Keep the network simple. I think I'll move some hosts so I can
eliminate some of the switches.
 2. Never let wireless laptops connect two-ways to the LAN.
 3. Cycle the power from time to time on the switches.

  -- Sally

--
Sally Shears (a.k.a. "Molly")

http://theWorld.com/~sshears

 
 
 

Linksys LAN -- network fails at a switch

Post by Heiko Voge » Thu, 03 Feb 2005 04:55:43



> I've had times when MACs began appearing on different sides of a switch
> for no apparent reason.  This can bring the switch(es) to their knees.
> Hard to "see" it without sniffing the wire and the problem source can
> be difficult to track down.

Could it be that someone on your network hooked up a sniffer and tries
to fool the switches by sending out packets with different MACs.
There exists a nifty tool for linux called "ettercap" which is able to
sniff in switched networks using the technique of ARP poisoning.

greetz,
haaner

 
 
 

Linksys LAN -- network fails at a switch

Post by Sally Shear » Sat, 05 Feb 2005 12:18:27





> > I've had times when MACs began appearing on different sides of a switch
> > for no apparent reason.  This can bring the switch(es) to their knees.
> > Hard to "see" it without sniffing the wire and the problem source can
> > be difficult to track down.

> Could it be that someone on your network hooked up a sniffer and tries
> to fool the switches by sending out packets with different MACs.
> There exists a nifty tool for linux called "ettercap" which is able to
> sniff in switched networks using the technique of ARP poisoning.

> greetz,
> haaner

Greetz to you Haaner --

Thanks for the suggestion, but no. That's not happening here.

  -- Sally

--
Sally Shears (a.k.a. "Molly")

http://theWorld.com/~sshears

 
 
 

1. linksys switch and internal network.

I'm having a problem creating a home network.

I have a linksys ezxs55w switch and two computers plugged into the switch.
My dsl modem is plugged into the switch also so I can have both machines on
the net at the same time.

I use win2k/suse on one machine and the other is just suse as a server. They
both use dhcp for ISP IP addressing.

If use traceroute to trace the route from one machine to the other, it shows
that the hops go to the gateway and then to the other machine.

The thing that I was wondering is if I can assign another internal IP addr
to each machine using ifconfig. Like assigning 192.168.100.1 to one (and
still have the ISP IP address asssigned to it) and the other would be
192.168.100.2

I have tried to staticly add the IP to the network scripts in the
/etc/sysconfig directory with no hope.

Both would need to see the internet.

thanks in advance.

craig.

2. module installation in TCP/IP stack error.

3. any help with LinkSys Etherfast 10/100 LAN Lan Card with SuSE 6.1?

4. Install question : boot disk

5. Easy way to switch between LAN and dial-up networks?

6. 3-button mouse-hack

7. Switching from PC-LAN to Linux tcp/ip network.

8. partition and swap space/

9. Linux Network down work with 1000 MBit Switch but with old 100 MBit Switch

10. Software for LinkSys Compact Auto Switch

11. Final solution: switched to LinkSys. it worked.

12. eth0 works with Linksys hub but not Netgear switch?!?

13. Linksys Router/Switch port forwarding