SSA Disk Cabling

SSA Disk Cabling

Post by Arsha » Wed, 15 Aug 2001 07:47:17



Hi there,

I inherited a two node cluster and am checking out the SSA disk cabling. It has
the following configuration. (Each box has two adapters).

Primary Box

1st Adapter                          2nd Adapter

A1--- 9                                A1---1
A2----16                               A2---8

Secondary Box

1st Adapter                          2nd Adapter

A1--- 12                               A1---4
A2----13                               A2---5

The disks are mirrored, front to Back i.e. pdisk 4-7 (Front) are mirrored on
pdisk 8-11 for production vg. Now if one of the adapters is failed in Primary
its been covered by the secondary adapter in Primary (above config.). What about
if a adapter fails in secondary, I don't think there is any redundancy there, if
one of the system fails then I think I will get link open errors (Some one in
IBM told me that failed link error will always happen, even if I make a two loop
system, in which the front loop is mirrored to the other loop disks) is that
so??? I also have the following queries.

1. Is this a good config. or should I go for a better one.
2. If I need to change cabling will it any way affect cluster or any thing (I
will do it after shutting down the cluster software, AIX and power off the
server).
3. Do someone has a better configuration and mailed that to me so I can check
that out.

Thanks in advance for your help.

Arshad.

 
 
 

SSA Disk Cabling

Post by Jose Pina Coelh » Wed, 15 Aug 2001 10:37:22



Box1:

Quote:> 1st Adapter                          2nd Adapter

> A1--- 9                                A1---1
> A2----16                               A2---8
Box2:
> 1st Adapter                          2nd Adapter

> A1--- 12                               A1---4
> A2----13                               A2---5

Looks quite alright, though there is some folklore against having the same
machine
twice in the same loop. (don't know if it was fixed).

Quote:> The disks are mirrored, front to Back i.e. pdisk 4-7 (Front) are mirrored
on
> pdisk 8-11 for production vg. Now if one of the adapters is failed in
Primary
> its been covered by the secondary adapter in Primary (above config.).

No need to mirror front-to-back, the configuration is symetric and
all the disks are on the same loop.

Quote:> What about if a adapter fails in secondary,

It's really symetric.  The SSA bay goes like this:

/o-#-1-2-3-4-#-o/o-#-5-6-7-8-#-o-#-9-10-11-12-#-o/-#-13-14-15-16-#-o
And 1 is looped back to 16

So you have Box1-1..4-Box2-5..8--Box1-9..12-Box2-13-15-back-to-box1

--
Doing AIX support was the most monty-pythonesque
activity available at the time.

 
 
 

SSA Disk Cabling

Post by Christer Pal » Wed, 15 Aug 2001 16:39:19



> Looks quite alright, though there is some folklore against having the same
> machine twice in the same loop. (don't know if it was fixed).

That "folklore" stems from basically two facts:

 1) You can't have both the A and the B ports of a given adapter in the
same loop.

 2) If a machine is turned off, the adapter will cause a break in the
loop. If a machine has more than one adapter in the loop, the loop will
consequently be broken in two places, and any disks* "in between"
the adapters may become unavailable to other hosts in the loop.

To avoid this, the two adapters are often placed right next to each
other on the loop, which causes another problem - now you cannot get the
aggregated bandwith of all four ports since one of the ports of each
adapter needs to talk "through" the other adapter.

--
Christer Palm

 
 
 

SSA Disk Cabling

Post by Simon Marches » Wed, 15 Aug 2001 18:20:25



> I inherited a two node cluster and am checking out the SSA disk cabling. It has
> the following configuration. (Each box has two adapters).

> Primary Box

> 1st Adapter                          2nd Adapter

> A1--- 9                                A1---1
> A2----16                               A2---8

> Secondary Box

> 1st Adapter                          2nd Adapter

> A1--- 12                               A1---4
> A2----13                               A2---5

> The disks are mirrored, front to Back i.e. pdisk 4-7 (Front) are mirrored on
> pdisk 8-11 for production vg. Now if one of the adapters is failed in Primary
> its been covered by the secondary adapter in Primary (above config.). What about
> if a adapter fails in secondary, I don't think there is any redundancy there, if
> one of the system fails then I think I will get link open errors (Some one in
> IBM told me that failed link error will always happen, even if I make a two loop
> system, in which the front loop is mirrored to the other loop disks) is that
> so??? I also have the following queries.

> 1. Is this a good config. or should I go for a better one.
> 2. If I need to change cabling will it any way affect cluster or any thing (I
> will do it after shutting down the cluster software, AIX and power off the
> server).
> 3. Do someone has a better configuration and mailed that to me so I can check
> that out.

The answer depends on your disk subsystems. If they are 7133-010 (drawer) or
7133-500 (tower), then the connectors are labelled Jn, so we assume not.

If they are 7133-020 or -600 (grey coloured subsystem usually with black cables) or
D40 or T40 (black subsystem usually with blue cables), then you have two loops. If
you don't know what that means, please check out the documentation available from
http://www.hursley.ibm.com/~ssa.

You will get the link open errors because to use this cabling the subsystem must
have been changed from the factory default of "bypass" to "forced inline" on each
pair of connectors 1,16 4,5 8,9 12,13.

Now, depending on the SSA adapters you have, you may be able to enhance this
configuration by configuring all the adapters in the same loop and changing the
subsystem back to bypass. If you do, then when you turn off one server, the
subsystem will loop back between the connector pairs and the single loop will be
reformed. You can see whether this will work by checking to see if you can see an
option in smit to set an SSA cluster number, which is used to enable adapter
failover also.

However, on the whole, what you have now works, the improvement you can get is small
and it will require some planning and some downtime to change.

My advice? If it ain't broke ...

 
 
 

SSA Disk Cabling

Post by Arshad She » Fri, 17 Aug 2001 04:14:59




> > Looks quite alright, though there is some folklore against having the same
> > machine twice in the same loop. (don't know if it was fixed).

> That "folklore" stems from basically two facts:

>  1) You can't have both the A and the B ports of a given adapter in the
> same loop.

>  2) If a machine is turned off, the adapter will cause a break in the
> loop. If a machine has more than one adapter in the loop, the loop will
> consequently be broken in two places, and any disks* "in between"
> the adapters may become unavailable to other hosts in the loop.

But in my configuration will I have any* disk when one of the
system fails?? since all our disks are on bypass mode, so I believe I
would not have any* disk when one of the system is down as they
can communicate with each other with the bypass switch. Am I rite??

Arshad

- Show quoted text -

Quote:

> To avoid this, the two adapters are often placed right next to each
> other on the loop, which causes another problem - now you cannot get the
> aggregated bandwith of all four ports since one of the ports of each
> adapter needs to talk "through" the other adapter.

 
 
 

SSA Disk Cabling

Post by Christer Pal » Fri, 17 Aug 2001 05:05:32



> But in my configuration will I have any* disk when one of the
> system fails?? since all our disks are on bypass mode, so I believe I
> would not have any* disk when one of the system is down as they
> can communicate with each other with the bypass switch. Am I rite??

I didn't review your specific configuration before I wrote that, so
those comments were in general.

You are right in that you can avoid breaking the loop in some cases by
using the bypass feature. In your case you don't have to bother too much
about that since you only have two adapters in each loop.

Just watch out for any unexpected bypass effects which may happen when
you have several loops in a 7133 drawer. In your case, you should
probably make sure that 8-9 and 1-16 are forced inline.

In your particular configuration you are not guarded against an adapter
failure at the SSA link level since you have the two adapters on
different loops. This is fine if you do LVM mirroring across the loops
and make sure you have runtime quorum checking turned off. (As a matter
of fact this may not be a very big deal - I've only seen an SSA adapter
fail once, and that one screwed up the loop anyway, so a redundant
adapter propably wouldn't have done any good.)

Depending on which adapters you have and whether or not you wish to do
SSA RAID, it is possible to get adapter redundancy in this configuration
by making one loop from the A ports of all four adapters, and another
loop from the B ports. For various reasons, however, I do personally
prefer the configuration you already have.

--
Christer Palm

 
 
 

1. Re-using SSA disks previously members of a SSA RAID array

On Tue, 16 Mar 1999 20:12:12 -0500 a carbon-based biped by the name of Lekic

Somewhere under the SSA Raid menu entry in the smit|Devices menu,
there is a point that will let you change the usage of a SSA disk.
If a disk has been a member of an array, it is probably listed as
'Array candidate disk' or 'Hot Spare'.
Change that to 'AIX system disk' for all the disks, and you should
be able to re-use the units as normal SSA disks.

Regards

Peter

2. DELUSER Command in Slackware ?

3. Adding disk into SSA disk arry

4. Where to get glibc2 source...

5. How do I delete a VG of SSA disks when the disks are no longer available?

6. lpr woes

7. Warning - problem SSA cables

8. Help: PS/2 mouse with XFree

9. SSA cable and adapter

10. SSA Cabling options

11. Source for SSA cables and adapters?

12. Trade: 8218 18GB SSA drives for 8209 9GB SSA drives

13. Useful SSA script (was: Mirrors Screwed up in SSA)