Very Computer

Hi there,

I inherited a two node cluster and am checking out the SSA disk cabling. It has
the following configuration. (Each box has two adapters).

Primary Box

1st Adapter                          2nd Adapter

A1--- 9                                A1---1
A2----16                               A2---8

Secondary Box

1st Adapter                          2nd Adapter

A1--- 12                               A1---4
A2----13                               A2---5

The disks are mirrored, front to Back i.e. pdisk 4-7 (Front) are mirrored on
pdisk 8-11 for production vg. Now if one of the adapters is failed in Primary
its been covered by the secondary adapter in Primary (above config.). What about
if a adapter fails in secondary, I don't think there is any redundancy there, if
one of the system fails then I think I will get link open errors (Some one in
IBM told me that failed link error will always happen, even if I make a two loop
system, in which the front loop is mirrored to the other loop disks) is that
so??? I also have the following queries.

1. Is this a good config. or should I go for a better one.
2. If I need to change cabling will it any way affect cluster or any thing (I
will do it after shutting down the cluster software, AIX and power off the
server).
3. Do someone has a better configuration and mailed that to me so I can check
that out.

Thanks in advance for your help.

Arshad.

Box1:

Quote:> 1st Adapter                          2nd Adapter

> A1--- 9                                A1---1
> A2----16                               A2---8
Box2:
> 1st Adapter                          2nd Adapter

> A1--- 12                               A1---4
> A2----13                               A2---5

Looks quite alright, though there is some folklore against having the same
machine
twice in the same loop. (don't know if it was fixed).

Quote:> The disks are mirrored, front to Back i.e. pdisk 4-7 (Front) are mirrored
on
> pdisk 8-11 for production vg. Now if one of the adapters is failed in
Primary
> its been covered by the secondary adapter in Primary (above config.).

No need to mirror front-to-back, the configuration is symetric and
all the disks are on the same loop.

Quote:> What about if a adapter fails in secondary,

It's really symetric.  The SSA bay goes like this:

/o-#-1-2-3-4-#-o/o-#-5-6-7-8-#-o-#-9-10-11-12-#-o/-#-13-14-15-16-#-o
And 1 is looped back to 16

So you have Box1-1..4-Box2-5..8--Box1-9..12-Box2-13-15-back-to-box1

--
Doing AIX support was the most monty-pythonesque
activity available at the time.

That "folklore" stems from basically two facts:

 1) You can't have both the A and the B ports of a given adapter in the
same loop.

 2) If a machine is turned off, the adapter will cause a break in the
loop. If a machine has more than one adapter in the loop, the loop will
consequently be broken in two places, and any disks* "in between"
the adapters may become unavailable to other hosts in the loop.

To avoid this, the two adapters are often placed right next to each
other on the loop, which causes another problem - now you cannot get the
aggregated bandwith of all four ports since one of the ports of each
adapter needs to talk "through" the other adapter.

--
Christer Palm

The answer depends on your disk subsystems. If they are 7133-010 (drawer) or
7133-500 (tower), then the connectors are labelled Jn, so we assume not.

If they are 7133-020 or -600 (grey coloured subsystem usually with black cables) or
D40 or T40 (black subsystem usually with blue cables), then you have two loops. If
you don't know what that means, please check out the documentation available from
http://www.hursley.ibm.com/~ssa.

You will get the link open errors because to use this cabling the subsystem must
have been changed from the factory default of "bypass" to "forced inline" on each
pair of connectors 1,16 4,5 8,9 12,13.

Now, depending on the SSA adapters you have, you may be able to enhance this
configuration by configuring all the adapters in the same loop and changing the
subsystem back to bypass. If you do, then when you turn off one server, the
subsystem will loop back between the connector pairs and the single loop will be
reformed. You can see whether this will work by checking to see if you can see an
option in smit to set an SSA cluster number, which is used to enable adapter
failover also.

However, on the whole, what you have now works, the improvement you can get is small
and it will require some planning and some downtime to change.

My advice? If it ain't broke ...

But in my configuration will I have any* disk when one of the
system fails?? since all our disks are on bypass mode, so I believe I
would not have any* disk when one of the system is down as they
can communicate with each other with the bypass switch. Am I rite??

Arshad

Quote:

> To avoid this, the two adapters are often placed right next to each
> other on the loop, which causes another problem - now you cannot get the
> aggregated bandwith of all four ports since one of the ports of each
> adapter needs to talk "through" the other adapter.

I didn't review your specific configuration before I wrote that, so
those comments were in general.

You are right in that you can avoid breaking the loop in some cases by
using the bypass feature. In your case you don't have to bother too much
about that since you only have two adapters in each loop.

Just watch out for any unexpected bypass effects which may happen when
you have several loops in a 7133 drawer. In your case, you should
probably make sure that 8-9 and 1-16 are forced inline.

In your particular configuration you are not guarded against an adapter
failure at the SSA link level since you have the two adapters on
different loops. This is fine if you do LVM mirroring across the loops
and make sure you have runtime quorum checking turned off. (As a matter
of fact this may not be a very big deal - I've only seen an SSA adapter
fail once, and that one screwed up the loop anyway, so a redundant
adapter propably wouldn't have done any good.)

Depending on which adapters you have and whether or not you wish to do
SSA RAID, it is possible to get adapter redundancy in this configuration
by making one loop from the A ports of all four adapters, and another
loop from the B ports. For various reasons, however, I do personally
prefer the configuration you already have.

--
Christer Palm