Very Computer

Hi

I have to set up an SSA disk array with the following config.

     2 x model D40 drawers and 2 Advanced-160 SSA adapters.

     D40-0    9,9,18,18  18,18,18,18    18,18,18,18  18,18,18,18
     D40-1    9,9,18,18  18,18,18,18    18,18,18,18  18,18,18,18

The system boots off internal disk.  The server is standalone and will
not need to share disks for clustering.  Th array is to house a fairly
large production Oracle database (80-100Gb) - It is the back end to
our general ledger.  I was thinking of building one drawer, wiring it
through card 0, and then mirroring the entire drawer onto the other
card/drawer combination.  Is this a good idea or would there be a
better combination.  I am more concerned about speed than redundancy.
(Each array will have a hot spare and I will be mirroring everything.)

Is there somewhere I can find a recommendation/manual on how I should
do the following;

1.  Best way to run the SSA cables.
2.  Should I stripe the data over all the 18GB disks, leaving the 9GB
disks for redo and rollback, and if I stripe, what size?

Any comments would be welcome

Thanks again

george

Just because they're smaller, doesn't mean they're faster, which is the
usual stated reason for using smaller disks for redo and rollback. Use
"lscfg -vl pdiskn" to determine the model number (something like
D**C-nnnn) and look on the IBM storage website to see what the specs are.
Remember that all the disks in an array should be the same size. But
there's no substitute for benchmarking your options with as near a real
workload as you can.

Fastest option and probably the easiest connect the d40's as 4 loops of 8
disks each 8 on it's own port of a ssa adapter.
now create raid10 devices 4 disks mirrored to 4 disks per loop,
Enable fast write cache.
Walk away and relax.
Put all the raid 10 disks into one volume group.
Now create the filesystems on all disks and use maximum spread.

Finished.
There is a pdf at IBM that explains what raid 10 is on ssa adapters.

First, I recommend having all disks in both drawers visible to BOTH
adapters.  This will provide maximum performance AND availability.  Even
though you plan to mirror (which I agree is a good idea), you should
also look at cabling to maximize availability.  Think in terms of the
most tolerance to the loss of ANY of the following: disk, adapter,
drawer.

I would have two loops.  Let's call them loop A and loop B.  Have each
adapter participate in BOTH loops.  So loop A could include ports A1 and
A2 in adapter 0, and ports A1 and A2 in adapter 1.  In the same way loop
B would include ports B1 and B2 in adapters 0 and 1. Each loop can be a
separate drawer.  So you would cable (for example) from port A1 on
adapter 0 though the front 8 drives in drawer 0 to port A1 on adapter
1.  Likewise, from port A2 on adapter 0 through the rear 8 drives on
drawer 0 to port A2 on adapter 1.  Similarly: B1 -> front drawer 1 -> B1
and B2-> rear drawer 1 -> B2.  This arrangement allows you to optimize
performance, and also allows a single adapter failure while maintaing
accessibility to ALL drives in both drawers.

In this arrangement you should mirror across loops, which is equivalent
to mirroring across drawers.  That way the loss of a drawer causes the
loss of just one of the two mirror copies, and all your data is still
available.  This assumes you are using LVM mirroring.  SSA adapter
mirroring will not work in this arrangement because all mirror copies in
adapter mirroring must be in the same loop -- a big negative.  If you
stripe you should keep all disks of a particular stripeset in a single
loop.  This is required if you do striping in the SSA adapter.  If you
do LVM striping (which is not a bad idea, IMHO) you'll have to take
particular care to make sure all stripe units for a particular striped
logical volume are in a single loop.  Ideally a stripe should involve
disks served primarily by each adapter, so that I/O to/from a stripe
will involve both adapters.  (A disk's "primary adapter" is generally
the adapter "closest" to the disk in the SSA loop).

An aside: I tend to favor doing my mirroring and striping using the LVM,
not the adapter.  I find this approach is more flexible and
(surprisingly) may perform slightly better (according to some of the
docs I've read).

Striping could definitely help your performance.  I don't have a
particluar stripe size recommendation, especially since I don't know
anything about administering Oracle.  But even un-optimized striping can
be beneficial if for no other reason than it helps balance I/O across as
many drives as possible (a good thing).

I recommend making a detailed drawing of all your drives and adapters
showing all interconnections.  This will help make sense of things and
will allow you to mentally experiment with "what if" ideas.

There are a number of good technical resources available, but they won't
(unfortunately) do the analysis for you.  The SSA docs are available at
http://www.storage.ibm.com/hardsoft/products/ssa/docs/index.html.  See,
for example, the "Advanced SerialRAID Adapters Technical Reference."
There are two extensive Redbooks available on the web covering LVM
topics: "AIX Logical Volume Manager from A to Z: Introduction and
Concepts" and "AIX Logical Volume Manager fro A to Z: Troubleshooting
and Commands."  Look for these at http://www.redbooks.ibm.com.  This is
also a nice page:
http://www.storage.ibm.com/hardsoft/products/ssa/docs/config_rules.html.

Regards,

Pittsburg State University    pittstate.edu
Pittsburg, KS

Hopefully you'll benefit from the following few tips:

1) read (browse through) the book: "Oracle 7 for UNIX Performance
Tuning Tips". The reason I recommend this book is because it is
available on-line at: http://www.wtainc.com/resource/1460.pdf
If you find the same book for Oracle 8 or 9 go for it. However, I
believe principle is the same.

Pay special attention to "Tunning Disk I/O" and then "Appendix 1: OFA
Rules".

2) Your DBAs are likely to tell you that all you have to do is put all
disks in a single volume group, create a /u01 filesystem and walk
away. This is wrong, and is a result of reading just Appendix 1 in the
book mentioned above. If you do that the system performance will be
lusy, to say the least.

3) Next, read "Monitoring and Managing SSA Disk Subsystems", Appendix
F: "Laying out Oracle Database on AIX Disks" at:
http://www.redbooks.ibm.com/redbooks/SG245251.html

Combine the information from 1, 2 and 3, get a good night sleep and it
will all be clearer than mud in the morning.

4) If you read any non-IBM book keep in mind that most DBAs are
trained and most of development is done on SUN platform and every
advice _should_ be taken with a grain of salt (and a few other
selected spices).

And whatever you do, have fun.

mj

PS: I addressed only performance issues. I believe there are already a
few posts regarding data availability using mirroring. Take that into
account too.