RA8000 performance.

RA8000 performance.

Post by d0 » Fri, 14 Jun 2002 06:25:12



We currently have an RA8000 attached to an ES-40 and using ADFVS.  The
HBA in the ES-40 is a kzpba-cb and the raid controller is an HSZ80.
There are a total of 16 drives in the cabinent(ds-rz1fc-vw 36.4GB 10k
rpm UW). We have hit a wall on performance after doubling the number of
users using the system. This wall is the throughput on the RA8000 and HBA.

It seems that after running various tests involving cp, mv, dd, cat,
etcetera and watching the iostats with monitor and using time and
calculating them I have noticed the raid is averaging about 19MB/s. This
is grossly under performing what it should be doing. The internals of
the raid claim a 40MB/s ability while the HBA claims 80MB/s. It seems
odd that we would not even be seeing half that. All drives in the raid
check out fine as well as the cabling, cache hits, fs etc appear ok. CPU
usage is not burdend and even if it was I'd think it would have little
performance hits on the likes of dd or cat. I could be wrong. These
tests were run between seperate HBAs to the raid and even just locally
on the raid all with about the same results. I used binary files varying
in size from 18MB to 380MB in the tests.

Does anyone have suggestions as to what might be configured wrong or
what needs to be done to up the performance of this system?  Or does
anyone else out there have stats on a similar setup and if so how do
they compare?

TIA

 
 
 

RA8000 performance.

Post by Quarter Horsem » Sat, 15 Jun 2002 02:07:55


What version, patch level of the operating system?

Are the HSZ80 disks JBODs or mirrors stripes or RAIDsets,
and if applicable, what's the stripe width or chunk size?

How are you using the HSZ80 writeback cache?

If you're not using JBODs, with respect to the HSZ80 SCSI
busses are you being "buss-smart" in arranging the spindles
inside the disk cab?  If so, and further, are you being
"buss-smart" in arranging stripes, mirrors, and RAIDsets?

How are the HSZ units partitioned, and what partitions are
you using for your tests?

You have only one HSZ80 controller, correct?

Any abnormal messages coming from the HSZ80 console?

Are you running the tests on an otherwise-quiet system?

Are you using LSM?

What's your AdvFS setup in terms of domains and filesets
involved in the IO tests?

Have you explored what the collect/collgui tool can tell you
about the running system, especially around active and wait
queues?  Do not underestimate the benefit of this tool.

What does "between separate HBAs" mean?  Is there a cp source
or target somewhere else, and might this be a bottleneck?

There's a start... hope it helps.

QH

 
 
 

RA8000 performance.

Post by d0 » Sun, 16 Jun 2002 01:17:35


Forgive me is I am not as fluent on the terms or if you leave me behind
on issues. I am by no means a Tru64 let alone ADVFS or Compaq RA systems
person. Anyhow, here goes:

Quarter Horseman wrote:
> What version, patch level of the operating system?

4.0e

AFAADVANCED435       installed  Advanced File System Advanced Utilities,
v4.0e, r435(General Applications)
AFAADVDAEMON435      installed  Advanced File System Daemon, v4.0e,
r435(General Applications)
AFAADVGUI435         installed  Advanced File System Graphical User
Interface, v4.0e, r435(General Applications)
AFAADVMAN435         installed  Advanced File System Advanced Utilities
Reference Pages, v4.0e, r435(General Applications)

OSFPAT00004600435    installed  Patch: AdvFS Boot Code Problem
Traversing Symbolic Links(Filesystem Patches)
OSFPAT00030300435    installed  Patch: AdvFS Not Ensuring Links to
/etc/fdmns Directory(Filesystem Patches)
OSFPAT00057800435    installed  Patch: advfs showfdmn command may core
dump(Filesystem Patches)

> Are the HSZ80 disks JBODs or mirrors stripes or RAIDsets,
> and if applicable, what's the stripe width or chunk size?

RAIDset level 5.

Chunk size is 128.

> How are you using the HSZ80 writeback cache?

Not sure what is meant by "How are you using...".  Its turned enabled.

WRITEBACK_CACHE is enabled
MAXIMUM_CACHED_TRANSFER=32
READAHEAD_CACHE is enabled
READ_CACHE is enabled

> If you're not using JBODs, with respect to the HSZ80 SCSI
> busses are you being "buss-smart" in arranging the spindles
> inside the disk cab?  If so, and further, are you being
> "buss-smart" in arranging stripes, mirrors, and RAIDsets?

Not sure if I am being smart or not. Apparently not too smart
considering I dont have this system running in top shape.

Name          Type                      Port Targ  Lun        Used by

------------------------------------------------------------------------------

DISK10000     disk                         1    0    0        R1

DISK10100     disk                         1    1    0        R1

DISK20000     disk                         2    0    0        R1

DISK20100     disk                         2    1    0        R1

DISK30000     disk                         3    0    0        R1

DISK30100     disk                         3    1    0        R1

DISK40000     disk                         4    0    0        R1

DISK40100     disk                         4    1    0        R1

DISK50000     disk                         5    0    0        R1

DISK50100     disk                         5    1    0        R1

DISK60000     disk                         6    0    0        R1

DISK60100     disk                         6    1    0        R1

Name          Storageset                     Uses             Used by

------------------------------------------------------------------------------

R1            raidset                        DISK10000        D100

                                              DISK10100

                                              DISK20000

                                              DISK20100

                                              DISK30000

                                              DISK30100

                                              DISK40000

                                              DISK40100

                                              DISK50000

                                              DISK50100

                                              DISK60000

                                              DISK60100

> How are the HSZ units partitioned, and what partitions are
> you using for your tests?

Name          Storageset                     Uses             Used by

------------------------------------------------------------------------------

R1            raidset                        DISK10000        D100

                                              DISK10100

                                              DISK20000

                                              DISK20100

                                              DISK30000

                                              DISK30100

                                              DISK40000

                                              DISK40100

                                              DISK50000

                                              DISK50100

                                              DISK60000

                                              DISK60100

For the tests I have used utils such as dd, cp, mv and applications as
well as defragment.

I have also used  VTDPY to monitor the raid during operations.

I do not think these tests are by any means a true "fair" picture but I
would think that I would still see better than 19MB/s.

Tests were run simply creating new files on the raid locally and going
between other devices on the machine.  All resulted in near identicle
throughput.

Is there a standard test I might run without taking the raid down?

I intend to try things like DILX on the controller but have yet to reach
a point that can be done easily.

> You have only one HSZ80 controller, correct?

Correct.

> Any abnormal messages coming from the HSZ80 console?

Only one thus far and just started appearing lastnight is:

"

Cache battery is near its end of life, it should be replaced SOON.  Run frutil-
to replace.

"

I have an order in for a replacement.

> Are you running the tests on an otherwise-quiet system?

Tried it both during production and off-hours(VERY little activity).

> Are you using LSM?

Not that I am aware of.  I do not think we have licensing for that...

> What's your AdvFS setup in terms of domains and filesets
> involved in the IO tests?

One advfs domain on the raid and local test were run with it.  Also we
have another RAID 450 also advfs and I ran tests from it to the RA8000
and visa versa.

> Have you explored what the collect/collgui tool can tell you
> about the running system, especially around active and wait
> queues?  Do not underestimate the benefit of this tool.

No I have not. I was not aware of this utility.

Only thing I have run with regard to that sort of thing is sys_check
-perf and took into account its suggestions.

> What does "between separate HBAs" mean?  Is there a cp source
> or target somewhere else, and might this be a bottleneck?

I have tried between the Host Bus Adapters of two raids and single
drives all on seperate HBAs.  ie: Between two KZPBA-CBs or CXs.  Of
course I also tried these test only running on the RA8000 and not going
between other devices.

> There's a start... hope it helps.

> QH

I appreciate all the help. I'm by NO means an expert here. I'd actually
say Im a novice and with regard to configs on these raids a total newbie.

BTW,
DISK60100     disk                         6    1    0        R1

           COMPAQ   BC036122C3       3B09

         Switches:

           NOTRANSPORTABLE

           TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ
negotiated)
           LOCAL

         Size: 71114123 blocks

         Configuration being backed up on this container

 
 
 

RA8000 performance.

Post by d0 » Sun, 16 Jun 2002 03:31:00


Another thing to note on is this raid is close to being maxed out.

40GB is available now out of 380 or so.

I assume this could also heavily impact performance?

 
 
 

RA8000 performance.

Post by Quarter Horsem » Tue, 18 Jun 2002 21:26:44


JN, thanks for the extra info.  I believe what you are seeing is
normal and that the RAIDset itself is the bottleneck, bringing along
with it cache flooding and possible IO retries.  This is a
farily-large set.  Remember that for every transfer the HSG80 must
calculate parity, divvy up the data and parity across all 12 spindles,
and complete it before it can attend to the next IO waiting in cache.
That's a lot of work.  Your disks are arranged correctly IO-wise in
the cab (fanned out across controller busses) but at least in a
textbook sense you have the possibility of contention from any
xx0xx/xx1xx pair in the RAIDset although the later cabs and
controllers are far less-susceptible to this.  In general no RAIDset
is going to perform as well as a plain disk or a stripe.  There are
special cases where it might.  Have you thought of RAID 0+1 as an
alternative to RAID 5?  RAID 5 is known as "poor man's fault
resilience" while RAID 0+1 is the performance alternative to RAID 5.
Just a thought.....

Hope this helps -- maybe Rollow will correct any mistakes I've made,
but I'm pretty sure I've got the bulk of it right.

QH

p.s.  Watch out for MAXIMUM_CACHED_TRANSFER which may hurt you on
larger block sizes.  The current value may be appropriate for your
production workload but may not be appropriate for the testing you're
doing (which is sort of a hint to be sure the testing you're doing is
relevant to the work that's going to get thrown at the cab in
production ;-) ).

QH

 
 
 

RA8000 performance.

Post by d0 » Wed, 19 Jun 2002 00:53:50



> JN, thanks for the extra info.  I believe what you are seeing is
> normal and that the RAIDset itself is the bottleneck, bringing along
> with it cache flooding and possible IO retries.  This is a
> farily-large set.  Remember that for every transfer the HSG80 must
> calculate parity, divvy up the data and parity across all 12 spindles,
> and complete it before it can attend to the next IO waiting in cache.
> That's a lot of work.  Your disks are arranged correctly IO-wise in
> the cab (fanned out across controller busses) but at least in a
> textbook sense you have the possibility of contention from any
> xx0xx/xx1xx pair in the RAIDset although the later cabs and
> controllers are far less-susceptible to this.  In general no RAIDset
> is going to perform as well as a plain disk or a stripe.  There are
> special cases where it might.  Have you thought of RAID 0+1 as an
> alternative to RAID 5?  RAID 5 is known as "poor man's fault
> resilience" while RAID 0+1 is the performance alternative to RAID 5.
> Just a thought.....

> Hope this helps -- maybe Rollow will correct any mistakes I've made,
> but I'm pretty sure I've got the bulk of it right.

> QH

> p.s.  Watch out for MAXIMUM_CACHED_TRANSFER which may hurt you on
> larger block sizes.  The current value may be appropriate for your
> production workload but may not be appropriate for the testing you're
> doing (which is sort of a hint to be sure the testing you're doing is
> relevant to the work that's going to get thrown at the cab in
> production ;-) ).

> QH

Thanks for the feedback. I will look further into optimizing the MCT for
our particular usage.  I suppose when they fill the raid up I will look
into rebuilding it with 0+1 in mind.
 
 
 

RA8000 performance.

Post by Islan » Wed, 24 Jul 2002 04:16:32


40mb/sec is the max rate (burst)
'around 20mb/dsec is to be expected

As for a 80mb/sec controller - only if running in ultra2 mode

HSZ80's are ultra wide (40mb/sec burst)

DT

--
David B Turner
Sales Dpt
Island Computers US Corporation
2700 Gregory Street
Suite 180
Savannah GA 31404
Tel: 912 447 6622
Fax: 912 201 0096

www.islandco.com
http://www.islandco.com/legal-email.htm

We sell Alpha's !
All emails are checked for Virus and Worms

Quote:> We currently have an RA8000 attached to an ES-40 and using ADFVS.  The
> HBA in the ES-40 is a kzpba-cb and the raid controller is an HSZ80.
> There are a total of 16 drives in the cabinent(ds-rz1fc-vw 36.4GB 10k
> rpm UW). We have hit a wall on performance after doubling the number of
> users using the system. This wall is the throughput on the RA8000 and HBA.

> It seems that after running various tests involving cp, mv, dd, cat,
> etcetera and watching the iostats with monitor and using time and
> calculating them I have noticed the raid is averaging about 19MB/s. This
> is grossly under performing what it should be doing. The internals of
> the raid claim a 40MB/s ability while the HBA claims 80MB/s. It seems
> odd that we would not even be seeing half that. All drives in the raid
> check out fine as well as the cabling, cache hits, fs etc appear ok. CPU
> usage is not burdend and even if it was I'd think it would have little
> performance hits on the likes of dd or cat. I could be wrong. These
> tests were run between seperate HBAs to the raid and even just locally
> on the raid all with about the same results. I used binary files varying
> in size from 18MB to 380MB in the tests.

> Does anyone have suggestions as to what might be configured wrong or
> what needs to be done to up the performance of this system?  Or does
> anyone else out there have stats on a similar setup and if so how do
> they compare?

> TIA