Very Computer

We've been running an Ultra 10 (Solaris 2.6, 1x UltraSparcII 440MHz
CPU, 2x 9Gb EIDE, 512Mb RAM) as a development platform for an
application that runs with Oracle 8.1.7 - performance is acceptable.

The test environment for the app is a Sun Blade 1000 (Solaris 2.8, 1x
UltraSparcIII 750MHz, 2x 36Gb FC-AL, 1Gb RAM), with exactly the same
DB configuration, and loads as the dev platform.

The intended production environment for the app will be a Sun Blade
1000 (Solaris 2.8, 2x UltraSparcIII 750MHz, 2x 36Gb FC-AL, 1Gb RAM).

All things being equal, can we assume that the Sun Blade configuration
will outperform the Ultra 10 ... ?

... well here's a tale:

A fairly disk intensive SQL query (run via sqlplus) to remove
duplicate postcodes from a table on the Ultra 10 takes approximately 2
minutes to return its record set.

On the test environment on the Sun Blade, with the UFS filesystems and
database tablespaces layout exactly the same as the Ultra 10, the same
query against the same volume of data takes 7 minutes.

Using top, sar, iostat, vmstat and mpstat all point to a massive
amount of waiting for io on the Sun Blade - generally 95-100% wio. On
the Ultra 10, WIO is maybe 50% for the duration of the query.

Our sys adm's have applied every patch cluster under the sun [sic] for
Solaris 8 on the Sun Blade, even HDD Firmware patches, switched UFS
logging on & off, low level disk scanning with format, etc, etc, while
the DBA's have been through both initora config's with a fine
toothcomb, set a huge sort_area_size in oracle, sized the SGA, set
/etc/system shared memory and semaphore settings accordingly. Even the
64-bit version of oracle was installed on the Sun Blade.

All to no avail: the Ultra 10 still seems to cut through the query so
much quicker than the Blade. Any improvement gained on the Sun Blade,
when the same config is made to the Ultra 10, well that just gets
quicker too.

The problem doesn't appear to be restricted to Oracle DB access
though: The Sun Blade seems to have a 'general sluggishness' about it.
If I perform the following command on the ultra 10 in a ufs file
system:-

   $ time tar xf lmbench-patch1.tar

   real    0m1.451s
   user    0m0.080s
   sys     0m0.290s

which unpacks 360 files into a tree under a directory LMbench. Then
deleting:

   $ time /bin/rm -rf LMbench/

   real    0m0.548s
   user    0m0.010s
   sys     0m0.090s

Running the same on the test Sun Blade, again the same tar file onto a
ufs file system:-

   $ time tar xf lmbench-2.0-patch1.tar

   real    0m4.670s
   user    0m0.020s
   sys     0m0.170s

and removing the resultant directory:-

   $ time /bin/rm -rf LMbench

   real    0m4.179s
   user    0m0.020s
   sys     0m0.070s

[reason I use /bin/rm is because i have alias rm='rm -i' in my
profile]

I know these are fairly crude tests, but it does illustrate that there
are real differences in doing the same commands on the same files on
the different machines: In both cases the real time taken is a lot
longer on the Sun Blade than Ultra 10, while the system and user time
are less. WIO stats jump to >95% ... Why should this be? At the time
of testing there was no load on either machine.

Unpacking & removing in a tmpfs filesystem (/tmp) is a lot lot
quicker, obviously because tmpfs is memory-based. Apart from having
logging on ufs in 2.8 (which whether switched on or off seems to make
no difference in above tests) what are the major differences between
ufs on 2.6 and 2.8.

These results have been bounced through to Sun, through our 3rd party
support agency, who have apparrently set up an Ultra 10 and Sun Blade
and have confirmed that their setup the Blade is better, and that our
issue is obviously down to the configuration or load on the machine.

So anyway, we've now taken delivery of the first Sun Blade to be used
in production, unpacked it, switched it on, installed 2.8 from fresh,
applied patches and run the same tar & rm tests... same preliminary
results as the test Sun Blade. Oracle has not been installed yet, so I
havent run the sql queries to see what it's like.

The obvious difference in the ultra 10 and the blade 1000, is the OS
used, 2.6 vs 2.8. I guess what I'm asking is there anything we have
glaringly missed out of the config of the Sun Blade? Is there such a
massive difference in 2.6 and 2.8 to cause this, or is the Sun Blade's
performance normal, and we have an (ahem) exceptionally fast Ultra 10?

If anyone can come up with a plausible explanation for this, or at
least a lead to work on,

TIA

Rich

[And just to throw a spanner in the works, our apps have been ported
to run on a Linux box (kernel 2.4.5, dual PIII 933MHz, 2x 9Gb SCSI 15K
rpm, 1Gb Ram, Oracle 8.1.7) and the app and all the tests run way, way
faster that the Ultra 10 and Sun Blade. Of course, all the usual
corporate reasons prevent us from using linux...]

[I will omit your comprehensive description.]

What I didn't catch from your post is whether you tried this same test
on Solaris 8 on an Ultra 10 or Solaris 2.6 on the Blade. This would
point a more accusing finger at either the hardware or the OS. (I
don't know if you can run 2.6 on the Blade.)

You may also want to try running pea.se in the SE Toolkit
(with "se -DWIDE pea.se") and monitor the charios column to see
if there is a significant difference in I/O on each OS.

Rich

---- Author, SE Toolkit
---- SE Toolkit.com                        http://www.setoolkit.com

Quote:

> What I didn't catch from your post is whether you tried this same test
> on Solaris 8 on an Ultra 10 or Solaris 2.6 on the Blade. This would
> point a more accusing finger at either the hardware or the OS. (I
> don't know if you can run 2.6 on the Blade.)

No, we've only used Solaris 2.6 on the Ultra 10, and only Solaris 2.8 on the
Sun Blade. I am lead to believe from our 3rd party support agency that 2.6
won't run on the Blade - I don't know if this is true of 2.7 though.
Timescales haven't permitted us to rebuild the dev machine with 2.8, and
funds (lack of) prevent us from getting hold of another ultra10 to play
with.

An interesting development has however arisen since my first post:
apparently Sun support didn't actually perform their tests on a Blade, but a
Fire 280-R instead, which for all intended purposes, shares the same
internals with the Blade. They have apparently re-run the tests on a Blade,
and  have reproduced the situation I described.

Quote:> You may also want to try running pea.se in the SE Toolkit
> (with "se -DWIDE pea.se") and monitor the charios column to see
> if there is a significant difference in I/O on each OS.

Latest SE Toolkit is already installed and running on the Ultra 10, and the
'virtual adrian' module is always lurking somewhere in the process list.
Unfortunately the toolkit isn't running on the Test box (blade), because the
BOFH's who control this machine are of the impression that 'SE Toolkit is
unstable and can crash Solaris' !!

From my experience, (a) SE Toolkit has never crashed our dev machine or any
other Solaris machine I've ever used, and (b) it's wise to keep the BOFH's
happy, so I never criticize their 'advice'...

Anyway, I will however run the pea module on the ultra10 and suggest again
that the toolkit is installed on the blade..

Thanks,

Rich

You may want to look hard at Solaris 9 x86 when it ships then. It claims
binary support for Linux apps.
--
'ric

 >

Quote:> An interesting development has however arisen since my first post:
> apparently Sun support didn't actually perform their tests on a Blade, but a
> Fire 280-R instead, which for all intended purposes, shares the same
> internals with the Blade. They have apparently re-run the tests on a Blade,
> and  have reproduced the situation I described.

This is a good step. A time consuming one but a good one.

Quote:> Latest SE Toolkit is already installed and running on the Ultra 10, and the
> 'virtual adrian' module is always lurking somewhere in the process list.
> Unfortunately the toolkit isn't running on the Test box (blade), because the
> BOFH's who control this machine are of the impression that 'SE Toolkit is
> unstable and can crash Solaris' !!

See http://www.setoolkit.com/prod.html.

BTW, I don't think v_a will tell you anything. I'm focusing on this problem
with pea.se to see there is significant differences in the amount of I/O
being performed on one machine versus the other.

Quote:> From my experience, (a) SE Toolkit has never crashed our dev machine or any
> other Solaris machine I've ever used, and (b) it's wise to keep the BOFH's
> happy, so I never criticize their 'advice'...

I'm not even going to start. I'll say something rude.

Quote:> Anyway, I will however run the pea module on the ultra10 and suggest again
> that the toolkit is installed on the blade..

As long as I'm in this part of the woods... I'd like to point out that
the UFS file system was an important advancement in technology.

ON A VAX 11/780.

I'm very much looking forward to the next generation file system on Solaris.

(mount -F ext3 ... hmmm).

Rich

---- Author, SE Toolkit
---- SE Toolkit.com                        http://www.setoolkit.com

Quote:>...
>The problem doesn't appear to be restricted to Oracle DB access
>though: The Sun Blade seems to have a 'general sluggishness' about it.
>If I perform the following command on the ultra 10 in a ufs file
>system:-

>   $ time tar xf lmbench-patch1.tar

>   real    0m1.451s
>   user    0m0.080s
>   sys     0m0.290s
>...

>Running the same on the test Sun Blade, again the same tar file onto a
>ufs file system:-

>   $ time tar xf lmbench-2.0-patch1.tar

>   real    0m4.670s
>   user    0m0.020s
>   sys     0m0.170s

well, maybe somehow DMA got turned off for the sunblade, or something silly
like that.
Cruddy disks, or substandard controller chip.

Ultra 10 uses SCSI, whereas sunblade uses FC-AL, right?

wonder what happens if you add a regular SCSI card to the sunblade,+
external disks, and try it again.

Quote:>[And just to throw a spanner in the works, our apps have been ported
>to run on a Linux box (kernel 2.4.5, dual PIII 933MHz, 2x 9Gb SCSI 15K
>rpm, 1Gb Ram, Oracle 8.1.7) and the app and all the tests run way, way
>faster that the Ultra 10 and Sun Blade. Of course, all the usual
>corporate reasons prevent us from using linux...]

So what happens when you put solaris x86 on that same box?

--
[Trim the no-bots from my address to reply to me by email!]
[ Do NOT email-CC me on posts. Pick one or the other.]

The word of the day is mispergitude

Quote:

> A fairly disk intensive SQL query (run via sqlplus) to remove
> duplicate postcodes from a table on the Ultra 10 takes approximately 2
> minutes to return its record set.

> On the test environment on the Sun Blade, with the UFS filesystems and
> database tablespaces layout exactly the same as the Ultra 10, the same
> query against the same volume of data takes 7 minutes.

You mention shared memory tuning in /etc/system, but I suspect
your problem is caused by write throttling on Solaris.
Have you added anything to tune UFS to /etc/system on the Blade?
For starters, try adding

set ufs:ufs_WRITES = 0

to /etc/system and reboot. Or if you want to try it without rebooting,
the following achieves the same result:

# adb -wk /dev/ksyms /dev/mem
ufs_WRITES/W 0
$q
#

There's no reason the Blade shouldn't be a lot faster on I/O than
the Ultra.

--
Stefaan (GPG Fingerprint 25D8 551B 4C0F BF73 3283 21F1 5978 D158 7539 76E4)
--
Please visit our Webster http://xxxxxxxx.xxxx.xxx, write or e-mail to X&x
promptly,if you are interested.And X&x shall be pleased to render you any
further services.                                      -- Spam from China

Wrong! Ultra 10 uses 5400 rpm E-IDE disks

Quote:> wonder what happens if you add a regular SCSI card to the sunblade,+
> external disks, and try it again.

Trouble is, we're near to over budget and too close to deadlines...

Quote:

> So what happens when you put solaris x86 on that same box?

The intel box was a loan box from Compaq (an ML-570, can't remember exact
model for sure), used to trial our app on Intel/Linux instead of
Sparc/Solaris, but we had to give it back. Our support agency flatly refused
to support Intel/Linux, so we said: 'Well what about Intel/Solaris?', which
they also flatly refused. They took no consideration of the
price/performance ratio's (much better) as compared to the blade machine,
just the fact that it was not the corporate standard, ie: Solaris on Sun, or
Windows NT on Intel. So, we couldn't justify using the Compaq, although the
database app and all disk-based performance was way way quicker, and it was
a cheaper machine too!

err, what? your "support agency"?

You're solaris admins, right? Install it yourself, do the benchmarks, then
take them to management.
It's a lot harder for them to turn down a proven (benchmark) winner that is
at least HALF the "standard" , vs non-standard.
linux is not anywhere in your stated standard. Solaris is, and intel
hardware is.

--
[Trim the no-bots from my address to reply to me by email!]
[ Do NOT email-CC me on posts. Pick one or the other.]

The word of the day is mispergitude