How to increase write speed to local hard drive?

How to increase write speed to local hard drive?

Post by Ross » Sat, 24 Jun 2006 23:01:23



Hi there,
I need to write about 4,000 files (the total size is just 500MB) to local
ufs partition.
But it takes about 35 minutes.
To delete them, it still takes about 20 minutes.
Is there a way to tune the local I/O to increase the write speed?
BTW, it's a Solaris 8 running on a Sun Fire server with SCSI hard drive in
standard configuration.
Thanks in advance,
Ross
 
 
 

How to increase write speed to local hard drive?

Post by greek_philosophi.. » Sat, 24 Jun 2006 23:09:41



> Hi there,
> I need to write about 4,000 files (the total size is just 500MB) to local
> ufs partition.
> But it takes about 35 minutes.
> To delete them, it still takes about 20 minutes.
> Is there a way to tune the local I/O to increase the write speed?
> BTW, it's a Solaris 8 running on a Sun Fire server with SCSI hard drive in
> standard configuration.
> Thanks in advance,
> Ross

Since it sounds like you are not keeping the files perhaps a tmpfs
filesystem
would be quicker?

.

 
 
 

How to increase write speed to local hard drive?

Post by Ross » Sun, 25 Jun 2006 00:22:00


Thanks Greek.
But I need to keep them for a while after reboot.
Are there some other things I can try (like enabling DMA)?
Thanks again,
Ross



>> Hi there,
>> I need to write about 4,000 files (the total size is just 500MB) to local
>> ufs partition.
>> But it takes about 35 minutes.
>> To delete them, it still takes about 20 minutes.
>> Is there a way to tune the local I/O to increase the write speed?
>> BTW, it's a Solaris 8 running on a Sun Fire server with SCSI hard drive
>> in
>> standard configuration.
>> Thanks in advance,
>> Ross

> Since it sounds like you are not keeping the files perhaps a tmpfs
> filesystem
> would be quicker?

> .

 
 
 

How to increase write speed to local hard drive?

Post by Rich Tee » Sun, 25 Jun 2006 01:18:00


Please don't top post.

Quote:> Thanks Greek.
> But I need to keep them for a while after reboot.
> Are there some other things I can try (like enabling DMA)?

Have you tried mounting the file system with the "noatime" option,
and/or enabled UFS logging?

You might also see some imporvment if you upgraded from Solaris 8.

--
Rich Teer, SCNA, SCSA, OpenSolaris CAB member

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-group.com/rich

 
 
 

How to increase write speed to local hard drive?

Post by Ross » Sun, 25 Jun 2006 05:58:28


Thanks Rite!
With noatime and logging options, the write time decreased from 35 minutes
to 20 minutes!
But it's still not good enough because for the same files on Linux machine
with IDE/Ext3 partition, it just takes 6 minutes!
Any more idea will be greatly appreciated,
Ross



> Please don't top post.

>> Thanks Greek.
>> But I need to keep them for a while after reboot.
>> Are there some other things I can try (like enabling DMA)?

> Have you tried mounting the file system with the "noatime" option,
> and/or enabled UFS logging?

> You might also see some imporvment if you upgraded from Solaris 8.

> --
> Rich Teer, SCNA, SCSA, OpenSolaris CAB member

> President,
> Rite Online Inc.

> Voice: +1 (250) 979-1638
> URL: http://www.rite-group.com/rich

 
 
 

How to increase write speed to local hard drive?

Post by Casper H.S. Di » Sun, 25 Jun 2006 17:54:49



>Thanks Rite!
>With noatime and logging options, the write time decreased from 35 minutes
>to 20 minutes!
>But it's still not good enough because for the same files on Linux machine
>with IDE/Ext3 partition, it just takes 6 minutes!
>Any more idea will be greatly appreciated,

Set the following kernel variable in /etc/system:

        set ufs:ufs_WRITES = 0

On a *quiescent* system, you can set this at runtime using:

        echo ufs_WRITES/0 | adb -wk

This disable the UFS write throttling feature.

Casper

 
 
 

How to increase write speed to local hard drive?

Post by tunl » Mon, 26 Jun 2006 06:02:06



> Thanks Rite!
> With noatime and logging options, the write time decreased from 35 minutes
> to 20 minutes!
> But it's still not good enough because for the same files on Linux machine
> with IDE/Ext3 partition, it just takes 6 minutes!
> Any more idea will be greatly appreciated,
> Ross

Ross,  The usual reason for a Linux PC to be a lot faster is that the
IDE  disk Write Cache   is enabled.  That  means that  the data is not
written to disk  until some time later.  If there was a power faliure
all the  8 MB of data  in the write cache  would be lost.

So you are tradeing data security  for speed.

To get a  a more  comparable base line  you should turn
the PC IDE disk write cache off  an remeasure the time
it takes to do the operations on linux.

One of the high points  in the feature list from solaris 8  to solaris
9
was UFS speed improvment although many of these imrovments was
backported to the later revisions  of Solaris8
the  02/02   ( feb, 2002 )  had many of them.

//Lars

 
 
 

How to increase write speed to local hard drive?

Post by Daniel Roc » Mon, 26 Jun 2006 08:58:19




>> Thanks Rite!
>> With noatime and logging options, the write time decreased from 35 minutes
>> to 20 minutes!
>> But it's still not good enough because for the same files on Linux machine
>> with IDE/Ext3 partition, it just takes 6 minutes!
>> Any more idea will be greatly appreciated,
>> Ross

> Ross,  The usual reason for a Linux PC to be a lot faster is that the
> IDE  disk Write Cache   is enabled.  That  means that  the data is not
> written to disk  until some time later.  If there was a power faliure
> all the  8 MB of data  in the write cache  would be lost.

Solaris also *always* enables the write cache for IDE disks - regardless
of the original setting:

http://cvs.opensolaris.org/source/xref/on/usr/src/uts/intel/io/dktp/c...
http://cvs.opensolaris.org/source/xref/on/usr/src/uts/intel/io/dktp/c...

If you want to forcibly turn write cache off you have to put:

set ata:ata_write_cache = -1

in /etc/system

--
Daniel

 
 
 

How to increase write speed to local hard drive?

Post by Robert Lawhea » Mon, 26 Jun 2006 09:17:52





>>> Thanks Rite!
>>> With noatime and logging options, the write time decreased from 35 minutes
>>> to 20 minutes!
>>> But it's still not good enough because for the same files on Linux machine
>>> with IDE/Ext3 partition, it just takes 6 minutes!
>>> Any more idea will be greatly appreciated,
>>> Ross

>> Ross,  The usual reason for a Linux PC to be a lot faster is that the
>> IDE  disk Write Cache   is enabled.  That  means that  the data is not
>> written to disk  until some time later.  If there was a power faliure
>> all the  8 MB of data  in the write cache  would be lost.

> Solaris also *always* enables the write cache for IDE disks - regardless
> of the original setting:

> http://cvs.opensolaris.org/source/xref/on/usr/src/uts/intel/io/dktp/c...
> http://cvs.opensolaris.org/source/xref/on/usr/src/uts/intel/io/dktp/c...

> If you want to forcibly turn write cache off you have to put:

> set ata:ata_write_cache = -1

> in /etc/system

Though it would be a little risky, and I'm not recommending it,
I believe that using Casper's "fastfs" would improve performance.
Google <solaris fastfs>.

Bob

 
 
 

How to increase write speed to local hard drive?

Post by Casper H.S. Di » Mon, 26 Jun 2006 18:56:00



>Solaris also *always* enables the write cache for IDE disks - regardless
>of the original setting:

Not in Solaris 8, 9 or 10.  Not on Solaris SPARC, either.

In those release it just keeps the default (which is off for most
Sun branded disks and ON for all other disks)

Quote:>http://cvs.opensolaris.org/source/xref/on/usr/src/uts/intel/io/dktp/c...
ata/ata_disk.c#130
>http://cvs.opensolaris.org/source/xref/on/usr/src/uts/intel/io/dktp/c...

ata/ata_disk.c#2919

Quote:>If you want to forcibly turn write cache off you have to put:
>set ata:ata_write_cache = -1
>in /etc/system

Which has no effect in Solaris 8 or 9 or 10 (except perhaps some
updates) or on SPARC.

ZFS will now enable write caches if it owns the whole disk and it
will flush the cache at the appropriate moments.

Casper

 
 
 

How to increase write speed to local hard drive?

Post by tunl » Mon, 26 Jun 2006 21:45:17





> >> Thanks Rite!
> >> With noatime and logging options, the write time decreased from 35 minutes
> >> to 20 minutes!
> >> But it's still not good enough because for the same files on Linux machine
> >> with IDE/Ext3 partition, it just takes 6 minutes!
> >> Any more idea will be greatly appreciated,
> >> Ross

> > Ross,  The usual reason for a Linux PC to be a lot faster is that the
> > IDE  disk Write Cache   is enabled.  That  means that  the data is not
> > written to disk  until some time later.  If there was a power faliure
> > all the  8 MB of data  in the write cache  would be lost.

> Solaris also *always* enables the write cache for IDE disks - regardless
> of the original setting:

> http://cvs.opensolaris.org/source/xref/on/usr/src/uts/intel/io/dktp/c...
> http://cvs.opensolaris.org/source/xref/on/usr/src/uts/intel/io/dktp/c...

> If you want to forcibly turn write cache off you have to put:

> set ata:ata_write_cache = -1

> in /etc/system

> --
> Daniel

The OP  is comparing   the  UFS performans on a Solaris 8 Sparc based
SCSI disk
with  the performance of EXT3  on a LINUX  PC  with IDE disks.

LInux wins here because data security is sacrificed to obtain speed on
IDE disks.

The OP will have to decide wheter he can live with loosing the  8 MB of
data in the
write cache which  LInux belives it has written to the disk but hasen't
 -
 in case of a power faliure, or if he cant live with it.
If he can't live with it, he'll have to turn off the LINUX PC IDE disk
write cache,
then his performance on linux will drop drastically.

 A system that have high requirements on data integrity/security cant
have
disks writecache active. since you are basically lying to the
operatingsystem
and saying that data has been written to disk when it hasen't - yet .

Then again if you have a high reliabilty  disksystem  like a Hitatchi
or EMC
SAN device which has its own internal battery backup you do trust the
disksystem to  cache and handle the data as soon as you have written it
away.

The OP's  ( and most users of PC based operating systems  )
 problem,  is that  the write cache on a small
IDE disk is usually not battery backuped - and will fail  with very
small power
fluctuations.

 
 
 

How to increase write speed to local hard drive?

Post by llotha » Mon, 26 Jun 2006 23:54:47


With 8 MB it seems that you are talking about the harddrive cache. This
cache will not be longer used then a few ms until one rotation is done.
IDE drives normally only use this for some cylinder read ahead, write
behind caching.

SCSI are more stupid here (called clever in the past where the OS were
stupid). But normally the time is very low  - measured in milliseconds.

 
 
 

How to increase write speed to local hard drive?

Post by Stefan Krüge » Tue, 27 Jun 2006 03:02:35



> ZFS will now enable write caches if it owns the whole disk and it
> will flush the cache at the appropriate moments.

hm let's assume solaris support's ZFS boot in the near future, i.e. you
can install right to a zfs /, /usr, /var etc.; what about swap? you need
a partition/slice for swap so zfs on such a system will never own the
whole disk, so no whole disk = no write cache = lower performance?

or are there plans to intergrate swap handling into zfs somehow?
encrypted swap with zfs-crypto, sounds nice, don't you think? ;)

 
 
 

How to increase write speed to local hard drive?

Post by Tim Bradsha » Tue, 27 Jun 2006 04:34:08



> With 8 MB it seems that you are talking about the harddrive cache. This
> cache will not be longer used then a few ms until one rotation is done.
> IDE drives normally only use this for some cylinder read ahead, write
> behind caching.

> SCSI are more stupid here (called clever in the past where the OS were
> stupid). But normally the time is very low  - measured in milliseconds.

And milliseconds count, in fact: it means that every time the system
does a write it needs to wait until the write is safely on disk if it
wants to be safe, which perhaps takes several ms, which might easily be
more than the time to actually write the data.

Of course, you can just elect not to be safe, but 8MB is quite a lot to
lose, even if you only lose it occasionally.  And with large numbers of
spindles you don't lose it only occasionally.

SCSI actually is much smarter, because the tagged-queuing stuff lets
the system have several writes in flight at once to the disk, with the
disk notifying when they complete, so it can often hide the latency,
without needing to rely on an unreliable write-cache (it's more-or-less
the same trick that processors do to hide latency).  I think that SATA
disks (and maybe just plain ATA ones) can do this now too.

--tim

 
 
 

How to increase write speed to local hard drive?

Post by Andrew Gabri » Tue, 27 Jun 2006 04:50:31




Quote:

> hm let's assume solaris support's ZFS boot in the near future, i.e. you
> can install right to a zfs /, /usr, /var etc.; what about swap? you need
> a partition/slice for swap so zfs on such a system will never own the
> whole disk, so no whole disk = no write cache = lower performance?

> or are there plans to intergrate swap handling into zfs somehow?
> encrypted swap with zfs-crypto, sounds nice, don't you think? ;)

ZFS supports creation of emulated volumes, which you can swap to.

--
Andrew Gabriel

 
 
 

1. Can SUSE 7.2 be installed from a local hard drive to a local hard drive

I have 2 hard drives hda and hdc on redhat 2.2.5-15 kernel.

I downloaded SuSE 7.2 and put the iso file on hda5, wrote
it to the device so I can mount the file iso.

I would like  the boot floopy to find /dev/hda5
instead of the cdrom to do the install from.

Can this be done? Redhat has a premade floppy for this
so I am hoping somewhere I can get this for Suse.

                           Thanks in advance to those who help

2. linux via pcmcia

3. how to increase single drive size with new hard drive

4. Korne Shell Commands?

5. How to increase the mksysb speed in mkcd with CD-R Drive ?

6. Powerbook (Lombard/Bronze) G3 1999 XFree

7. Increasing Data Size Beyone my Hard Drive

8. ftp and telnet for DOS

9. Increasing Hard Drives

10. alternative way to increase hard drive size?

11. RAID 5 hard drive size increase

12. copying hard drive to hard drive

13. Clustering - Network Speed vs Local Bus Speed