Very Computer

I have a 450 server that uses a DLT IV tape unit to backup files using
ufsdump (Level 0 for weekly).  The system is running Oracle.

The odd thing is: about 3 weeks ago, a full system backup (Oracle
shutdown, of course) runs for 3 1/2 hours max.  Now it is 6 hours
without the database size changing!

The only thing that changed was the addition of a file system to backup
- /opt (around 200 mega bytes).  Currently, there are 12 filesystems to
backup versus the original 11 filesystems.  Oddly, the throughput has
gone down by 2/3 (6000+ kbytes/sec originally to 2000+ kbytes/sec) for
the Oracle filesystems.  Again, there were no increases in the database
size.

Any idea on what is going on? Where to start troubleshooting?

Thanks.

The funny thing is that throughput dives only in certain filesystems.
The filesystems associated with Solaris has throughput in the 7000+ Kbps
while the ones in associated with Oracle drops to 2000+ kbps.

I will try your suggestion.

Thanks.

Quote:>I have a 450 server that uses a DLT IV tape unit to backup files using
>ufsdump (Level 0 for weekly).  The system is running Oracle.

>The odd thing is: about 3 weeks ago, a full system backup (Oracle
>shutdown, of course) runs for 3 1/2 hours max.  Now it is 6 hours
>without the database size changing!

>The only thing that changed was the addition of a file system to backup
>- /opt (around 200 mega bytes).  Currently, there are 12 filesystems to
>backup versus the original 11 filesystems.  Oddly, the throughput has
>gone down by 2/3 (6000+ kbytes/sec originally to 2000+ kbytes/sec) for
>the Oracle filesystems.  Again, there were no increases in the database
>size.

This may have many reasons.....

A point to start from may be a read speed test. Run:

        star -cM -time -C /mount-point . > /dev/null

to check the read speed with a speed optimized backup program.

ftp://ftp.berlios.de/pub/star/alpha/

--



URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

Quote:>   I have a 450 server that uses a DLT IV tape unit to backup files using
>   ufsdump (Level 0 for weekly).  The system is running Oracle.

        Orace rulez!

Quote:

>   The odd thing is: about 3 weeks ago, a full system backup (Oracle
>   shutdown, of course) runs for 3 1/2 hours max.  Now it is 6 hours
>   without the database size changing!

>   Again, there were no increases in the database size.

        Then check the UFS filesystems.

        find / -type f -size +60000 -ls

        That'll show files about 30 Mb and larger. As a matter of course, delete any
        core files you find before backing up (or put on a separate tape.)

-am     ? 2003

Quote:> I have a 450 server that uses a DLT IV tape unit to backup files using
> ufsdump (Level 0 for weekly).  The system is running Oracle.

> The odd thing is: about 3 weeks ago, a full system backup (Oracle
> shutdown, of course) runs for 3 1/2 hours max.  Now it is 6 hours
> without the database size changing!

Do a "ufsdump" on each filesystem, but replace your tape device
with "/dev/null".  (Be very careful to leave "u" out of the options,
otherwise this will mess up your /etc/dumpdates!)

This will tell you whether there is trouble reading the filesystems
quickly enough.  (It will show you the average data speed in
ufsdump's output.)  If there isn't trouble reading the filesystems,
you can assume it's probably trouble writing to the tape.

Also, when you added /opt, I assume you changed around the
partitions a little.  Is it possible that /etc/fstab has an
error or that your backup script is still using the
old /dev/rdsk/c?t?d?s? entries?  If so, it could be the
case that you're actually backing up the wrong devices
and writing too much data for that reason.

   - Logan

Quote:>  I have a 450 server that uses a DLT IV tape unit to backup files using
>  ufsdump (Level 0 for weekly).  The system is running Oracle.

>  The odd thing is: about 3 weeks ago, a full system backup (Oracle
>  shutdown, of course) runs for 3 1/2 hours max.  Now it is 6 hours
>  without the database size changing!

>  The only thing that changed was the addition of a file system to backup
>  - /opt (around 200 mega bytes).  Currently, there are 12 filesystems to
>  backup versus the original 11 filesystems.  Oddly, the throughput has
>  gone down by 2/3 (6000+ kbytes/sec originally to 2000+ kbytes/sec) for
>  the Oracle filesystems.  Again, there were no increases in the database
>  size.

>  Any idea on what is going on? Where to start troubleshooting?

Did you remount oracle filesystem with 'forcedirectio' option ?
Can you show output of 'mount' command ?

przemol

Here is part of the mount output:

I am getting largefile errors.  The disk are mirrored using DiskSuite.
Again, I have not seen this errors before:

/ on /dev/md/dsk/d150
read/write/setuid/intr/largefiles/onerror=panic/dev=1540096 on Sat Se
p  6 12:55:42 2003
/usr on /dev/md/dsk/d151
read/write/setuid/intr/largefiles/onerror=panic/dev=1540097 on Sat
 Sep  6 12:55:43 2003
/proc on /proc read/write/setuid/dev=37c0000 on Sat Sep  6 12:55:41 2003
/dev/fd on fd read/write/setuid/dev=3880000 on Sat Sep  6 12:55:44 2003
/etc/mnttab on mnttab read/write/setuid/dev=3980000 on Sat Sep  6
12:55:46 2003
/var on /dev/md/dsk/d154
read/write/setuid/intr/largefiles/onerror=panic/dev=154009a on Sat
 Sep  6 12:55:47 2003
/var/run on swap read/write/setuid/dev=1 on Sat Sep  6 12:55:47 2003
/tmp on swap read/write/setuid/dev=2 on Sat Sep  6 12:55:52 2003
/u00 on /dev/md/dsk/d350
read/write/setuid/intr/largefiles/onerror=panic/dev=154015e on Sat
 Sep  6 12:55:52 2003
/syncsort_tmp on /dev/md/dsk/d250
read/write/setuid/intr/largefiles/onerror=panic/dev=15400
fa on Sat Sep  6 12:55:52 2003
/u01 on /dev/md/dsk/d450
read/write/setuid/intr/largefiles/onerror=panic/dev=15401c2 on Sat
 Sep  6 12:55:52 2003
/ftphome on /dev/md/dsk/d253
read/write/setuid/intr/largefiles/onerror=panic/dev=15400fd on
 Sat Sep  6 12:55:52 2003
/u02 on /dev/md/dsk/d550
read/write/setuid/intr/largefiles/onerror=panic/dev=1540226 on Sat
 Sep  6 12:55:52 2003

Any ideas?

Thanks

Quote:>  Here is part of the mount output:

>  I am getting largefile errors.  [ ... ]

Where ?

Quote:>  / on /dev/md/dsk/d150
>  read/write/setuid/intr/largefiles/onerror=panic/dev=1540096 on Sat Se

The above doesn't mean you have errors. It shows how filesystem behaves
when an error occur.

przemol

My mistake.  I read it wrongly.

There is no problem with the filesystem.  I will look at the other
answers and figure out where the throughput slows down.

Thanks.

Quote:> Here is part of the mount output:

> I am getting largefile errors.  The disk are mirrored using DiskSuite.

> / on /dev/md/dsk/d150
> read/write/setuid/intr/largefiles/onerror=panic/dev=1540096 on Sat Se

No, you're not. "onerror=panic" is the default, and means that if the
kernel detects an inconsistency in the filesystem it'll panic. There are
other options - see the mount_ufs(1M) manpage

                 onerror = action
                       This option specifies the action  that  UFS
                       should  take  to  recover  from an internal
                       inconsistency on  a  file  system.  Specify
                       action  as  panic,  lock,  or umount. These
                       values cause a forced  system  shutdown,  a
                       file  system lock to be applied to the file
                       system, or the file system to  be  forcibly
                       unmounted,  respectively.  The  default  is
                       panic.

 > Again, I have not seen this errors before:

Odd, the output of mount has looked like that since Solaris 8 was released.

--
Tony

My mistake.  I read it wrongly.

There is no problem with the filesystem.  I will look at the other
answers and figure out where the throughput slows down.

Thanks.