Converting from sd?? devices (veritas?) to find source of the bottlenecks.

Converting from sd?? devices (veritas?) to find source of the bottlenecks.

Post by Eric Harl » Fri, 19 Jul 2002 21:42:47



Hardware - big Sun Solaris 2.6 with EMC drives.  20+ processors.
Veritas.  Hardware under lock and key at undisclosed location with
guards and security with those little red laser beams to detect people
getting too close.  Or so I've heard. <g>

Problem - High wio.  Nobody seems to know why and no one has the time
to look for the source of the problem.  Oracle is probably the
culprit, but which database and which tables aren't known.  DBA too
busy or doesn't know how to look at the problem.

Me - Access to the machine (telnet) trying to find the source of the
problem.  I have regular user privies.

iostat -x gives me numbers like this - and there's a whole lot more
(pages and pages)

sd168        0.2    0.2    7.4    1.4  0.0  0.0   13.0   0   0
sd169        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
sd170        3.9    0.1   63.2    1.1  0.0  0.0    3.6   0   1
sd171        6.7    0.6  101.1    5.3  0.0  0.0    3.6   0   2
sd172        6.8    0.2  102.0    1.8  0.0  0.0    3.6   0   2
sd173        6.8    0.1  102.5    1.5  0.0  0.0    3.5   0   2
sd174        6.7    0.1  101.8    1.4  0.0  0.0    3.6   0   2

For those devices being used, I'd like to find out which tables are
causing problems.  This gives me sd devices which I assume are created
by Veritas.  I can't map them back to the mounted file system.

With Oracle, I have table information -
/DB42/PROJ/ORACLE/tab00/PROJ_tab01.dbf is on
/dev/vx/dsk/proj-table/vol01
So my goal is to find out that if the disk activity is on
/dev/vx/dsk/proj-table/vol01, I can narrow down the activity to a set
of tables, but I only have the device info from iostat.

What I want to know is if I see that device sd174 has a lot of
traffic, I want to know which database table(s) are getting the
traffic.  I want to avoid Oracle stats because I want disk activity
and I think the Oracle stats may not reflect disk activity if the
tables are cached.  Besides, I don't have DBA rights.

Is there a way to convert between the devices reported by iostat and
find out where they are in the filesystem so I can figure out what's
causing all the io activity?

 -Eric

 
 
 

Converting from sd?? devices (veritas?) to find source of the bottlenecks.

Post by Anthony Mandi » Fri, 19 Jul 2002 22:04:05



> Problem - High wio.  Nobody seems to know why and no one has the time
> to look for the source of the problem.  Oracle is probably the
> culprit, but which database and which tables aren't known.

        Oracle IS the problem.

Quote:> DBA too busy or doesn't know how to look at the problem.

        Oracle DBAs are like MS MCSEs. Its his problem, let him deal
        with it.

Quote:> For those devices being used, I'd like to find out which tables are
> causing problems.  This gives me sd devices which I assume are created
> by Veritas.

        Nope, the sd format is an old BSDism. You can map them back.

Quote:> I can't map them back to the mounted file system.

        I posted a little shell script here last week. Search for it.

Quote:> With Oracle, I have table information -
> /DB42/PROJ/ORACLE/tab00/PROJ_tab01.dbf is on
> /dev/vx/dsk/proj-table/vol01
> So my goal is to find out that if the disk activity is on
> /dev/vx/dsk/proj-table/vol01, I can narrow down the activity to a set
> of tables, but I only have the device info from iostat.

        Veritas has its own version of iostat. Use that instead.

-am     ? 2002

 
 
 

Converting from sd?? devices (veritas?) to find source of the bottlenecks.

Post by Andy Lennar » Fri, 19 Jul 2002 22:41:14




Quote:>Hardware - big Sun Solaris 2.6 with EMC drives.  20+ processors.
>Veritas.  Hardware under lock and key at undisclosed location with
>guards and security with those little red laser beams to detect people
>getting too close.  Or so I've heard. <g>

>Problem - High wio.  Nobody seems to know why and no one has the time
>to look for the source of the problem.  Oracle is probably the
>culprit, but which database and which tables aren't known.  DBA too
>busy or doesn't know how to look at the problem.

>Me - Access to the machine (telnet) trying to find the source of the
>problem.  I have regular user privies.

>iostat -x gives me numbers like this - and there's a whole lot more
>(pages and pages)

>sd168        0.2    0.2    7.4    1.4  0.0  0.0   13.0   0   0
>sd169        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
>sd170        3.9    0.1   63.2    1.1  0.0  0.0    3.6   0   1
>sd171        6.7    0.6  101.1    5.3  0.0  0.0    3.6   0   2
>sd172        6.8    0.2  102.0    1.8  0.0  0.0    3.6   0   2
>sd173        6.8    0.1  102.5    1.5  0.0  0.0    3.5   0   2
>sd174        6.7    0.1  101.8    1.4  0.0  0.0    3.6   0   2

>For those devices being used, I'd like to find out which tables are
>causing problems.  This gives me sd devices which I assume are created
>by Veritas.  I can't map them back to the mounted file system.

>With Oracle, I have table information -
>/DB42/PROJ/ORACLE/tab00/PROJ_tab01.dbf is on
>/dev/vx/dsk/proj-table/vol01
>So my goal is to find out that if the disk activity is on
>/dev/vx/dsk/proj-table/vol01, I can narrow down the activity to a set
>of tables, but I only have the device info from iostat.

>What I want to know is if I see that device sd174 has a lot of
>traffic, I want to know which database table(s) are getting the
>traffic.  I want to avoid Oracle stats because I want disk activity
>and I think the Oracle stats may not reflect disk activity if the
>tables are cached.  Besides, I don't have DBA rights.

>Is there a way to convert between the devices reported by iostat and
>find out where they are in the filesystem so I can figure out what's
>causing all the io activity?

> -Eric

is iostat -xn what you want?

maybe RMCtaz can help too...

--

 
 
 

Converting from sd?? devices (veritas?) to find source of the bottlenecks.

Post by Darren Dunha » Sat, 20 Jul 2002 02:16:28



> Problem - High wio.
> iostat -x gives me numbers like this - and there's a whole lot more
> (pages and pages)
> sd168        0.2    0.2    7.4    1.4  0.0  0.0   13.0   0   0
> sd169        0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
> sd170        3.9    0.1   63.2    1.1  0.0  0.0    3.6   0   1
> sd171        6.7    0.6  101.1    5.3  0.0  0.0    3.6   0   2
> sd172        6.8    0.2  102.0    1.8  0.0  0.0    3.6   0   2
> sd173        6.8    0.1  102.5    1.5  0.0  0.0    3.5   0   2
> sd174        6.7    0.1  101.8    1.4  0.0  0.0    3.6   0   2
> For those devices being used, I'd like to find out which tables are
> causing problems.  This gives me sd devices which I assume are created
> by Veritas.  I can't map them back to the mounted file system.

'iostat -x' and 'iostat -nx' give the same order, but -nx has the
cxtxdxsx name rather than the sdxx name.  

From there, you can use 'vxprint -ht <volume>' to see which disks are
used by which volumes.

Quote:> With Oracle, I have table information -
> /DB42/PROJ/ORACLE/tab00/PROJ_tab01.dbf is on
> /dev/vx/dsk/proj-table/vol01
> So my goal is to find out that if the disk activity is on
> /dev/vx/dsk/proj-table/vol01, I can narrow down the activity to a set
> of tables, but I only have the device info from iostat.

If you can convince the system administrator to make 'vxstat' setuid
(it's a pretty innocuous program), you can get more direct stats of the
volumes.

vxstat -g <diskgroup> -i 5 -vp <volume>

would give direct stats on that volume.

Quote:> What I want to know is if I see that device sd174 has a lot of
> traffic, I want to know which database table(s) are getting the
> traffic.  I want to avoid Oracle stats because I want disk activity
> and I think the Oracle stats may not reflect disk activity if the
> tables are cached.  Besides, I don't have DBA rights.
> Is there a way to convert between the devices reported by iostat and
> find out where they are in the filesystem so I can figure out what's
> causing all the io activity?

iostat -nx <-> vxprint -ht

or read vxstat directly.

--

Unix System Administrator                    Taos - The SysAdmin Company
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

 
 
 

1. How to find performance bottlenecks in device driver?

I'm interested in tracking down performance bottlenecks in a device
driver (running under SunOS 5.1).  I'm planning on instrumenting the
driver at various points to collect timing info and a smattering of other
statistics.  

I'm looking for advice in describing a general approach to instrumenting the
kernel, particularly when looking for places to improve performance.  Are
there any pre-existing packages out there for doing this (perhaps a set of
routines for collecting timing info and function call frequency, and
postprocessing software for summarizing timing data)?  Are there any papers
available that might give some kind of guidance in this area?

Thanks in advance for any info provided.  I will gladly summarize to the
net should anyone indicate interest (assuming, of course, I receive any
responses).

best regards,
mb
--

Disclaimer:  booloo speaks for booloo and no other.

2. OpenBSD 2.9 as home-router/firewall with a 56k external modem

3. Can any open source compiler convert C++ source to C source?

4. Did LILO trash my boot sector??????

5. I/O or CPU bottleneck at Solaris 8 / Oracle / Veritas 3.5-mp3 server ??

6. Using a mic on a Thinkpad

7. how to reliably convert between sd* and c* disk names?

8. Development libraries in XFree-3.3.1

9. Finding Source of Device Interrupts

10. INN can't keep up; how to find bottleneck?

11. (How to) Convert root from Sun DiskSuite to Veritas Volume Manager & OS Upgrade

12. Minimum system requirements - finding bottleneck

13. Finding LAN Bottleneck