Controlling pci bus utilization

Controlling pci bus utilization

Post by Greg Menk » Sun, 13 Feb 2005 01:43:14



Hi,

I have an Ultra 5 400mhz running Solaris 9 I'm using to work up a disk
array.  I have a pretty basic mirrored configuration going on the array,
but when I do something disk intensive, like a big ufsrestore to test
it, the foreground tasks become quite unresponsive.  I have FSS enabled
with my user's project at a 100:1 share relative to the rest of the
system- configuration performing as expected via prstat- so its not as
if the root ufsrestore is using up the cpu from a scheduler standpoint.
My guess is the heavy pci traffic from the disk i/o is slowing down the
video & network i/o, etc..

It seems like FSS does have some indirect effect on the problem.  If I
run mozilla or do something else in my project that involves a lot of
cpu-time, responsiveness improves.  Presumably this is the FSS giving me
my 100:1 cpu-share, the big disk job being throttled down, easing the
pci utilization.  However if all I'm doing is little fiddly stuff, then
my project starts to bog down.  iostat shows the varying disk throughput
that I would expect in this situation.

So I was wondering if there are tools to limit bus bandwidth on a
project if not process basis beyond tweaking process priorities.  I mean
something like diffserv statistical limits that reduce the scheduling of
the task.  If Solaris 10 has features to do something like this, it
wouldn't bother me to upgrade this machine.

Thanks,

Greg

 
 
 

Controlling pci bus utilization

Post by ba.. » Sun, 13 Feb 2005 02:46:03


I assume you're not attempting to access other data on the same
arrays, right?  Otherwise you could be stuck waiting for other IOs
to complete...

Depending on the number of drives in the array, you could be
also getting interrupt bound.  You can examine this using kstats; we
starting keeping interrupt statistics on Solaris 9.

You can delay processes doing arbitrary things with dtrace
if a "home grown" solution helps.

Solaris 10 is considerably more efficient doing IO than Solaris 9, esp.
on Sparc as we are now able to avoid mapping/remapping pages in
the page cache into the kernel's address space.  We're still working on
getting this enabled for amd64 (need 64 bit kernel address space for
this).
This may or may not help your problem depending on it's source.

Solaris 10 is also much more accurate doing process accounting; it
correctly
accounts for processes running in sync with the clock in contrast to
previous
releases, which could miss such activitivies entirely.

- Bart

 
 
 

Controlling pci bus utilization

Post by Greg Menk » Sun, 13 Feb 2005 06:27:18



> I assume you're not attempting to access other data on the same
> arrays, right?  Otherwise you could be stuck waiting for other IOs
> to complete...

Thats right- the array is sitting there soaking up a ufsrestore while I
dink around with mozilla, a few ssh sessions and a SunPCI board all
running off the built-in ide drive.

Quote:> Depending on the number of drives in the array, you could be
> also getting interrupt bound.  You can examine this using kstats; we
> starting keeping interrupt statistics on Solaris 9.

10 drives on 1 diff scsi channel.  The drive config is mirrored; 2
submirrors of 4 drives each, 2 drives reserved as hotswap.  I'll have a
look at kstat- I've not used it much.  

Once I'm happy the array is all worked up, it'll go into production on a
dual 450 Ultra 80 w/ 2 diff channels, 5 drives on each w/ the same
mirror config as above.

Quote:> Solaris 10 is considerably more efficient doing IO than Solaris 9, esp.
> on Sparc as we are now able to avoid mapping/remapping pages in
> the page cache into the kernel's address space.  We're still working on
> getting this enabled for amd64 (need 64 bit kernel address space for
> this).
> This may or may not help your problem depending on it's source.

> Solaris 10 is also much more accurate doing process accounting; it
> correctly
> accounts for processes running in sync with the clock in contrast to
> previous
> releases, which could miss such activitivies entirely.

Sounds like I should give it a whirl on this box.  I'm really digging it
at home on my test box.

Thanks,

Gregm

 
 
 

1. Theoretical Question on SBUs & PCI Bus bandwidth utilization

Hey,  I have often wondered about the bus bandwidth.  How would one tell
that the bus was overwhelmed by the other hardware and requests.  If I have
a bunch of SCSI disks, mutliple processors, multiple nics and heavy load,
how would I determine if I was running out of bandwidth on the bus (lets say
I'm all on one sbus).  Is there some proc tool or sar like tool that would
point to that?

--
Peter Kahn
Software Epidemiologist

http://www.meetingmaker.com

2. squid

3. PCI bus vs. S-bus

4. Zenith 1024x768

5. VL-Bus Vs. PCI bus

6. Dual boot with nt - ntfs and linux

7. Tuning PCI latency/max_grant or priority settings on Sun Blade 150 PCI-bus

8. Performance after stripe increase

9. "PCI BIOS passed nonexistant PCI bus 0"

10. hauppauge wintv pci fm - sound over pci bus

11. Does a 4422 pci cards really need a 66 Mhz PCI bus ?

12. 3 PCI buses, not all found in /proc/pci or lspci, but scanpci sees

13. How to control bandwidth utilization?