how to reduce file-system cache size

how to reduce file-system cache size

Post by Frank Zeroc » Tue, 21 Apr 1998 04:00:00



hi all,

because of  some performance trouble with our Linux-Box (2xPII 300,
512MB,
2.0.33/2.1.89/2.1.96) I want to reduce the file-system cache size.
What should I do, so that Linux only use for example 32MB for cache
buffers
and not all available  memory?

best regards
frank

 
 
 

how to reduce file-system cache size

Post by Peter A Fei » Tue, 21 Apr 1998 04:00:00



> hi all,

> because of  some performance trouble with our Linux-Box (2xPII 300,
> 512MB,
> 2.0.33/2.1.89/2.1.96) I want to reduce the file-system cache size.
> What should I do, so that Linux only use for example 32MB for cache
> buffers
> and not all available  memory?

Linux will release this memory as it's needed.  There's no way (or
need) to fix this limit, to the best of my knowledge.  What sort of
problems are you having & why do you think it's a lack of memory,
esp. when you've got 512 MB?

--
Peter Fein
1005 E. 60th St.
Chicago, IL 60637
773-834-6206

http://pfein.home.ml.org/

 
 
 

how to reduce file-system cache size

Post by Frank Zeroc » Tue, 21 Apr 1998 04:00:00




> > hi all,

> > because of  some performance trouble with our Linux-Box (2xPII 300,
> > 512MB,
> > 2.0.33/2.1.89/2.1.96) I want to reduce the file-system cache size.
> > What should I do, so that Linux only use for example 32MB for cache
> > buffers
> > and not all available  memory?

> Linux will release this memory as it's needed.  There's no way (or
> need) to fix this limit, to the best of my knowledge.  What sort of
> problems are you having & why do you think it's a lack of memory,
> esp. when you've got 512 MB?

the problem is the following:
I have one program which accesses the hard-disk very often.
This program runs 10 to 20 times slower after a large disk-read is done,
which enlarged the fs-cache-buffer. (for ex. after a cp)
The problem disappears, if I reduce the memory to 64MB (boot option
mem=64M).
 
 
 

how to reduce file-system cache size

Post by Paul Flinder » Tue, 21 Apr 1998 04:00:00




> > Linux will release this memory as it's needed.  There's no way (or
> > need) to fix this limit, to the best of my knowledge.  What sort of
> > problems are you having & why do you think it's a lack of memory,
> > esp. when you've got 512 MB?

> the problem is the following:
> I have one program which accesses the hard-disk very often.
> This program runs 10 to 20 times slower after a large disk-read is done,
> which enlarged the fs-cache-buffer. (for ex. after a cp)

I don't see that reducing the FS cache would help - the copy would
flush the data that you are interested in out of a small cache just as
effectively as out of a large one.

Are you updating a single file (or set of files) - if so maybe
mmap+mlock would provide a solution.

Quote:> The problem disappears, if I reduce the memory to 64MB (boot option
> mem=64M).

Curious - if you weren't running PII's I'd ask whether all of your RAM
is being cached.

--
Paul

 
 
 

how to reduce file-system cache size

Post by Frank Sweetse » Tue, 21 Apr 1998 04:00:00



> I have one program which accesses the hard-disk very often.
> This program runs 10 to 20 times slower after a large disk-read is done,
> which enlarged the fs-cache-buffer. (for ex. after a cp)
> The problem disappears, if I reduce the memory to 64MB (boot option
> mem=64M).

ah.... it sounds like you're motherboard isn't caching past 64M.  this
could either be because you need more cache, or your motherboard could
simply not support it.  if you do a dejanews or altavista search, you
should be able to find some patches that will let you use this higher,
slower memory as a swapdisk instead of as normal ram.  it won't be as fast
as caching the full memory, but it will be faster than it is now, as it'll
keep stuff out of the slow memory as much as possible.

--
Frank Sweetser rasmusin at wpi.edu fsweetser at blee.net | PGP key available
paramount.res.wpi.net RH 5.0 kernel 2.0.33/2.1.97   i586 | at public servers
"...[Linux's] capacity to talk via any medium except smoke signals."
(By Dr. Greg Wettstein, Roger Maris Cancer Center)

 
 
 

how to reduce file-system cache size

Post by Mark Tranchan » Tue, 21 Apr 1998 04:00:00



> the problem is the following:
> I have one program which accesses the hard-disk very often.
> This program runs 10 to 20 times slower after a large disk-read is done,
> which enlarged the fs-cache-buffer. (for ex. after a cp)
> The problem disappears, if I reduce the memory to 64MB (boot option
> mem=64M).

Are you sure all 512MB is being cached by the L2 cache?

If not, look at http://fatale.res.cmu.edu/~keryan/patches/ .

Mark.

 
 
 

how to reduce file-system cache size

Post by Robert Hyat » Tue, 21 Apr 1998 04:00:00


: hi all,

: because of  some performance trouble with our Linux-Box (2xPII 300,
: 512MB,
: 2.0.33/2.1.89/2.1.96) I want to reduce the file-system cache size.
: What should I do, so that Linux only use for example 32MB for cache
: buffers
: and not all available  memory?

: best regards
: frank

Why would you want to do this?  Anytime you need the memory, Linux
will flush anything needing flushing and then give you that memory.

But when you don't need it, it certainly makes file I/O go like blazes,
speed-wise...

--
Robert Hyatt                    Computer and Information Sciences

(205) 934-2213                  115A Campbell Hall, UAB Station
(205) 934-5473 FAX              Birmingham, AL 35294-1170

 
 
 

how to reduce file-system cache size

Post by Kaz Kylhe » Tue, 21 Apr 1998 04:00:00





>> I have one program which accesses the hard-disk very often.
>> This program runs 10 to 20 times slower after a large disk-read is done,
>> which enlarged the fs-cache-buffer. (for ex. after a cp)
>> The problem disappears, if I reduce the memory to 64MB (boot option
>> mem=64M).

>ah.... it sounds like you're motherboard isn't caching past 64M.  this

I can't believe anyone would manufature such garbage nowadays.

But we are talking Wintel...

 
 
 

how to reduce file-system cache size

Post by Paul Flinder » Tue, 21 Apr 1998 04:00:00






> >> I have one program which accesses the hard-disk very often.
> >> This program runs 10 to 20 times slower after a large disk-read is done,
> >> which enlarged the fs-cache-buffer. (for ex. after a cp)
> >> The problem disappears, if I reduce the memory to 64MB (boot option
> >> mem=64M).

> >ah.... it sounds like you're motherboard isn't caching past 64M.  this

> I can't believe anyone would manufature such garbage nowadays.

Of the Triton chipsets only the HX will cache more than 64Mb in the L2
cache and that needs an external Tag RAM.

However the original poster said he had a PII based system - PIIs
cache 512Mb

 
 
 

how to reduce file-system cache size

Post by Frank Sweetse » Tue, 21 Apr 1998 04:00:00






> >> I have one program which accesses the hard-disk very often.
> >> This program runs 10 to 20 times slower after a large disk-read is done,
> >> which enlarged the fs-cache-buffer. (for ex. after a cp)
> >> The problem disappears, if I reduce the memory to 64MB (boot option
> >> mem=64M).

> >ah.... it sounds like you're motherboard isn't caching past 64M.  this

> I can't believe anyone would manufature such garbage nowadays.

> But we are talking Wintel...

you want garbage?  how 'bout the fact that the PII's can't handle >512M -
period!  any data written to address N where N >= 512M is also written to
addres N - 512M - which will obviously have disasterous results...

--
Frank Sweetser rasmusin at wpi.edu fsweetser at blee.net | PGP key available
paramount.res.wpi.net RH 5.0 kernel 2.0.33/2.1.97   i586 | at public servers
Woody: How would a beer feel, Mr. Peterson?
Norm:  Pretty nervous if I was in the room.
                -- Cheers, Loverboyd

 
 
 

how to reduce file-system cache size

Post by Frank Zeroc » Wed, 22 Apr 1998 04:00:00



> > the problem is the following:
> > I have one program which accesses the hard-disk very often.
> > This program runs 10 to 20 times slower after a large disk-read
> > is done,
> > which enlarged the fs-cache-buffer. (for ex. after a cp)

> I don't see that reducing the FS cache would help - the copy would
> flush the data that you are interested in out of a small cache just as
> effectively as out of a large one.

That's the point: If the FS cache is large then the buffers filled by
the
cp command were not flushed out and  the performance is poor. Keeping
the cache size small by reducing the whole memory or run a small program
which
allocates and writes all 512MB in order to flush the FS buffers solves
the
problem.

Quote:

> > The problem disappears, if I reduce the memory to 64MB (boot option
> > mem=64M).

> Curious - if you weren't running PII's I'd ask whether all of your RAM
> is being cached.

 All the RAM is cached. I'm really sure :-) (Board: ASUS P2L97-DS)
 
 
 

how to reduce file-system cache size

Post by Paul Flinder » Wed, 22 Apr 1998 04:00:00




> > > the problem is the following:
> > > I have one program which accesses the hard-disk very often.
> > > This program runs 10 to 20 times slower after a large disk-read
> > > is done,
> > > which enlarged the fs-cache-buffer. (for ex. after a cp)

> > I don't see that reducing the FS cache would help - the copy would
> > flush the data that you are interested in out of a small cache just as
> > effectively as out of a large one.

> That's the point: If the FS cache is large then the buffers filled
> by the cp command were not flushed out and the performance is
> poor. Keeping the cache size small by reducing the whole memory or
> run a small program which allocates and writes all 512MB in order to
> flush the FS buffers solves the problem.

Mmmm.... I see what's happening.

Ok - if you do something like

cp -a <some 500Mb tree> <somewhere else>
run your app

with lots of ram then you may see something as you describe as the
system writes pages dirtied by the cp out to disk as your application
starts.

However - if you copy a 500Mb tree then 1Gig of disk I/O must be done
sometime - if you time the cp with 512Meg you should find it faster
than with 64 because fewer of the disk writes will have been completed
when cp exits.

If you time the sequence "cp ...;sync" it should be about the same
with 64Mb as with 512 (disk I/O being the limiting factor and you
force all the disk I/O to be included in the time).

So - if the situation above is the one that you are in try:

cp -a <some 500Mb tree> <somewhere else>
sync
run your app

The cp+sync will complete in more-or-less the same time as with 64Mb
and your application will find lots of pages cached but re-usable
without further I/O.

If, however, you are doing something like this

start app
(performance OK)
                          cp .....
(performance reduced)

Then the problem is cp "stealing" memory pages from your application -
you can't really avoid this since you've just massively increased the
resources that you need and the system has done its best. However in
this case I can't see that performance with 64 would be better in
absolute terms (maybe better in relative terms because your app goes
from OK(paging) to OK(paging) with 64 and from FAST(everything in mem)
to OK (paging) with 512 which you then perceive as a slowdown).

If you are in this scenario then look at what the mlock call does -
you can use it to "fix" some pages in memory and they won't get paged
out when you do the cp.

Filesystem cached pages go through the same cache as virtual memory so
reducing the amount of cache available wouldn't solve your problem.

--
Paul

 
 
 

how to reduce file-system cache size

Post by Frank Zeroc » Wed, 22 Apr 1998 04:00:00



> If, however, you are doing something like this

> start app
> (performance OK)
>                           cp .....
> (performance reduced)

Yes, it's mainly this third scenario but:

start app
(Performance OK)
     cp .... &
(Performance very poor (but this seems reasonable)
     ... cp finished
(Performance still reduced !)

I had no clue for this strange behaviour, so I wrote a program
(disktest)
which creates a large file (kill.me). Then it makes random reads and
writes to that file. The following happens:

start app
(Perf. OK)
     disktest
(Perf. very poor)
     ...disktest finished
(Perf. reduced)
     del kill.me
(Perf. OK !!)

If only 64MB are installed, the performance reaches normal level shortly
after disktest exits.

Quote:> If you are in this scenario then look at what the mlock call does -
> you can use it to "fix" some pages in memory and they won't get paged
> out when you do the cp.

I can't try mlock --- the app. came without source :-|
 
 
 

how to reduce file-system cache size

Post by Paul Flinder » Wed, 22 Apr 1998 04:00:00




> > If, however, you are doing something like this

> > start app
> > (performance OK)
> >                           cp .....
> > (performance reduced)

> Yes, it's mainly this third scenario but:

> start app
> (Performance OK)
>      cp .... &
> (Performance very poor (but this seems reasonable)

Here you'll be reading stuff from disk and probably paging some of
your application out.

Quote:>      ... cp finished
> (Performance still reduced !)

With lots of RAM cp will finish with most of the data it wrote still
in memory - as your application runs and touches pages these will have
to be written out before your app can be paged back in hence the
slowdown after cp has finished.

Quote:> I had no clue for this strange behaviour, so I wrote a program
> (disktest)
> which creates a large file (kill.me). Then it makes random reads and
> writes to that file. The following happens:

> start app
> (Perf. OK)
>      disktest
> (Perf. very poor)
>      ...disktest finished
> (Perf. reduced)

OK - up to this point you're in the same situation as with the cp

Quote:>      del kill.me
> (Perf. OK !!)

Now you've just marked all the dirty pages belonging to kill.me as not
dirty anymore. When you need them they can just be zeroed out and
given to you (or filled from disk) without needing to write their
contents out.

Quote:

> If only 64MB are installed, the performance reaches normal level shortly
> after disktest exits.

I'd still like to see some absolute figures - I'm pretty sure that
with 64Mb performance will be uniformly lower than with 512.

Quote:

> > If you are in this scenario then look at what the mlock call does -
> > you can use it to "fix" some pages in memory and they won't get paged
> > out when you do the cp.

> I can't try mlock --- the app. came without source :-|

Ahh, I see <insert lecture about why source availability is a "good thing">

OK then either add a sync after the copy - this will write all the dirty
pages out and they'll then be available again. Performance will still be
low during the cp+sync and you'll still take a hit as the system pages the
app back in but it should all be over fairly quickly.

If the application is dynamically loaded you may be able to use LD_PRELOAD
to wrap brk/sbrk and mmap calls and lock the processes memory in RAM

The unified cache is good when you don't know in advance whether programs
will use lots of memory or do lots of block I/O but it hurts in cases such
as yours when you want to attach a specific priority to the use of memory
amongst processes.

Maybe the answer is a memory priority value similar to process "niceness"
which would allow you to run your process at a higher memory priority then
usual.

--
Paul

 
 
 

how to reduce file-system cache size

Post by Robert Hyat » Wed, 22 Apr 1998 04:00:00


:> If, however, you are doing something like this
:>
:> start app
:> (performance OK)
:>                           cp .....
:> (performance reduced)
:>
:>
: Yes, it's mainly this third scenario but:

: start app
: (Performance OK)
:      cp .... &
: (Performance very poor (but this seems reasonable)
:      ... cp finished
: (Performance still reduced !)

I tried to reproduce this and could not.  running xosview shows what
percent of memory is used for cache.  running a big app after a big
copy will drive the cache way down, down to the point that your program
is resident and the file-cache pages are mostly gone...  at least on my
system...

when I fire up the big app, the cache is cleared...  without a bunch
of disk i/o since the dirty file pages are written out along the way
to avoid gross file inconsistencies...

--
Robert Hyatt                    Computer and Information Sciences

(205) 934-2213                  115A Campbell Hall, UAB Station
(205) 934-5473 FAX              Birmingham, AL 35294-1170

 
 
 

1. reduce file system size in my AIX RS/6000

hi Everybody:
        I am trying to get some free disk space at rootvg for my system
upgrade(from 4.1.5 to 4.2). The filesystem I want to shrink size is
/home. Instead of make mksysb and change data in the file /image.data
and reinstall my machine (according to the book AIX System Management
Guide: Operating system and Devices, page 6-24) Can I just back up /home
filesystem, remove it and re-create it in a smaller size?

Thanks a lot!
Yang

2. 'at' runs job, cron doesn't!

3. server-status no longer works after switch to Solaris-2.8 apache

4. reduce /usr file system size

5. <curses.h> EASY question!!

6. How to find the file system cache size on a sparc 10

7. Q. About Msg Queues

8. size of the file system cache on solaris 2.6

9. File System Cache Size?

10. Reduce file system file

11. Reducing the size of a log file (without interrupting a running application) (Bourne shell)

12. reduce the file size