Returning memory to system? (SIZE)

Returning memory to system? (SIZE)

Post by Valliappa Lakshman » Thu, 17 Apr 1997 04:00:00



I have a memory-intensive real-time algorithm that runs continously.
The memory it uses is malloced and freed (actually 'new'ed and
'delete'd but you get the idea).

Now, after maybe several hours, I find that the "SIZE" of the program
as given by the UNIX utility "top" has increased. The memory being
actually used by the system ("RES" in top) remains within the normal
bounds.  SIZE continues to increase, with no set pattern, while "RES"
remains within range.

I have run the program under purify but even when the SIZE has increased,
there are absolutely no leaks. Not even 4 bytes!

Questions:

1) Why could this (SIZE increases, RES okay) happen? How can we fix it?

2) Hypothesis:  maybe the memory is getting fragmented and so, the program
keeps continously adding to its SIZE in order to get the required memory.
How do I _really return_ allocated memory to the system instead of free-ing
but tagging it for future use by the program?

thanks for any suggestions!

 
 
 

Returning memory to system? (SIZE)

Post by Fletcher.Gl.. » Thu, 17 Apr 1997 04:00:00


I don't know which flavor of UNIX you are using, but on some, malloc()
does not do garbage collection.  In Solaris 2.x there is the choice
of using libmalloc which is announced as "space efficient, but slower".



Quote:>I have a memory-intensive real-time algorithm that runs continously.
>The memory it uses is malloced and freed (actually 'new'ed and
>'delete'd but you get the idea).

>Now, after maybe several hours, I find that the "SIZE" of the program
>as given by the UNIX utility "top" has increased. The memory being
>actually used by the system ("RES" in top) remains within the normal
>bounds.  SIZE continues to increase, with no set pattern, while "RES"
>remains within range.

>I have run the program under purify but even when the SIZE has increased,
>there are absolutely no leaks. Not even 4 bytes!

>Questions:

>1) Why could this (SIZE increases, RES okay) happen? How can we fix it?

>2) Hypothesis:  maybe the memory is getting fragmented and so, the program
>keeps continously adding to its SIZE in order to get the required memory.
>How do I _really return_ allocated memory to the system instead of free-ing
>but tagging it for future use by the program?

>thanks for any suggestions!


 
 
 

Returning memory to system? (SIZE)

Post by Andrew Giert » Thu, 17 Apr 1997 04:00:00


writes:

 Valliappa> 1) Why could this (SIZE increases, RES okay) happen? How
 Valliappa> can we fix it?

That may be an artifact of the malloc() implementation; some systems
seem to be biased towards grabbing more memory from the O/S rather
than reusing space already freed.

 Valliappa> 2) Hypothesis: maybe the memory is getting fragmented and
 Valliappa> so, the program keeps continously adding to its SIZE in
 Valliappa> order to get the required memory.  How do I _really
 Valliappa> return_ allocated memory to the system instead of free-ing
 Valliappa> but tagging it for future use by the program?

Fragmentation can cause the memory used to be much larger than expected,
yes.

As for actually returning free memory to the system, one way to do
that is to provide your own specialised allocator that uses mmap() to
attach blocks of memory to the process (either by specifying MAP_ANON
or by opening /dev/zero and mapping that). These allocations can be
returned to the system simply by unmapping the blocks. You need to map
fairly large chunks and then carve them up for your own use; use
sysconf(_SC_PAGE_SIZE) to get the system page granularity.

--
Andrew.

comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>

 
 
 

Returning memory to system? (SIZE)

Post by Bill Lan » Thu, 17 Apr 1997 04:00:00




>I have a memory-intensive real-time algorithm that runs continously.
>The memory it uses is malloced and freed (actually 'new'ed and
>'delete'd but you get the idea).

>Now, after maybe several hours, I find that the "SIZE" of the program
>as given by the UNIX utility "top" has increased. The memory being
>actually used by the system ("RES" in top) remains within the normal
>bounds.  SIZE continues to increase, with no set pattern, while "RES"
>remains within range.

>I have run the program under purify but even when the SIZE has increased,
>there are absolutely no leaks. Not even 4 bytes!

>Questions:

>1) Why could this (SIZE increases, RES okay) happen? How can we fix it?

>2) Hypothesis:  maybe the memory is getting fragmented and so, the program
>keeps continously adding to its SIZE in order to get the required memory.
>How do I _really return_ allocated memory to the system instead of free-ing
>but tagging it for future use by the program?

>thanks for any suggestions!

To return memory to the system you will have to write your own memory handling
routines that allocate a chunk of memory from the system and then allocates
desired memory from the chunk (have to do it that way since memory allocation
from the system is slow), when a chunk consists of only free memory then your
routine can return it back to the system.

Of course in the process of writing your own memory allocator you may end up
solving the problem of getting more and more memory from the system making it
unnecessary for you to return memory to the system.

    Bill Lanam

 
 
 

Returning memory to system? (SIZE)

Post by Mariusz Zyd » Thu, 24 Apr 1997 04:00:00




[>...what are you talking about....what's this got to do with C ???????

------- Just because YOU don't know, doesn't mean it does not make
sense. In fact, it makes perfect sense to me, and it has to do with
C (it's a C program that has a problem with it).
--
 Mariusz Zydyk                           http://www.ucalgary.ca/~mszydyk/

 
 
 

Returning memory to system? (SIZE)

Post by Geor » Thu, 24 Apr 1997 04:00:00



> I have a memory-intensive real-time algorithm that runs continously.
> The memory it uses is malloced and freed (actually 'new'ed and
> 'delete'd but you get the idea).

> Now, after maybe several hours, I find that the "SIZE" of the program
> as given by the UNIX utility "top" has increased. The memory being
> actually used by the system ("RES" in top) remains within the normal
> bounds.  SIZE continues to increase, with no set pattern, while "RES"
> remains within range.

> I have run the program under purify but even when the SIZE has increased,
> there are absolutely no leaks. Not even 4 bytes!

> Questions:

> 1) Why could this (SIZE increases, RES okay) happen? How can we fix it?

> 2) Hypothesis:  maybe the memory is getting fragmented and so, the program
> keeps continously adding to its SIZE in order to get the required memory.
> How do I _really return_ allocated memory to the system instead of free-ing
> but tagging it for future use by the program?

> thanks for any suggestions!

...what are you talking about....what's this got to do with C ???????
....message ends.....
 
 
 

Returning memory to system? (SIZE)

Post by Volker Hetze » Thu, 24 Apr 1997 04:00:00



> I have a memory-intensive real-time algorithm that runs continously.
> The memory it uses is malloced and freed (actually 'new'ed and
> 'delete'd but you get the idea).

> Now, after maybe several hours, I find that the "SIZE" of the program
> as given by the UNIX utility "top" has increased. The memory being
> actually used by the system ("RES" in top) remains within the normal
> bounds.  SIZE continues to increase, with no set pattern, while "RES"
> remains within range.

> I have run the program under purify but even when the SIZE has increased,
> there are absolutely no leaks. Not even 4 bytes!

> Questions:
> 2) Hypothesis:  maybe the memory is getting fragmented and so, the program
> keeps continously adding to its SIZE in order to get the required memory.
> How do I _really return_ allocated memory to the system instead of free-ing
> but tagging it for future use by the program?

What about a custom memory allocator for a single/a few class(es)?
Identify the objects that are most often created and deleted and rewrite
new and delete for them. (You have checked that the number of objets
remains
roughly constant over time, haven't you?)

You do this by using the global new to allocate a chunk of chars and
then
you can manage this memory on your own.
implement a myClass::new and a myclass::delete to return a free adress
when called (and flag id busy) and to flag it free in case of delete.
You don't need to worry about constructors and destructors.
When someone says "myClass *X(new myClass(1,2,3)", myClass's own new
will only be called to provide the memory.

 
 
 

Returning memory to system? (SIZE)

Post by Lawrence Kir » Fri, 25 Apr 1997 04:00:00





>[>...what are you talking about....what's this got to do with C ???????

>------- Just because YOU don't know, doesn't mean it does not make
>sense. In fact, it makes perfect sense to me, and it has to do with
>C (it's a C program that has a problem with it).

However the problem is beyond the scope of the C language, it has to do
with the memory allocation strategy used by the particular implementation
which the C language doesn't specify.

--
-----------------------------------------


-----------------------------------------

 
 
 

Returning memory to system? (SIZE)

Post by juli » Sun, 27 Apr 1997 04:00:00


Hi Valliappa,


> 1) Why could this (SIZE increases, RES okay) happen? How can we fix it?

just to say that once i had the same problem on Digital Unix.

I worked hard looking for bad use of *alloc() and free() in the code,
then found the problem: the program was a server handling one request at
a time (no fork()'s), something like:

for(;;) {
 newsock = accept(...);
 infile = fdopen(newsock,...);
 outfile = fdopen(newsock,...);
 --- process request ---
 close(newsock);

Quote:}

Well, i had to add fclose(infile), fclose(outfile)!

Cheers, julio