Memory Leak ??? PROBLEM with MALLOC and FREE ????

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Ramesh Nataraja » Sun, 01 Apr 2001 23:55:24



HI Friends,

I have a problem with using malloc and free

I try to allocate memory using malloc and freed it up.
The code worked perfectly well. When I checked with
purify for memory leaks, it showed no leak.

But my problem is, the process's "data" size increases with
every malloc but does not get decreased when freed. I came
to know about this when i monitored the process by using TOP command.

Also when i used "truss" on the executable produced with malloc &
subsequently freeing it ,I found an unusual thing.

MAlloc allocates memory using brk system call, but the corresponding
free does not call any system call to reduce the memory and hence
the data size increases with every malloc.

Is this a bug in malloc and free implementation?

I am using solaris 2.6 and gcc compiler.

Any reason for this behaviour?

Ramesh

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Tom St Deni » Mon, 02 Apr 2001 00:14:08



Quote:> HI Friends,

> I have a problem with using malloc and free

> I try to allocate memory using malloc and freed it up.
> The code worked perfectly well. When I checked with
> purify for memory leaks, it showed no leak.

> But my problem is, the process's "data" size increases with
> every malloc but does not get decreased when freed. I came
> to know about this when i monitored the process by using TOP command.

> Also when i used "truss" on the executable produced with malloc &
> subsequently freeing it ,I found an unusual thing.

> MAlloc allocates memory using brk system call, but the corresponding
> free does not call any system call to reduce the memory and hence
> the data size increases with every malloc.

> Is this a bug in malloc and free implementation?

Perhaps, or maybe you need to shrink your heap.  In windows it's done
automagically (not in MS-DOS though).

Essentially you allocate mem, which in turn allocates heap from the OS.  You
then "free()" it and that changes the free mem in your programs heap.  Your
libc must not resize the heap though...

I dunno, that might not be it but a good guess.

Tom

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by George Gran » Mon, 02 Apr 2001 05:54:06



Quote:>But my problem is, the process's "data" size increases with
>every malloc but does not get decreased when freed. I came
>to know about this when i monitored the process by using TOP command.
...
>MAlloc allocates memory using brk system call, but the corresponding
>free does not call any system call to reduce the memory and hence
>the data size increases with every malloc.
...
>Ramesh

Yes, this does happen. It's by design. I'm sure one of the Gurus on here will be
able to tell you why it's "A Good Idea", but I don't really know why.

On AIX, you can use "disclaim" to give the system memory back, I'm not sure what
it would be called on other kinds of UNIX though.

-George

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Alain Magloir » Mon, 02 Apr 2001 08:02:08




>>But my problem is, the process's "data" size increases with
>>every malloc but does not get decreased when freed. I came
>>to know about this when i monitored the process by using TOP command.
> ...
>>MAlloc allocates memory using brk system call, but the corresponding
>>free does not call any system call to reduce the memory and hence
>>the data size increases with every malloc.
> ...
>>Ramesh
> Yes, this does happen. It's by design. I'm sure one of the Gurus on here will be
> able to tell you why it's "A Good Idea", but I don't really know why.

It is costly?
For example an allocator in platform XXX can get new memory by calling
mmap() which return memory in pagesize of 4k.
So if you ask for 20, 30 45, 56 78 12, 14, 13, 12, 12, 24 bytes, the allocator
will slice the 4k.  So unless you free the entire memory and let the allocator
coalesce the space it can not unmap() and give the memory back.

In this case it is not about "A Good Idea" but whether it is actually
possible/practicle to do it.  Solaris and Linux for big chunks try
to coalesce and give back the memory.  See you manual pages for
specific platforms.

Quote:> On AIX, you can use "disclaim" to give the system memory back, I'm not sure what
> it would be called on other kinds of UNIX though.
> -George

--
alain
 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by David Schwart » Mon, 02 Apr 2001 09:02:20



> >But my problem is, the process's "data" size increases with
> >every malloc but does not get decreased when freed. I came
> >to know about this when i monitored the process by using TOP command.

        The process' "data" size is a measure of how much *virtual* memory it's
using. Normally virtual memory is not a scarce resource, so you
shouldn't care.

Quote:> >Also when i used "truss" on the executable produced with malloc &
> >subsequently freeing it ,I found an unusual thing.

> >MAlloc allocates memory using brk system call, but the corresponding
> >free does not call any system call to reduce the memory and hence
> >the data size increases with every malloc.

        Yes, if malloc uses 'brk', then this will happen.

Quote:> >Is this a bug in malloc and free implementation?

        No. Again, virtual memory is normally not a scarce resource.

Quote:> No. It's characteristic of just about all malloc implementations
> on every Unix system there has ever been.

        Not so. There are plenty of malloc implementations that use 'mmap' to
get memory instead of 'sbrk'.

Quote:> The memory returned by free() is made available to satisfy subsequent
> malloc() requests, but it is not returned to the kernel.

        It was never taken from the kernel so it needn't be returned. The
kernel always controls all physical memory unless you take specific
steps to lock it in place.

        DS

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Lawrence Kir » Mon, 02 Apr 2001 04:26:30




Quote:>HI Friends,

>I have a problem with using malloc and free

>I try to allocate memory using malloc and freed it up.
>The code worked perfectly well. When I checked with
>purify for memory leaks, it showed no leak.

>But my problem is, the process's "data" size increases with
>every malloc but does not get decreased when freed. I came
>to know about this when i monitored the process by using TOP command.

This is quite normal. The C language says that freed memory is made
available for further allocation to the program. Some malloc()
implementations can release freed memory to the OS in some circumstances,
some never do. Some peop,e would argue that the latter are more correct
in terms of the C language specification. What should happen is that
freed memory may be allosed again in a later *alloc() call if there
is enough contiguous memory available and it meets any other requirements
of the allocator's strategy.

Quote:>Also when i used "truss" on the executable produced with malloc &
>subsequently freeing it ,I found an unusual thing.

>MAlloc allocates memory using brk system call, but the corresponding
>free does not call any system call to reduce the memory and hence
>the data size increases with every malloc.

Again that is quite normal.

Quote:>Is this a bug in malloc and free implementation?

No. You could argue that there was a bug if something like this
kept on eating more and more memory:

   for (i = 0; i < 1000000; i++) {
       void *p = malloc(10000);
       free(p);
   }

Quote:>I am using solaris 2.6 and gcc compiler.

>Any reason for this behaviour?

It is just the way many malloc libraries are implemented, and it is
valid.

--
-----------------------------------------


-----------------------------------------

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Dan P » Mon, 02 Apr 2001 11:41:42





>>>But my problem is, the process's "data" size increases with
>>>every malloc but does not get decreased when freed. I came
>>>to know about this when i monitored the process by using TOP command.
>> ...
>>>MAlloc allocates memory using brk system call, but the corresponding
>>>free does not call any system call to reduce the memory and hence
>>>the data size increases with every malloc.
>> ...
>>>Ramesh

>> Yes, this does happen. It's by design. I'm sure one of the Gurus on here will be
>> able to tell you why it's "A Good Idea", but I don't really know why.

>It is costly?
>For example an allocator in platform XXX can get new memory by calling
>mmap() which return memory in pagesize of 4k.
>So if you ask for 20, 30 45, 56 78 12, 14, 13, 12, 12, 24 bytes, the allocator
>will slice the 4k.  So unless you free the entire memory and let the allocator
>coalesce the space it can not unmap() and give the memory back.

>In this case it is not about "A Good Idea" but whether it is actually
>possible/practicle to do it.  Solaris and Linux for big chunks try
>to coalesce and give back the memory.  See you manual pages for
>specific platforms.

Giving back the memory to the OS creates a conformance problem.

    The free function causes the space pointed to by ptr to be
    deallocated, that is, made available for further allocation.

When reading this statement from the C standard, keep in mind the fact
that the C standard describes the behaviour of a single program, not of
the whole system.

If the memory freed by the program is given back to the OS, the OS might
give it to another program, so the program that deallocated it in the
first place may not be able to reuse it, although the above quote seems
to guarantee that it could reuse it.

Dan
--
Dan Pop
CERN, IT Division

Mail:  CERN - IT, Bat. 31 1-014, CH-1211 Geneve 23, Switzerland

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Micah Cowa » Mon, 02 Apr 2001 12:39:05






> >>>But my problem is, the process's "data" size increases with
> >>>every malloc but does not get decreased when freed. I came
> >>>to know about this when i monitored the process by using TOP command.
> >> ...
> >>>MAlloc allocates memory using brk system call, but the corresponding
> >>>free does not call any system call to reduce the memory and hence
> >>>the data size increases with every malloc.
> >> ...
> >>>Ramesh

> >> Yes, this does happen. It's by design. I'm sure one of the Gurus on here will be
> >> able to tell you why it's "A Good Idea", but I don't really know why.

> >It is costly?
> >For example an allocator in platform XXX can get new memory by calling
> >mmap() which return memory in pagesize of 4k.
> >So if you ask for 20, 30 45, 56 78 12, 14, 13, 12, 12, 24 bytes, the allocator
> >will slice the 4k.  So unless you free the entire memory and let the allocator
> >coalesce the space it can not unmap() and give the memory back.

> >In this case it is not about "A Good Idea" but whether it is actually
> >possible/practicle to do it.  Solaris and Linux for big chunks try
> >to coalesce and give back the memory.  See you manual pages for
> >specific platforms.

> Giving back the memory to the OS creates a conformance problem.

>     The free function causes the space pointed to by ptr to be
>     deallocated, that is, made available for further allocation.

> When reading this statement from the C standard, keep in mind the fact
> that the C standard describes the behaviour of a single program, not of
> the whole system.

> If the memory freed by the program is given back to the OS, the OS might
> give it to another program, so the program that deallocated it in the
> first place may not be able to reuse it, although the above quote seems
> to guarantee that it could reuse it.

In that case, I doubt there is any such thing as a conforming
implementation.

Micah

--
A gentle answer turns away wrath, but a harsh word stirs up anger.
                                        Proverbs 15:1

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Logan Sh » Mon, 02 Apr 2001 15:25:19



>Giving back the memory to the OS creates a conformance problem.

>    The free function causes the space pointed to by ptr to be
>    deallocated, that is, made available for further allocation.

>When reading this statement from the C standard, keep in mind the fact
>that the C standard describes the behaviour of a single program, not of
>the whole system.

>If the memory freed by the program is given back to the OS, the OS might
>give it to another program, so the program that deallocated it in the
>first place may not be able to reuse it, although the above quote seems
>to guarantee that it could reuse it.

So, you're saying the standard says that in this code

        void *p;

        p = malloc (10000);
        free (p);
        p = malloc (10000);

that the second malloc should never fail under any circumstance.
(Assuming that the code isn't threaded or something.)

While I agree that what you quoted could be read to say that, I'm not
sure I believe that's the only valid interpretation.  In fact, I'd say
the text you quoted is just plain ambiguous.  It does not specify to
*what* it's made available for further allocation.

In case anyone cares, before virtual memory was commonplace, it was a
pretty common thing for malloc() and free() to work against a memory
pool that was common to all processes.  This was true even on systems
that supported multitasking.  (At least, it was true on the Amiga.)

  - Logan
--
whose?  my  your   his  her   our   their   _its_
who's?  I'm you're he's she's we're they're _it's_

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Dan P » Mon, 02 Apr 2001 15:32:57




>>Giving back the memory to the OS creates a conformance problem.

>>    The free function causes the space pointed to by ptr to be
>>    deallocated, that is, made available for further allocation.

>>When reading this statement from the C standard, keep in mind the fact
>>that the C standard describes the behaviour of a single program, not of
>>the whole system.

>>If the memory freed by the program is given back to the OS, the OS might
>>give it to another program, so the program that deallocated it in the
>>first place may not be able to reuse it, although the above quote seems
>>to guarantee that it could reuse it.

>So, you're saying the standard says that in this code

>    void *p;

>    p = malloc (10000);
>    free (p);
>    p = malloc (10000);

>that the second malloc should never fail under any circumstance.
>(Assuming that the code isn't threaded or something.)

>While I agree that what you quoted could be read to say that, I'm not
>sure I believe that's the only valid interpretation.  In fact, I'd say
>the text you quoted is just plain ambiguous.  It does not specify to
>*what* it's made available for further allocation.

As I've already explained, the standard describes the behaviour of a
C program, not of a whole system.  Therefore, there is no ambiguity.

Dan
--
Dan Pop
CERN, IT Division

Mail:  CERN - IT, Bat. 31 1-014, CH-1211 Geneve 23, Switzerland

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Dan P » Mon, 02 Apr 2001 15:34:25




>> Giving back the memory to the OS creates a conformance problem.

>>     The free function causes the space pointed to by ptr to be
>>     deallocated, that is, made available for further allocation.

>> When reading this statement from the C standard, keep in mind the fact
>> that the C standard describes the behaviour of a single program, not of
>> the whole system.

>> If the memory freed by the program is given back to the OS, the OS might
>> give it to another program, so the program that deallocated it in the
>> first place may not be able to reuse it, although the above quote seems
>> to guarantee that it could reuse it.

>In that case, I doubt there is any such thing as a conforming
>implementation.

Why?  Implementations where nothing is returned back to the OS until
program termination are quite common.

Dan
--
Dan Pop
CERN, IT Division

Mail:  CERN - IT, Bat. 31 1-014, CH-1211 Geneve 23, Switzerland

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Logan Sh » Tue, 03 Apr 2001 04:40:27





>>>Giving back the memory to the OS creates a conformance problem.

>>>    The free function causes the space pointed to by ptr to be
>>>    deallocated, that is, made available for further allocation.

>>>When reading this statement from the C standard, keep in mind the fact
>>>that the C standard describes the behaviour of a single program, not of
>>>the whole system.
>>While I agree that what you quoted could be read to say that, I'm not
>>sure I believe that's the only valid interpretation.  In fact, I'd say
>>the text you quoted is just plain ambiguous.  It does not specify to
>>*what* it's made available for further allocation.
>As I've already explained, the standard describes the behaviour of a
>C program, not of a whole system.  Therefore, there is no ambiguity.

It's not because I haven't read your argument that I don't agree; it's
because I wasn't convinced by it.

Languages must be implemented on real systems.  Languages place certain
constraints on real systems.  Therefore, language standards make
statements about not only the language but also about the real systems
on which they might be implemented.

Furthermore, the authors of the standard would (hopefully) have the
good sense to understand that programs written in a language will be
interacting with the rest of the system.  To neglect this reality would
not be helpful.  (Imagine I am writing a standard describing how pilots
should behave, and I say, "after your airplane has cleared the area,
the runway is made available for further use.")

When reading/interpreting a standard about something that interacts
with other things, I would operate under the assumption that the
standard would make statements about both its main topic and the things
that it would interact with.

Does the C language standard explicitly state that I should do
otherwise?  If so, is it unambiguous that it applies to the statement
you originally quoted?  If so, then you are right.  If not, then I
maintain that it's ambiguous.

  - Logan
--
whose?  my  your   his  her   our   their   _its_
who's?  I'm you're he's she's we're they're _it's_

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by David Schwart » Tue, 03 Apr 2001 04:44:06



> Giving back the memory to the OS creates a conformance problem.

>     The free function causes the space pointed to by ptr to be
>     deallocated, that is, made available for further allocation.

> When reading this statement from the C standard, keep in mind the fact
> that the C standard describes the behaviour of a single program, not of
> the whole system.

> If the memory freed by the program is given back to the OS, the OS might
> give it to another program, so the program that deallocated it in the
> first place may not be able to reuse it, although the above quote seems
> to guarantee that it could reuse it.

        Hahahaha! That's really funny. This is the same as the argument that
any system that implements 'kill -9' isn't conformant since the standard
doesn't say your program can just randomly stop running.

        DS

 
 
 

Memory Leak ??? PROBLEM with MALLOC and FREE ????

Post by Ramesh Nataraja » Tue, 03 Apr 2001 13:21:08


Thanks for the reply.

This leads to another question:

assume i allocate 4k bytes using malloc
and I free it up.

As per what i can comprehend the os would have allocated a
segment of some predefined k ( say 4k) to the process due to
this malloc operation. Free would have freed that 4k bytes, but
still the process has that region attached to it.

So can I assume my second and subsequent mallocs ( totalling < 4k) will
always not fail?

Ramesh
--

 
 
 

1. Unix memory malloc & Free problem

Hi All.

I wrote an application on Solaris2.5 using malloc & free function.

I'm tracking the application memory status using top & proctool
services.

It seems that even when I'm releasing  an allocated memory block using
the free command
the application doesn't increase the total free memory of the O.S and
doesn't decrease the application
total memory.

I know that on Unix , the free memory block is marked as free and still
owned to the application
but on the other hand it looks like the application in 'eating' memory.

My questions is :
 - How can I cause my application to return the free memory block to the
O.S so the total free
memory of the system will be increased by each free I make.

- Is there a way to know the REAL memory of the application because so
far  top & proctool are also counting
memory blocks I already free.

Any help will be great full !

--
Eyal Haviv.
Product Manager.
Computer Associates - Security-7(software) Ltd.
Phone : +972-4-9592101.

2. determining directories.

3. Memory problem : malloc() & free()

4. RedHat 6.1 can not see my two ne2000 cards.

5. malloc / memory-leak - check

6. cannot compile term in SCO-unix

7. Communicating Unix to PC, via TCP/IP

8. malloc/free and memory used by a process

9. RSS growing when I malloc() and free() memory on AIX4.3.2

10. Q: What (free) debuggers are available w/memory leak testing

11. Free Virutal Memory is 1% even though Total Free Virtual Memory is above 1GB

12. memory leaks by shared memory and fork