Memory fragmentation with realloc()

Memory fragmentation with realloc()

Post by Emil Zanetti » Wed, 09 Jul 1997 04:00:00



I have found that excessive use of realloc() causes memory
fragmentation.

As an example, I have a program that uses a rotary buffer (about 400
elements deep). Each element starts at 32 bytes, but more memory
is realloc()ed as needed. This realloc happens slowly (sometimes 2
bytes at a time), and the buffers can sometimes exceed 1024 bytes.
Since I'm using a rotary buffer, each element would eventually get
flushed and restart at 32 bytes. Therefore the program should
eventually reach a "high water mark" in memory usage, and never
go beyond that. Unfortunately, after adding fragmentation to the
equation, this ends in a program that keeps chewing more memory until
the OS crashes!

The best solution I've been able to come up with is to replace the
realloc()s with a malloc, memcpy, and free. So instead of just
trying to expand, or shrink the memory block in its spot, the block
will actually be reallocated, hopefully filling in the memory gaps.

I've been told the QNX4.24 will have better memory management.

Has anyone else come up against this problem? Does anyone have a
better solution?

--
Emil Zanettin
"If you're looking for trouble, you've come to the right place!"

www:    http://www.anca.com.au, http://www.netlink.com.au/~zanettin

 
 
 

Memory fragmentation with realloc()

Post by Mitchell Schoenbr » Wed, 09 Jul 1997 04:00:00


: Since I'm using a rotary buffer, each element would eventually get
: flushed and restart at 32 bytes. Therefore the program should
: eventually reach a "high water mark" in memory usage, and never
: go beyond that. Unfortunately, after adding fragmentation to the
: equation, this ends in a program that keeps chewing more memory until
: the OS crashes!

This suggests one of two things.  Either your description is not
correct, or there is a bug in realloc().  Let me describe
a worst case scenerio in which there are at most n elements
that need to be allocated, with a maximum size of m.

|---- 1 ----|  free (m-1)  |----- 2 ---- |  free (m-1) ......|---- n ----|

This diagram would suggest that the maximum space needed would be
n*m + (n-1) * (m-1) bytes.  Of course some allocation units have
some overhead which I'll call X, which might get multiplied by
(2n-1).

If your program blows allocates more than this amount of memory,
it indicates a bug.  The best thing to do in that case would be
to try to create code as simple as possible that shows the problem,
and post it in the QUICS quics.experts.c conference.

: The best solution I've been able to come up with is to replace the
: realloc()s with a malloc, memcpy, and free.

One would have to guess that this is approximately what realloc()
does.

: So instead of just
: trying to expand, or shrink the memory block in its spot, the block
: will actually be reallocated, hopefully filling in the memory gaps.

: I've been told the QNX4.24 will have better memory management.

This doesn't look like a QNX problem.  It looks like a Watcom
problem.

: Has anyone else come up against this problem? Does anyone have a
: better solution?

If your problem is the fragmentation of memory there are two
directions to look in.  One is to see if the requirements of
your application suggest a better solution.  If not, and/or
you need a general solution, you might try a technique that
allows compacting of memory.  This of course is at the
expense of speed.
--


 
 
 

1. Dos file system memory leak and Memory fragmentation

   Hi, we are developing an application which continuously writes image
files on to an internal SCSI  disc which uses a DOS file system on it.
We use VxWorks 5.1 on a 68040 board. We used HeapTrace to and found
that the DOS file system leaks 80 bytes for every open() call.  After a
running for a while, the system memory get very fragmented, memShow()
shows the available memory around say 1.7MB and the maximum contiguous
space is only sa y 60K.

   This is bad because our DOS cluster size is around 63K and an open()
on a DOS file requires a contiguous space of the cluster size, 63k!
Consequently our system failed when the max contiguous space gets below
the DOS cluster size.

I have the following question:

    1. Is there any fix for the DOS open() leaking memory ?
    2. Is there any other way to manage the system memory to avoid memory
    fragmentation ?
    3. Any other suggestions to solve my problem.

    Thanks in advance for any help

---
-------------------------------------------------------
 Hao Nguyen, CSIRO Div of Manufacturing Tech,
 Locked Bag No. 9, Preston, Vic, 3072, Australia.

 Phone: 03 662 7762
 Fax: 03 662 7853

2. Strange 5333 access violation error on closing application on Win nt/2000/XP

3. Memory allocation and realloc

4. Which IOS version is better ?

5. realloc freed memory

6. Nathan Mates

7. Problem: fragmentation of free memory

8. Install a lot of files

9. Subject: Vxworks 5.3.1 - memory fragmentation

10. VxWorks - dosFs1 memory Leaks and fragmentation

11. Memory fragmentation errors?

12. memory fragmentation

13. memory fragmentation using open() , close()