Very Computer

Hi.

        Is there a memory allocation bug in gcc?  I'm running Linux
1.2.13, gcc 2.7 and libc 5.0.9.  If I compile and run the small
program below, malloc (or new, if I use g++) manages to allocate much
more memory that is available (including virtual memory).  For
example, here I have 40MB of physical memory and 32MB of swap space,
but malloc manages to allocate 1143 blocks of 1MB before returning
NULL!!!  Where is this memory coming from?

------------%< Beginning of code %<----------
#include <stdlib.h>

#define NB 120
#define BLOCK 1024*1024

char *iv[NB];

main()
{
  int i,j;

  for (i=0; i<NB; i++) {
    printf("%d  ", i);
    iv[i] = (char *)malloc(BLOCK);
    if (iv[i]==NULL) {
      printf("out of memory \n");
      exit(-1);
    }
  }
  for (;;) {
    for (i=0; i<NB; i++)
      for (j=0; j<BLOCK; j++)
        iv[i][j] = iv[i][j] + 1;
  }

Quote:}

------------%< end of code %<-----------

        On the code above, if I set NB to anything below 1143, the
program runs without segmentation faults or other errors.  Linux
starts to swap heavily, and becomes almost unusable, but doesn't die.

        However, if I request only one large block, malloc correctly
returns NULL if the block is bigger than the available memory.

        Is this a known bug?  Must I upgrade immediatly?

        Many thanks in advance.

Celso.

You stumbled onto a policy feature of the compiler library (I think) malloc function
known as "on-demand" allocation.  This feature is non-POSIX and you can crash the
system by doing what you did, i.e allocating a gigabyte and then trying to use it.
It is harmless until you actually address more memory than you have.
To prevent this you would use kernel quotas or turn off
the feature entirely (may have to recompile gcc, I'm not sure).

Quote:>Hi.
>    Is there a memory allocation bug in gcc?  

No.  Linux, prior to 2.0, as well as many other Unixes will allow
memory allocations to succeed whether or not there is the actual memory
to fulfill the requests, as they only make the memory available
when the memory gets *used*.  Apparently, it was decided that Linux
2.0 wouldn't let you do that anymore.

It turns out that that broke some programs (such as the java compiler)
that relied on the old behavior since they wastefully malloc'd memory
that they never used.  For these programs to work, you now need to set
aside extra (never used) swap space.  Of course, on my machine, I've
got 128MB of swap, so I never worry.  (Unless I'm running my *
simulations again... :-)

--Joe

--
============= Joe Zbiciak =============    Just then the floating disembodied

 - -http://www.veryComputer.com/~im14u2c/ -
======#include <std_disclaimer.h>======     "Everything you know is wrong!"

--
============= Joe Zbiciak =============    Just then the floating disembodied

 - -http://www.veryComputer.com/~im14u2c/ -
======#include <std_disclaimer.h>======     "Everything you know is wrong!"

I believe that linux malloc will allocate as much as you like, up to the
maximum address space the processor allows.  It's only when you try to
USE the memory that you the system page-faults it in.

Quote:> ------------%< Beginning of code %<----------
> #include <stdlib.h>

> #define NB 120
> #define BLOCK 1024*1024

> char *iv[NB];

> main()
> {
>   int i,j;

>   for (i=0; i<NB; i++) {
>     printf("%d  ", i);
>     iv[i] = (char *)malloc(BLOCK);
>     if (iv[i]==NULL) {
>       printf("out of memory \n");
>       exit(-1);
>     }
>   }

You didn't mention if anything over 1143 causes malloc() to return NULL
at this point or not.  I've run similar tests and I don't recall finding
an upper limit, but I might not have gone up high enough.

Quote:>   for (;;) {
>     for (i=0; i<NB; i++)
>       for (j=0; j<BLOCK; j++)
>         iv[i][j] = iv[i][j] + 1;
>   }
> }

At this point, when you finally exhaust your real virtual memory, you
should get a virtual memory exhausted result.

Have you let this program run through the entire set of iv[] pointers?
I suspect it would start getting rather slow as more and more swapping
takes place.

Quote:>         Is this a known bug?  Must I upgrade immediatly?

I'm not sure it IS a bug.

Just my two cents.

Marc

--
Marc L. Allen
Schlumberger Technologies
Staff Engineer

(Gad! What a boring signature!!)