maximum memory allocation

maximum memory allocation

Post by Nicolas Devillar » Tue, 11 Aug 1998 04:00:00



Hello:

I need to be able to acccess the maximum amount of memory under as many
Unix flavours as possible. For some reason, using malloc() and consorts
will only allocate RAM on most Unixes and eventually fail when physical
memory is exhausted (why not allocate further into swap space?? That's a
question left to each Unix flavour guru). Thus the need for some more
elaborate (portable) methods.

From what I could find on the subject, it is possible to allocate memory
directly in the swap area, by using mmap() on /dev/zero, something like:

void * devzero_malloc(size_t s)
{
    char * c ;
    int fd = open("/dev/zero", O_RDWR) ;
    c = mmap(0,s,PROT_READ|PROT_WRITE,MAP_PRIVATE,fd,0) ;
    close(fd) ;
    if (c==(char*)-1) {
        perror("mmap") ;
        return NULL ;
    }
    return (void*)c ;

Quote:}

void devzero_free(void * p, size_t s)
{
    munmap(p,s);

Quote:}

(the above code is just provided for clarity purposes and has not been
tested)

However:

- mmap() is not available on all Unixes, but that is not a real issue
  since all modern Unixes tend to have it these days.
- /dev/zero is also not available everywhere (HPUX, Ultrix do not have
  it). Anyone know a replacement for /dev/zero providing read/write
  functionalities, on HPUX?
- There is apparently a problem of definition. On the man page
  for /dev/zero, under Solaris and Linux it is possible to read:

  % man zero
  [...]
    Reads from a zero special file always return a  buffer  full
    of zeroes.  The file is of infinite length.

    Writes to a zero special file are always successful,
    but the data written is ignored.
  ---

  ... which brings in the following question: if all data written to
this device are ignored as quoted here in the manual page, how is it
possible that I can write to it and retrieve these values afterwards
(try it!)?? The malloc() implementation from Doug Lea included in the
GNU libc uses the fact that data can be retrieved from /dev/zero to
actually perform some kind of memory allocation in the swap areas, as in
the above functions. Is it safe to read/write data this way? Either the
manual page or the GNU malloc are wrong then!

Last attempt at getting some more memory: create a file in the current
working directory with the requested size rounded up to the correct page
size, and mmap() it. This is dirty and may leave temporary files if the
process does not abort correctly, but it also more likely to work since
users in need of a lot of memory usually have a lot of disk space too
(and disk is ever cheaper). Another issue is to make sure that enough
space is available on the filesystem in a portable way over Unix
flavours, but this can be achieved using 'df' tricks. This solution has
the advantage that the maximum allocatable amount of memory is limited
by the filesystem limitations only. But is it a safe solution? Objections
someone?

Anyone has a hint about the safety of mmap'ing /dev/zero? About the fact
that data can be safely located there? Is there any extended malloc()
implementation floating around making use of the last quoted solution?

Many thanks for helping,
Nicolas

 
 
 

maximum memory allocation

Post by Dr. Michael Alber » Thu, 13 Aug 1998 04:00:00



> Hello:

> I need to be able to acccess the maximum amount of memory under as many
> Unix flavours as possible. For some reason, using malloc() and consorts
> will only allocate RAM on most Unixes and eventually fail when physical

Malloc and other library calls allocate virtual memory.

I am not sure exactly how much memory you need. I will assumed that you
need a few gigabytes.  In this case, it's not a question of malloc or
/dev/zero, but whether your system uses 32-bit or 64-bit pointers.  At
32-bit's, you can only address (name) 4-Gigabytes of memory, no matter
how you allocate it.  There are several possibilities:

Some systems (like SGI?) currently have both 32-bit and 64-bit API's,
selected by choice of compiler flag and LD_LIBRARY_PATH.

If not, the you have to look at memory a few gigabytes at a time. If your
system supports large files, you could have one large file, which you
map and unmap a piece at a time (ugh) or if you don't have large files
you could have multiple files which you map in one at a time.

Both are painful.  Some system also have proprietary mechanisms whereby
a user program can use more than 4Gigabytes of memory, but it still only
see's 4-Gigabtyes at a time, and there is a special system call to
change which 4-Gig's are seen.

Best wishes,
  Mike

 
 
 

maximum memory allocation

Post by Andrew Giert » Thu, 13 Aug 1998 04:00:00


 Nicolas> Hello:

 Nicolas> I need to be able to acccess the maximum amount of memory
 Nicolas> under as many Unix flavours as possible. For some reason,
 Nicolas> using malloc() and consorts will only allocate RAM on most
 Nicolas> Unixes and eventually fail when physical memory is exhausted

You are misinterpreting your results. malloc() will allow memory to be
allocated up to a specified per-process resource limit, which on
modern systems is set via a limit/ulimit command in the shell or using
getrlimit()/setrlimit(). (On older systems it may be a kernel tunable
parameter.)

On some systems, there is a separate limit for "maximum virtual space
used per-process", which counts not only the malloc() heap but also
mapped files, shared memory, anon segments etc.

Oh, and watch out for systems with early swap reservation. On such
systems if inadequate amounts of swap space are configured, it may be
impossible to use all available physical RAM.

By far the simplest solution to your problem is to ensure that your
system limits are set high enough for your requirements.

 Nicolas> - /dev/zero is also not available everywhere (HPUX, Ultrix
 Nicolas> do not have it). Anyone know a replacement for /dev/zero
 Nicolas> providing read/write functionalities, on HPUX?

MAP_ANON flag to mmap().

 Nicolas> - There is apparently a problem of definition. On the man page
 Nicolas>   for /dev/zero, under Solaris and Linux it is possible to read:
 [...]
 Nicolas> Is it safe to read/write data this way?

yes.

 Nicolas> Either the manual page or the GNU malloc are wrong then!

The manual page is referring to the device's behaviour with regard to
read() and write(), not what happens when you use it with mmap().

--
Andrew.

comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>
                           or <URL: http://www.whitefang.com/unix/>

 
 
 

1. Maximum memory allocation for DB2?

AIX 5.2, P690 LPAR w/ 3Gb RAM:

DB2 7.2 fp7 is to be installed - DBA has asked how much memory he can
use in the DB2 memory allocation?

Is there a recommended amount which I should reserve for the O/S and
houskeeping duties (backups, etc)?

Is there an easy way to work out what I should let him have for DB2?

Cheers

2. Problem mounting Partition..... Newbie !

3. Memory allocation and de-allocation

4. help wifi problem

5. memory allocation by OS memory manager

6. htaccess problems witApache 1.3 upgrade

7. Infinite memory?!? Memory allocation bug with gcc?

8. [PATCH] 0/7 2.5.35 SCSI multi-path

9. suppress allocation warnings for radix-tree allocations

10. Still problems with memory allocations

11. Allocation of memory failed

12. memory alloc failuer : __alloc_pages: 1-order allocation failed.

13. memory allocation problems(brk fails with EGAIN, ENOMEM etc.,