Very Computer

I need to be able to acccess the maximum amount of memory under as many
Unix flavours as possible. For some reason, using malloc() and consorts
will only allocate RAM on most Unixes and eventually fail when physical
memory is exhausted (why not allocate further into swap space?? That's a
question left to each Unix flavour guru). Thus the need for some more
elaborate (portable) methods.

From what I could find on the subject, it is possible to allocate memory
directly in the swap area, by using mmap() on /dev/zero, something like:

void * devzero_malloc(size_t s)
{
    char * c ;
    int fd = open("/dev/zero", O_RDWR) ;
    c = mmap(0,s,PROT_READ|PROT_WRITE,MAP_PRIVATE,fd,0) ;
    close(fd) ;
    if (c==(char*)-1) {
        perror("mmap") ;
        return NULL ;
    }
    return (void*)c ;

Quote:}

Quote:}

However:

- mmap() is not available on all Unixes, but that is not a real issue
  since all modern Unixes tend to have it these days.
- /dev/zero is also not available everywhere (HPUX, Ultrix do not have
  it). Anyone know a replacement for /dev/zero providing read/write
  functionalities, on HPUX?
- There is apparently a problem of definition. On the man page
  for /dev/zero, under Solaris and Linux it is possible to read:

  % man zero
  [...]
    Reads from a zero special file always return a  buffer  full
    of zeroes.  The file is of infinite length.

    Writes to a zero special file are always successful,
    but the data written is ignored.
  ---

  ... which brings in the following question: if all data written to
this device are ignored as quoted here in the manual page, how is it
possible that I can write to it and retrieve these values afterwards
(try it!)?? The malloc() implementation from Doug Lea included in the
GNU libc uses the fact that data can be retrieved from /dev/zero to
actually perform some kind of memory allocation in the swap areas, as in
the above functions. Is it safe to read/write data this way? Either the
manual page or the GNU malloc are wrong then!

Last attempt at getting some more memory: create a file in the current
working directory with the requested size rounded up to the correct page
size, and mmap() it. This is dirty and may leave temporary files if the
process does not abort correctly, but it also more likely to work since
users in need of a lot of memory usually have a lot of disk space too
(and disk is ever cheaper). Another issue is to make sure that enough
space is available on the filesystem in a portable way over Unix
flavours, but this can be achieved using 'df' tricks. This solution has
the advantage that the maximum allocatable amount of memory is limited
by the filesystem limitations only. But is it a safe solution? Objections
someone?

Anyone has a hint about the safety of mmap'ing /dev/zero? About the fact
that data can be safely located there? Is there any extended malloc()
implementation floating around making use of the last quoted solution?

Many thanks for helping,
Nicolas

I am not sure exactly how much memory you need. I will assumed that you
need a few gigabytes.  In this case, it's not a question of malloc or
/dev/zero, but whether your system uses 32-bit or 64-bit pointers.  At
32-bit's, you can only address (name) 4-Gigabytes of memory, no matter
how you allocate it.  There are several possibilities:

Some systems (like SGI?) currently have both 32-bit and 64-bit API's,
selected by choice of compiler flag and LD_LIBRARY_PATH.

If not, the you have to look at memory a few gigabytes at a time. If your
system supports large files, you could have one large file, which you
map and unmap a piece at a time (ugh) or if you don't have large files
you could have multiple files which you map in one at a time.

Both are painful.  Some system also have proprietary mechanisms whereby
a user program can use more than 4Gigabytes of memory, but it still only
see's 4-Gigabtyes at a time, and there is a special system call to
change which 4-Gig's are seen.

Best wishes,
  Mike

 Nicolas> Hello:

 Nicolas> I need to be able to acccess the maximum amount of memory
 Nicolas> under as many Unix flavours as possible. For some reason,
 Nicolas> using malloc() and consorts will only allocate RAM on most
 Nicolas> Unixes and eventually fail when physical memory is exhausted

You are misinterpreting your results. malloc() will allow memory to be
allocated up to a specified per-process resource limit, which on
modern systems is set via a limit/ulimit command in the shell or using
getrlimit()/setrlimit(). (On older systems it may be a kernel tunable
parameter.)

On some systems, there is a separate limit for "maximum virtual space
used per-process", which counts not only the malloc() heap but also
mapped files, shared memory, anon segments etc.

Oh, and watch out for systems with early swap reservation. On such
systems if inadequate amounts of swap space are configured, it may be
impossible to use all available physical RAM.

By far the simplest solution to your problem is to ensure that your
system limits are set high enough for your requirements.

 Nicolas> - /dev/zero is also not available everywhere (HPUX, Ultrix
 Nicolas> do not have it). Anyone know a replacement for /dev/zero
 Nicolas> providing read/write functionalities, on HPUX?

MAP_ANON flag to mmap().

 Nicolas> - There is apparently a problem of definition. On the man page
 Nicolas>   for /dev/zero, under Solaris and Linux it is possible to read:
 [...]
 Nicolas> Is it safe to read/write data this way?

yes.

 Nicolas> Either the manual page or the GNU malloc are wrong then!

The manual page is referring to the device's behaviour with regard to
read() and write(), not what happens when you use it with mmap().

--
Andrew.

comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>
                           or <URL: http://www.whitefang.com/unix/>