Reserve/commit memory management strategy for UNIX

Reserve/commit memory management strategy for UNIX

Post by Casper H.S. D » Sat, 17 Sep 1994 17:52:04




Quote:>It appears that it is possible to provide roughly this type of
>functionality on UNIX using mmap(2) and lseek(2).  In this case, you
>could mmap(2) a large chunk of virtual address space, without
>allocating any backing store in the corresponding file.  SIGBUS should
>occur when a process accesses a portion of the address space that is
>not in backing store, and the signal handler could then extend the
>backing store.

If you mmap a file in a region bigger than a the file, you will indeed
get a SIGBUS when you touch a page at the end of the file.
Alternatively, you can create a large empty file (a file that
consists of a hole by just setting the size big enough) and use that.
In that case the SIGBUS is generated when the disk is full, storage
allocation is automatic.  More modern versions of Mmap() come with
the MAP_NORESERVE (MAP_AUTORESERV) option.  This allows you to map
anonymous memory (/dev/zero on Suns) without reserving the backing
store.  Also some systems have MAP_AUTOGROW which will automatically
grow the object when needed.

Only when you want to grow an object and don't have MAP_AUTOGROW, you'll
get a SIGBUS signal when it is possible to perform an action to remedy
the problem.  In all other cases SIGBUS means out of disk space or
out of swap space.

Quote:>However, it seems that this approach is fraught with peril under UNIX
>since SIGBUS may mean other things besides "no backing store."  Has
>anyone ever used this approach successful on UNIX?

I've only used mmap with a "holey" file and in that case I used
SIGBUS to detect a full disk.

On many unixes it is possible to specify get more information about how
thw SIGBUS happened (which address, whether it was an allignement error
or an address error, etc).

Casper

 
 
 

Reserve/commit memory management strategy for UNIX

Post by Douglas C. Schmi » Sat, 17 Sep 1994 10:27:15


Hi,

        Windows NT provides an interesting mechanism for implementing
dynamically sized data structures.  Basically the scheme works as
follows:

        1. You can "reserve" a large chunk of virtual address space,
           without "committing" any backing store

        2. If a process or thread accesses a virtual address that
           isn't yet covered by backing store Windows NT
           raises a particular exception.

        3. The exception handler can then commit the backing
           store and restart the instruction (which should
           now succeed).

It appears that it is possible to provide roughly this type of
functionality on UNIX using mmap(2) and lseek(2).  In this case, you
could mmap(2) a large chunk of virtual address space, without
allocating any backing store in the corresponding file.  SIGBUS should
occur when a process accesses a portion of the address space that is
not in backing store, and the signal handler could then extend the
backing store.

However, it seems that this approach is fraught with peril under UNIX
since SIGBUS may mean other things besides "no backing store."  Has
anyone ever used this approach successful on UNIX?

        Thanks,

                Doug
--

Department of Computer Science, Washington University
St. Louis, MO 63130. Work #: (314) 935-7538; FAX #: (314) 935-7302
http://www.cs.wustl.edu/~schmidt/

 
 
 

Reserve/commit memory management strategy for UNIX

Post by Eric W Burg » Sun, 18 Sep 1994 05:21:36


Since SVR2 (at least, possibly earlier), a memory page isn't really
allocated until you write to it.  A quick test is to repetitively
malloc buffers until you run out of memory.  If the O/S only a phantom
allocation, your program will execute in milliseconds.  If it does
real allocation, it could take seconds.  The proof is in rerunning
your program, only this time writting data to the allocated pages --
it will run MUCH slower.

--
--  Eric William Burger       --  +1 301/417-0700(v) 0707(f) --
--  George Mason University   --  SITE PhD Program           --
--  The Telephone Connection  --  Director of Engineering    --
The views expressed above do not necessarily reflect the views
of George Mason University or The Telephone Connection.


:       1. You can "reserve" a large chunk of virtual address space,
:          without "committing" any backing store

:       2. If a process or thread accesses a virtual address that
:          isn't yet covered by backing store Windows NT
:          raises a particular exception.

:       3. The exception handler can then commit the backing
:          store and restart the instruction (which should
:          now succeed).

: It appears that it is possible to provide roughly this type of
: functionality on UNIX using mmap(2) and lseek(2).  In this case, you
: could mmap(2) a large chunk of virtual address space, without
: allocating any backing store in the corresponding file.  SIGBUS should
: occur when a process accesses a portion of the address space that is
: not in backing store, and the signal handler could then extend the
: backing store.

: However, it seems that this approach is fraught with peril under UNIX
: since SIGBUS may mean other things besides "no backing store."  Has
: anyone ever used this approach successful on UNIX?

 
 
 

Reserve/commit memory management strategy for UNIX

Post by Steve Ra » Sat, 17 Sep 1994 22:21:10



Quote:>Hi,

>    Windows NT provides an interesting mechanism for implementing
>dynamically sized data structures.  Basically the scheme works as
>follows:

>    1. You can "reserve" a large chunk of virtual address space,
>       without "committing" any backing store

>    2. If a process or thread accesses a virtual address that
>       isn't yet covered by backing store Windows NT
>       raises a particular exception.

>    3. The exception handler can then commit the backing
>       store and restart the instruction (which should
>       now succeed).

>It appears that it is possible to provide roughly this type of
>functionality on UNIX using mmap(2) and lseek(2).  In this case, you
>could mmap(2) a large chunk of virtual address space, without
>allocating any backing store in the corresponding file.  SIGBUS should
>occur when a process accesses a portion of the address space that is
>not in backing store, and the signal handler could then extend the
>backing store.

>However, it seems that this approach is fraught with peril under UNIX
>since SIGBUS may mean other things besides "no backing store."  Has
>anyone ever used this approach successful on UNIX?

Hi Doug,

I'm not an expert on VM, but I believe SVR4 already does this
sort of thing.  When you malloc an extent of memory, backing
store is reserved, but not allocated.  When you page fault on
unallocated pages in the range of memory allocated, the backing
store is actually allocated.

Steve Rago

 
 
 

Reserve/commit memory management strategy for UNIX

Post by Terry Lambe » Mon, 19 Sep 1994 11:29:51



] Since SVR2 (at least, possibly earlier), a memory page isn't really
] allocated until you write to it.

Do you mean "by definition" or "implementation defined"?  If "by definition",
then I think you meant to say SVR3.

SVR2 was the last version that did not reuire hardware virtual memory
(an mmu for implementation).

SRV2 was the last real UNIX to run on the 68010 (and was the last one I
could port to my Amiga 1000 with a 68010 because of that.  8-)).

                                        Terry Lambert

---
Any opinions in this posting are my own and not those of my present
or previous employers.

 
 
 

Reserve/commit memory management strategy for UNIX

Post by domingos begal » Wed, 21 Sep 1994 01:25:53



Quote:>Since SVR2 (at least, possibly earlier), a memory page isn't really
>allocated until you write to it.  A quick test is to repetitively
>malloc buffers until you run out of memory.  If the O/S only a phantom
>...

i believe with svr4 (at least irix) memory needs to be allocated for
a process to run,  if there is not enough, malloc gets NULL.  irix 5.x
provides a "vswap" variable to emulate the old behaviour.

--

"Fear is a very important feeling to have.  It helps you stay together.
It helps you race longer... and live longer."  Ayrton Senna (1960-1994)