mmap as a shm replacement

mmap as a shm replacement

Post by Andrew Giert » Sat, 15 Mar 1997 04:00:00



[The original poster messed up the Newsgroups line, so I'm fixing it
for him and quoting his post in full.]

 Derek> I'm trying to decide if mmap is a good replacement for sysv
 Derek> shared memory (shmget/shmat).  The way I see it, mmap()ing a
 Derek> file MAP_SHARED allows me to have a permanent shared memory
 Derek> region with backing store that's kept in the filesystem.

 Derek> so a call to
 Derek>    id = shmget(ftok(filename))
 Derek>    shmat(id,...)

 Derek> could be replaced with
 Derek>    filedes = open(filename)
 Derek>    mmap(filedes, MAP_SHARED, ...)
 Derek>    close(filedes)

 Derek> I'd of course encapsulate this into an internal library to
 Derek> simplify.

 Derek> There may be some problems with this, such as files staying
 Derek> around longer than they should, but that's a (notorious)
 Derek> problem with sysv shared memory regions as well--and they
 Derek> exist outside of the normal filesystem namespace.

 Derek> Anyway, enough intro.  My question is: is this a reasonable
 Derek> thing to do?  Looking at various man pages, there's some
 Derek> discrepancy about what mmap does in different situations,
 Derek> which I can deal with.  I'm just not sure if these pages are
 Derek> really shared in the same way shm* memory is.  Does a process
 Derek> maping a file shared actually map to the _same_ physical pages
 Derek> as all the other processes mapping to the same file, or are
 Derek> they really mapped into each processes space, with some
 Derek> filesystem trickery thing going on behind the scenes to make
 Derek> them _look_ shared?

The answer is, of course, "it depends".

I know of one system where mmap() is implemented using a page-alias
scheme where every time any page in the mapping is accessed from a
different process, it takes a fast page fault and the system fiddles
with the mappings to effectively "move" the physical page to the new
process (and the same happens again when another process touches the
page). (This impacts only performance, it's entirely transparent, and
there is a workaround if necessary. This is for AIX on certain hardware
combinations where hardware page aliasing isn't allowed.)

Other systems may, or may not, have similar problems. HP-UX, for example,
always uses the same physical pages for shared mappings, but if a large
number of different mappings are in use, it may need to shuffle protection
keys around all the time. (This restriction applies equally to mappings
done by shmat(), though.)

Intel-based systems will generally share the pages directly, since the 386
MMU doesn't have any restrictions that would prevent it. (Both the POWER
and HP-PA, as I understand it, have inverted page tables and virtual-tagged
cache, which makes page sharing more difficult.)

--
Andrew.

 
 
 

mmap as a shm replacement

Post by Derek Inksette » Sat, 15 Mar 1997 04:00:00




>  Derek> I'm trying to decide if mmap is a good replacement for sysv
>  Derek> shared memory (shmget/shmat).  The way I see it, mmap()ing a
>  Derek> file MAP_SHARED allows me to have a permanent shared memory
>  Derek> region with backing store that's kept in the filesystem.
[SNIP]
>  Derek>                         I'm just not sure if these pages are
>  Derek> really shared in the same way shm* memory is.  Does a process
>  Derek> maping a file shared actually map to the _same_ physical pages
>  Derek> as all the other processes mapping to the same file, or are
>  Derek> they really mapped into each processes space, with some
>  Derek> filesystem trickery thing going on behind the scenes to make
>  Derek> them _look_ shared?

> The answer is, of course, "it depends".

> I know of one system where mmap() is implemented using a page-alias
> scheme where every time any page in the mapping is accessed from a
> different process, it takes a fast page fault and the system fiddles
> with the mappings to effectively "move" the physical page to the new
> process (and the same happens again when another process touches the
> page). (This impacts only performance, it's entirely transparent, and
> there is a workaround if necessary. This is for AIX on certain hardware
> combinations where hardware page aliasing isn't allowed.)
> Other systems may, or may not, have similar problems. HP-UX, for example,
> always uses the same physical pages for shared mappings, but if a large
> number of different mappings are in use, it may need to shuffle protection
> keys around all the time. (This restriction applies equally to mappings
> done by shmat(), though.)

> Intel-based systems will generally share the pages directly, since the 386
> MMU doesn't have any restrictions that would prevent it. (Both the POWER
> and HP-PA, as I understand it, have inverted page tables and virtual-tagged
> cache, which makes page sharing more difficult.)

OK, but don't those systems have the same problems with any kind of page
sharing?  I.e. won't shared memory behave the same way?  The thing I was
having a problem with was that I haven't found any reference to using
mmap in this way.  Even Stevens in _APUE_ only mentions using mmap as an
alternative to traditional file I/O amd briefly mentions using
MAP_ANONYMOUS|MAP_SHARED to share pages between related processes.

Derek
--

Systems Consultant                 http://www.dct.com/~derek


 
 
 

mmap as a shm replacement

Post by George Bal » Sun, 16 Mar 1997 04:00:00



> >  Derek> I'm trying to decide if mmap is a good replacement for sysv
> >  Derek> shared memory (shmget/shmat).  The way I see it, mmap()ing a
> >  Derek> file MAP_SHARED allows me to have a permanent shared memory
> >  Derek> region with backing store that's kept in the filesystem.
> [SNIP]
> >  Derek>                         I'm just not sure if these pages are
> >  Derek> really shared in the same way shm* memory is.  Does a process
> >  Derek> maping a file shared actually map to the _same_ physical pages
> >  Derek> as all the other processes mapping to the same file, or are
> >  Derek> they really mapped into each processes space, with some
> >  Derek> filesystem trickery thing going on behind the scenes to make
> >  Derek> them _look_ shared?

My experience is with SunOS4 and 5 and here the answer is that if the
pages come from the same file they will use the same physical pages.

The VM system keep s a track of the contents of each physical page via
the page structure, the contents are identified by (vnode,offset). When
the system takes a page fault, the kernel checks to see if the required
page is in memory by looking for a page with the right (vnode,offset)
identity (this is obtained by examining the mmaped segments from the
kernel's representaiton of the process address space). This can be done
quickly through a hashing algorithm. If the page is already there, then
a mapping is introduced into the context for the process and a ref count
on the page is incremented.

In SunOS 4 & 5 the shm_ facility is (to the best of my knowledge)
implemented using the same underlying technology as mmap. The biggest
difference is that to share memory between unrelated processes using
mmap, you need a filesystem object to act as backing for the segment.
The shm_ functions allow a region of anonymous memory to be used. It is
possible to mmap anonymous memory through the /dev/zero special file,
but this can only be shared among related processes (ie descended from a
common parent, where the original mmap is done).

Hope this helps

George

 
 
 

mmap as a shm replacement

Post by Andrew Gabri » Sun, 16 Mar 1997 04:00:00




Quote:

>>  Derek> I'm trying to decide if mmap is a good replacement for sysv
>>  Derek> shared memory (shmget/shmat).  The way I see it, mmap()ing a
>>  Derek> file MAP_SHARED allows me to have a permanent shared memory
>>  Derek> region with backing store that's kept in the filesystem.

>The thing I was
>having a problem with was that I haven't found any reference to using
>mmap in this way.  Even Stevens in _APUE_ only mentions using mmap as an
>alternative to traditional file I/O amd briefly mentions using
>MAP_ANONYMOUS|MAP_SHARED to share pages between related processes.

One problem that springs to mind is lack of semaphores, pretty essential
in most shared memory schemes.

You could use the SysV IPC semaphores (but then why not use it's shared
memory?), or you could use pthread_mutex with PTHREAD_PROCESS_SHARED on
systems with POSIX threads (or non-POSIX equivalents), or you could use
files (urgh)...

--


 
 
 

mmap as a shm replacement

Post by Jens-Uwe Mag » Sun, 16 Mar 1997 04:00:00






> >  Derek> I'm trying to decide if mmap is a good replacement for sysv
> >  Derek> shared memory (shmget/shmat).  The way I see it, mmap()ing a
> >  Derek> file MAP_SHARED allows me to have a permanent shared memory
> >  Derek> region with backing store that's kept in the filesystem.
> [SNIP]
> >  Derek>                         I'm just not sure if these pages are
> >  Derek> really shared in the same way shm* memory is.  Does a process
> >  Derek> maping a file shared actually map to the _same_ physical pages
> >  Derek> as all the other processes mapping to the same file, or are
> >  Derek> they really mapped into each processes space, with some
> >  Derek> filesystem trickery thing going on behind the scenes to make
> >  Derek> them _look_ shared?

> > The answer is, of course, "it depends".

> > I know of one system where mmap() is implemented using a page-alias
> > scheme where every time any page in the mapping is accessed from a
> > different process, it takes a fast page fault and the system fiddles
> > with the mappings to effectively "move" the physical page to the new
> > process (and the same happens again when another process touches the
> > page). (This impacts only performance, it's entirely transparent, and
> > there is a workaround if necessary. This is for AIX on certain hardware
> > combinations where hardware page aliasing isn't allowed.)

> > Other systems may, or may not, have similar problems. HP-UX, for example,
> > always uses the same physical pages for shared mappings, but if a large
> > number of different mappings are in use, it may need to shuffle protection
> > keys around all the time. (This restriction applies equally to mappings
> > done by shmat(), though.)

> > Intel-based systems will generally share the pages directly, since the 386
> > MMU doesn't have any restrictions that would prevent it. (Both the POWER
> > and HP-PA, as I understand it, have inverted page tables and virtual-tagged
> > cache, which makes page sharing more difficult.)

> OK, but don't those systems have the same problems with any kind of page
> sharing?  I.e. won't shared memory behave the same way?  The thing I was
> having a problem with was that I haven't found any reference to using
> mmap in this way.  Even Stevens in _APUE_ only mentions using mmap as an
> alternative to traditional file I/O amd briefly mentions using
> MAP_ANONYMOUS|MAP_SHARED to share pages between related processes.

These systems attempt to make sure that memory mappings in all processes
use the same virtual address in all processes. For example AIX reserves a
segment of the address space where it places mapped files or shared memory
segments used by shmat. Mmap allows a much finer grain control of the
pages of a process's address space, and a lot of the programs that use
mmap will have the same physical page attached to a different virtual
addresses. I generally use shmat on HP-UX and AIX and mmap on all other
systems. Actually AIX complicates this by allowing only 10 shmat shared
segments, so one has to fall back to mmap anyways if one uses more mapped
files.

HP-UX does not implement a special mapping facility as AIX via shmat does,
but instead places additional restrictions on where mmap will place the
mappings. We do have a program that does CRC checksum files, and on HP-UX
it did fail mysteriously if the file was a program actually executing. As
a program executing is already mapped and the CRC program did already have
a mapping at that address (namely the CRC program itself), it always
received an error on attempting to mmap that file.
_______________________________________________________________________


Brahmsstr. 3                    Tel.: +49 511 660238

 
 
 

mmap as a shm replacement

Post by Bob Withe » Sun, 16 Mar 1997 04:00:00



says...



> >>  Derek> I'm trying to decide if mmap is a good replacement for sysv
> >>  Derek> shared memory (shmget/shmat).  The way I see it, mmap()ing a
> >>  Derek> file MAP_SHARED allows me to have a permanent shared memory
> >>  Derek> region with backing store that's kept in the filesystem.

> >The thing I was
> >having a problem with was that I haven't found any reference to using
> >mmap in this way.  Even Stevens in _APUE_ only mentions using mmap as an
> >alternative to traditional file I/O amd briefly mentions using
> >MAP_ANONYMOUS|MAP_SHARED to share pages between related processes.

> One problem that springs to mind is lack of semaphores, pretty essential
> in most shared memory schemes.

> You could use the SysV IPC semaphores (but then why not use it's shared
> memory?), or you could use pthread_mutex with PTHREAD_PROCESS_SHARED on
> systems with POSIX threads (or non-POSIX equivalents), or you could use
> files (urgh)...

How about an advisory file lock on the underlaying mapped file?

--

Bob Withers              Do or do not, there is no try.

 
 
 

mmap as a shm replacement

Post by Doug Siebe » Sun, 16 Mar 1997 04:00:00



>One problem that springs to mind is lack of semaphores, pretty essential
>in most shared memory schemes.

Check out msem_lock/msem_unlock on HP-UX, AIX 4.x, and I believe Digital
Unix.  It offers extremely fast semaphores based in memory mapped files.
If there is no contention for the semaphore, there is no system call overhead
at all (at least on HP-UX, I cannot speak for anything else)  You don't get
much more efficient than that -- certainly better than the SysV semaphore
calls which always incur a system call overhead, and vastly superior to
normal lockf/flock file locking.

--
Douglas Siebert                Director of Computing Facilities

Ack!  My reality check just bounced!!

 
 
 

mmap as a shm replacement

Post by Bryan O'Sulliva » Sun, 16 Mar 1997 04:00:00


g> It is possible to mmap anonymous memory through the /dev/zero
g> special file, but this can only be shared among related processes
g> (ie descended from a common parent, where the original mmap is
g> done).

The file descriptor in question can be passed between unrelated
processes in the usual way.

        <b

--
Let us pray:



 
 
 

mmap as a shm replacement

Post by Scott Schwar » Sun, 16 Mar 1997 04:00:00


| You could use the SysV IPC semaphores (but then why not use it's shared
| memory?), or you could use pthread_mutex with PTHREAD_PROCESS_SHARED on
| systems with POSIX threads (or non-POSIX equivalents), or you could use
| files (urgh)...

Or you could use pipes.  A pipe with one byte in it works like a
binary semaphore.

 
 
 

mmap as a shm replacement

Post by Paul David Fo » Tue, 18 Mar 1997 04:00:00





> >>  Derek> I'm trying to decide if mmap is a good replacement for sysv
> >>  Derek> shared memory (shmget/shmat).  The way I see it, mmap()ing a
> >>  Derek> file MAP_SHARED allows me to have a permanent shared memory
> >>  Derek> region with backing store that's kept in the filesystem.

> >The thing I was
> >having a problem with was that I haven't found any reference to using
> >mmap in this way.  Even Stevens in _APUE_ only mentions using mmap as an
> >alternative to traditional file I/O amd briefly mentions using
> >MAP_ANONYMOUS|MAP_SHARED to share pages between related processes.

> One problem that springs to mind is lack of semaphores, pretty essential
> in most shared memory schemes.

When I tried to get Sybase running under Sol2, I screamed at the
system because of the SysV IPC requirements. In the end I knocked
up a pre-load shared lib which allows a program (i.e. sybase) to
use mmap-ed memory instead of SysV IPC memory without any recompilation
needed.

Its not complete but its good enough for Sybase to work with.
(I didnt implement semaphores because Sybase didnt need them).

If anyones interested post a message here. If need be I can pass
it on to someone who can shove it somewhere useful.

_______________________________________________________
| Paul David Fox          Lehman Brothers.             |


| ######################  Tel: +44 171 601 0011 x6025  |
This message is boring.

 
 
 

mmap as a shm replacement

Post by Derek Inksette » Tue, 18 Mar 1997 04:00:00



> g> It is possible to mmap anonymous memory through the /dev/zero
> g> special file, but this can only be shared among related processes
> g> (ie descended from a common parent, where the original mmap is
> g> done).

> The file descriptor in question can be passed between unrelated
> processes in the usual way.

Are you sure?  I thought mapping /dev/zero created a special mapping,
unrelated to the file descriptor.  In fact my test programs seem to
indicate that shared descriptors to /dev/zero do not cause mmap to share
pages.

What system exhibits this behavior?

Derek
--

Systems Consultant                 http://www.dct.com/~derek

 
 
 

mmap as a shm replacement

Post by Casper H.S. Dik - Network Security Engine » Tue, 18 Mar 1997 04:00:00



Quote:>When I tried to get Sybase running under Sol2, I screamed at the
>system because of the SysV IPC requirements. In the end I knocked
>up a pre-load shared lib which allows a program (i.e. sybase) to
>use mmap-ed memory instead of SysV IPC memory without any recompilation
>needed.

Interesting that it complained.  The SystemV shared memory is quite hard
to remove from Solaris (it's in the core OS) but it may need to have
some paremeters tuned.  It may very well be faster than using mmap()ed
memory.  Typical Sybase configurations may need to have the system V
shared memory parameters increased.

Quote:>Its not complete but its good enough for Sybase to work with.
>(I didnt implement semaphores because Sybase didnt need them).

They're also in the system.

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

 
 
 

mmap as a shm replacement

Post by Thor Lancelot Sim » Tue, 18 Mar 1997 04:00:00




Quote:

>In SunOS 4 & 5 the shm_ facility is (to the best of my knowledge)
>implemented using the same underlying technology as mmap. The biggest
>difference is that to share memory between unrelated processes using
>mmap, you need a filesystem object to act as backing for the segment.

That's why /tmp is a memory filesystem.

--

   But as he knew no bad language, he had called him all the names of common
 objects that he could think of, and had screamed: "You lamp!  You towel!  You
 plate!" and so on.              --Sigmund Freud

 
 
 

mmap as a shm replacement

Post by Nick Nevi » Wed, 19 Mar 1997 04:00:00


     > When I tried to get Sybase running under Sol2, I screamed at the
     > system because of the SysV IPC requirements. In the end I knocked
     > up a pre-load shared lib which allows a program (i.e. sybase) to
     > use mmap-ed memory instead of SysV IPC memory without any recompilation
     > needed.

     > Its not complete but its good enough for Sybase to work with.
     > (I didnt implement semaphores because Sybase didnt need them).

     > If anyones interested post a message here. If need be I can pass
     > it on to someone who can shove it somewhere useful.

I'm interested.

Thanks, Nick.

 
 
 

mmap as a shm replacement

Post by Paul David Fo » Thu, 20 Mar 1997 04:00:00




> >When I tried to get Sybase running under Sol2, I screamed at the
> >system because of the SysV IPC requirements. In the end I knocked
> >up a pre-load shared lib which allows a program (i.e. sybase) to
> >use mmap-ed memory instead of SysV IPC memory without any recompilation
> >needed.

> Interesting that it complained.  The SystemV shared memory is quite hard
> to remove from Solaris (it's in the core OS) but it may need to have
> some paremeters tuned.  It may very well be faster than using mmap()ed
> memory.  Typical Sybase configurations may need to have the system V
> shared memory parameters increased.

I didnt say SHM/SEM werent in the kernel (I know its difficult to
remove them). What I'm saying is that some magic incantation
in /etc/system wasnt there. (This was under
Solaris 2.4 on a crappy SparcClassic with 24MB of RAM; it might
have been that Solaris scales the resources too low in that
configuration). I
didnt have time to figure out what was wrong and doing the
shlib made things easy.

The interesting thing was why Sybase didnt appear to need SEMs.
From what I could see Sybase uses this mmap-ed memory as a way
of communicating run-time stats. Its just a pain that Sybase
didnt work 'out of the box' and even more a shame that
Sun doesnt document all the config options one can stuff in
/etc/system.

(How many mails have you answered Casper about the magic
names in /etc/system over the years?)

I can understand why they aren't documented but it is a nuisance
as that is one of the worst areas of Solaris.

Quote:> >Its not complete but its good enough for Sybase to work with.
> >(I didnt implement semaphores because Sybase didnt need them).

> They're also in the system.

Yup - but that didn't help me.

Quote:> Casper
> --
> Expressed in this posting are my opinions.  They are in no way related
> to opinions held by my employer, Sun Microsystems.
> Statements on Sun products included here are not gospel and may
> be fiction rather than truth.

--
_______________________________________________________
| Paul David Fox          Lehman Brothers.             |


| ######################  Tel: +44 171 601 0011 x6025  |
This message is boring.
 
 
 

1. mmap as a shm replacement

I'm trying to decide if mmap is a good replacement for sysv shared
memory (shmget/shmat).  The way I see it, mmap()ing a file MAP_SHARED
allows me to have a permanent shared memory region with backing store
that's kept in the filesystem.

so a call to
   id = shmget(ftok(filename))
   shmat(id,...)

could be replaced with
   filedes = open(filename)
   mmap(filedes, MAP_SHARED, ...)
   close(filedes)

I'd of course encapsulate this into an internal library to simplify.

There may be some problems with this, such as files staying around
longer than they should, but that's a (notorious) problem with sysv
shared memory regions as well--and they exist outside of the normal
filesystem namespace.

Anyway, enough intro.  My question is: is this a reasonable thing to
do?  Looking at various man pages, there's some discrepancy about what
mmap does in different situations, which I can deal with.  I'm just not
sure if these pages are really shared in the same way shm* memory is.
Does a process maping a file shared actually map to the _same_ physical
pages as all the other processes mapping to the same file, or are they
really mapped into each processes space, with some filesystem trickery
thing going on behind the scenes to make them _look_ shared?

Derek (Remove NOSPAM from address in replies)
--

Systems Consultant                 http://www.dct.com/~derek

2. ld.so problem with using Xfree86+Xwrapper.

3. SHM vs. MMAP?

4. Help with SiS 648 chipset and agpgart

5. advantage of mmap files over SysV shm

6. tar & mt

7. POSIX shm and mmap on Solaris problem -- a bug?

8. I hate Linus Torvalds, revisted

9. mmap vs. shm*

10. IPC: mmap or shm or what?

11. mmap: using it to mmap Queue Data structure

12. ALSA + mmap or OSS emulation + mmap producing stutering sound

13. MMAP cache problem in 2.4.x, Re: [parisc-linux] unreliable mmap (fwd)