Can mlock prevent system flash dirty pages to HD when mmap a regular file ?

Can mlock prevent system flash dirty pages to HD when mmap a regular file ?

Post by s.. » Thu, 14 Aug 1997 04:00:00



   > A SHARED mapped file _always_ flushes the relevant segments
   > of memory to the file, because other processes have to be
   > able to see the changes made by the writing process.

   I am not quite sure about the implementation of mmap, but it
   seems the system just map the file buffer address to process
address. If it's true, all the processes  map the same buffer
   by the same file name, and all the changes will be done directly on
the same buffer ( same shared  memory area ). So the change
   will be seen at once by every processes using mmap. I think
   the system write the buffer to HD for 3 reasons:
   * Let other process which do not using mmap know the changes.
   * Not enough core.
   * Prevent crash or similar things.

   Then what we need to do is to prevent system write content ( dirty
pages) from file buffer to HD.  Is the mlock, plock useful for this ?

   I like shmget, shmat too, but they are not so good if the
   programmer can not change the cooperating application ( unrelated,
not generate by fork() ) much from source code.

   Any suggestion ?  Thanks in advanced

  John Sun  

============= CUT===================
I have a data share model, I think it's a very common one, but I
want to find out a solution maybe unusual.

----------------------------------------------------------------
One process keep updating the shared data area
(when changing, it lock the data area),

Other processes read the shared data by making a copy data(i)
at time i.
----------------------------------------------------------------

I know we can have several solutions for this:

-------------------------------------
Allocate an anonymous memory area by
mmap( ..., -1,0)         (BSD Style), or
mmap(..., /dev/zero, 0)  (SUN Style), or

Using a regular data file, and we don't need to copy data from file
buffer to kernel buffer, by
mmap(..., a_regular_file,0)  

Or schmget, shmat(), shmdt() (SysV-based).
-------------------------------------

IPC schmget stuff is not easy to handle, especially when we need to
reuse some hard_to_change code or no_source_code application which can
only access share data by normal files operation.

The method using mmap(..., /dev/zero) or mmap(..., -1,0) to generate
anonymous memory area can not work for unrelated processes ( only
child which was created by fork() will inherits the mapped region ),
and it is not portable to non SVR4 systems.

Things will become easier if we

file_handler = open( a_regular_file )
using mmap(..., MAP_SHARE, file_handler, 0) for data_update process,
using mmap(..., MAP_PRIVATE, file_handler,0) for data_read process.

All the processes only need to know a file name even they come from
different application and designed by different programmer.

But my problem came:

* We got some disk I/O from time to time because the system update the
regular file automatically. Can I eliminate those disk I/O and speed
up data access by mlock, pin, munpin ?   Because all the share data
are frequently changed temperary data, we don't need to write it to
HD.   Is there any tool can trace the flash/swap/sync action for
special file ( not a statistics for all the files )  and let us know
whether memory lock works  ?

* How does the mmap works ?  Map the address of file buffer to the
process address ?  Will the normal file operation (write, close,
fclose) from other process cause data being flash to harddisk ?  If it
did,  this easy_to_handle solution fail.

* How does mmap with MAP_PRIVATE work ?  Make a copy from the mapped
memory area or directly from the file on harddisk.  If the latter is
true, I think MAP_SHARE + memcpy will be better for those data_read
processes.

PS, I have a quick look at Apache ( a free web server ) code for
scoreboard file. It seems it use separate implementation for install
option "share memory" and "regular file". When using "share memory"
option, it call functions something like
write_core_data_to_file() to write data to a regular file.

I think a "Virtual File" stay in core will be better (if we can make
it possible) for this scoreboard like situation, because it will make
it easier for any "extention", "Add-On", cooperate program to
handle the data in file.

Any suggestion or successful story ?  Thanks in advanced.

John Sun
Rapid Prototyping New Baby

 
 
 

Can mlock prevent system flash dirty pages to HD when mmap a regular file ?

Post by s.. » Fri, 15 Aug 1997 04:00:00



> On some systems, this doesn't require writing to the disk.  The internal
> implementation of read() and write() for regular files uses the VM
> subsystem, so they'll see the mapping.  Conversely, write() just writes
> to a kernel memory buffer that's mapped into the file,
> and the VM system eventually flushes it to disk.

     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      Is that you meant even mlock() or plock() can not prevent this ?
Thanks in advanced.    

> --

> BBN Corporation, Cambridge, MA
> Support the anti-spam movement; see <http://www.cauce.org/>
> Please don't send technical questions directly to me, post them to newsgroups.

-------------------------- cut ---------------------
Quote:>   > A SHARED mapped file _always_ flushes the relevant segments
>   > of memory to the file, because other processes have to be
>   > able to see the changes made by the writing process.

>   I am not quite sure about the implementation of mmap, but it
>   seems the system just map the file buffer address to process
>address. If it's true, all the processes  map the same buffer
>   by the same file name, and all the changes will be done directly on
>the same buffer ( same shared  memory area ). So the change
>   will be seen at once by every processes using mmap. I think
>   the system write the buffer to HD for 3 reasons:
>   * Let other process which do not using mmap know the changes.
>   * Not enough core.
>   * Prevent crash or similar things.

>   Then what we need to do is to prevent system write content ( dirty
>pages) from file buffer to HD.  Is the mlock, plock useful for this ?

>   I like shmget, shmat too, but they are not so good if the
>   programmer can not change the cooperating application ( unrelated,
>not generate by fork() ) much from source code.

>   Any suggestion ?  Thanks in advanced

>  John Sun  

>============= CUT===================
>I have a data share model, I think it's a very common one, but I
>want to find out a solution maybe unusual.

>----------------------------------------------------------------
>One process keep updating the shared data area
>(when changing, it lock the data area),

>Other processes read the shared data by making a copy data(i)
>at time i.
>----------------------------------------------------------------

>I know we can have several solutions for this:

>-------------------------------------
>Allocate an anonymous memory area by
>mmap( ..., -1,0)         (BSD Style), or
>mmap(..., /dev/zero, 0)  (SUN Style), or

>Using a regular data file, and we don't need to copy data from file
>buffer to kernel buffer, by
>mmap(..., a_regular_file,0)  

>Or schmget, shmat(), shmdt() (SysV-based).
>-------------------------------------

>IPC schmget stuff is not easy to handle, especially when we need to
>reuse some hard_to_change code or no_source_code application which can
>only access share data by normal files operation.

>The method using mmap(..., /dev/zero) or mmap(..., -1,0) to generate
>anonymous memory area can not work for unrelated processes ( only
>child which was created by fork() will inherits the mapped region ),
>and it is not portable to non SVR4 systems.

>Things will become easier if we

>file_handler = open( a_regular_file )
>using mmap(..., MAP_SHARE, file_handler, 0) for data_update process,
>using mmap(..., MAP_PRIVATE, file_handler,0) for data_read process.

>All the processes only need to know a file name even they come from
>different application and designed by different programmer.

>But my problem came:

>* We got some disk I/O from time to time because the system update the
>regular file automatically. Can I eliminate those disk I/O and speed
>up data access by mlock, pin, munpin ?   Because all the share data
>are frequently changed temperary data, we don't need to write it to
>HD.   Is there any tool can trace the flash/swap/sync action for
>special file ( not a statistics for all the files )  and let us know
>whether memory lock works  ?

>* How does the mmap works ?  Map the address of file buffer to the
>process address ?  Will the normal file operation (write, close,
>fclose) from other process cause data being flash to harddisk ?  If it
>did,  this easy_to_handle solution fail.

>* How does mmap with MAP_PRIVATE work ?  Make a copy from the mapped
>memory area or directly from the file on harddisk.  If the latter is
>true, I think MAP_SHARE + memcpy will be better for those data_read
>processes.

>PS, I have a quick look at Apache ( a free web server ) code for
>scoreboard file. It seems it use separate implementation for install
>option "share memory" and "regular file". When using "share memory"
>option, it call functions something like
>write_core_data_to_file() to write data to a regular file.

>I think a "Virtual File" stay in core will be better (if we can make
>it possible) for this scoreboard like situation, because it will make
>it easier for any "extention", "Add-On", cooperate program to
>handle the data in file.

>Any suggestion or successful story ?  Thanks in advanced.

>John Sun
>Rapid Prototyping New Baby


 
 
 

Can mlock prevent system flash dirty pages to HD when mmap a regular file ?

Post by Barry Margoli » Fri, 15 Aug 1997 04:00:00



>   the system write the buffer to HD for 3 reasons:
>   * Let other process which do not using mmap know the changes.

On some systems, this doesn't require writing to the disk.  The internal
implementation of read() and write() for regular files uses the VM
subsystem, so they'll see the mapping.  Conversely, write() just writes to
a kernel memory buffer that's mapped into the file, and the VM system
eventually flushes it to disk.

--

BBN Corporation, Cambridge, MA
Support the anti-spam movement; see <http://www.cauce.org/>
Please don't send technical questions directly to me, post them to newsgroups.

 
 
 

Can mlock prevent system flash dirty pages to HD when mmap a regular file ?

Post by Rory Fost » Fri, 15 Aug 1997 04:00:00


|>    > A SHARED mapped file _always_ flushes the relevant segments
|>    > of memory to the file, because other processes have to be
|>    > able to see the changes made by the writing process.
|>
|>    Then what we need to do is to prevent system write content ( dirty
|> pages) from file buffer to HD.  Is the mlock, plock useful for this ?
|>

If I understand correctly, you wish to prevent mmap pages from being flushed
to disk.  The mlock and plock calls, however, are routines that prevent
pages from being paged-out or swapped-out.  Dirty pages to write-back or
write-through is like an orange.  And pages to page-out or swap-out is an apple.

In the rest of your article, you eliminate shared memory segments, but this
could have been a good choice.  If you're able to update the application
enough to use mmap, why can't you update the application to use shared memory?

Well, good luck.  As you pointed out, there are several ways to solve the
problem.  Each has its own set of implications and limitations.

--
Rory Foster

 
 
 

1. Bios extension (Datalight Flash FX) prevents boot up from HD

I'm using an embedded PC board MediaGX Processor board
(www.arcom.co.uk)  using the Cyrix GX processor.  It has on board flash
which can be driven by the bios extension Flash FX from Datalight
(www.datalight.com).

Originally I could not boot beyond the:

lilo: linux
Loading........

and then, quite by accident, I installed a hardware module which shorted
some of the ISA address lines which prevented the BIOS extension from
loading (presumably because the 55AA magic number of the extension could
not be found), and then, without the bios extension, linux had no
trouble booting.

I'm informed by the Flash FX docs that the extension replaces the int
13h disk i/o so that the flash can be read/written like a normal drive.
Problem is, this confused linux boot up like I said.  I see from the
kernel boot sources that int 13h is used extensively.

My question is
        a) how can I stop the extension from loading (without resorting to
shorting address lines)

or preferably
        b) how can I convince the linux booter to talk to the hard drive..  I
don't understand why it can load the bootstrap from the HD okay and then
hangs.  If it's of any consequence: the flash is located at 0xDC000.

Cheers

Andy

-

2. Help PCMCIA & Xircom 10/100 +56K ==> /dev/modem always busy

3. Flash ROM Driver / Flash File System

4. A GOOD 56K modem?

5. File System Found Dirty

6. Help Sendmail/Relay Host or DNS

7. Enhanced Filing system - file system like DEC Advanced File system for Linux

8. Micro$oft Netmeeting thru linux router

9. File systems dirty? read this!

10. file system still dirty, please rerun fsck

11. Novice Question: Dirty File System

12. 2.4.19 place buffer dirtied in truncate() on inode's dirty data list

13. Flash File System