Advice on two management of massive files between two processes

Advice on two management of massive files between two processes

Post by Logan Sh » Fri, 27 Oct 1995 04:00:00

>    I would like to map the file in parts and process the file,
>    delete the part of the file consumed and continue processing
>    using the disk as a buffer between the process that is consuming
>    the data and the process that is producing the trace data.

>    I would like an outline for an apporach to solving this probelm
>    since the trace can enever be store completely and I need to
>    map segements of the file to run the simualtor on the data.

So what you want to do is map PORTIONS of a file into memory?  I've
never done it, but according to the SunOS 4.1.3 manual page for mmap(),
it can be done.  A snippet:

             caddr_t mmap(addr, len, prot, flags, fd, off)
             caddr_t addr;
             size_t len;
             int prot, flags, fd;
             off_t off;

             mmap() establishes a mapping between the  process's  address
             space  at  an  address pa for len bytes to the memory object
             represented by fd at off for len bytes.  The value of pa  is
             an  implementation-dependent  function of the parameter addr
             and values of flags, further described below.  A  successful
             mmap()  call  returns  pa as its result.  The address ranges
             covered by [pa, pa + len) and [off, off + len) must be legi-
             timate  for  the  possible (not necessarily current) address

So, presumably "len" is the number of bytes that actually get mapped
between the file and memory and "off" is the place where the mapping
starts in the file.  Hypothetically, then, you ought to be able to
just give successive calls to mmap() with a different value for
off (followed by appropriate calls to munmap()), and you have what
you want.

BTW, you're dealing with 300 MEGAbytes of data, not a 300 GIGAbytes (as
your article said), right?  If you're really dealing with 300GB of
data, though: your question takes on a whole new meaning.  You want to
take the data in memory and then mmap() it to a file for a while (so
the other process can use it), then throw it away and mmap() some more
data into the file, and so on.  In that case, why not just use plain
old shared memory?  As the data is collected, the first process writes
it into memory, then the second process processes it, then the first
process writes more (or use a double-buffering strategy so that the
first process can always be writing if you're collecting data in real
time).  Or is there some kind of minimum amount of data that the
second process needs to deal with at once, where that minimum is
more than the swap space?

  - Logan
                   =-- do not send me any junk e-mail --=


1. One Process on the machine having two IP address and two ethernet interface.

Hi~ all.. :)

My machine has two ethernet cards. its ethernet cards are assigned IP
address each.
One server process works on my machine.
If this server process wanna deal with packets from all two interfaces, two
that is, clients can be on two interfaces.

Is it possible?
If not, Does it need each server process according to each IP. need two
server process ?

Help me..
Beg your advice..
Happy Happy day..

| PC ??? ?3? ?3Y ???? ?? ?? IP ?? A.A.A.A ? A.A.A.B ?
?? PC ?? ??3a? ?-1? ??? ????? ???.
?-1?? ????? ??????? ????e??? A.A.A.A, A.A.A.B ? ?e????-
??? ?? ???.
? ?-1?? ??? LAN ?e?? ????? 1T?? ?3?? ? ?? ????? ? ??.

.. ???? ?y?o???? ?-1??| ??? ? ??????~?
(1) ??3a? ?-1?? ?3? IP ??? ???? ??????~?

(2) ??? ???? ??, ?3? ?-1? ?????o?| ? IP ?? ??????

??e o??1??3?.

2. perl5.003 binary available?

3. How to setup two terminals, two keyboards, two mouses?

4. HELP linux/cdrom (i forgot to add my email adress)

5. two routers (two wans) connecting to one lan using two nics.

6. VTxxx Dumb Terminals

7. two dsl connections, two routers, dual nics on linux box , want to run two websites

8. Mailer 1.3 problems

9. How to get file modification time in seconds/milli-seconds?

10. how to join two files using two columns as key

11. How to merge two columns of numbers from two different files?

12. Adding a second hard drive -- file system layout advice?

13. Two processes redirects output to the same file