Very Computer

Thought I'd throw this out as I'm sure I'm not the only one to have to
have solved this...

I have a stream of data arriving in one a pipe via process A -- about 50M/day
I want to buffer this data to a large disk buffer.
Multiple readers want to get at that circular buffer and read data.

In memory, this would be easy, with a large number of threads and their
condition variables this would be easy.

I first thought I'd divide the buffer into say 1000000 "pages".  The writer
could lock any page as it was accessing/updating it.  Readers would have to
aquire a lock to read the page.  

Still, readers need to know where a writer is, so I need an index of some
sort.  This needs to be on disk as well, but I thought I'd mmmap that into core.
After the writer started work on a page, it would msync() the shared map.

Still, locking is an issue.  How do I keep the index pages consistent without
hitting the disk continuouly and using lock() style code.  Again, condition
variables work, but I need one million of them.

Any ideas?

 John> I have a stream of data arriving in one a pipe via process A --
 John> about 50M/day I want to buffer this data to a large disk
 John> buffer.  Multiple readers want to get at that circular buffer
 John> and read data.

 John> In memory, this would be easy, with a large number of threads
 John> and their condition variables this would be easy.

 John> I first thought I'd divide the buffer into say 1000000 "pages".
 John> The writer could lock any page as it was accessing/updating it.
 John> Readers would have to aquire a lock to read the page.

 John> Still, readers need to know where a writer is, so I need an
 John> index of some sort.  This needs to be on disk as well, but I
 John> thought I'd mmmap that into core.  After the writer started
 John> work on a page, it would msync() the shared map.

 John> Still, locking is an issue.  How do I keep the index pages
 John> consistent without hitting the disk continuouly and using
 John> lock() style code.  Again, condition variables work, but I need
 John> one million of them.

This looks perfectly solvable to me using nothing more than the usual
fcntl() locking functions.

How about this:

  writer always holds an exclusive lock on the element which it will
  write next

  readers can get the writer's position using F_GETLK

  readers hold shared locks for at least the first unread element;
  this automatically handles waking up readers when data arrives, and
  also blocks the writer when the queue becomes full

--
Andrew.

: Thought I'd throw this out as I'm sure I'm not the only one to have to
: have solved this...
:
:
: I have a stream of data arriving in one a pipe via process A -- about 50M/day
: I want to buffer this data to a large disk buffer.
: Multiple readers want to get at that circular buffer and read data.
:

If you are interested in a commercial product that I wrote and am open to
extention of the features to include multiple-reader circular queue functionality,
check out:

http://www.compgen.com/widgets

The end of this page describes the SMQ FIFO library.

For what OS is this?

Curt Smith