Very Computer

I am trying to make a language processor POSIX.1-1990 compliant as part of
my M.S. C.S. thesis project, and have hit a sticky spot.

I have a parent process that needs to pass messages of arbitrary length to
and from multiple child processes.  The current implementation is using
SysV message queues.

The only POSIX.1 mechanisms that are available to me are pipes, and FIFOs;
I have written an implementation of binary semaphores based on POSIX.1
advisory locks (thanks to a suggestion in W. Richard Stevens' book!).

The type and length of each message is known before the writer has to write
the message.

The hard part seems to be making sure that the reader gets a complete message,
and NO MORE than a single complete message.  If I could solve this, I could
create a linked-list of received messages, and provide a function call to
either fetch a message from the list, or block until another process writes
a message.

Any thoughts on how I should approach this?

Thanks,
--Malcolm

---
--------------------------------------------------------------------------------

Motorola ASIC, Design Automation   Phone:   (602) 491-7182
OACS Tools and Architecture

[...]

Make the parent process arrange for a pipe connection to each child.  Handle
the communication back to the parent through a single FIFO.

Quote:>The type and length of each message is known before the writer has to write
>the message.
>The hard part seems to be making sure that the reader gets a complete message,
>and NO MORE than a single complete message.

There are at least 3 strategies that might be best in different situations:
(1) use fixed length read()s and writes()s, passing structs that are
padded as necessary. (2) use some delimiter known not to be in the
data. (3) write the size ahead of the data.  In all cases when writing
to the FIFO the entire message must be passed in a single write()
and must be less than PIPE_MAX.  If you expect these things to be
flying around fairly fast you might want to do the read()s into a
large buffer to reduce the system call overhead since your chosen
strategy will let you sort it out in user space.

Les Mikesell

No disagreement, but maybe a clarification. What if there is
outstanding data written to the pipe but not yet read. Does the
PIPE_MAX limit still apply? In other words, does the write block until
it can do a PIPE_MAX write atomically? Commonsense says yes, but...

M.

That's what PIPE_MAX always meant under unix.  I don't know what
the posix spec actually says, but FIFOs would be pretty useless
if you can't count on this behaviour.  Also, you will probably
want to have the process reading the FIFO also open it for
write access so you don't see 0-length read()s if it has been
opened for writing and then closed by all the writers.  On some
versions opening for r/w access will also prevent the usual
blocking when you open a FIFO, but on others you may still have to
twiddle O_NDELAY for the open and clear it with fcntl() if you
want the reads to block.

Les Mikesell

AAAAGGGHHH!!!!  Twelve years later and we're still screwing around with PIPES!
NT deserves to win if we can't do any better than this......

Contract Programmer   | TEL (708) 632-7885   | obscured by the thrill I got
Motorola Cellular     | Arlington Hgts, Il.  | from getting it to work at all.'
Infrastructure Group  |----------------------|-----  anon product review

Quote:>>Make the parent process arrange for a pipe connection to each child.  Handle
>>the communication back to the parent through a single FIFO.

>>There are at least 3 strategies that might be best in different situations:
>AAAAGGGHHH!!!!  Twelve years later and we're still screwing around with PIPES!
>NT deserves to win if we can't do any better than this......

Does that mean you are of the opinion that you should build everything
possible into the kernel instead of providing simple mechanisms that
let user programs do as much or as little as they need?

Les Mikesell

Quote:>>>I have a parent process that needs to pass messages of arbitrary length to
>>>and from multiple child processes.  The current implementation is using
>>>SysV message queues.

The only thing I would add to this discussion, is that _arbitrary_  (which I
take to mean of any length and possibly "infinite"/ _reeaal_ long) is
going to be an impossible requirement to fulfill.  There is _always_ going to
be some upper bound on the size of a message that can be handled by a
system (the theoretical biggest message = size of disk + size of core -
size of both the processes).  So basically, if you are dealing with a message
subsystem you always need to keep in mind the upper limit on the size of
the message(s) you can deal with.  Of course sometimes it isn't obvious that
your message might exceed the  subsystem's limit (PIPE_MAX or whatever).....

To kick a dead goat then, there is _always_ going to a limit on how big your
message can be. If I am pointing out the obvious then sue me, but when I came
to this realization one night it was a little painful.  Of course, things like
message subsystems is what keeps us software types in jobs.

Regards,
-cme

But once again you have the option of rolling your own boundaries using
a scheme that allows the chunks to be fragmented and re-assembled by
your application instead of putting that cruft in the kernel.  Or it
may turn out to be useful to write the huge items to disk files and
just pass the names through the pipes.  If you like your queues to
survive a reboot you have to write them to disk anyway.  Or if you want
speed you might blow things through a shared memory segment using the
pipes to pass around the offset and size of filled/empty spots.

Les Mikesell

What about the simple solution of specifying the pipe length
when it is created. In this case, you can sacrify more
memory when you need it. And for the cases in which you
don't have a preference and don't mind to use the default
size choosen by the OS vendor, you can use the constant

#define VENDOR_X_RECOMANDED_MAX_PIPE_SIZE  PIPE_MAX

This option of specifying the pipe size is avialable for example
on OS9 (but don't see it as an OS9 recomandation ;-)).

And regarding the message vs. stream debate, IMHO it would
message level handling standard in the OS (or the standard library
or whatever).

Tal

--

 =======================================================================

 =======================================================================

: I am trying to make a language processor POSIX.1-1990 compliant as part of
: my M.S. C.S. thesis project, and have hit a sticky spot.
:
: The only POSIX.1 mechanisms that are available to me are pipes, and FIFOs;
: I have written an implementation of binary semaphores based on POSIX.1
: advisory locks (thanks to a suggestion in W. Richard Stevens' book!).
:
: The type and length of each message is known before the writer has to write
: the message.
:
: The hard part seems to be making sure that the reader gets a complete message,
: and NO MORE than a single complete message.  If I could solve this, I could
: create a linked-list of received messages, and provide a function call to
: either fetch a message from the list, or block until another process writes
: a message.
:
: Any thoughts on how I should approach this?

you can of course write your own messaging protocol
<SOM>,length,Message
-- but as that is so obvious I guess are hoping that there are some
system call which you can use.

I have no idear if STREAM pipes are POSIX.1-1990,
however they would forfill your requirements in one of two ways.

EITHER using the getmsg, putmsg etc, function calls.

OR you could use ioctal() call and push "ptem" (pseudo-terminal hardware
emulation module) and "ldterm" standard terminal line discipline) onto the
pipe to create a terminal interface then if ICANON is set and "\n"
can be use as a record marker the terminal modules will give you the
data a "line" at a time.

Even if you can't use these mechanisms I think that a browse through
the manual pages on putmsg etc, may help you with your own design.

TTFN Philip