fifosophical question ...

fifosophical question ...

Post by lijnz.. » Fri, 03 Sep 1993 02:16:50



Dear netters !

I am puzzled by fifo's, or named pipes. To paraphrase Magritte, cela n'est
pas une pipe, but then what is it? What can you use it for?  Is it a just a
neat trick, or sometimes indispensable? Could you use it for
eg. self-decompressing files, or something? Why do they always seem to have
length 0? Do these pipe-thingies reside on the disk, and can you share them
through NFS? comp.unix.questions-4 says something obscure about using a named
pipe as a .plan file, but that isn't quite clear. I am just asking out of
curiosity, just wasting bandwidth and time 8-).  And yes, i tried to rtfm,
but tfm does not explain a thing about fifos. Am i asking a faq ? or am i
asking too much ? ( 8 no that's not a smiley 8 question marks so far ...8-)
yes that was one. Should it be a faq ? (9) Anyone out there ? (10: i stop)
 Thanks,
                                                                Philip

 
 
 

fifosophical question ...

Post by damer.. » Sat, 04 Sep 1993 05:55:09


: Dear netters !

: I am puzzled by fifo's, or named pipes. To paraphrase Magritte, cela n'est
: pas une pipe, but then what is it? What can you use it for?  Is it a just a
: neat trick, or sometimes indispensable? Could you use it for
: eg. self-decompressing files, or something? Why do they always seem to have
: length 0? Do these pipe-thingies reside on the disk, and can you share them
: through NFS? comp.unix.questions-4 says something obscure about using a named
: pipe as a .plan file, but that isn't quite clear. I am just asking out of
: curiosity, just wasting bandwidth and time 8-).  And yes, i tried to rtfm,
: but tfm does not explain a thing about fifos. Am i asking a faq ? or am i
: asking too much ? ( 8 no that's not a smiley 8 question marks so far ...8-)
: yes that was one. Should it be a faq ? (9) Anyone out there ? (10: i stop)
:  Thanks,
:   Philip --

A FIFO is a pipe that acts like a file and a file that acts like a
pipe.  Like a file,  it has a name,  so that it can be used by
non-related processes for Inter Process Communications.  But it
acts like a pipe:

    1. It must be opened at both ends before it comes open;
    2. Reads and writes block until both ends are open;
    3. It blocks when full;

It is very useful for scheduling;  requests are written (an atomic
operation) to the open fifo,  and a second process reads the fifo to
get its work.  Lp uses it.

Fifos can be used with vi to provide advanced editing functionality.
For instance,  I use a program that supplies boilerplate based upon
a few characters and the language being used (an if in a shell script
expands to if [ test ];then ~;else ~;fi ,  while an if in a 'C'
program becomes if (!) { ~ } else { ~ } (supply your own newlines).

I developed a program that had drop down menus called from within
vi.

Dan Mercer                                            Applications + Plus

======================================================================
About a year ago I told the following joke which got a strong positive
reaction from only 1 in 5 (all males my age - 40+) and shrugs from the
rest.  I wonder if it is more relevant now?

"What does Mogadishu mean in Somali?"  Answer - "Saigon"

 
 
 

fifosophical question ...

Post by Jim Kiss » Sat, 04 Sep 1993 17:09:02


: Dear netters !

: I am puzzled by fifo's, or named pipes. To paraphrase Magritte, cela n'est
: pas une pipe, but then what is it?

A fifo is a First In First Out file (stack/object etc.)  You cannot seak
on it. You can write to it up to a limit of about 5K (may be more now
but this was the limit in Ver 7 & Sys III).  The data you write to it
can be read in the byte order that you wrote it.

: What can you use it for?

The named pipe was one of the first IPC (Inter Process Communication)
facilities available on Unix.  (I know Ver 7 had the multiplexed file
system but it wasn't very wide spread)  It pre-dates Shared memory and Queues

Consider

ls -lt|head

This pipes the output of the ls command into the head command.  The
pipe used is not a named pipe but behaves exactly like a named pipe
would.  You could get the same result with something like the following
commands.

mknod -p my_named_pipe

head -t < my_named_pipe&
ls -lt > my_named_pipe

The syntax may be incorrect but the ideas are exactly what the shell
does when you use the '|' command/pragma/object .....

: Is it a just a
: neat trick, or sometimes indispensable? Could you use it for
: eg. self-decompressing files, or something?

It depends if you write multi-process applications.  If you don't, you
don't need IPC. If you do write multi-process applications, pipes can
be used for communication between processed. Named pipes allow two
non-related (don't have the same parent) processes to have a rendezvous
point. (yes I know you can do it with sockets, but when I started
programming on Unix, local area networks didn't exist).  The first process
to open the pipe can attemp to write.  A failure indicates that no one
is ready to read.  You can use two pipes to provide bi-directional
communications.

: Why do they always seem to have
: length 0?

If you haven't written anything to them, there is not anything in them,
therefore length == 0.

: Do these pipe-thingies reside on the disk, and can you share them
: through NFS?

All pipe-thingies are part of the file system.  Named pipes have an
external/viewable mark on the file system.

No, the last time I looked, NFS did not support pipes.

: comp.unix.questions-4 says something obscure about using a named
: pipe as a .plan file, but that isn't quite clear. I am just asking out of
: curiosity, just wasting bandwidth and time 8-).  And yes, i tried to rtfm,
: but tfm does not explain a thing about fifos. Am i asking a faq ? or am i
: asking too much ? ( 8 no that's not a smiley 8 question marks so far ...8-)
: yes that was one. Should it be a faq ? (9) Anyone out there ? (10: i stop)
:  Thanks,
:                                                               Philip

Pipes are the oldest (I think) inter process communication technology in
the Unix operating system.  I first seen Named Pipes in Sys III c 1981.
There are other and in many cases better IPC methods available in BSD/
AIX/SYS V.4....... than named pipes.

-------------------------------------------------------------------------------
Jim Kissel                            Telephone +44 344 863 222
Siemens Nixdorf Information Systems                 344 850 461 (Direct line)
Systems Development Group             Fax       +44 344 850 452

Oldbury, Bracknell, Berkshire         UUCP       ....{ukc,athen}!sni!jlk
RG12 4FZ  Great Britain              

Have you noticed that no one talks about Pavlov's cat?
-------------------------------------------------------------------------------

 
 
 

fifosophical question ...

Post by Alvin W. L » Sun, 05 Sep 1993 04:51:58




>: Is it a just a
>: neat trick, or sometimes indispensable? Could you use it for
>: eg. self-decompressing files, or something?

>It depends if you write multi-process applications.  If you don't, you
>don't need IPC. If you do write multi-process applications, pipes can
>be used for communication between processed. Named pipes allow two
>non-related (don't have the same parent) processes to have a rendezvous
>point. (yes I know you can do it with sockets, but when I started
>programming on Unix, local area networks didn't exist).  The first process
>to open the pipe can attemp to write.  A failure indicates that no one
>is ready to read.  You can use two pipes to provide bi-directional
>communications.

There are programs out there which does not read from stdin/write to
stdout.  They expect a specific filename from the command line.  I
find named pipes extremely handy when the file I want to operate on
are hugh and I don't have diskspace to store the uncompressed version.
In that case, I'll simply:

Read from compressed file:

        % some_program file=my_named_pipe &
        % zcat some_file.Z > my_named_pipe

Write to compressed file:

        % compress -c my_named_pipe > some_other_file.Z &
        % some_other_program file=my_named_pipe
--
 Alvin W. Law .............................................. Oracle Corporation
 Senior Software Engineer ...................... 300 Oracle Parkway, Box 659306
 Manufacturing Applications .......................... Redwood Shores, CA 94065

 
 
 

fifosophical question ...

Post by Peter Stee » Sat, 11 Sep 1993 02:09:28



>A FIFO is a pipe that acts like a file and a file that acts like a
>pipe.  Like a file,  it has a name,  so that it can be used by
>non-related processes for Inter Process Communications.  But it
>acts like a pipe:
>    1. It must be opened at both ends before it comes open;
>    2. Reads and writes block until both ends are open;

What happens if two processes try to write to the same fifo
file when there is process in place to read from the pipe?
Both processes would go into a write-blocked state, but which
one gets to write into the fifo first when the processes
become unblocked?

Quote:>    3. It blocks when full;

When is a fifo considered to be full?

Quote:>Fifos can be used with vi to provide advanced editing functionality.
>For instance,  I use a program that supplies boilerplate based upon
>a few characters and the language being used (an if in a shell script
>expands to if [ test ];then ~;else ~;fi ,  while an if in a 'C'
>program becomes if (!) { ~ } else { ~ } (supply your own newlines).
>I developed a program that had drop down menus called from within
>vi.

Are you saying you developed a program that provides vi with
drop menus? That would be something I'd like to see...
--

Acadia Univ., Wolfville, NS, Canada B0P 1X0  902-542-2201  Fax: 902-542-4364
 
 
 

fifosophical question ...

Post by damer.. » Sat, 11 Sep 1993 07:47:12


: >A FIFO is a pipe that acts like a file and a file that acts like a
: >pipe.  Like a file,  it has a name,  so that it can be used by
: >non-related processes for Inter Process Communications.  But it
: >acts like a pipe:
:  
: >    1. It must be opened at both ends before it comes open;
: >    2. Reads and writes block until both ends are open;
:  
: What happens if two processes try to write to the same fifo
: file when there is process in place to read from the pipe?
: Both processes would go into a write-blocked state, but which
: one gets to write into the fifo first when the processes
: become unblocked?

Whoever was first on the queue.  And his write will be fulfilled, even
if it blocks the queue,  guaranteeing the atomicity of the write.
For instance, if the first write was 10K,  and the PIPSIZ = 8K,  the
firts 8K would be written,  the pipe would block until read,  then the
next 2K would be written,  then the other process would become
unblocked.  It is important,  when multiple processes write to the
same pipe,  that some form of protocol be used,  even if it is only
line blocking (which can be very effective).

:  
: >    3. It blocks when full;
:  
: When is a fifo considered to be full?

From man 2 write:

     If the file being written is a pipe (or FIFO), the system-dependent
     maximum number of bytes that it can store is given by PIPSIZ (defined
     in <sys/inode.h>).  The minimum value of PIPSIZ on any HP-UX system is
     8192.  When writing a pipe, the following conditions apply:

          If the O_NDELAY or O_NONBLOCK file status flag is set:

               If nbyte is less than or equal to PIPSIZ and sufficient room
               exists in the pipe or FIFO, the write() succeeds and returns
               the number of bytes written;

               If nbyte is less than or equal to PIPSIZ but insufficient
               room exists in the pipe or FIFO, the write() returns having
               written nothing.  If O_NONBLOCK is set, -1 is returned and
               errno is set to EAGAIN.  If O_NDELAY is set, 0 is returned.

               If nbyte is greater than PIPSIZ and the pipe or FIFO is
               full, the write returns having written nothing.  If
               O_NONBLOCK is set, -1 is returned and errno is set to
               EAGAIN.  If O_NDELAY is set, 0 is returned.

               If nbyte is greater than PIPSIZ, and some room exists in the
               pipe or FIFO, as much data as fits in the pipe or FIFO is
               written, and write() returns the number of bytes actually
               written, an amount less than the number of bytes requested.

:  
: >Fifos can be used with vi to provide advanced editing functionality.
: >For instance,  I use a program that supplies boilerplate based upon
: >a few characters and the language being used (an if in a shell script
: >expands to if [ test ];then ~;else ~;fi ,  while an if in a 'C'
: >program becomes if (!) { ~ } else { ~ } (supply your own newlines).
:  
: >I developed a program that had drop down menus called from within
: >vi.
:  
: Are you saying you developed a program that provides vi with
: drop menus? That would be something I'd like to see...
: --

: Acadia Univ., Wolfville, NS, Canada B0P 1X0  902-542-2201  Fax: 902-542-4364

Vi was only used as an editor within the program.  The menus could be
used to browse data structures in a coredump and output text in a more
readable fashion.  The user would hit a function key to get the menus
which would cause vi to write to an input pipe and then block on a
:r infifo

When the user was done prowling the menus and had selected his
function,  the background process would write to info and,  voila,  it
wound up back in the editor for browsing.  Their he user could mangle
the returned data into other commands also accessible through fifos.

Unfortunately,  the code isn't mine to share and I doubt if it's even
being used anymore.  The business unit it was developed for having
fallen on hard times.

--
Dan Mercer                                            Applications + Plus

======================================================================
About a year ago I told the following joke which got a strong positive
reaction from only 1 in 5 (all males my age - 40+) and shrugs from the
rest.  I wonder if it is more relevant now?

"What does Mogadishu mean in Somali?"  Answer - "Saigon"

 
 
 

fifosophical question ...

Post by Christoph Badu » Mon, 13 Sep 1993 09:06:40



>For instance, if the first write was 10K,  and the PIPSIZ = 8K,  the
>firts 8K would be written,  the pipe would block until read,  then the
>next 2K would be written,  then the other process would become
>unblocked.

This isn't guaranteed.  In fact, it doesn't work as you describe on
most Unix Versions.
--

Personally, I don't care whether someone is cool enough to quote Doug
Gwyn--I only care whether Doug Gwyn is cool enough to quote. -- Larry Wall

 
 
 

fifosophical question ...

Post by damer.. » Wed, 15 Sep 1993 01:02:00



: >For instance, if the first write was 10K,  and the PIPSIZ = 8K,  the
: >firts 8K would be written,  the pipe would block until read,  then the
: >next 2K would be written,  then the other process would become
: >unblocked.

: This isn't guaranteed.  In fact, it doesn't work as you describe on
: most Unix Versions.
: --

: Personally, I don't care whether someone is cool enough to quote Doug
: Gwyn--I only care whether Doug Gwyn is cool enough to quote. -- Larry Wall
I was describing an ordinary write,  of course.  If you are suggesting
that ordinary writes can be interleaved,  that would violate the
atomicity of the write.  Different rules do exist if O_NDELAY or
O_NONBLOCK file status flag is set.  To excerpt from HP-UX man pages

write (2)...

      If the file being written is a pipe (or FIFO), the system-dependent
      maximum number of bytes that it can store is given by PIPSIZ (defined
      in <sys/inode.h>).  The minimum value of PIPSIZ on any HP-UX system is
      8192.  When writing a pipe, the following conditions apply:

           If the O_NDELAY or O_NONBLOCK file status flag is set:

                If nbyte is less than or equal to PIPSIZ and sufficient room
                exists in the pipe or FIFO, the write() succeeds and returns
                the number of bytes written;

                If nbyte is less than or equal to PIPSIZ but insufficient
                room exists in the pipe or FIFO, the write() returns having
                written nothing.  If O_NONBLOCK is set, -1 is returned and
                errno is set to EAGAIN.  If O_NDELAY is set, 0 is returned.

                If nbyte is greater than PIPSIZ and the pipe or FIFO is
                full, the write returns having written nothing.  If
                O_NONBLOCK is set, -1 is returned and errno is set to
                EAGAIN.  If O_NDELAY is set, 0 is returned.

                If nbyte is greater than PIPSIZ, and some room exists in the
                pipe or FIFO, as much data as fits in the pipe or FIFO is
                written, and write() returns the number of bytes actually
                written, an amount less than the number of bytes requested.

           If the O_NDELAY and O_NONBLOCK file status flags are clear:

                The write() always executes correctly (blocking as
                necessary), and returns the number of bytes written.

I believe this i sthe same way it ran on the NCR SYS V systems I ran
on in my previous job.  We considered using FIFO's in our current
project as an IPC mechanism (used sockets instead) and ran extensive
tests (mega-megabytes under all load conditions) to make sure the
software agreed with the manpage (not always guaranteed :-) ).  We
were satisfied that no interleaving could occur.

--
Dan Mercer                                            Applications + Plus

======================================================================
About a year ago I told the following joke which got a strong positive
reaction from only 1 in 5 (all males my age - 40+) and shrugs from the
rest.  I wonder if it is more relevant now?

"What does Mogadishu mean in Somali?"  Answer - "Saigon"

 
 
 

fifosophical question ...

Post by Alan Wats » Wed, 15 Sep 1993 10:00:57





>>>For instance, if the first write was 10K,  and the PIPSIZ = 8K,  the
>>>firts 8K would be written,  the pipe would block until read,  then the
>>>next 2K would be written,  then the other process would become
>>>unblocked.

>>This isn't guaranteed.  In fact, it doesn't work as you describe on
>>most Unix Versions.

>I was describing an ordinary write,  of course.  If you are suggesting
>that ordinary writes can be interleaved,  that would violate the
>atomicity of the write.  Different rules do exist if O_NDELAY or
>O_NONBLOCK file status flag is set.  To excerpt from HP-UX man pages
>[...]

Welcome to the reality of a million and one versions of UNIX.  To
excerpt from the OSF/1 man pages (although it is the same on Ultrix and
also, from my reading, SunOS):

   Write requests to a pipe (or FIFO) are handled the same as a regular
   file with the following exceptions:

   [...]

   +  If the size of the write() request is less than or equal to the
      value of the PIPE_BUF system variable, the write() function is
      guaranteed to be atomic. The data is not interleaved with data
      from other processes doing writes on the same pipe.  Writes of
      greater than PIPE_BUF bytes can have data interleaved, on
      arbitrary boundaries, with writes by other processes, whether or
      not O_NONBLOCK or O_NDELAY are set.

If you need (portable) atomic operations, don't use FIFOs and don't use
NFS.

 
 
 

fifosophical question ...

Post by damer.. » Fri, 17 Sep 1993 07:37:14





: >>>For instance, if the first write was 10K,  and the PIPSIZ = 8K,  the
: >>>firts 8K would be written,  the pipe would block until read,  then the
: >>>next 2K would be written,  then the other process would become
: >>>unblocked.
: >
: >>This isn't guaranteed.  In fact, it doesn't work as you describe on
: >>most Unix Versions.
: >
: >I was describing an ordinary write,  of course.  If you are suggesting
: >that ordinary writes can be interleaved,  that would violate the
: >atomicity of the write.  Different rules do exist if O_NDELAY or
: >O_NONBLOCK file status flag is set.  To excerpt from HP-UX man pages
: >[...]

: Welcome to the reality of a million and one versions of UNIX.  To
: excerpt from the OSF/1 man pages (although it is the same on Ultrix and
: also, from my reading, SunOS):

:    Write requests to a pipe (or FIFO) are handled the same as a regular
:    file with the following exceptions:

:    [...]

:    +  If the size of the write() request is less than or equal to the
:       value of the PIPE_BUF system variable, the write() function is
:       guaranteed to be atomic. The data is not interleaved with data
:       from other processes doing writes on the same pipe.  Writes of
:       greater than PIPE_BUF bytes can have data interleaved, on
:       arbitrary boundaries, with writes by other processes, whether or
:       not O_NONBLOCK or O_NDELAY are set.

: If you need (portable) atomic operations, don't use FIFOs and don't use
: NFS.

Ugh!!!!  Must have been the influence of IBM ;-)
--
Dan Mercer                                            Applications + Plus

======================================================================
About a year ago I told the following joke which got a strong positive
reaction from only 1 in 5 (all males my age - 40+) and shrugs from the
rest.  I wonder if it is more relevant now?

"What does Mogadishu mean in Somali?"  Answer - "Saigon"

 
 
 

fifosophical question ...

Post by Christoph Badu » Fri, 17 Sep 1993 20:03:09





>: >For instance, if the first write was 10K,  and the PIPSIZ = 8K,  the
>: >firts 8K would be written,  the pipe would block until read,  then the
>: >next 2K would be written,  then the other process would become
>: >unblocked.
>: This isn't guaranteed.  In fact, it doesn't work as you describe on
>: most Unix Versions.
>I was describing an ordinary write,  of course.

I know.  I was responding to precisely that case.

Quote:>  If you are suggesting
>that ordinary writes can be interleaved,  that would violate the
>atomicity of the write.

The atomicity of writes to pipes was never guaranteed for arbitrarily
large writes.  It's usually only guaranteed for writes of up to PIPSIZ
bytes, but you won't find that explicitly stated in most man pages.
POSIX requires writes of up to PIPE_BUF[*] bytes to be atomic.

The relevant paragraph of you qoute form the write(2) man page is:

Quote:>           If the O_NDELAY and O_NONBLOCK file status flags are clear:
>                The write() always executes correctly (blocking as
>                necessary), and returns the number of bytes written.

Note that this doesn't explicitly guarantee atomicity.  It just say
that write() will write all the data it is told to write.  And the bit
about "blocking as necessary" is a strong hint that the write is not
atomic.  (Blocking translates to sleeping in the kernel and if more
than one process is sleeping on the same channel all of them will be
awakened and the order in which they are run is random.)

Quote:>I believe this i sthe same way it ran on the NCR SYS V systems I ran
>on in my previous job.

That's right.

Quote:>  We considered using FIFO's in our current
>project as an IPC mechanism (used sockets instead) and ran extensive
>tests (mega-megabytes under all load conditions) to make sure the
>software agreed with the manpage (not always guaranteed :-) ).  We
>were satisfied that no interleaving could occur.

The you were very lucky.  To make writes of aribtrary sizes to pipes
atomic would require a major redesign of the relevant kernel modules
in System V.

[*]  I can see how patchconf() can return the value "indetermined" for
_PC_PATH_BUF, which would give you atomic write to pipes and FIFOs for
arbitrarily large writes.
--

Personally, I don't care whether someone is cool enough to quote Doug
Gwyn--I only care whether Doug Gwyn is cool enough to quote. -- Larry Wall

 
 
 

1. fifosophical question ...

Posted this one before to comp.unix.shells, but  .questions seems more
appropriate ...

Dear netters !

I am puzzled by fifo's, or named pipes. To paraphrase Magritte, cela n'est
pas une pipe, but then what is it? What can you use it for?  Is it a just a
neat trick, or sometimes indispensable? Could you use it for
eg. self-decompressing files, or something? Why do they always seem to have
length 0? Do these pipe-thingies reside on the disk, and can you share them
through NFS? comp.unix.questions-4 says something obscure about using a named
pipe as a .plan file, but that isn't quite clear. I am just asking out of
curiosity, just wasting bandwidth and time 8-).  And yes, i tried to rtfm,
but tfm does not explain a thing about fifos. Am i asking a faq ? or am i
asking too much ? ( 8 no that's not a smiley 8 question marks so far ...8-)
yes that was one. Should it be a faq ? (9) Anyone out there ? (10: i stop)
 Thanks,
                                                                Philip

2. OpenVPN Problems PLZ Help !!! TIA

3. Mini-Linux Distribution Questions + General Questions + UMSDOS Questions

4. Problems with a NIC, I think.

5. Questions, questions questions

6. Win98 not connecting to RH 5.2 Samba

7. 3 issues --- 1-Gnome Question, 2-CDRom Detect Question, 3-IMLIB Question

8. Printing from HPs to Suns

9. Questions Questions Questions ???

10. Questions, Questions, and more Questions..

11. 3 issues --- 1-Gnome Question, 2-CDRom Detect Question, 3-IMLIB Question

12. Newbie Questions (ICEWM Background, ppp question, mem > 64mb)

13. sar Question (probably a newbie question)