How to elminate pipe buffering?

How to elminate pipe buffering?

Post by Jan Pospis » Tue, 16 Jan 1996 04:00:00



Hi,

My problem:
I have two processes pSender and pReceiver coupled by a pipe.
  pSender | pReceiver
Any time pSender is killed anyway, pReceiver looses about
4 kBytes of data from pSender (if pSender writes to stdout the
data is visible). So I guess it is lost somewhere in the internal
pipe buffer(?).
Is there a kernel parameter or a shell option or any other solution
to eliminate the pipe buffering?. I'm running HP-UX 9.07 and ksh.
Please help. Thanx.
  Jan
P.S. Only the source code of pReceiver is available.
--

+------------------------------------------------------------------------+
|                                                                        |

| company:    Siemens AG            www:     [soon]                      |
| department: ANL TD E 44           tel:     +49 911 895 4928            |
|                                                                        |
+-[ Only those who dream can fly. ]--------------------------------------+

 
 
 

How to elminate pipe buffering?

Post by Stephen Bayn » Fri, 19 Jan 1996 04:00:00


: Hi,

: My problem:
: I have two processes pSender and pReceiver coupled by a pipe.
:   pSender | pReceiver
: Any time pSender is killed anyway, pReceiver looses about
: 4 kBytes of data from pSender (if pSender writes to stdout the
: data is visible). So I guess it is lost somewhere in the internal
: pipe buffer(?).
: Is there a kernel parameter or a shell option or any other solution
: to eliminate the pipe buffering?. I'm running HP-UX 9.07 and ksh.
: Please help. Thanx.
:   Jan
: P.S. Only the source code of pReceiver is available.
: --

If psender is using the C io library to write its output then you are
likly to be loosing the data in the libraries buffering rather than the
pipe. (Writing to stdout is buffered by default, if the output is a terminal
it is line buffered, otherwise it is block buffered.) You could try
replacing the pipe by a pseudo tty to trick psender into thinking it has
a terminal as output and using less buffering.

--

Philips Semiconductors Ltd
Southampton                                 My views are my own.
United Kingdom

 
 
 

How to elminate pipe buffering?

Post by Tom La » Fri, 19 Jan 1996 04:00:00



> I have two processes pSender and pReceiver coupled by a pipe.
>   pSender | pReceiver
> Any time pSender is killed anyway, pReceiver looses about
> 4 kBytes of data from pSender (if pSender writes to stdout the
> data is visible). So I guess it is lost somewhere in the internal
> pipe buffer(?).

The data's probably waiting in pSender's internal stdio buffer.  Doing
fflush() on the file would force the data out to the kernel, whereupon it
would be available to pReceiver.  Changing the stdio buffering mode
to line-buffered or non-buffered would let this happen automatically.
But by default, stdio will buffer 8K or so at a time, in order to
avoid excessive kernel calls.

Quote:> P.S. Only the source code of pReceiver is available.

In that case I'm afraid you're screwed.  You can't solve this problem
without changing the internal behavior of the pSender process; it's not
the kernel's fault, but rather a question of how the stdio library
behaves inside pSender.

                        regards, tom lane

 
 
 

How to elminate pipe buffering?

Post by Barry Margol » Sun, 21 Jan 1996 04:00:00




>You could try
>replacing the pipe by a pseudo tty to trick psender into thinking it has
>a terminal as output and using less buffering.

I believe there is a utility command called "pty" in the comp.sources.unix
archive that can be used for this.  You would write

pty pSender | pReceiver
--
Barry Margolin
BBN PlaNET Corporation, Cambridge, MA

Phone (617) 873-3126 - Fax (617) 873-6351

 
 
 

How to elminate pipe buffering?

Post by Neal P. Murp » Tue, 23 Jan 1996 04:00:00




>>Hi,
>>My problem:
>>I have two processes pSender and pReceiver coupled by a pipe.
>>  pSender | pReceiver
>>Any time pSender is killed anyway, pReceiver looses about
>>4 kBytes of data from pSender (if pSender writes to stdout the
>>data is visible). So I guess it is lost somewhere in the internal
>>pipe buffer(?).
>>Is there a kernel parameter or a shell option or any other solution
>>to eliminate the pipe buffering?. I'm running HP-UX 9.07 and ksh.
>>Please help. Thanx.
>>  Jan
>>P.S. Only the source code of pReceiver is available.
>>--
>>+------------------------------------------------------------------------+
>>|                                                                        |

>>| company:    Siemens AG            www:     [soon]                      |
>>| department: ANL TD E 44           tel:     +49 911 895 4928            |
>>|                                                                        |
>>+-[ Only those who dream can fly. ]--------------------------------------+
>You could use setbuf() to unbuffer the stream.  I presume you are
>using a file pointer to reference the pipe.  This has several
>restrictions as documented in the man pages for printf etc where they
>cannot be used.
>A better alternative would be to use fflush(fp) after each output
>operation in pSender.

I believe the problem lies in the fact that around 4K can be buffered in the
pipe itself, i.e., between the sender and the receiver.

Perhaps an inelegant solution might be to have the pReceiver be a child of
another process that *also* has the pipe open, but doesn't read from it until
the child dies. Thus, after the child dies, any data still in the pipe would
be shipped *back* to the sender. Of course, this isn't fool-proof; the only
way to do that would be to rewrite the receiver to acknowlege data received.

Fest3er
--
The Microsoft thirty bladed swiss army knife with corkscrew and magnifying lens
[is] inappropriate for all of my word processing needs.

  - Stephanie Evans

 
 
 

How to elminate pipe buffering?

Post by Ron McOu » Tue, 23 Jan 1996 04:00:00



>Hi,
>My problem:
>I have two processes pSender and pReceiver coupled by a pipe.
>  pSender | pReceiver
>Any time pSender is killed anyway, pReceiver looses about
>4 kBytes of data from pSender (if pSender writes to stdout the
>data is visible). So I guess it is lost somewhere in the internal
>pipe buffer(?).
>Is there a kernel parameter or a shell option or any other solution
>to eliminate the pipe buffering?. I'm running HP-UX 9.07 and ksh.
>Please help. Thanx.
>  Jan
>P.S. Only the source code of pReceiver is available.
>--
>+------------------------------------------------------------------------+
>|                                                                        |

>| company:    Siemens AG            www:     [soon]                      |
>| department: ANL TD E 44           tel:     +49 911 895 4928            |
>|                                                                        |
>+-[ Only those who dream can fly. ]--------------------------------------+

You could use setbuf() to unbuffer the stream.  I presume you are
using a file pointer to reference the pipe.  This has several
restrictions as documented in the man pages for printf etc where they
cannot be used.

A better alternative would be to use fflush(fp) after each output
operation in pSender.

 
 
 

How to elminate pipe buffering?

Post by Ron McOu » Wed, 24 Jan 1996 04:00:00



>>  Jan
>>P.S. Only the source code of pReceiver is available.

Sorry I missed this part of the info on the buffering problem.
Quote:>You could use setbuf() to unbuffer the stream.  I presume you are
>using a file pointer to reference the pipe.  This has several
>restrictions as documented in the man pages for printf etc where they
>cannot be used.
>A better alternative would be to use fflush(fp) after each output
>operation in pSender.

 
 
 

How to elminate pipe buffering?

Post by Jimmy Aitk » Tue, 12 Mar 1996 04:00:00


        [ wants to have unbuffered input on prog2 when
          prog1 | prog2 is run ]

Quote:>As I recall there is no solution if you don't have the source to
>prog1.  The pipe isn't producing the buffering directly, chances are
>the streamed stdio library calls in prog1 are (printf, puts etc.).
>The choice of output device detected by the library will affect the
>streaming.  If a tty type device is sensed then the output will be
>line buffered ("\n" will cause a flush).  Connecting to a pipe is
>detected as not a tty.

I meant to post my reply, but just mailed it off.  This can be done
without any change to the code is you use 'expect'.  There's an
example of exactly this is Don Libe's book 'Exporing Expect'.  The
code is: (page 299)
#!/usr/local/bin/expect
eval spawn -noecho $argv
set timeout -1
expect

You would use this as:
unbuffer prog1 | prog2

To say nothing of the rest of the uses to which the excellent 'expect'
program can be put.

Cheers,
Jimmy
--


 __                           Key ID is: EEAB8101
 \/ http://www.peter.com
`Rrrrum Ti-Tum Ti-Tum Ti-Tum, Rrrrum Ti-Tum Ti Ta-Ta,
 Rrrrum Ti-Tum Ti-Tum Ti-Tum, Rrrrum Ti Tiddly Ta...'

 
 
 

How to elminate pipe buffering?

Post by Wolfgang Den » Wed, 13 Mar 1996 04:00:00




>>I have a similar problem, but I haven't read all messages of this
>>thread.
>>Could someone mail me the original post with its answer?
>>Anyway, my problem is the following:
>>I type at the command line :   prog1 | prog2
>>where prog1 is a program of which I have not the source.
>>I need to processing the output of prog1 as fast as possible, so the
>>buffering introduced by pipe is unacceptable for me.
>>Is there any solution?
>As I recall there is no solution if you don't have the source to
>prog1.  The pipe isn't producing the buffering directly, chances are

There is a solution, and it's name is "expect".

Have a look in the examples that come with it,  there  is  one  named
"unbuffer" - guess what it does.

Since it's so short - here it comes:

        # Description: unbuffer stdout of a program
        # Author: Don Libes, NIST

         eval spawn -noecho $argv
         set timeout -1
         expect

Wolfgang Denk



The important thing about being a leader is not being right or wrong,
but being *certain*.                    - Terry Pratchett, _Truckers_

 
 
 

How to elminate pipe buffering?

Post by Ron McOu » Wed, 20 Mar 1996 04:00:00



>I meant to post my reply, but just mailed it off.  This can be done
>without any change to the code is you use 'expect'.  There's an
>example of exactly this is Don Libe's book 'Exporing Expect'.  The
>code is: (page 299)
>#!/usr/local/bin/expect
>eval spawn -noecho $argv
>set timeout -1
>expect
>You would use this as:
>unbuffer prog1 | prog2
>To say nothing of the rest of the uses to which the excellent 'expect'
>program can be put.
>Cheers,
>Jimmy
>--


> __                           Key ID is: EEAB8101
> \/ http://www.peter.com
>`Rrrrum Ti-Tum Ti-Tum Ti-Tum, Rrrrum Ti-Tum Ti Ta-Ta,
> Rrrrum Ti-Tum Ti-Tum Ti-Tum, Rrrrum Ti Tiddly Ta...'

Thanks - I have never used expect but I should look into it.
 
 
 

1. pipe/buffer-command for buffering large amounts of I/O


I don't know whether this is just what you want, but maybe you could try:

        cp /dev/null FILE
        tail -f FILE | command_2 &
        command1  >> FILE

You did say that storing the data on the disk was acceptable.

I'm sending followups out of comp.lang.perl.misc.
--


2. monitor problems

3. Sound Dropouts

4. dump pipe gzip pipe ssh pipe dd... blocksize?

5. Unable to allocate semaphores

6. Elminating Nonreentrant Library Calls in Multithreaded Programs

7. Looking forward to Apple's MacOX X

8. Buffers vs child pipe to stdout

9. Limited buffer size using pipes?

10. Pipes & buffering?...

11. FIFO pipe buffer questions

12. pipe buffer size