Very Computer

Hi, all:

I have a question about performance and implementation about File I/O vs.
Piping.

Right now I have a program of A which produces  output file A.output. I have
a program B reading A.output to do some postprocessing. So the flow looks like
this:

             A --> A.output --> B

Since A.output if very large ( ~100MB ), it takes a long time to write and
read the file. I am looking into the possibility to set up a pipe between A
and B to transfer the data. I guess I can use either socket or pipe to do
it. Anyone having the experience how much performance will I gain and which
way is easier to implement( socket or stream ) ?

Now suppose A is a lot faster than B, I think it will result in a lot of
data storing in an internal buffer. Will it cause some memory problem?

Thank you very much for your answer.

Haizhou Chen

Quote:>Now suppose A is a lot faster than B, I think it will result in a lot of
>data storing in an internal buffer. Will it cause some memory problem?

--

BBN Corporation, Cambridge, MA
Support the anti-spam movement; see <http://www.cauce.org/>

Hi, all:

I have a question about performance and implementation about File I/O vs.
Piping.

Right now I have a program of A which produces  output file A.output. I have
a program B reading A.output to do some postprocessing. So the flow looks
like
this:

             A --> A.output --> B

Since A.output if very large ( ~100MB ), it takes a long time to write and
read the file. I am looking into the possibility to set up a pipe between A
and B to transfer the data. I guess I can use either socket or pipe to do
it. Anyone having the experience how much performance will I gain and which
way is easier to implement( socket or stream ) ?

Now suppose A is a lot faster than B, I think it will result in a lot of
data storing in an internal buffer. Will it cause some memory problem?

Thank you very much for your answer.

Haizhou Chen

It sounds like your data-processing is, or can be, a multi-stage
process.  In this case, it is very usual in Unix programs to take your
input from stdin and send your output to stdout, for many reasons:-

1) You can always get output to file by doing
        programA > output-file-name

2) You can build up longer pipes
        programA | programB | programC | ...

3) You can have alternative processing
        programA | programB | programD | ...

Addtitionally it it very useful to have both the input and output
human-readable text.  The advantages are:-

1) You're more likely to be able to tell if it's wrong

2) You can generate test-cases by hand

3) You can process output with the usual Unix tools (awk, sed, grep,
more, perl, and so on).

        A | B

You can set up named pipes, or sockets, but this seems overkill for what
you are describing. It is likley that it will be quite a lot faster than
writing to a disk file as the pipe contents are usually held totally in
memory, but anything which produces 100MB of data is not going to be
very fast.

Quote:>Now suppose A is a lot faster than B, I think it will result in a lot of
>data storing in an internal buffer. Will it cause some memory problem?

Icarus

%Hi, all:

%I have a question about performance and implementation about File I/O vs.
%Piping.

%Right now I have a program of A which produces  output file A.output. I have
%a program B reading A.output to do some postprocessing. So the flow looks
%like
%this:

%             A --> A.output --> B

%Since A.output if very large ( ~100MB ), it takes a long time to write and
%read the file. I am looking into the possibility to set up a pipe between A
%and B to transfer the data. I guess I can use either socket or pipe to do
%it. Anyone having the experience how much performance will I gain and which
%way is easier to implement( socket or stream ) ?

All else being equal, a pipe ought to be faster than the file, since you
avoid doing actual disk I/O. On most BSD-derived systems, a pipe is simply
a Unix domain socket, so there will be no difference between pipe & socket.

%Now suppose A is a lot faster than B, I think it will result in a lot of
%data storing in an internal buffer. Will it cause some memory problem?

No, but it will cause A to slow down to match B's speed. The pipe will only
buffer so much data, after which it will block the writing process and
prevent it from continuing. When the pipe buffer gets empty enough the
writer will be allowed to resume. (This is assuming that you have not
issued an ioctl to use non-blocking mode.)
--
Howard Chu                              Principal Member of Technical Staff

Adverti*ts proof-read for US$100 per word. Submission of your ad to my
email address constitutes your acceptance of these terms.

Quote:> It sounds like your data-processing is, or can be, a multi-stage
> process.  In this case, it is very usual in Unix programs to take your
> input from stdin and send your output to stdout, for many reasons:-

  Typical scientific DATA processing involves programs operating on binary
  data files (somewhat large), and either altering them in place, or
  producing a separate output file. Sometimes extra files are used as
  "control files". Almost always stdin is used for control, and stdout
  for diagnostics and status info.

----------------------------------------------------------------------------
Lucio Chiappetti - IFCTR/CNR - via Bassini 15 - I-20133 Milano (Italy)      
----------------------------------------------------------------------------
Fuscim donca de Miragn        E tornem a sta scio' in Bregn                
Che i fachign e i cortesagn   Magl' insema no stagn begn                    
Drizza la', compa' Tapogn                            (Rabisch, II 41, 96-99)
----------------------------------------------------------------------------
For more info : http://www.ifctr.mi.cnr.it/~lucio/personal.html            
----------------------------------------------------------------------------

I have in mind a case study of a program which worked on external
files and took control information from stdin.

Unfortunately it was worked on by a succession of people whose primary
insterest was not the correctness of the program (but was instead the
physics of the end result), and it reached the stage where the program
required thirty megabytes of _control_ information as input for each
run of the simulation.  This input was itself crucial to the correct
operation of the program, but impossible to check.  Of course this an
extreme (but genuine) example.

For a (IMHO) better way of doing things, see Jon Bentley's column in
the June 1987 issue of "Communications of the ACM", which is also
collected in his book, ISBN 0-201-11889-0.