Capturing process output through pipe in a non-atomic fashion

Capturing process output through pipe in a non-atomic fashion

Post by Chris Cor » Mon, 19 May 2003 12:46:50



I'm definitely a Linux coding newbie but I haven't seen this addressed
in the FAQ so hopefully you guys can help me out.

I have a parent process that forks() off a child, directs the child's
stdout and stderr to a parent-owned pipe, and then execs() an
arbitrary program. Obviously this is the kind of thing that is done
all the time in Linux-land. :-)

Because I need to reserve the right to kill the child if it's around
too long, the parent needs to read() the pipe in a non-blocking
fashion. I choose to use poll() to help facilitate this but I suspect
that using fcntl() to put the pipes into non-blocking mode would have
worked too (is one way better than the other?). All of this works
fine.

The problem is when the output from the child process is intermittent.
What I'm currently seeing is that I can read nothing from the pipe
until the child completes. In my test case the amount of output is
very small (a K or so) and the child is writing it to stdout slowly,
say a line every few seconds. I really need my parent process to read
the lines as they are generated by the child, and not all in one bunch
at the end.

Any help or advice would be greatly appreciated. Thanks!

- Chris

 
 
 

Capturing process output through pipe in a non-atomic fashion

Post by Sybren Stuve » Mon, 19 May 2003 18:54:10


Chris Corry enlightened us with:

Quote:> The problem is when the output from the child process is intermittent.
> What I'm currently seeing is that I can read nothing from the pipe
> until the child completes. In my test case the amount of output is
> very small (a K or so) and the child is writing it to stdout slowly,
> say a line every few seconds. I really need my parent process to read
> the lines as they are generated by the child, and not all in one bunch
> at the end.

Are you sure you print out multiple *lines* from the child? Since stdout
is linebuffered, you'll need to output a '\n' after each burst of
output. Alternatively, there should be a way to make the pipe not
line-buffered - can anyone shed some light on this?

Sybren
--
| Somebody ought to cross ball point pens with coat hangers so that |
| the pens will multiply instead of disappear.                      |
\-------------------------------------------------------------------/
If you have to use Outlook, fix it: http://flash.to/oe-quotefix/

 
 
 

Capturing process output through pipe in a non-atomic fashion

Post by Russell Sha » Mon, 19 May 2003 14:16:41



> I'm definitely a Linux coding newbie...


 
 
 

Capturing process output through pipe in a non-atomic fashion

Post by Kevin Easto » Mon, 19 May 2003 19:23:44



> Chris Corry enlightened us with:
>> The problem is when the output from the child process is intermittent.
>> What I'm currently seeing is that I can read nothing from the pipe
>> until the child completes. In my test case the amount of output is
>> very small (a K or so) and the child is writing it to stdout slowly,
>> say a line every few seconds. I really need my parent process to read
>> the lines as they are generated by the child, and not all in one bunch
>> at the end.

> Are you sure you print out multiple *lines* from the child? Since stdout
> is linebuffered, you'll need to output a '\n' after each burst of
> output. Alternatively, there should be a way to make the pipe not
> line-buffered - can anyone shed some light on this?

It's not the pipe that's line-buffered, it's the stdout stream in the
child process.  And it's probably not line-buffered either - if libc
detects that stdout is connected to a pipe, it may change the
buffering mode to fully-buffered.

stdout can be made unbuffered by the following call:

#include <stdio.h>
setvbuf(stdout, NULL, _IONBF, 0);

...but this must be done in the code of the child process that is
exec()ed, because library constructs like stdio do not survive across
exec().

        - Kevin.

 
 
 

1. Capturing process output through pipe in a non-atomic fashion

I'm definitely a Unix coding newbie but I haven't seen this addressed
in any of the FAQs so hopefully you guys can help me out.

I have a parent process that forks() off a child, directs the child's
stdout and stderr to a parent-owned pipe, and then execs() an
arbitrary program -- a pretty common and mundane activity. :-)

Because I need to reserve the right to kill the child if it's around
too long, the parent needs to read() the pipe in a non-blocking
fashion. I choose to use poll() to help facilitate this but I suspect
that using fcntl() to put the pipes into non-blocking mode would have
worked too (is one way better than the other?). All of this works
fine.

The problem is when the output from the child process is intermittent.
What I'm currently seeing is that I can read nothing from the pipe
until the child completes. In my test case the amount of output is
very small (a K or so) and the child is writing it to stdout slowly,
say a line every few seconds (a make might be a good example). I
really need my parent process to read the lines as they are generated
by the child, and not all in one bunch at the end. Since the child can
be any arbitrary program, any solution that relies on modifying the
child doesn't work for me.

It should be noted that I had a version of this that used popen() and
worked perfectly. I had to take this more manual route once I realized
I needed to obtain the pid of the child process (in order to perform
timeout kill()s). (I really don't want to have to track down Linux
popen() source code to figure this out.)

Any help or advice would be greatly appreciated. Thanks!

- Chris

2. Terror in New York und Washington

3. remove mixture of non-atomic operations with page->flags which requires atomic operations to access

4. Solaris 2.5.1 to 2.6 Upgrade Slow? (AutoLayout phase)

5. How to pass control data to other process thru stream pipes

6. Looking for a good terminal program

7. Make objrmap mapcount non-atomic

8. Multi-cast problems

9. Move "used FPU status" into new non-atomic thread_info->status field.

10. non-atomic test victim #1

11. capture of data through a modem by a non-interactive process ??

12. Piping slow output to a process