bash redirection stdout and stderr - pipeline command causes subshell execution

bash redirection stdout and stderr - pipeline command causes subshell execution

Post by S. Anthony Sequeir » Sun, 04 Sep 2005 07:05:50



Hi,

Hope I piqued your interest with the subject.
I will try to be clear.

I have a function which executes several commands.

It is called thus:

{ function 3>&1 1>&2 2>&3 | tee function.err } &> function.log

This works fine, inasmuch it produces a logfile with just the errors, and
another with both stdout and stderr interleaved, for further investigation
of anything in the error log.

The problem is when one or more of the commands sets a variable which should
be available further down the script, or when a cd is done within the
function, again required later.

My research (man pages and google) indicate that a pipe symbol causes
commands to be executed in a subshell.  This causes any environmental
alterations in the function not to be propagated to the rest of the script.

I have been attempting to find a construct that will duplicate the logging
facility, without operating in a subshell, in vain.  I can place any such
commands outside the function call, but this is not ideal, as I lose the
logging.

Any ideas please?

Cheers.
--
Tony

 
 
 

bash redirection stdout and stderr - pipeline command causes subshell execution

Post by en.. » Sun, 04 Sep 2005 17:09:38


If you know in advance what pipes you need, you can have the entire
script run inside a pair of braces (or anything equivalent), and apply
a bunch of redirections to the construct, like this:

exec 4>&1 5>&1    # save the original stdout and stderr
{
  {
    exec 6>&1 7>&2  # Save the pipes 6->file1&2 and 7->file1
    {
      {
        echo "This goes to file3"
        echo "This goes to file3 and file4" >&2

        # swap 1&2 and 4&5
        exec 3>&1 1>&4 4>&3 3>&2 2>&5 5>&3

        echo "This goes to orig-stdout"
        echo "This goes to orig-stderr" >&2

        command1
        command2 args >&4 2>&5  # stdout to file3 and file4,
                                # stderr to file4.
        command3      >&6 2>&7  # stdout to file1 and file2,
                                # stderr to file2.
        command4
      } 3>&2 2>&1 1>&3 | tee file4
    } &> file3
  } 3>&2 2>&1 1>&3 | tee file2

Quote:} &> file1

However, if the need to open files arises dynamically, I do not know.
Oh, yes, you can probably do something like this.
...
let n++
mknod /tmp/mypipe$n-totee p
exec           9>proc$n.log
tee proc$n.err </tmp/mypipe$n-totee >&9
func           >&9 2>/tmp/tmp/mypipe$n-totee

Provided your platform supports named pipes.
I guess you will have to experiment a little (or read op a lot :) ) to
determine if the tee process terminates when the func closes its files.

I hope this helps
Enrique

 
 
 

bash redirection stdout and stderr - pipeline command causes subshell execution

Post by S. Anthony Sequeir » Sun, 04 Sep 2005 17:24:04



> If you know in advance what pipes you need, you can have the entire
> script run inside a pair of braces (or anything equivalent), and apply
> a bunch of redirections to the construct, like this:

Enrique,

Many thanks for your answer, allow me to read, digest and test, I will reply
then.

Regards.
--
Tony

 
 
 

bash redirection stdout and stderr - pipeline command causes subshell execution

Post by S. Anthony Sequeir » Sun, 04 Sep 2005 18:31:10



> If you know in advance what pipes you need, you can have the entire
> script run inside a pair of braces (or anything equivalent), and apply
> a bunch of redirections to the construct, like this:

Hi again,

I have played with your script, but I cannot find a way to adapt it to my
situation.  I'm probably not thinking straight.  I'm still musing over the
named pipes though, maybe there is some way to adapt them.

I'll try and make things clearer.

I have a generic script, which assists me in building software packages.

It goes like this (roughly, there is more to it):

Fprepare() {
  command 1 &&
  command 2 &&
  cd whereever &&
  VAR='required all thru the script'

Quote:}

Fconfigure() {
  configure &&
  VAR2='required all thru the script'

Quote:}

Fmake() {
  make

Quote:}

Finstall() {
  make install

Quote:}

{ Fprepare 3>&1 1>&2 2>&3 | tee Fprepare.err } &> Fprepare.log
# error checking

{ Fconfigure 3>&1 1>&2 2>&3 | tee Fconfigure.err } &> Fconfigure.log
# error checking

{ Fmake 3>&1 1>&2 2>&3 | tee Fmake.err } &> Fmake.log
# error checking

{ Finstall 3>&1 1>&2 2>&3 | tee Finstall.err } &> Finstall.log
# error checking

What I require is, any environmental alteration within the functions, should
be available to the other functions, whilst keeping the separate log and
error files.

I'm not sure it is achievable without placing the cds and the variable
setting commands outside the functions, in which case logging is lost.
--
Tony

 
 
 

bash redirection stdout and stderr - pipeline command causes subshell execution

Post by Netocra » Mon, 05 Sep 2005 12:18:17



> Hi,

> Hope I piqued your interest with the subject.

You did.  I've just become aware of this issue myself.

Quote:> I will try to be clear.

> I have a function which executes several commands.

> It is called thus:

> { function 3>&1 1>&2 2>&3 | tee function.err } &> function.log

> This works fine, inasmuch it produces a logfile with just the errors,
> and another with both stdout and stderr interleaved, for further
> investigation of anything in the error log.

> The problem is when one or more of the commands sets a variable which
> should be available further down the script, or when a cd is done within
> the function, again required later.

> My research (man pages and google) indicate that a pipe symbol causes
> commands to be executed in a subshell.  This causes any environmental
> alterations in the function not to be propagated to the rest of the
> script.

Bash's manpage documents this behaviour for "simple command[s] other than
a builtin or shell function" and also "[c]ommand substitution, commands
grouped with parentheses, and asynchronous commands ...  Builtin commands
that are invoked as part of a pipeline are also executed in a subshell
environment."

This is pretty similar to the POSIX shell spec at
http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.ht...
which is not surprising given that bash is a POSIX shell.

However the POSIX spec adds:
"Additionally, each command of a multi-command pipeline is in a subshell
environment; as an extension, however, any or all commands in a pipeline
may be executed in the current environment."

A little ambiguous for a specification... and yes, bash chooses not to
offer this extension as you have discovered.

This leads to undesirable inconsistency.  The fish shell deals with this
issue by never forking a new shell instance, thus ensuring consistency and
slightly better (but probably unnoticably) execution times.  If you want
a distinct subshell environment in fish, you must explicitly run a new
shell interpreter.

You might find fish useful in your situation, with the caveat that
redirecting fds other than 0, 1, 2 is somewhat broken in the current
release.  Also redirection currently works at pipeline level rather than
individual process level, so you'd have to wrap the function call
redirections in another block, something like:
{ { function 3>&1 1>&2 2>&3 } | tee function.err } &> function.log
except that the fish syntax is slightly different - it doesn't support
braces so you'd have to use an "if true" block.  There's discussion of
adding a block operator.

Even though written after the standard, fish deviates from POSIX in many
other ways, largely to improve and simplify shell syntax from the author's
point of view; his philosophy is that for portable POSIX scripting bash is
available and therefore fish is free to improve on POSIX's syntax.  Its
focus is on user-friendliness and interactive use and it contains some
neat ideas with development ongoing.  One useful addition to fish is
variable scoping; scope is local by default.

Quote:> I have been attempting to find a construct that will duplicate the
> logging facility, without operating in a subshell, in vain.  I can place
> any such commands outside the function call, but this is not ideal, as I
> lose the logging.

You could pass the logging filenames into the function and perform the
redirections and tee on each command within the function, but that would
be pretty inconvenient.  Perhaps other shells (ksh, zsh) have chosen not
to fork in this situation and you could use one of them instead, but I
suspect that they would do the same.

Quote:> Any ideas please?

Well you needn't perform the environment-changing commands within the
function; in your later post you say that logging would be lost in that
case, but what logging do you want to perform on changing a directory or
setting a variable?  And if you do need to log such things, what's
stopping you?

Quote:> Cheers.

--
http://members.dodo.com.au/~netocrat
 
 
 

bash redirection stdout and stderr - pipeline command causes subshell execution

Post by S. Anthony Sequeir » Mon, 05 Sep 2005 20:13:30



> Bash's manpage documents this behaviour for "simple command[s] other than
> a builtin or shell function" and also "[c]ommand substitution, commands
> grouped with parentheses, and asynchronous commands ...  Builtin commands
> that are invoked as part of a pipeline are also executed in a subshell
> environment."

[snip excellent treatise on bash, POSIX and fish]

Quote:> You could pass the logging filenames into the function and perform the
> redirections and tee on each command within the function, but that would
> be pretty inconvenient.  Perhaps other shells (ksh, zsh) have chosen not
> to fork in this situation and you could use one of them instead, but I
> suspect that they would do the same.

I had considered this, but as you mentioned, it is extremely inconvenient.

Quote:>> Any ideas please?

> Well you needn't perform the environment-changing commands within the
> function; in your later post you say that logging would be lost in that
> case, but what logging do you want to perform on changing a directory or
> setting a variable?  And if you do need to log such things, what's
> stopping you?

I wanted the the logging to give me stderr only in one file and stdout +
stderr in another.  I gave this up in the end, and created a function for
such constructs (environment manipulation) that redirected stderr and
stdout to a single file.

Many thanks for your answer.
--
Tony