comp.unix.shell FAQ - Answers to Frequently Asked Questions

comp.unix.shell FAQ - Answers to Frequently Asked Questions

Post by Joe Halp » Sat, 31 Jul 2004 06:40:11



The FAQ has been posted here, and is also available at
http://home.comcast.net/~j.p.h/

Joe

 
 
 

comp.unix.shell FAQ - Answers to Frequently Asked Questions

Post by Joe Halp » Sat, 31 Jul 2004 06:40:10


Archive-name: unix-faq/shell/sh
Posting-Frequency: monthly
Version: $Id: cus-faq.html,v 1.6 2004/05/02 20:56:41 jhalpin Exp jhalpin $
Maintainer: Joe Halpin

This FAQ list contains the answers to some Frequently Asked Questions
often seen in comp.unix.shell. It spells "unix" in lower case letters
to avoid arguments about whether or not Linux, FreeBSD, etc are
unix. That's not the point of this FAQ, and I'm ignoring the issue.

This document as a whole is Copyright (c) 2003 Joe Halpin. It may be
copied freely. Exceptions are noted in individual answers.

Suggestions, complaints, et al, should be sent to the maintainer at
j....@comcast.net or posted to comp.unix.shell

There are two levels of questions about shells.

One is the use of the shell itself as an interface to the operating
system. For example, "how do I run a program in the background, and go
on with other things?". Or "how do I setup environmental variables
when I log in?".

The other level is how to write shell scripts. This often involves
having the shell execute unix utilities to perform part of the work
the shell script needs to accomplish, and requires knowledge of these
utilities, which isn't nominally in the scope of shell
programming. However, unless the question involves something other
than standard unix utilities, it should be included in this FAQ.

Standard unix utilities are defined by either POSIX or the Single Unix
Specification. These are now joined and are normally abbreviated as
"POSIX/SUS". This specification can be found at

http://www.opengroup.org/onlinepubs/007904975/toc.htm

The man pages found on that web page define standard behavior for any
given utility (or the shell itself). However, you should also check
the man page on your system for any utility or shell you need to
use. There isn't always a perfect correspondence between the standard
and a particular implementation (in fact, I'm not sure there's any
case in which they perfectly correspond).

There is also an Austin Group FAQ, which describes the standardization
effort in more detail ls
at

http://www.opengroup.org/austin/faq.html

Other good web sites that provide information about shells and shell
programming (including OS utilities) include:

http://www.shelldorado.com/
http://cfaj.freeshell.org/shell/
http://www.faqs.org/faqs/by-newsgroup/comp/comp.unix.shell.html

This FAQ is available at
http://home.comcast.net/~j.p.h/
http://www.newsville.com/cgi-bin/getfaq?file=comp.unix.shell/comp.uni...

---------------------------
The predictable legal stuff

---------------------------
The answers given in this FAQ list are provided with the best
intentions, but they may not be accurate for any particular
shell/os. They may be completely wrong for any shell/os. If you don't
test the answers, that's a bug in your procedures.

There are no guarantees for the answers or recommendations given in
this document. In fact, I don't even claim to have tested any or all
of them myself. Many of the answers here have been contributed by one
or more regular participants in the newsgroup, who I believe to be
competent (certainly more competent than I am), but THERE ARE NO
GUARANTEES.

Did I really need to make that all uppercase? Hopefully not, but there
are a lot of lawyers around with too much time on their hands, so I
want to make it clear that THERE ARE NO GUARANTEES
about the accuracy of answers in this FAQ list. This is, hopefully, an
aid to people trying to learn shell programming, but it is NOT a
supported product. You have to figure out for yourself whether or not
the answers here work for what you're trying to do.

Under no circumstances will the maintainer of this FAQ list, or any
contributors to it, be held liable for any mistakes in this
document. If the answers work for you, well and good. If not, please
tell me and I'll modify them appropriately so that this will be more
useful.

If you don't agree to that, don't read any farther than this. Reading
beyond this point indicates your agreement.

If you do test the answers and find a problem, please send email to
the maintainer (see above), so it can be corrected, or (preferably)
post a question to the newsgroup so it can be discussed and corrected
if there's a problem.

A number of people have contributed to this FAQ, knowingly or
unknowingly. Some of the answers were taken from previous postings in
the group, and other people contributed questions and answers
directly to the maintainer, which you are welcome to do as well.

Among the contributors is Heiner Steven, who also provided the
momentum to get this FAQ list started. He maintains a web site about
shell programming that has a lot of good stuff in it.

http://www.shelldorado.com/

======================================================================

CONTENTS:

0   COPYING

0a. Glossary
    Google
    POSIX/SUS ("the standard")
    UUOC
    dotfile
    portable
    race condition
    shebang
    shells
    top-posting

0b. Notes about using echo
1.  How can I send e-mails with attached files?
2.  How can I generate random numbers in shell scripts?
3.  How can I automatically transfer files using FTP with error checking?
4.  How can I remove whitespace characters within file names?
5.  How can I automate a telnet session?
6.  How do I do date arithmetic?
7.  Why did someone tell me to RTFM?
9.  How do I create a lock file?
10. How can I convert DOS text files to unix, and vice versa?
11. How can a shell prompt be set up to change the title of xterm?
12. How do I batch an FTP download/upload?
13. How do I get the exit code of cmd1 in cmd1|cmd2
14. Why do I get "script.sh: not found"
15. Why doesn't echo do what I want?
16. How do I loop through files with spaces in their name?
17. how do I change my login shell?
18. When should I use a shell instead of perl/python/ruby/tcl...
19. Why shouldn't I use csh?
20. How do I reverse a file?
21. How do I remove last n lines?
22. how do I get file size, or file modification time?
23. How do I get a process id given a process name? Or, how do I find out if a process is still running, given a process ID?
24. How do I get a script to update my current environment?
25. how do I rename *.foo to *.bar?
26. How do I use shell variables in awk scripts
27. How do I input the user with a timeout?
28. How do I get one character input from the user?
29. why isn't my .profile read?
30. why do I get "[5" not found in "[$1 -eq 2]"?
31. How do I exactly display the content of $var (with a \n appended).
32. How do I exactly display the content of $var (without a \n appended).
33. How do I split a pathname into the directory and file?
34. How do I make an alias take an argument?
35. How do I deal with a file whose name begins with a weird character
36. Why do I lose the value of global variables that are set in a loop
Appendix A: Some example scripts
Appendix B: References. These correspond with numbers in square
            brackets (e.g. [1]) which may appear in the text.

======================================================================

ANSWERS

0  COPYING

   Some contributors may copyright their submissions and license them
   differently than this document.

   [1] Chris F.A. Johnson. Examples marked with COPYING[1] were
       contributed by Chris F.A. Johnson. He has copyrighted these
       examples, and licensed them under the GNU General Public
       License (GPL). Copying them directly into another script will
       cause that script to also come under the GPL. For details see

       http://www.fsf.org/licenses/licenses.html

0a.Glossary

   -------------------------------
   Google

      Google is one of the search engines on the Internet. It took
      over dejanews some years ago, and now is the standard reference
      when directing someone to a past thread one some topic. This is
      a very good place to start when researching a question about
      shell programming (and just about anything else).

        http://groups.google.com/advanced_group_search

   -------------------------------
   POSIX/SUS ("the standard")

      POSIX (Portable Operating System Interface) and SUS (Single Unix
      Specification) have been joined into one standard. This is what
      people usually mean when they refer to "the standard" in
      discussions about unix. When people in this group refer to the
      POSIX shell, they are talking about the shell prescribed by this
      specification. You can find this standard at

        http://www.opengroup.org/onlinepubs/007904975/toc.htm

   -------------------------------
   UUOC

      This is short for "Useless use of cat". It's used to point out
      that some example script has used cat when it could have used
      redirection instead. It's more efficient to redirect input than
      it is to spawn a process to run cat. For example

        $ cat file | tr -d 'xyz'

      runs two processes, one for cat and one for tr. This is less
      efficient than

        $ tr -d 'xyz' < file

      In general, "cat file | somecommand" can be more efficiently
      replaced by "somecommand < file"

      or (especially for multi-file input)

        $ somecommand file [file ...]

      but check the man page for "somecommand" to find out if it will
      accept this syntax.

      For more details about this, as well as other things like it, see
        http://rhols66.adsl.netsonic.fi/era/unix/award.html

   -------------------------------
   dotfile

      This refers to a file which starts with '.' (a dot). These files
      are not shown in directory listings without the -a (or -A in
      newer versions of ls - check the man page on your system) option
      to ls. Often they are configuration files, subdirectories used
      by applications to store configuration files, NFS swap files, et
      al.

   -------------------------------
   portable

      The word "portable" means different things to different people,
      in
...

read more »

 
 
 

comp.unix.shell FAQ - Answers to Frequently Asked Questions

Post by Stephane CHAZELA » Sat, 31 Jul 2004 19:38:32


2004-07-29, 21:40(+00), Joe Halpin:
> The FAQ has been posted here, and is also available at
> http://home.comcast.net/~j.p.h/

[...]

some comments.

Why not splitting it so that parts are less than 60kB?

[...]

> Version: $Id: cus-faq.html,v 1.7 2004/05/31 14:24:00 jhalpin Exp $

The text version is 1.6

Note that everywhere "zsh 4.1" is mentionned in that faq, it should be
replaced with "zsh 4.1 and above" (latest version is 4.2.x).

> 2. How can I generate random numbers in shell scripts?
[...]
>    c. Use a utility such as awk(1), which has random number generation
>       included. This approach is the most portable between shells and
>       operating systems.

>         awk 'BEGIN {srand();print rand()}'

>       However, note that if you call this line more than once within
>       the same second, you'll get the same number you did the previous
>       time.

Won't work with old awks though (same as in 6.h)

[...]

> 4. How can I remove whitespace characters within file names?

>    File names in unix can contain all kinds of whitespace characters,
>    not just spaces. The following examples only work with spaces,
>    adjust accordingly.

>    a. Use the substitution capabilities of awk, sed, et al.
[...]
>    b. Use the substitution capabilities of the shell if it has
>       them. Check the man page for your shell (probably under a
>       section named something like "Parameter expansion") to see. For
>       example:

>       f=${filename// /_}

>       With zsh:

>       autoload -U zmv
>       zmv '* *' '$f:gs/ /_/'

It should be noted that the zmv solution renames the files (call mv
internally and adress several problems that may arise) while the other
solutions only update a variable (and then, renaming the files may
involve a quite complicated script to do it reliably).

[...]

> 6. How do I do date arithmetic?
[...]
>    g. Arbitrary date arithmetic
[...]
>       Another possibility is given in the examples section, from

>         http://groups.google.com/groups?selm=n6d6zalnpk.fsf%40ogion.it.jyu.fi

See also:

http://groups.google.com/groups?selm=slrnbvnhu2.3v.stephane.chaze...@spam.is.invalid

[...]

> 20. How do I reverse a file?
[...]
>     Also, nl can be used as

>     nl -ba -d'
>     '

>     i.e. NL as the delimiter.
>     You may also be able to use nl with the -p option, which is POSIX,
>     and widely available, going back to AT&T SysV/386 R3.2 from 1988
>     or thereabouts.

That latter part is irrelevant to the question. -p is of no use here as
it doesn't prevent the page splitting.

> 21. how do I remove the last n lines?
[...]
>     AWK solutions:

1

>        awk 'NR<=(count-12)' count="`awk 'END{print NR}' file`" file

2

>        awk 'NR>n{print a[NR%n]} {a[NR%n]=$0}' n=12 file

>        awk 'BEGIN{n=12} NR>n{print a[NR%n]} {a[NR%n]=$0}' file

It should be noted that 2 is in every case much more efficient than
the other ones (it runs only one awk and read the file only once).

1 (and the non-awk solutions) might only be prefered if <n> is
very big (worth several MB) as the 2 solution holds <n> lines in
memory.

[...]

>       sed -n -e :a -e '1,12{N;ba' -e '}' -e 'P;N;D' file

>       The last solution is basically same algorithm as the rolling
>       array awk solutions, and shares with them the advantage that
>       the file is only read once - they will even work in a pipe.

But many sed implementations have a low limit on what it can
hold in its pattern space (in memory) so that one can't be used
for big values of <n>.

[...]

>     Using GNU dd:

>       ls -l file.txt | {
>         IFS=" "
>         read z z z z sz z
>         last=`tail -10 file.txt | wc -c`
>         dd bs=1 seek=`expr $sz - $last` if=/dev/null of=file.txt
>       }

>       This is different than other solutions in that, rather than
>       creating a new, shorter file it overwrites the trailing lines
>       in the original file with nulls.

No, it doesn't fill with 0. It truncate(2)s the file. It may
reveal the most efficient solution for very big files (and
relatively small values of <n>) as it doesn't read the whole
file (just the trailing part) and just does a ftruncate system
call.

You could do the same with perl with a bit of programming.

[...]

> 25. How do I rename *.foo to *.bar?

[...]

The mmv, ren, rename and zmv solutions should be put first as they are
the simplest and most reliable ones.

For the other ones, a "use it at your own risk" or "make a
backup of your data before" or "prepend echo to the moving
command first" warnings would be welcome.

See also the "-n" option to zmv.

[...]

>       for file in *.foo

for file in ./*.foo

(or use -- in every command below).

>       do
>         newfile=`basename "$file" .foo`.bar
>         [ -f "$file" ] || continue
>         [ -f "$newfile" ] && continue ## or deal with it another way

What if "$newfile" is a directory?

ls -d "$newfile" > /dev/null 2>&1 && continue

>         mv "$file" `basename "$file" .foo`.bar

mv "$file" "$newfile"

> 26. How do I use shell variables in awk scripts

>     Depending on the version of awk being used, either use the -v
>     command line option,

>       $ awk -v var=xxx '{print $0,var}'

>     or add the variable after the command, as in

>       $ awk '{print $0,var}' var=xxx file

>     Note that using the latter syntax var will not be available in the
>     BEGIN section.

Also beware of the \n, \t... expanded.

A third (and more reliable solution) is to use the ARGV special awk
array.

> 28. How do I get one character input from the user?

>     In bash this can be done with the "-n" option to read.
>     In ksh93 it's read -N
>     In zsh it's read -k

>     More portably:

>     OLDSTTY=$(stty -g)  # save our terminal settings
>     stty cbreak  # enable independent processing of each input character
>     ONECHAR=$(dd bs=1 count=1 2>/dev/null)  # read one byte from standard in
>     stty $OLDSTTY  # restore the terminal settings

stty "$OLDSTTY"

[...]

> 33. How do I split a pathname into the directory and file?

>     The most portable way of doing this is to use the external
>     commands dirname(1) and basename(1), as in

>       pathname='/path/to/some/file'
>       dir=`dirname $pathname`
>       file=`basename $pathname`

dir=`dirname "$pathname"`
file=`basename "$pathname"`

>     However, since this executes an external command, it's slower than
>     using shell builtins (if your shell has them)

and fails if the {dir,base}name ends in NL characters.

>     For ksh, bash, and
>     POSIX shells the following will do the same thing more
>     efficiently:

and zsh.

>       pathname=/path/to/some/file
>       file=${pathname##*/}

>     To get the directory using the shell builtin, you should first
>     ensure that the path has a '/' in it.

>       case $pathname in
>         */*) dir=${pathname%/*};;
>         *)   dir=''
>       esac

In zsh, (abd csh, tcsh), you have

${pathname:h} (head) ${pathname:t} (tail).

[...]

> 34. How do I make an alias take an argument?

>     In Bourne-derived shells aliases cannot take arguments, so if you
>     need to be able to do that, define a shell function rather than
>     an alias.

In no shell actually (in csh it's just a trick that makes look like it
takes an argument), an alias is just... an alias.

--
Stephane

 
 
 

comp.unix.shell FAQ - Answers to Frequently Asked Questions

Post by j.. » Sun, 01 Aug 2004 23:26:45



> 2004-07-29, 21:40(+00), Joe Halpin:
> > The FAQ has been posted here, and is also available at
> > http://home.comcast.net/~j.p.h/
> [...]

> some comments.

> Why not splitting it so that parts are less than 60kB?

Mostly I don't have the time to redo it. I'm working pretty much seven
days a week now, my wife thinks I should spend some time with her too,
etc, etc, etc.

Heiner is looking into setting up a wiki of some kind and making the
faq more of a collaboration. If he can get that setup I'll be happy to
give up the copyright and make this a group effort. I think that would
be better anyway.

Thanks for your other comments, I'll try and incorporate them this
weekend.

Joe
--
It's the RINSE CYCLE!!  They've ALL IGNORED the RINSE CYCLE!!
  - yow

 
 
 

1. comp.unix.shell FAQ - Answers to Frequently Asked Questions

The FAQ has been posted here, and is also available at
http://home.comcast.net/~j.p.h/

It's been split into three sections now, with the addition of the
recent contribution by Ed Morton.

Joe

2. connecting Linux to a Win98 proxy

3. - - - The Disk Zapper Data Security Device - - -

4. ifconfig and do_ioctl

5. ATI Mach64 GX

6. comp.unix.shell FAQ - Answers to Frequently Asked Questions Appendix A and B