comp.unix.shell FAQ - Answers to Frequently Asked Questions

comp.unix.shell FAQ - Answers to Frequently Asked Questions

Post by Joe Halp » Wed, 31 Dec 2003 05:14:49



The FAQ has been posted here, and is also available at
http://home.comcast.net/~j.p.h/

Joe

 
 
 

comp.unix.shell FAQ - Answers to Frequently Asked Questions

Post by Joe Halp » Wed, 31 Dec 2003 05:14:49


Archive-name: unix-faq/shell/sh
Posting-Frequency: monthly
Version: $Id: cus-faq,v 1.12 2003/11/30 20:19:58 jhalpin Exp jhalpin $
Maintainer: Joe Halpin

This FAQ list contains the answers to some Frequently Asked Questions
often seen in comp.unix.shell. It spells "unix" in lower case letters
to avoid arguments about whether or not Linux, FreeBSD, etc are
unix. That's not the point of this FAQ, and I'm ignoring the issue.

This document as a whole is Copyright (c) 2003 Joe Halpin. It may be
copied freely. Exceptions are noted in individual answers.

Suggestions, complaints, et al, should be sent to the maintainer at
j....@comcast.net or posted to comp.unix.shell

There are two levels of questions about shells.

One is the use of the shell itself as an interface to the operating
system. For example, "how do I run a program in the background, and go
on with other things?". Or "how do I setup environmental variables
when I log in?".

The other level is how to write shell scripts. This often involves
having the shell execute unix utilities to perform part of the work
the shell script needs to accomplish, and requires knowledge of these
utilities, which isn't nominally in the scope of shell
programming. However, unless the question involves something other
than standard unix utilities, it should be included in this FAQ.

Standard unix utilities are defined by either POSIX or the Single Unix
Specification. These are now joined and are normally abbreviated as
"POSIX/SUS". This specification can be found at

http://www.opengroup.org/onlinepubs/007904975/toc.htm

The man pages found on that web page define standard behavior for any
given utility (or the shell itself). However, you should also check
the man page on your system for any utility or shell you need to
use. There isn't always a perfect correspondence between the standard
and a particular implementation (in fact, I'm not sure there's any
case in which they perfectly correspond).

There is also an Austin Group FAQ, which describes the standardization
effort in more detail ls
at

http://www.opengroup.org/austin/faq.html

Other good web sites that provide information about shells and shell
programming (including OS utilities) include:

http://www.shelldorado.com/
http://cfaj.freeshell.org/shell/
http://www.faqs.org/faqs/by-newsgroup/comp/comp.unix.shell.html

This FAQ is available at
http://home.comcast.net/~j.p.h/
http://www.newsville.com/cgi-bin/getfaq?file=comp.unix.shell/comp.uni...

---------------------------
The predictable legal stuff

---------------------------
The answers given in this FAQ list are provided with the best
intentions, but they may not be accurate for any particular
shell/os. They may be completely wrong for any shell/os. If you don't
test the answers, that's a bug in your procedures.

There are no guarantees for the answers or recommendations given in
this document. In fact, I don't even claim to have tested any or all
of them myself. Many of the answers here have been contributed by one
or more regular participants in the newsgroup, who I believe to be
competent (certainly more competent than I am), but THERE ARE NO
GUARANTEES.

Did I really need to make that all uppercase? Hopefully not, but there
are a lot of lawyers around with too much time on their hands, so I
want to make it clear that THERE ARE NO GUARANTEES
about the accuracy of answers in this FAQ list. This is, hopefully, an
aid to people trying to learn shell programming, but it is NOT a
supported product. You have to figure out for yourself whether or not
the answers here work for what you're trying to do.

Under no circumstances will the maintainer of this FAQ list, or any
contributors to it, be held liable for any mistakes in this
document. If the answers work for you, well and good. If not, please
tell me and I'll modify them appropriately so that this will be more
useful.

If you don't agree to that, don't read any farther than this. Reading
beyond this point indicates your agreement.

If you do test the answers and find a problem, please send email to
the maintainer (see above), so it can be corrected, or (preferably)
post a question to the newsgroup so it can be discussed and corrected
if there's a problem.

A number of people have contributed to this FAQ, knowingly or
unknowingly. Some of the answers were taken from previous postings in
the group, and other people contributed questions and answers
directly to the maintainer, which you are welcome to do as well.

Among the contributors is Heiner Steven, who also provided the
momentum to get this FAQ list started. He maintains a web site about
shell programming that has a lot of good stuff in it.

http://www.shelldorado.com/

======================================================================

CONTENTS:

0   COPYING

0a. Glossary
    Google
    POSIX/SUS ("the standard")
    UUOC
    dotfile
    portable
    race condition
    shebang
    shells
    top-posting

0b. Notes about using echo
1.  How can I send e-mails with attached files?
2.  How can I generate random numbers in shell scripts?
3.  How can I automatically transfer files using FTP with error checking?
4.  How can I remove whitespace characters within file names?
5.  How can I automate a telnet session?
6.  How do I do date arithmetic?
7.  Why did someone tell me to RTFM?
9.  How do I create a lock file?
10. How can I convert DOS text files to unix, and vice versa?
11. How can a shell prompt be set up to change the title of xterm?
12. How do I batch an FTP download/upload?
13. How do I get the exit code of cmd1 in cmd1|cmd2
14. Why do I get "script.sh: not found"
15. Why doesn't echo do what I want?
16. How do I loop through files with spaces in their name?
17. how do I change my login shell?
18. When should I use a shell instead of perl/python/ruby/tcl...
19. Why shouldn't I use csh?
20. How do I reverse a file?
21. How do I remove last n lines?
22. how do I get file size, or file modification time?
23. How do I get a process id given a process name? Or, how do I find out if a process is still running, given a process ID?
24. How do I get a script to update my current environment?
25. how do I rename *.foo to *.bar?
26. How do I use shell variables in awk scripts
27. How do I input the user with a timeout?
28. How do I get one character input from the user?
29. why isn't my .profile read?
30. why do I get "[5" not found in "[$1 -eq 2]"?
31. How do I exactly display the content of $var (with a \n appended).
32. How do I exactly display the content of $var (without a \n appended).
33. How do I split a pathname into the directory and file?

Appendix A: Some example scripts
Appendix B: References. These correspond with numbers in square
            brackets (e.g. [1]) which may appear in the text.

======================================================================

ANSWERS

0  COPYING

   Some contributors may copyright their submissions and license them
   differently than this document.

   [1] Chris F.A. Johnson. Examples marked with COPYING[1] were
       contributed by Chris F.A. Johnson. He has copyrighted these
       examples, and licensed them under the GNU General Public
       License (GPL). Copying them directly into another script will
       cause that script to also come under the GPL. For details see

       http://www.fsf.org/licenses/licenses.html

0a.Glossary

   -------------------------------
   Google

      Google is one of the search engines on the Internet. It took
      over dejanews some years ago, and now is the standard reference
      when directing someone to a past thread one some topic. This is
      a very good place to start when researching a question about
      shell programming (and just about anything else).

        http://groups.google.com/advanced_group_search

   -------------------------------
   POSIX/SUS ("the standard")

      POSIX (Portable Operating System Interface) and SUS (Single Unix
      Specification) have been joined into one standard. This is what
      people usually mean when they refer to "the standard" in
      discussions about unix. When people in this group refer to the
      POSIX shell, they are talking about the shell prescribed by this
      specification. You can find this standard at

        http://www.opengroup.org/onlinepubs/007904975/toc.htm

   -------------------------------
   UUOC

      This is short for "Useless use of cat". It's used to point out
      that some example script has used cat when it could have used
      redirection instead. It's more efficient to redirect input than
      it is to spawn a process to run cat. For example

        $ cat file | tr -d 'xyz'

      runs two processes, one for cat and one for tr. This is less
      efficient than

        $ tr -d 'xyz' < file

      In general, "cat file | somecommand" can be more efficiently
      replaced by "somecommand < file"

      or (especially for multi-file input)

        $ somecommand file [file ...]

      but check the man page for "somecommand" to find out if it will
      accept this syntax.

   -------------------------------
   dotfile

      This refers to a file which starts with '.' (a dot). These files
      are not shown in directory listings without the -a (or -A in
      newer versions of ls - check the man page on your system) option
      to ls. Often they are configuration files, subdirectories used
      by applications to store configuration files, NFS swap files, et
      al.

   -------------------------------
   portable

      The word "portable" means different things to different people,
      in different situations, which is to say, there isn't one
      definition of "portable".

      At one extreme, a portable script is one which will work under
      any shell, on any operating system. At this end of the spectrum,
      there is no such thing as a portable shell script (some
      operating systems don't even have
...

read more »

 
 
 

comp.unix.shell FAQ - Answers to Frequently Asked Questions

Post by Tapani Tarvaine » Wed, 31 Dec 2003 19:39:15



> Archive-name: unix-faq/shell/sh
> Posting-Frequency: monthly
> Version: $Id: cus-faq,v 1.12 2003/11/30 20:19:58 jhalpin Exp jhalpin $

Just picking on one question:

Quote:> 21. how do I remove the last n lines?

>     First we need to tell the code how many lines we want to cut
>     from the bottom of a file.

>       X=10

>     Then We can do this:

>       echo $(head -$(( $(wc -l < file ) - $X )) file) >\
>       $$ && cat $$ > file && rm $$  

>        The break down:  
>        1) $(wc -l < file)
>           Find out how many lines are in the file. Need to use
>           redirection so wc won't print the file name.
>        2) $(( $lines_in_file - $X ))
>           Take the output from step one and do some math to find out
>           how many lines we want to have when all is said and done.
>        3) head -$lines_when_said_and_done file
>           extract all but the unwanted lines from the file
>        4) echo $all_but_the_unwanted_lines_from_the_file > $$
>           this puts those lines into a temp file that has the name of
>           the pid of the current shell.
>        5) && cat $$ > file
>           if everything has worked so far then cat the temp file into
>           the original file.  This is better than mv or cp because it
>           insures that the permissions of the temp file do not
>           override with the perms of the original file.
>        6) && rm $$
>           Remove the temp file.

I *hate* using echo there: it can break the file if there
are special characters or if it is too big, and IMHO it just
makes it harder to understand, as it's totally unnecessary.
Also, the old head syntax is somewhat out-of-place among
otherwise modern constructs. Thus I'd suggest changing that to

       head -n $(( $(wc -l < file ) - $X )) file >$$ \
         && cat $$ >file && rm $$  

and combining comments 3 & 4 (and renumbering the rest):

        3) head -$lines_when_said_and_done file
           extracts all but the unwanted lines from the file,
           and >$$ puts those lines into a temp file that has
           the name of the pid of the current shell.

It might also be a good idea to make all solutions uniform at least in
file names and line counts, e.g., always take input from file named
'file' and have result written to stdout, and keep the number of lines
removed always as 12 (using 12 in the above as well, or 10 in the all
the others).

Quote:>     NAWK solution:
>       $file=whatever
>       nawk -v count="`nawk 'END{print NR}' $file`" 'NR<(count-11)' $file))

That is actually broken (typo?), the trailing )) must be removed.
It is also perhaps somewhat confusing to call this "nawk" solution, as it
will work as well with any newer awk, although admittedly not with classic
awk - but it takes only a slight change will make it compatible with it, too
(only the -v switch has to be removed).
Also, it may be confusing that even though we're deleting 12 lines,
the only number in there is 11; using <= would make it clearer.

So, a suggested rewrite for the awk section:

     AWK solutions:

       awk 'NR<=(count-12)' count="`awk 'END{print NR}' file`" file

       awk 'NR>n{print a[NR%n]} {a[NR%n]=$0}' n=12 file

       awk 'BEGIN{n=12} NR>n{print a[NR%n]} {a[NR%n]=$0}' file

I would also consider dropping one of the latter two, as they're
essentially identical (the former is easier when the line count is
in a variable, so I'd prefer it) - perhaps a pointer to Q26 might
be useful.

A brief explanation how these work might also be useful, e.g.,
"Whenever a line is read, the line that came 12 lines ago
is printed, and then overwritten with the newly read line,
using an rolling array indexed 0..11."

Quote:>     Command solution:
>       Using the last with the '-f' option to do something it wasn't
>       intended for.

Is it really useful to give FAQ answers that are only hints to
something that may not even be possible to do (system-dependent)?
I'd drop this, or at least explain it more.

Quote:>     $SHELL/sed/mv solution:
>       for F in * (or however you get a list of files))) ; do
>           L=`wc -l <$F`
>           DL=`expr $L - 12`
>           sed "$DL,\$d" $F >new$F
>           mv new$F $F
>       done

That's broken, in that it removes 13 last lines instead of 12.
I would also make it similar to the others (the for loop only confuses
things here), i.e.,

        L=`wc -l <file`
        DL=`expr $L - 11`
        sed "$DL,\$d" file    

or simpler, more readable (IMHO--at least the 12 is there as such)
and slightly more efficient, too:

        L=`wc -l <file`
        DL=`expr $L - 12`
        sed "${DL}q" file    

It could also be given as a one-liner,

        sed "`expr \`wc -l <file\` - 12`q" file

or with modern shells

        sed "$((`wc -l <file` - 12))q" file

A pure sed solution is also possible, e.g.,

        sed -n -e :a -e '1,12{N;ba' -e '}' -e 'P;N;D' file

with basically same algorithm as the rolling array awk solutions, and
shares with them the advantage that the file is only read once - they
will even work in a pipe.

Quote:>     $SHELL/head solution:
>        i=`wc -l filename | cut -c1-8`
>        head -n `expr $i - 12` filename

This is unnecessarily complicated, and actually so close to the
well-explained first solution I'd drop this altogether, or
alternatively make it work with pre-POSIX stuff (and discard the
unnecessary cut):

        i=`wc -l <file`
        head -`expr $i - 12` file

Then this should be immediately after the first solution to
highlight the differences.

Quote:>     PERL solution:

If a perl solution is provided at all I'd recommend John Krahn's


which is functionally equivalent with the two later awk versions and
the pure sed solution (and will also work in a pipe).

Quote:>     Using GNU dd:

>       ls -l file.txt | {
>         IFS=" "
>         read z z z z sz z
>         last=`tail -10 file.txt | wc -c`
>         dd bs=1 seek=`expr $sz - $last` if=/dev/null of=file.txt
>       }

Perhaps a word of explanation would be helpful here, as it is
functionally different from the others: instead of creating a
new, shorter file it overwrites the trailing lines with nulls.

--
Tapani Tarvainen

 
 
 

comp.unix.shell FAQ - Answers to Frequently Asked Questions

Post by j.. » Thu, 01 Jan 2004 10:21:28




> > Archive-name: unix-faq/shell/sh
> > Posting-Frequency: monthly
> > Version: $Id: cus-faq,v 1.12 2003/11/30 20:19:58 jhalpin Exp jhalpin $

> Just picking on one question:

[...]

Thanks. I'll try to rework that soon.

Joe