NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by Ken Abrahamse » Sun, 31 Dec 1899 09:00:00



When doing shell scripting and needing to manipulate (sed, grep, etc) file
contents, what are generally accepted good shell programming practices
determining when to assign the contents of a file to a shell variable versus
creating a tmp file and using the tmp to do all the work in?

Avoiding tmp files means avoiding duplicate tmp file naming problems, and
the cleanup of these tmp files but what type of problems would be created by
using shell variables, especially if the contents of these shell variables
were loaded from a 'large' file. If there are size thresholds (i.e., it's a
bad idea to use a shell variable if the data is over xxxx kilobytes, but OK
if under this), what do people find these thresholds to be? Any other
pointers / recommendations would be appreciated.

I'm trying to do better scripting, but find no information on these type of
'style' guidelines.

Thanks for your comments!
ken

 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by -ljl » Sun, 31 Dec 1899 09:00:00




Quote:> When doing shell scripting and needing to manipulate (sed, grep, etc)
file
> contents, what are generally accepted good shell programming practices
> determining when to assign the contents of a file to a shell variable
versus
> creating a tmp file and using the tmp to do all the work in?

> Avoiding tmp files means avoiding duplicate tmp file naming problems,

I not an expert by any sense of the word but this can be solved by
using the "PID of the current process", reserved shell variable $$.
Each process is unique.
  tmp=sort$$
  sort table > $tmp
  mv $tmp table
This is an example from "Portable Unix".  Of course the last line
could delete '$tmp', or whatever.

Quote:> bad idea to use a shell variable if the data is over xxxx kilobytes,

but OK

I've looked at lot of other peoples work and large amounts of data went
into files and < 100 (or so) characters went into variables.

Quote:> I'm trying to do better scripting, but find no information on these
> type of 'style' guidelines.

I've never seen this question raised before.  Also, I presume variables
are faster than files.

--
Louis-ljl-{ Louis J. LaBash, Jr. }

Sent via Deja.com http://www.deja.com/
Before you buy.

 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by Matthew Land » Sun, 31 Dec 1899 09:00:00



> When doing shell scripting and needing to manipulate (sed, grep, etc) file
> contents, what are generally accepted good shell programming practices
> determining when to assign the contents of a file to a shell variable versus
> creating a tmp file and using the tmp to do all the work in?

> Avoiding tmp files means avoiding duplicate tmp file naming problems, and
> the cleanup of these tmp files but what type of problems would be created by
> using shell variables, especially if the contents of these shell variables
> were loaded from a 'large' file. If there are size thresholds (i.e., it's a
> bad idea to use a shell variable if the data is over xxxx kilobytes, but OK
> if under this), what do people find these thresholds to be? Any other
> pointers / recommendations would be appreciated.

> I'm trying to do better scripting, but find no information on these type of
> 'style' guidelines.

> Thanks for your comments!
> ken

The use of temp files is good to avoid.  You can get around the uniqueness a
by using $$ (the PID of the current shell) as part of the temp file's name.
But using a temp file, means doing disk I/O.  Memory access is hundreds if not
thousands of times faster than magnetic media.  If the data isn't huge, stick
it in a variable.  If it is huge, it is probably already in a file.  If not
(like a `find /` type command), avoid both variables and temp files and use
pipes.  If the data has to be accessed time and time again, and is large,
a temp file might be the way to go.  My general rule is to keep a variable
witin the arg limit (32K) or within some editors max line size limit (2048).

 - Matt
--
_______________________________________________________________________

   << Comments, views, and opinions are mine alone, not IBM's. >>

 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by Grant Edwar » Sun, 31 Dec 1899 09:00:00



>The use of temp files is good to avoid.  

Sure, as long it doesn't make your script more complex.

Quote:>You can get around the uniqueness a by using $$ (the PID of the
>current shell) as part of the temp file's name. But using a
>temp file, means doing disk I/O. Memory access is hundreds if
>not thousands of times faster than magnetic media.  

Under any decent OS (Linux included), operations on a tempfile
are, in practice, done on a block of memory in the buffer
cache.  If it is short-lived, the data may never get flushed
out to a platter at all. I doubt that the speed difference is
really noticable for most shell-scripts.

Do whatever is simplest and easiest to understand.

First make it work.  Then make if faster -- but only if you
have to.

--
Grant Edwards                   grante             Yow!  But they went to MARS
                                  at               around 1953!!
                               visi.com            

 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by Barry Margoli » Sun, 31 Dec 1899 09:00:00




>Under any decent OS (Linux included), operations on a tempfile
>are, in practice, done on a block of memory in the buffer
>cache.  

Except that closing a file typically starts flushing it to disk.  What you
describe is likely only if /tmp is mounted on a ram-disk (e.g. Solaris
"tmpfs").

Quote:>     If it is short-lived, the data may never get flushed
>out to a platter at all. I doubt that the speed difference is
>really noticable for most shell-scripts.

>Do whatever is simplest and easiest to understand.

>First make it work.  Then make if faster -- but only if you
>have to.

That's the best advice.

In response to the original poster's question, the answer really depends on
the machine's configuration and load.  If you use variables, but the
machine doesn't have enough physical RAM to keep it all in memory (which
depends on what other processes are also competing for RAM) it will have to
be flushed to the swap partition.  These days memory is cheap, so
well-configured systems have over 100 MB of RAM (and big servers may exceed
1 GB), so if the system isn't overloaded you can probably get away with
10's of MB being kept in shell variables.  However, if you keep modifying
it and storing the results in another variable, it will multiply (I don't
know how good the garbage collection of most shell implementations are --
it's probably not usually a big issue).

--

Genuity, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by John Hasle » Sat, 26 Aug 2000 09:07:01


Quote:Grant Edwards writes:
> First make it work.  Then make if faster

And more secure.  Shellscripts that use tempfiles can create security
holes.
--
John Hasler

Dancing Horse Hill
Elmwood, Wisconsin
 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by Grant Edwar » Sat, 26 Aug 2000 10:43:59





>>Under any decent OS (Linux included), operations on a tempfile
>>are, in practice, done on a block of memory in the buffer
>>cache.  

>Except that closing a file typically starts flushing it to disk.  

I don't think Linux acts that way.  For smaller programs I can run makes
that generated handfuls of temp files, and if the machine isn't busy it all
happens in memory with almost no disk activity at all.

Quote:>What you describe is likely only if /tmp is mounted on a ram-disk (e.g.
>Solaris "tmpfs").

--
Grant Edwards                   grante             Yow!  Isn't this my STOP?!
                                  at              
                               visi.com            
 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by Christopher Brow » Sat, 26 Aug 2000 12:53:57


Centuries ago, Nostradamus foresaw a time when Barry Margolin would say:



>>Under any decent OS (Linux included), operations on a tempfile
>>are, in practice, done on a block of memory in the buffer
>>cache.  

>Except that closing a file typically starts flushing it to disk.  What you
>describe is likely only if /tmp is mounted on a ram-disk (e.g. Solaris
>"tmpfs").

Linux does not have an exact equivalent to tmpfs, and thus while
operations on files in /tmp do get cached for _read_ purposes, there is
some degree of "write-thru" on output.

In other words, the data does get shoved out to disk.

Quote:>>         If it is short-lived, the data may never get flushed
>>out to a platter at all. I doubt that the speed difference is
>>really noticable for most shell-scripts.

>>Do whatever is simplest and easiest to understand.

>>First make it work.  Then make if faster -- but only if you
>>have to.

>That's the best advice.

Indeed.  Code that works, albeit not with optimal performance, is better
than code that you have tried, unsuccessfully, to optimize.

Quote:>In response to the original poster's question, the answer really depends on
>the machine's configuration and load.  If you use variables, but the
>machine doesn't have enough physical RAM to keep it all in memory (which
>depends on what other processes are also competing for RAM) it will have to
>be flushed to the swap partition.  These days memory is cheap, so
>well-configured systems have over 100 MB of RAM (and big servers may exceed
>1 GB), so if the system isn't overloaded you can probably get away with
>10's of MB being kept in shell variables.  However, if you keep modifying
>it and storing the results in another variable, it will multiply (I don't
>know how good the garbage collection of most shell implementations are --
>it's probably not usually a big issue).

The other issue worth a _little_ thought is that peppering /tmp
with temporary files may represent a security hole, as it is pretty
common for those files to be world-readable.

The "ultimate" way of resolving this is to open an output file,
unlink it, and only _then_ start writing to it.  All I/O has to
take place within the process that holds the file descriptor,
because the file at that point becomes anonymous, and will disappear
as soon as it is closed.

Alternatively, one might create a directory
  /tmp/whatever.$$ (process ID)
and, before doing anything else, do:
  chmod 700 /tmp/whatever.$$
At that point, anything created in that directory is private to the
given user, which is quite a lot better than it being open to the
world.
--

:FATAL ERROR -- ERROR IN ERROR HANDLER

 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by Barry Margoli » Sat, 26 Aug 2000 13:04:10




>The "ultimate" way of resolving this is to open an output file,
>unlink it, and only _then_ start writing to it.  All I/O has to
>take place within the process that holds the file descriptor,
>because the file at that point becomes anonymous, and will disappear
>as soon as it is closed.

The OP was asking about shell scripting, and it isn't very easy to do that
in scripts.  I guess it can be done with:

exec >/tmp/tempfile.$$
exec </tmp/tempfile.$$
rm /tmp/tempfile.$$
...

but I haven't tried it.

--

Genuity, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by John Hasle » Sun, 31 Dec 1899 09:00:00


Quote:Christopher Browne writes:
> The "ultimate" way of resolving this is to open an output file, unlink
> it, and only _then_ start writing to it.

In a shell script?  Even in C this is not secure against race conditions.
The best thing to do in a shell script is to use the tempfile or mktemp
utilities.
--
John Hasler

Dancing Horse Hill
Elmwood, Wisconsin
 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by Phi » Sun, 31 Dec 1899 09:00:00



Quote:>Grant Edwards writes:
>> First make it work.  Then make if faster

>And more secure.  Shellscripts that use tempfiles can create security
>holes.

They can be used properly if permissions are set properly on relevant
directories and files.
Phil.
 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by Ken Pizzi » Sun, 31 Dec 1899 09:00:00



>I not an expert by any sense of the word but this can be solved by
>using the "PID of the current process", reserved shell variable $$.
>Each process is unique.

... provided that you are working on a disk that is not shared.
Things like NFS*up the "$$ is unique among running
processes" assumption.

                --Ken Pizzini

 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by marktchand.. » Sun, 31 Dec 1899 09:00:00


Some consider Perl a shell scripting language of sorts.  If you use
Perl you can open a file and operate on it line by line, or character by
character, or whatever.  You can also forget sed and grep, and use the
Perl functionality.  THIS is the way that I would go.

Already the $$ has been pointed out, so no need to rehash that.

You can try some clever combinations of "cat" and "|" and ">" to read
the file and write back out to it:  e.g.

   cat bob.txt | sort -r > bob.txt

mtc



Quote:> When doing shell scripting and needing to manipulate (sed, grep, etc)
file
> contents, what are generally accepted good shell programming practices
> determining when to assign the contents of a file to a shell variable
versus
> creating a tmp file and using the tmp to do all the work in?

> Avoiding tmp files means avoiding duplicate tmp file naming problems,
and
> the cleanup of these tmp files but what type of problems would be
created by
> using shell variables, especially if the contents of these shell
variables
> were loaded from a 'large' file. If there are size thresholds (i.e.,
it's a
> bad idea to use a shell variable if the data is over xxxx kilobytes,
but OK
> if under this), what do people find these thresholds to be? Any other
> pointers / recommendations would be appreciated.

> I'm trying to do better scripting, but find no information on these
type of
> 'style' guidelines.

> Thanks for your comments!
> ken

Sent via Deja.com http://www.deja.com/
Before you buy.
 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by spi.. » Sun, 31 Dec 1899 09:00:00



Quote:> Linux does not have an exact equivalent to tmpfs, and thus while
> operations on files in /tmp do get cached for _read_ purposes, there is
> some degree of "write-thru" on output.
> In other words, the data does get shoved out to disk.

There's no reason why linux shouldn't mimic this though.
Linux does have Ramdisk capabilities, after all...
--
______________________________________________________________________________

|Andrew Halliwell BSc(hons)| "The day Microsoft makes something that doesn't |
|            in            |  suck is probably the day they start making     |
|     Computer science     |  vacuum cleaners" - Ernst Jan Plugge            |
------------------------------------------------------------------------------
 
 
 

NEWBIE-Shell scripting - When to use script variable vs. create tmp file???

Post by Martien Verbrugg » Sun, 31 Dec 1899 09:00:00


On Sat, 26 Aug 2000 13:21:09 GMT,

Quote:

> Some consider Perl a shell scripting language of sorts.  If you use

I don't think you're quite right here.

Some people do consider Perl to be a scripting language, but I don't
think I've ever heard anyone call it a _shell_ scripting language before
now. Many discussions go on regularly about whether it is a scripting
language or a programming language, and people get in knots trying to
find out what the difference betweenthose two is. However, a shell is a
shell, and perl ain't one of those.

Maybe you know another 'some' than I do, and in that case you are right,
but they are still wrong.

Quote:> Perl you can open a file and operate on it line by line, or character by
> character, or whatever.  You can also forget sed and grep, and use the
> Perl functionality.  THIS is the way that I would go.

Perl is a good language for many system administration tasks, and often
you'll find that shell scripts that use a lot of sed, grep and awk will
benefit considerably from a rewrite in Perl. However, don't make the
mistake of using Perl when you need to call large numbers of external
programs. While perl can easily do tat, shells are normally better at
it.

Some things work well in shells, some in Perl, and some in C.  I believe
that that was more or less your point as well, but I couldn't just let
you call Perl a shell scripting language, now, could I? :)

Martien
--
Martien Verbruggen              | Since light travels faster than
Interactive Media Division      | sound, isn't that why some people
Commercial Dynamics Pty. Ltd.   | appear bright until you hear them
NSW, Australia                  | speak?

 
 
 

1. Copy files using filenames from text files with shell script or bash script

How can I do this????

I have the file "lista.txt", and this file have the next information:
     one.txt
     two.txt
     five.txt
    six.txt
-------------------
This files exist in the same directory that "list.txt", I would like to
copy this files to other directory

Using Windows the batch file would be like this:  for /f "tokens=*" %i
in (list.txt) do copy "%i" destdir\

How can I do something like that in REDHAT??

Best Regards

2. letters after the device

3. How to pass a variable from a shell script to another shell script...

4. IP forwarding problem

5. Using a variable instead of creating a file in script

6. Mirroring: Paralel scheduling policy

7. file contents into variable, or global variable from shell script

8. USB printer

9. Create a file with a CGI script under /tmp on the web server ?

10. Passing a variable between shell script and perl script

11. Passing a variable from a Perl script to a shell script

12. Newbie help with Shell scripts (using Korn shell)

13. Running scripts using variables in a parent script