Very Computer

Hi,
I posted here a while back, asking for help on a script to delete
users' CACHE files, the location of these being $HOME/USERS/DEFAULT/CACHE

What I wanted to do, wasn't to delete the CACHE directory, but to remove
the files *within* the directory.

After a lot of help from various folk around here, I came up with this:

#!/bin/sh
LOCATION=USERS/DEFAULT/CACHE/
for LOCATION in `ls /home` ; do
rm /home/$LOCATION/USERS/DEFAULT/CACHE/*
   done

I installed that into /usr/local/bin, ran it, and hey presto, saw
/home go down to 75% in a matter of seconds.

Today, I come in, and find a problem report, saying that /home is
once more at 100%.  I check the disk, and sure enough, it's full.

I find this strange, as the script is in root's crontab, but just
to be sure, I connect, and run the script manually.

I then run a find for any files still in the users' CACHE, and see that
some of the them have not been deleted !!

Could someone shed some light on why a script that worked before,
should suddenly stop working ..?

Thanks.

--
Desmond Coughlan    Network Engineer    Forum des Images    Paris    France
***************************************************************************
The views expressed in these articles are my own, and do not necessarily
reflect the views of the Forum des Images.
***************************************************************************

Do any of the filenames begin with "."?  The * wildcard doesn't match those
filenames, you have to use /home/$LOCATION/USERS/DEFAULT/CACHE/.* to get
them.

--

GTE Internetworking, Powered by BBN, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

[snip]

Quote:> >Could someone shed some light on why a script that worked before,
> >should suddenly stop working ..?
> Do any of the filenames begin with "."?  The * wildcard doesn't match those
> filenames, you have to use /home/$LOCATION/USERS/DEFAULT/CACHE/.* to get
> them.

No, they're all mostly gif files, but the names are in upper case ...

Other than that, whatever I do, the script does not see them ...  :-(

--
Desmond Coughlan    Network Engineer    Forum des Images    Paris    France
***************************************************************************
The views expressed in these articles are my own, and do not necessarily
reflect the views of the Forum des Images.
***************************************************************************

Could it be that the cache files were in use and locked at the time
the rm command ran?  You said that "some" were not removed.  Does this
mean that some were?  If so, you might want to look into other issues
than the script syntax.

 - Matt

--

  AIX and HACMP Certified Specialist    | |  / \ |\| |  \. ,_|    _| --
  / Comments, views, and opinions \     | |_/ ^ \|   | ) | |      \, *_)
  \ are mine alone, not IBM's.    /     |___|/~\_\_|\|__/|_|        \(

This line doesn't do anything useful: you set LOCATION again in the next
line.

Quote:>for LOCATION in `ls /home` ; do
>rm /home/$LOCATION/USERS/DEFAULT/CACHE/*
>   done

Does it produce any errors at all?  You might be running into argument
length limits, but the shell would complain about that.

(A common beginner's mistake is to "shut up" cron jobs by appending
`>/dev/null 2>&1'.  Needless to say, this is almost always a bad idea:
it makes it far too easy to break a script and not discover about it for
months afterwards.  It can be a bit more work to do something sensible
with the output, but it's well worth it in the long run.)

A couple of possible debugging techniques: i) use `sh -x' to see exactly
what commands are executed; ii) replace `rm' with `echo' to ensure that
your glob really does find the files you think it should.

Tim.
--
Tim Goodwin   | "If you don't know what closures are, you probably don't
Leicester, UK | want to know what closures are." -- Larry Wall

Hmm ... would that affect root's ability to hose the cache ?  In any
case, no, as of five minutes ago, when I last tried the script, no
one is in the building, and I get the same result.  :-(

--
Desmond Coughlan    Network Engineer    Forum des Images    Paris    France
***************************************************************************
The views expressed in these articles are my own, and do not necessarily
reflect the views of the Forum des Images.
***************************************************************************

Quote:> This line doesn't do anything useful: you set LOCATION again in the next
> line.
> >for LOCATION in `ls /home` ; do
> >rm /home/$LOCATION/USERS/DEFAULT/CACHE/*
> >   done

Here is the script, as it stands at this time, and as I run it :

#!/bin/sh
cd /home
for LOCATION in `ls /export/home/` ; do
rm /export/home/$LOCATION/USERS/DEFAULT/CACHE/*
done

Quote:> Does it produce any errors at all?  You might be running into argument
> length limits, but the shell would complain about that.

It does.  I get this (snipped) :

rm: cannot remove `/export/home/chailley/USERS/DEFAULT/CACHE/*': No such file or directory
rm: cannot remove `/export/home/chalab_l/USERS/DEFAULT/CACHE/*': No such file or directory
./purge: /bin/rm: Argument list too long

... and so on ...

[snip]

This is now critical, as :

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             1.4G   17M  1.3G   1% /
/dev/sda2             1.9G  388M  1.4G  21% /usr
/dev/sda3             2.4G  244M  2.0G  11% /var
/dev/sda4             2.5G  114k  2.4G   0% /tmp
/dev/sdb2             8.1G  7.6G     0 100% /export
/dev/sdc1             8.2G  6.1G  1.7G  78% /cache

--
Desmond Coughlan    Network Engineer    Forum des Images    Paris    France
***************************************************************************
The views expressed in these articles are my own, and do not necessarily
reflect the views of the Forum des Images.
***************************************************************************

A lucky "guess" :-).

Quote:>rm: cannot remove `/export/home/chailley/USERS/DEFAULT/CACHE/*': No such file or directory

OK, that implies that there are no files at all in that user's cache
directory.  You can silence this error by using `rm -f'.

Quote:>./purge: /bin/rm: Argument list too long

And this means that some user has so many files that they exceed your
system's limit on command line arguments.  The most general solution
to this is to use xargs(1); in this case in combination with find(1).

    #! /bin/sh
    for LOCATION in `ls /export/home/` ; do
            find /export/home/$LOCATION/USERS/DEFAULT/CACHE -type f -print |xargs rm -f
    done

Tim.
--
Tim Goodwin   | "If you don't know what closures are, you probably don't
Leicester, UK | want to know what closures are." -- Larry Wall

I would say to be as close to rm $DIR/* the find command should be kept
from possibly doing recursive searches.

find /export/home/$LOCATION/USERS/DEFAULT/CACHE/* -prune -type f -print |xargs
rm -f

 - Matt

--

  AIX and HACMP Certified Specialist    | |  / \ |\| |  \. ,_|    _| --
  / Comments, views, and opinions \     | |_/ ^ \|   | ) | |      \, *_)
  \ are mine alone, not IBM's.    /     |___|/~\_\_|\|__/|_|        \(

That wildcard will run into the same argument length limits that the
original script had.  You have to *not* use a wildcard to avoid it.

Actually, the following variant of the original script might work OK:

for LOCATION in /export/home/*/USERS/DEFAULT/CACHE; do
  cd $LOCATION
  rm -f *
done

This doesn't have the /export/home/<username>/DEFAULT/CACHE prefix on every
filename, so the total length of the argument list will be significantly
reduced.

--

GTE Internetworking, Powered by BBN, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

Doing a 'find | xargs rm' on directories that users are able to write into
is a Bad Idea, security wise.  You can minimize the exposure somewhat *if*
you have a 'find' *and* 'xargs' that both support the 'trailing null ends
string' scheme (I believe the GNU versions provide this, but I forget the
flag - '-0' maybe?).

Consider a user who does the following:

cd ~/USERS/DEFAULT/CACHE
mkdir 'foobar^J'   (with a trailing linefeed)
cd foobar*
touch vmunix

Find will output this:

USERS/DEFAULT/CACHE/foobar    (note that trailing linefeed ;)
/vmunix

Guess what file you just kissed bye-bye? ;)

You can improve this somewhat by using 'find -flags -exec rm {} \;',
but this is still subject to a race condition. This problem was noted
on the Bugtraq mailing list *way* back on May 21, 1996, and beaten to
death over the following few days.  Given that the use of -exec to get
around the 'find | xargs' hole was widespread in 1996, the original xargs
problem must be even more ancient than that (I seem to remember having seen
it back in the late 80's, but can't prove it...)