Very Computer

by **Frank Gec** » Fri, 01 Feb 2002 00:35:39

I was wondering what is considered the best technique for determining if
you have new files in a directory as I can think of a couple but they
don't seem that elegant. I recall that there as a real good way to do
this but I have forgotten (signals maybe?). What have others done?

Some idea's are:
1. Every so often(use timer/sleep) get the modified date on all files in
a directory keeping track of the last time you did it and process the
ones with a greater date (use stat or directly read the directory
file?).
2. Instead of a program(C/C++) use a shell to do the same thing.

by **John Gordon,217-352-6511x7418,CEERD-CF-** » Fri, 01 Feb 2002 00:50:45

> I was wondering what is considered the best technique for determining if
> you have new files in a directory as I can think of a couple but they
> don't seem that elegant. I recall that there as a real good way to do
> this but I have forgotten (signals maybe?). What have others done?

there is no general-purpose way of detecting new files.

when you say "new file", what do you mean? a file that wasn't there
before, or an existing file that has been updated?

Quote:> Some idea's are:
> 1. Every so often(use timer/sleep) get the modified date on all files in
> a directory keeping track of the last time you did it and process the
> ones with a greater date (use stat or directly read the directory
> file?).
> 2. Instead of a program(C/C++) use a shell to do the same thing.

these would work if you mean "new file" in the second sense above.

---
John Gordon "No Silicon Heaven? Preposterous! Where would

by **Frank Gec** » Fri, 01 Feb 2002 01:14:56

A new file/one could be over written but not added too. The files will be
placed in the directory via ftp, rcp etc. Do I need to worry about locking as I
could process an incomplete file if it is still in transit.

> > I was wondering what is considered the best technique for determining if
> > you have new files in a directory as I can think of a couple but they
> > don't seem that elegant. I recall that there as a real good way to do
> > this but I have forgotten (signals maybe?). What have others done?

> there is no general-purpose way of detecting new files.

> when you say "new file", what do you mean? a file that wasn't there
> before, or an existing file that has been updated?

> > Some idea's are:
> > 1. Every so often(use timer/sleep) get the modified date on all files in
> > a directory keeping track of the last time you did it and process the
> > ones with a greater date (use stat or directly read the directory
> > file?).
> > 2. Instead of a program(C/C++) use a shell to do the same thing.

> these would work if you mean "new file" in the second sense above.

> ---
> John Gordon "No Silicon Heaven? Preposterous! Where would

by **John Gordon,217-352-6511x7418,CEERD-CF-** » Fri, 01 Feb 2002 02:28:06

> A new file/one could be over written but not added too. The files will
> be placed in the directory via ftp, rcp etc. Do I need to worry about
> locking as I could process an incomplete file if it is still in
> transit.

yes, you must worry about processing an incomplete file. i don't think
standard file locking will work for you -- files incoming via FTP or
SCP aren't locked, are they?

you have two options here:

1. do you have control of the process that writes the files? if so,
write the file to a temporary name and rename the file to its "real"
name when finished. this way, whenever a file exists with the "real"
name, you know it is finished writing. (if you can't rename files,
write a second dummy file after the first file. if this dummy file
exists, you know the first file is finished.)

2. if you don't have control of the writing process, then your best bet
is to check the timestamp of the file, and assume that files older
than, say, thirty seconds are finished writing.

---
John Gordon "No Silicon Heaven? Preposterous! Where would

by **Ed L Cashi** » Fri, 01 Feb 2002 05:13:45

> > I was wondering what is considered the best technique for determining if
> > you have new files in a directory as I can think of a couple but they
> > don't seem that elegant. I recall that there as a real good way to do
> > this but I have forgotten (signals maybe?). What have others done?

> there is no general-purpose way of detecting new files.

There's fam, though.

--
--Ed L Cashin | PGP public key:

by **Frank Gec** » Fri, 01 Feb 2002 05:46:01

fam? Not familiar with it. Could not find any info/man page.

Thanks,

Frank

> > > I was wondering what is considered the best technique for determining if
> > > you have new files in a directory as I can think of a couple but they
> > > don't seem that elegant. I recall that there as a real good way to do
> > > this but I have forgotten (signals maybe?). What have others done?

> > there is no general-purpose way of detecting new files.

> There's fam, though.

> --
> --Ed L Cashin | PGP public key:

by **Ed L Cashi** » Sat, 02 Feb 2002 09:18:32

[on top, moved to bottom]

> > > > I was wondering what is considered the best technique for determining if
> > > > you have new files in a directory as I can think of a couple but they
> > > > don't seem that elegant. I recall that there as a real good way to do
> > > > this but I have forgotten (signals maybe?). What have others done?

> > > there is no general-purpose way of detecting new files.

> > There's fam, though.
> fam? Not familiar with it. Could not find any info/man page.

Sorry for the late follow up. It is the file alteration monitor (I
think), and it was a nice feature in IRIX. SGI started to move toward
Linux, and now there's some fam stuff for linux.

The idea is that the kernel provides some kind of callback facility
and a fam deamon accepts monitoring requests from interested
processes.

e.g., a file manager GUI thing might say to the fam daemon, "Hey, I
want to know if /home/joe changes!" The fam daemon says to
the kernel, "Let me know if directory /home/joe changes." Joe backs
up his .bashrc and the kernel notifies famd, which notifies the GUI.

--
--Ed L Cashin | PGP public key:

by **Andy Isaacs** » Sat, 02 Feb 2002 10:49:40

>Sorry for the late follow up. It is the file alteration monitor (I
>think), and it was a nice feature in IRIX. SGI started to move toward
>Linux, and now there's some fam stuff for linux.

>The idea is that the kernel provides some kind of callback facility
>and a fam deamon accepts monitoring requests from interested
>processes.

>e.g., a file manager GUI thing might say to the fam daemon, "Hey, I
>want to know if /home/joe changes!" The fam daemon says to
>the kernel, "Let me know if directory /home/joe changes." Joe backs
>up his .bashrc and the kernel notifies famd, which notifies the GUI.

FreeBSD does something along these lines using kernel event queues.
http://www.FreeBSD.org/cgi/man.cgi?query=kevent
http://www.freebsd.org/cgi/cvsweb.cgi/src/usr.bin/tail/forward.c?rev=...

I believe the other BSDs have them as well.

-andy

Very Computer

Checking for new files in a directory

Checking for new files in a directory

Checking for new files in a directory

Checking for new files in a directory

Checking for new files in a directory

Checking for new files in a directory

Checking for new files in a directory

Checking for new files in a directory

Checking for new files in a directory