:> I have a very large file, our web server access_log which is around 320 MB
:> and another one which is around 290 MB. I need to erase the first
:> few (around 1 million) lines of the top of the file. I tried using vi,
:> but it gave me an error message and crashed. I tried to use a few other
:> editors like, pico or jot and they did the same thing. I was wondering if
:> anyone has any suggestion how to edit these large files.
:This will keep just the last 100000 lines of the file:
:last -100000 filename >/path/to/new_file
:mv /path/to/new_file filename
I would modify that slightly to
last -100000 filename >/path/to/new_file
cat /path/to/new_file > filename
as this will preserve the ownership and permissions, where Art's version
However, I would also worry about locking. If there is any chance that
the server is going to be writting new records while you are working with
the file, then you must take a different approach in order to not
lose the newest records. You must either stop the server, or else convince
it to start writing to a new log file on the fly.
Unfortunately the NCSA 1.4 server doesn't offer a way to change the
log file on the fly, so I usually stop the server. [I haven't had time
to install the 1.5 server.] I haven't ever looked to see if the
netscape servers can handle starting a new file.
A number of applications will close and re-open their log file if you
send them a HUP signal. If your WWW server will do that, then you can [e.g.]
mv filename /path/to/new_file/on/same/filesystem
killall -HUP server_name
This works because when you 'mv' on the same filesystem, only the
directory entry changes, and open connections are preserved.
The process will continue to write to the now-renamed file until
it is told otherwise. Then when it gets the HUP signal, it will close
that file and open 'filename' again and write to that... but that's
now a different file than before.
Note: Not all programs are set up to use the HUP signal in this manner!
Generally speaking, programs that are not set up to treat HUP this way,
will die when sent a HUP signal -- so don't do your experimentation
during a mission-critical time!