Log file rotation - what a nightmare with Apache :-(

Log file rotation - what a nightmare with Apache :-(

Post by Richard Llo » Tue, 04 May 1999 04:00:00



Looking at the Apache 1.3.6 documentation, I see that Apache can accept
three signals: TERM (to kill it completely), HUP (restart now), USR1
(graceful restart).

While these are fine for config tweaks or full starts/stops, the majority
of sites just need a way for Apache to use a new rotated log file (i.e. a
zero-length file that's been created with the same name after the currently
open log file has been moved to another name (e.g. renamed to
access_log.19990503 or whatever), typcially on a daily/weekly/monthly
log rotation basis.

Hence, I propose that Apache accepts another signal (e.g. SIGUSR2) that
does the following:

* Tell httpd children to stop logging (but they should continue Web serving
  as normal) - don't know if you've got a "logging" flag present for the
  children, but you'd need one added if there isn't.
* For each site/virtual host served:
  - Immediately close the currently open log files (typcially access_log and
    error_log).
  - Immediately attempt to re-open log files (I don't know if Apache stores
    the full pathanmes of each log file in memory, but it would need to in
    order to re-open the files without re-reading the server config again
    [which is what the HUP and USR1 signals do already anyway and is "risky"]).
* Tell httpd children to start logging again.

Adding support for this would be excellent - midnight log file rotation would
result in no Web server restarts and only a few log lines dropped because of
the logging stop/start period. As it stands, I have to do a "risky' Web
restart each midnight and it's bitten us on more than one occasion when it's
failed to restart for various reasons.

BTW, before you ask me why I don't use "rotatelogs", it's because we log
our virtual hosts to separate log files (for later analysis by Analog)
and hence we'd have to have hundreds of "rotatelogs" processes running piping
the logs through !

On a personal note, I think Apache should consider building log file rotation
code into the core distribution rather than some kludgy "rotatelogs" program.
This would include directives to determine the filename extension to be
added to the rotated log (e.g. .YYYYDDMM would mean "move to
access_log.19990503" (this would be at 12.00am midnight on the morning
of 4th May 1999 of course ! i.e. it would be "yesterday's date"), plus
directives to determine log file rotation frequency (in
years/months/days/hours) and time (defaulting to midnight - can you really see
anyone needing non-midnight rotation though ? Perhaps only for testing). It
would be nice if it could optionally gzip the rotated log too.

If Apache did all of that, then it wouldn't need that log stop/start signal
that I proposed :-) As in incentive, you can configure Microsoft IIS server
to do this log file rotation automatically - if MS can do it, why can't
Apache ?!


Connect,                     WWW: http://www.connect.org.uk/
Liverpool University,  
Merseyside, England,
Great Britain.


Connect,                     WWW: http://www.connect.org.uk/
Liverpool University,  
Merseyside, England,
Great Britain.

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Alex Brow » Wed, 05 May 1999 04:00:00



> Hence, I propose that Apache accepts another signal (e.g. SIGUSR2) that
> does the following:

Well then, ehy not write it all up yourself, make the necessary amendments to the
documentation, and then submit it to the Apache Group as an addition to the server?

Failing that, appreciate the fact that it does a fantastic job for what is a free
server.

A.

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Tony Finc » Wed, 05 May 1999 04:00:00



>Hence, I propose that Apache accepts another signal (e.g. SIGUSR2) that
>does the following:

>* Tell httpd children to stop logging (but they should continue Web serving
>  as normal) - don't know if you've got a "logging" flag present for the
>  children, but you'd need one added if there isn't.

Losing log information might be regarded as a bad thing.

Quote:>* For each site/virtual host served:
>  - Immediately close the currently open log files (typcially access_log and
>    error_log).
>  - Immediately attempt to re-open log files (I don't know if Apache stores
>    the full pathanmes of each log file in memory, but it would need to in
>    order to re-open the files without re-reading the server config again
>    [which is what the HUP and USR1 signals do already anyway and is "risky"]).
>* Tell httpd children to start logging again.

Apache's child processes currently inherit their log file descriptos
from the parent process, so this mechanism changes the way things work
in a possibly problematic way -- you'd have an FD for each child,
rather than one FD shared by all of them. Note that the parent process
runs as root wheras the children are unprivileged, so existing working
setups may break with this scheme.

Quote:>Adding support for this would be excellent - midnight log file rotation would
>result in no Web server restarts and only a few log lines dropped because of
>the logging stop/start period. As it stands, I have to do a "risky' Web
>restart each midnight and it's bitten us on more than one occasion when it's
>failed to restart for various reasons.

If you change the configuration then it should be tested so that you
are confident a midnight restart will work. If you do that then
SIGUSR1 is adequate.

Quote:>BTW, before you ask me why I don't use "rotatelogs", it's because we log
>our virtual hosts to separate log files (for later analysis by Analog)
>and hence we'd have to have hundreds of "rotatelogs" processes running piping
>the logs through !

We have had problems with large numbers of log file descriptors and so
we just have one access log for each server written to disk via
something like rotatelogs. Behind that there's some complicated stuff
that does reverse DNS lookups, accumulates stats, splits logs per
virtual host, and rewrites them according to the customer's
specification. (We use a mass virtual hosting system -- no
<VirtualHost> sections -- so we have to do something like this.)

Tony.
--

Arthur: "Oh, that sounds better, have you worked out the controls?"
Ford:   "No, we just stopped playing with them."

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Richard Llo » Wed, 05 May 1999 04:00:00




> Well then, ehy not write it all up yourself, make the necessary amendments to
> the documentation, and then submit it to the Apache Group as an addition to
> the server?

Because I have a full-time job that's why :-) It would require several weeks
of sussing out a) how to write a module (it's not easy !), b) writing one,
c) testing it, d) documenting it and e) passing on to the Apache folks to work
in it.

It would be better if someone familiar with the Apache coding wrote this
because it impacts on the whole logging system.


Quote:>If you change the configuration then it should be tested so that you
>are confident a midnight restart will work. If you do that then
>SIGUSR1 is adequate.

You are therefore implying that if we have to change a configuration, we
have to restart the Web server before midnight (i.e. in prime time, unless
you do it from a home Net connection an hour before...), which for a heavily
loaded Web server with a lot of virtual hosts would be unacceptable.
Better that the midnight log rotation doesn't have to re-read the config
*at all*.

Quote:>We have had problems with large numbers of log file descriptors

We've been OK - we just bumped up the maximum number of open files allowed
in the kernel and that does the trick nicely.

Quote:> and so we just have one access log for each server written to disk via
> something like rotatelogs.

I'm not a fan of writing all your virtual hosts' logging to one file myself -
there's always a risk that if that one goes wrong, you don't get any logging
at all..

BTW, the reason I threw in the bit about log file rotation is that many
sites (ours included) seem to have home-brew log file rotation scripts
for their Apache setup, whereas I think a built-in one (that can be
disabled of course) with enough flexibility could cater for 95% of sites
that run Apache out there. "Re-inventing the wheel" seems to be a phrase
that springs to mind...


Connect,                       WWW: http://www.connect.org.uk/
3, Brownlow Street,
Liverpool University,
Merseyside, UK. L69 3GL

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Darren Co » Thu, 06 May 1999 04:00:00


Is there a place where users can propose features they'd like to see
in Apache? This newsgroup seems like a good place, but they often get
"stop moaning, it's free" replies like below. They also might get
missed by the Apache developers who don't read all messages.

Can www.apche.org run php3 scripts? If so, I could create a simple
form that allows people to submit their suggestions, and displays them
on a web page. Alternatively an email address/mailing list that people
send their suggestions to.

Darren

---------------



>> Hence, I propose that Apache accepts another signal (e.g. SIGUSR2) that
>> does the following:
>Well then, ehy not write it all up yourself, make the necessary amendments to the
>documentation, and then submit it to the Apache Group as an addition to the server?
>Failing that, appreciate the fact that it does a fantastic job for what is a free
>server.
>A.

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Marc Slemk » Thu, 06 May 1999 04:00:00



Quote:>Looking at the Apache 1.3.6 documentation, I see that Apache can accept
>three signals: TERM (to kill it completely), HUP (restart now), USR1
>(graceful restart).
>While these are fine for config tweaks or full starts/stops, the majority
>of sites just need a way for Apache to use a new rotated log file (i.e. a
>zero-length file that's been created with the same name after the currently
>open log file has been moved to another name (e.g. renamed to
>access_log.19990503 or whatever), typcially on a daily/weekly/monthly
>log rotation basis.
>Hence, I propose that Apache accepts another signal (e.g. SIGUSR2) that
>does the following:
>* Tell httpd children to stop logging (but they should continue Web serving
>  as normal) - don't know if you've got a "logging" flag present for the
>  children, but you'd need one added if there isn't.
>* For each site/virtual host served:
>  - Immediately close the currently open log files (typcially access_log and
>    error_log).
>  - Immediately attempt to re-open log files (I don't know if Apache stores
>    the full pathanmes of each log file in memory, but it would need to in
>    order to re-open the files without re-reading the server config again
>    [which is what the HUP and USR1 signals do already anyway and is "risky"]).
>* Tell httpd children to start logging again.

You can not do this.  Impossible.  It is essential that the Apache parent
open the logs as the user it is started as, not as the user it runs as.
Otherwise your logs are generaly useless and very subject to all sorts
of tampering.

In any case, why go to all that complexity?  If you want to do
that, simply have all the child processes reopen the logfiles when
given a specific signal.  They have to do it anyway to make your scheme
work, and trying to add some magical logging flag that somehow gets
set in the children just complicates things for no reason.

Quote:>Adding support for this would be excellent - midnight log file rotation would
>result in no Web server restarts and only a few log lines dropped because of
>the logging stop/start period. As it stands, I have to do a "risky' Web
>restart each midnight and it's bitten us on more than one occasion when it's
>failed to restart for various reasons.

Then you need to fix your restart procedure.  If you are wildly deploying
changes to config files that may break on restart without ensuring they
work when deployed, you are setting yourself up for a big fall whenever
it is restarted.  Seriously, it is very bad practice to have deployed
config changes that are just sitting there waiting for something completely
unrelated to happen that will end up breaking your entire server for
non-obvious reasons.

It really doesn't make sense to build a lot of complex functionality
into the server when each person will want it to work slightly
differently and it can be done just fine outside of the server.

Quote:>BTW, before you ask me why I don't use "rotatelogs", it's because we log
>our virtual hosts to separate log files (for later analysis by Analog)
>and hence we'd have to have hundreds of "rotatelogs" processes running piping
>the logs through !

[...]

Quote:>If Apache did all of that, then it wouldn't need that log stop/start signal
>that I proposed :-) As in incentive, you can configure Microsoft IIS server
>to do this log file rotation automatically - if MS can do it, why can't
>Apache ?!

Because Microsoft thinks that everything needs to be done by a
monolithic monster that is horribly complex, prone to failure, and almost
impossible to understand.

"Microsoft does it so everyone else needs to" is really one of the worst
arguments to use to try to argue a technical point.

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Alex Brow » Thu, 06 May 1999 04:00:00





> > Well then, ehy not write it all up yourself, make the necessary amendments to
> > the documentation, and then submit it to the Apache Group as an addition to
> > the server?

> Because I have a full-time job that's why :-)

And how well do I know you Richard?  Hmmm, perhaps I wasn't the man to tell that to
...

Quote:> It would require several weeks
> of sussing out a) how to write a module (it's not easy !),

Well, a colleague has just started from scratch with far less programming ability
than yourself and is getting on top of it after a fortnight, so until you try ...

Quote:> >We have had problems with large numbers of log file descriptors

> We've been OK - we just bumped up the maximum number of open files allowed
> in the kernel and that does the trick nicely.

I'm trying to solve nightmare problems with file descriptors at the moment, which

Quote:> > and so we just have one access log for each server written to disk via
> > something like rotatelogs.

> I'm not a fan of writing all your virtual hosts' logging to one file myself -
> there's always a risk that if that one goes wrong, you don't get any logging
> at all..

which is why you have to be _VERY_ careful that you get it right, presumably, and
have it independently read through and tested by colleagues each time.

A.

--
Alex Brown
Webmaster, Cable Internet

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Richard Llo » Fri, 07 May 1999 04:00:00




> Then you need to fix your restart procedure.  If you are wildly deploying
> changes to config files that may break on restart without ensuring they
> work when deployed, you are setting yourself up for a big fall whenever
> it is restarted.

I think my point was that I don't want a restart of the Web server (i.e.
a config re-read) "automatically" at midnight. This is because, on our
platform (HP-UX), we're prone to a "can't bind to port 80" problem, that
even a hard restart doesn't fix (the server needs a reboot to clear the
problem !). This is what "bites" us during midnight restarts, not the
change in the config.

Quote:> It really doesn't make sense to build a lot of complex functionality
> into the server when each person will want it to work slightly
> differently and it can be done just fine outside of the server.

As far as log file rotation goes, if you allow the rotation time,
the frequency, the filename extension of the rotated log (to allow the
time/date/month/year to be included) and the optional compression to be
specified, then that probably covers 95% of users needs (I strongly dispute
that "each person will want to it work slightly differently" - this is
what configuration options are for !).  The only thing I can think of in
addition to all of this is possible further processing on the rotated logs -
the most common being "logresolve" (i.e. substituting numeric IPs with their
named equivalent, but this is indeed something that can be done separately
from log rotation and shouldn't be built into Apache) and maybe log purging
(again, a simple "find . -type f -mtime +31 -exec rm -f {} \;" command is
good enough, so you don't need that in Apache either).

Quote:> Because Microsoft thinks that everything needs to be done by a
> monolithic monster that is horribly complex, prone to failure, and almost
> impossible to understand.

Microsoft get almost everything wrong, but building in log file rotation
into the server was actually something they got right for once :-)


Connect,                       WWW: http://www.connect.org.uk/
3, Brownlow Street,
Liverpool University,
Merseyside, UK. L69 3GL

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Andre » Fri, 07 May 1999 04:00:00


Try this, it works:

mv /path/to/old/log/file /path/to/new/log/file
touch /path/to/new/log/file
kill -HUP <pid of apache>

the last line can be done using the file that contains the pid of apache...

eg.

kill -HUP `cat /path/to/httpd.pid`

or if you wanted, you could grep through your ps aux (althogh the previous
is better)

kill -HUP `ps aux | grep httpd | tail -1`

Oh the power! :)

You can get past without HUP'ing Apache...  It is needed.

It shouldn't hurt you. If you have problems restarting than it is likely
your .conf files are screwed and you would have had problems down the line
anyways. Fix 'em now while you can.

Good luck with it!

Andrew

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Richard Llo » Sat, 08 May 1999 04:00:00




> Try this, it works:

> mv /path/to/old/log/file /path/to/new/log/file
> touch /path/to/new/log/file
> kill -HUP <pid of apache>

That's the "restart now" signal (as opposed to kill -USR1, which is the
"graceful restart" signal) and it's what we've determined is the "safest"
midnight restart when rotating logs. A full stop/start (kill -TERM and then
re-run the httpd from scratch) causes occasional "can't bind to port 80"
problems for us on HP-UX [requiring a server reboot to clear the condition]
and we found that "kill -USR1" takes "too long" (up the length of your
timeout param to stop using the old config/httpd's).

The *big* problem with using "kill -HUP" is that for a config with a large
number of virtual hosts like ours, it can literally take 2 minutes or so
for even the -HUP option to restart the Web server. This is why I'm not
keen on anything that has to re-read the config from scratch when rotating
logs at midnight.

Quote:> It shouldn't hurt you. If you have problems restarting than it is likely
> your .conf files are screwed and you would have had problems down the line
> anyways. Fix 'em now while you can.

Nope, as I said, it's not the conf files - it's the "can't bind to port 80"
problem and having the Web server not serving for up to 2 minutes at around
midnight because of (over 500 !) virtual hosts. This is why I want a signal
that just closes and opens the log files and doesn't nothing else - this
would be a matter of seconds instead of minutes and also shouldn't need to
drop any existing connections (i.e. should be completely unnoticeable to
the outside world, unlike the -HUP option).


Connect,                       WWW: http://www.connect.org.uk/
3, Brownlow Street,
Liverpool University,
Merseyside, UK. L69 3GL

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Alan J. Flavel » Sat, 08 May 1999 04:00:00



> The *big* problem with using "kill -HUP" is that for a config with a large
> number of virtual hosts like ours, it can literally take 2 minutes or so
> for even the -HUP option to restart the Web server.

Perhaps you want to consider moving to the "mass virtual hosting"
configuration option.  Of course, there are some issues with that, like
not having separate log files: you have to assess your best compromise.
 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Andre » Sun, 09 May 1999 04:00:00


You could always start multiple daemons, each with their own minimalist
config file....

Your dog, your choice.

Quote:> The *big* problem with using "kill -HUP" is that for a config with a large
> number of virtual hosts like ours, it can literally take 2 minutes or so
> for even the -HUP option to restart the Web server.

 
 
 

Log file rotation - what a nightmare with Apache :-(

Post by Kai Schaetz » Tue, 11 May 1999 04:00:00


I have been wondering all time back why people use rotatelogs or the Apache
documentation recommends not just "deleting" log files with an rm -f but
killing and restarting httpd.
I'm doing our daily log rotation with a small shell script which simply moves
the log files to a new location, starts the stats program, then after return
renames the old daily logs with a timestamp and gzips them up. Each httpd
serves at least 100 virtual hosts. No problems at all with this setup for
more than a year now. The server creates new "0" bytes logs for all virtual
hosts with the first request to any of the virtual hosts.

Kai

--

Kai Sch?tzl, Berlin, Germany
IE-Center: http://ie5.de & http://msie.winware.org
ClubWin - Help for Windows Users: http://www.clubwin.com

 
 
 

1. Apache 1.3.3 log file rotation leaves open file descriptors

In Apache 1.3.3 (Solaris) we are using the USR1 signal (or HUP) to rotate the
log files daily.

Our rotation strategy is:
1) rename log file
2) issue USR1

What appears to be happening though, is that Apache is not closing the file
descriptors for the old log files. (We eventually run out of file descriptors -
in just over a month)  and lsof shows that Apache still has all the old log
files open.

Anyone else noticed this behaviour?

Thanks,
Mark Hume

2. USB ZIP 100 to extract data?

3. ANNOUNCE: an alternative log file rotation program for Apache

4. Ultra2 SCSI?

5. Log Rotation for suexec log file

6. /etc/system settings

7. Apache log rotation script

8. wu-ftpd: anonymous users logged off after password

9. Apache log rotation

10. Apache - Log Rotation ?!

11. Apache & log rotation

12. Apache 1.3 log rotation and multiple users

13. Piped Log Rotations in apache