Very Computer

I need to set up some new log reporting for a server which
handles about 4 million hits a month. Logs are currently kept
in sets of 2 days of activity.

and doing dns lookups would take forever to do on a routine
basis without some sort of effective cacheing.

any suggestions for "big" logs..I'm getting ready to look at mkstats
and something called analog i think..

Sonny

: any suggestions for "big" logs..I'm getting ready to look at mkstats
: and something called analog i think..

As the author of MKStats, I'll tell you that you'll probably run into
problems trying to run my program on very large log files.  It wasn't
designed from the beginning to handle huge log files, because it uses
perl and stores a lot of information in memory.

The advantage is that you get more information about your usage than
anything other program can offer you.  The disadvantage is that it's a
memory hog :)

I'm working on a way to avoid this problem and use a custom database
method of storing the information.  If I could guarantee that everyone
was running SQL server, this would be easy :)
But unfortunately, all I can assume is that the user is running perl, and
I want to keep the program completely cross-platform and independent of
other programs or databases.  So it makes it a little harder :)

Analog is very fast, and will probably handle your amount of info easily,
but won't offer the same kind of information as mkstats.  Other than
that, your best bet is to hook into a database and use a program that can
use that info.

--
Matt Kruse

---------------------------+--------------------------------------------
 MKStats: WWW log analysis |        "First try to Understand.
      www.mkstats.com      |         Then try to be Understood."

-----BEGIN PGP SIGNED MESSAGE-----

Nothing above this line is part of the signed message.

Benjamin Franz

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQCVAwUBMooauOjpikN3V52xAQHpDQQAg66sR/80Boq1IKdaWmJCRXbyTKy0g65I
GFpfQeT2gd83GAI9kTkja+jpaEtO/4XXBv7wlfw+2stEW7M/hyeduOu4kB3RRN4c
CMlASv5jmgH0odBh1qjeUIsAl2vUBhZUJVjdX9BVP2E9OO1j0QN3cEBupKVKsGoN
dV3tDFbReos=
=GqbZ
-----END PGP SIGNATURE-----

I always run logresolve first then analog, mkstats or wwwstats on the
result file, depending on which my customer prefers.  I find that though
analog is screamingly fast, but the detail of mkstats or wwwstats is
really important to clients who want to know what the usage patterns
are.

--
Jay Thorne                http://net.result.com/
President, The Net Result Systems * Services  Telephone:(604) 220 2504
WWW & Internet Systems Consultant.

I've used Analog on our log files.  We get about the same activity
that you describe, and run reports monthly.  Analog is quite swift
in its processing.

Analog, BTW, does cache DNS records although I no longer recall the
details involved.

        - Andrew

---
 -----------------------------------------------------------
| Andrew Gideon              |   TAG Online inc.            |
| Consultant                 |   539 Valley Road            |
|                            |   Upper Montclair, N.J.      |
| Tel: (201) 783-5583        |                     07043    |
| Fax: (201) 783-5334        |                              |

 -----------------------------------------------------------

Quote:>any suggestions for "big" logs..I'm getting ready to look at mkstats
>and something called analog i think..

Cheers!
Greg

--

Web Services Analyst, UNIX Delivery Sys & Services     |  te kea, ka pai.
Boeing Information and Support Services                |
Disclaimer: My opinions are not necessarily the same as Boeing's.

|> [...]
|> >
|> >Analog is very fast, and will probably handle your amount of info easily,
|> >but won't offer the same kind of information as mkstats.  Other than
|> >that, your best bet is to hook into a database and use a program that can
|> >use that info.
|> >
|> [...]
|>
|> I've used Analog on our log files.  We get about the same activity
|> that you describe, and run reports monthly.  Analog is quite swift
|> in its processing.
|>
|> Analog, BTW, does cache DNS records although I no longer recall the
|> details involved.
|>
Yes, indeed it does, but it's a very slow hash algorithm used inside
of analog.
So we at EUnet Germany are using the 'logresolve' utility coming with
apache to resolve the logs daily and then generating a daily / weekly
and/or monthly report with analog from this saved log files.

Jan
--

Dipl.-Inform. Jan Wedekind              
Emil-Figge-Str. 80, D-44227 Dortmund    The opinions in this mail / article
Tel/Fax +49-231-972 -00/-1188           are my own and not those of EUnet GmbH.

By the way, I'm somewhat surprised that Jan Wedekind described analog's hash
algorithm as "very slow", but I'm always open to suggestions for improvement
if anyone has better ideas.

--

  Stochastic Networks Group, Statistical Laboratory,
  16 Mill Lane, Cambridge, CB2 1SB, England    Tel.: +44 1223 337955
  "Collection of rent is subject to Compulsive Competitive Tendering" Cam. City

Quote:>>any suggestions for "big" logs..I'm getting ready to look at mkstats
>>and something called analog i think..
>Analog seems to be the best program for running through huge numbers of
>hits.  I've abandoned everything else.

Greetz...

Luuk

:I need to set up some new log reporting for a server which
:handles about 4 million hits a month. Logs are currently kept
:in sets of 2 days of activity.
:
:and doing dns lookups would take forever to do on a routine
:basis without some sort of effective cacheing.
:
:any suggestions for "big" logs..I'm getting ready to look at mkstats
:and something called analog i think..

We have about 250 Virtual hosts.

I use apache's custom log format to include '%v' - the virtual host
of the server - and the referer in a single large log file.

I do not have apache resolve the logs - I post-process them with
logresolve.

The key turned out to be to do the following :-

1. grep out each Virtual host to its own file
2. run logresolve on this (good locality compared to the single composite
   log). Great speedups by taking this step.
3. run analog on the result.

Cheers,       Andy!

#! /bin/sh
#
# process http logs
#
# daily script run as the www server
#

#         http://www.veryComputer.com/
# For:
#         Rocky Mountain Internet http://www.veryComputer.com/
#
# May safely be run many times, but not around the witching hour
# of root's rotation ..
#

GZIP=/usr/local/bin/gzip
BINDIR=/www/apache-ssl
LOGDIR=/var/log/apache-ssl-logs

#
# root turns the log over to this file - now static
#

LASTLOG=$LOGDIR/access_old

# Backups, gzipped
#

ARCHIVE=/var/log/old-apache-logs

#
# site root directories are assumed to be of the form
#
# $SITEROOT/www.wizzy.com/index.html
#

SITEROOT=/www

# Roll time back 12 hours to get the month yesterday ..
monthday=`TZ=GMT+19 date +%b-%d`
month=`TZ=GMT+19 date +%b`

onevirtualsite() {
    SITE=$1
    SITEDIR=$SITEROOT/$SITE/statistics
    [ ! -d $SITEDIR ] && return                 # outta here
    [ ! -f $SITEDIR/analog.conf ] && return     # outta here

    LOG=$SITEDIR/$month.html

    grep " $SITE " $LASTLOG | \
        $BINDIR/logresolve | \
        $GZIP --stdout > $SITEDIR/access_log.$monthday.gz

    $GZIP --decompress --stdout $SITEDIR/access_log.${month}-??.gz | \
        $BINDIR/analog +g$SITEDIR/analog.conf \
            +C"REFEXCLUDE http://$SITE/*" - > $LOG.new

    mv -f $LOG.new $LOG

Quote:}

$GZIP --stdout $LASTLOG > $ARCHIVE/access_log.${monthday}.gz

# virtual IPs

cd $SITEROOT

for d www.* ; do
  onevirtualsite $d
done

exit 0