Passive http traffic logger using TCPDUMP

Passive http traffic logger using TCPDUMP

Post by Ia » Mon, 03 Apr 2000 05:00:00

 I run a medium sized Windows NT network; my users are sharing
internet access through a 10Mb gateway via a switch.  I would like to
log the various web sites that my users access by username in a SQL
database, but I don't want to setup a proxy server to gather the
needed information.
  I recently set up a MS proxy server (without separating the network)
which is logging all proxy requests to an ODBC compliant database.  So
far it has worked.  The only problem I am having is that users can
choose weather or not they want to go through the proxy or straight
through to the Internet.
  I have been using TCPDUMP to gather HTTP GET, POST etc. requests
using the following perl script:


$LIMIT = shift || 5000;

open (STDIN,"/usr/sbin/tcpdump -lnx -s 1024 dst port 80 |");
while (<>) {
    if (/^\S/) {
        last unless $LIMIT--;
        while ($packet=~/(GET|POST|).+/g)  {
            print "$client -> $host\t$&\n";
        undef $client; undef $host; undef $packet;
        ($client,$host) = /(\d+\.\d+\.\d+\.\d+).+ >
            if /P \d+:\d+\((\d+)\)/ && $1 > 0;
    next unless $client && $host;
    $packet .= $_;


The problems that I am running into are:

1.  Works great for real-time gathering but I am need of piping the
STDOUT into a database using MS Access as the front end
2. Using MS Proxy, the domain\username is attached to the website that
he/she visited.  I cannot do this with the script. I only get the
source IP.
3.  Using MS Proxy, not only is the destination IP logged, but so is
the FQDN e.g. --> as opposed to -->
4.  I don't want to physically separate the Internet from the network
in order to log HTTP traffic.
5. I don't like having to deal with licensing and maintaining a MS SQL
database/proxy server/server

  I need some insight on where to go next.  How can I get
domain\username to be attached to the FQDN and URI of the site visited
and logged in a SQL database?

If I type nbtstat -a <ip_addr> on a NT machine, I get the NetBIOS name
of the Domain, workstation (HEX03) and username (HEX06) currently on
the workstation at the time of the transfer.  Is there a similiar
command using Linux that I can include in the above script?

Which database server should I use?  I was thinking about MySQL.  What
would be the best way to port the script's data into the database's
table?  Can I use MS Access as a front end to a MySQL database?

Or am I going about this the wrong way?