Very Computer

Howdy all!!  First off let me tell you that I'm new to shell
scripting.  I have been trying to get the log file from IPTraff to
give me such results as total bytes and bps sent from and to our
dialup & adsl customers as well as our web hosting servers based on
TCP protocols. I'm using the ip_traffic log produced from IPTraf that
looks like this:

Fri May 10 19:18:40 2002; ******** IP traffic monitor started ********
Fri May 10 19:18:40 2002; TCP; eth0; 1500 bytes; from
xxx.xxx.xxx.208:47727 to xxx.xxx.xxx176:25; first packet
Fri May 10 19:18:40 2002; TCP; eth0; 49 bytes; from
xxx.xxx.xxx.162:110 to xxx.xxx.xxx.99:1029; first packet
Fri May 10 19:18:40 2002; TCP; eth0; 40 bytes; from xxx.xxx.xxx.176:25
to xxx.xxx.xxx.208:47727; first packet
Fri May 10 19:18:40 2002; TCP; eth0; 1500 bytes; from
xxx.xxx.xxx.139:80 to xxx.xxx.xxx.233:2162; first packet

The script that I have been working with uses grep and awk since it's
faster that the loops that I started out with even though there are a
lot of lines.  (I'll post it below.)

My problem is I can get through the script without errors, but the
results are wacky.  Says that the in and out bytes are the same.  If
someone could look this over and see what they think that would be
greatly appreciated.

Another question if you don't mind.  The standard redirect "cat file1
| awk something > file2" seems to be deleting the info in file1.  Is
that what sould be happening?  My could worker says not, but ever time
I have run the thing it does.

Again thanks for all you help.
JIM

######################################!/bin/sh

CFGDIR=/usr/local/iptraf/config
DATADIR=/usr/local/iptraf/data
DBDIR=/usr/local/iptraf/db
WEBDIR=/usr/local/iptraf/www
LOGDIR=/var/log/iptraf
DATE=`/bin/date`

#make sure log is present
if test -f $LOGDIR/ip_traffic.log; then
        :
else
        exit
fi

#create data file for tcp/udp services
/bin/cat $LOGDIR/ip_traffic.log | /bin/sed -e "s/;//g" | /bin/sed -e
"s/:/ /g" | awk '{print $10,$13,$14,$16,$17}' > $LOGDIR/ip_trafficdata

test -f $DATADIR/rentalservers && /bin/rm -f $DATADIR/rentalserversin
test -f $DATADIR/dialup && /bin/rm -f $DATADIR/dialupin
test -f $DATADIR/adsl && /bin/rm -f $DATADIR/adslin
test -f $DATADIR/rentalservers && /bin/rm -f $DATADIR/rentalserversout
test -f $DATADIR/dialup && /bin/rm -f $DATADIR/dialupout
test -f $DATADIR/adsl && /bin/rm -f $DATADIR/adslout

test -f $DATADIR/rentalservershttpout && /bin/rm -f
$DATADIR/rentalservershttpout
test -f $DATADIR/rentalserversftpout && /bin/rm -f
$DATADIR/rentalserversftpout
test -f $DATADIR/rentalserversmailout && /bin/rm -f
$DATADIR/rentalserversmailout
test -f $DATADIR/rentalservershttpin && /bin/rm -f
$DATADIR/rentalservershttpin
test -f $DATADIR/rentalserversftpin && /bin/rm -f
$DATADIR/rentalserversftpin
test -f $DATADIR/rentalserversmailin && /bin/rm -f
$DATADIR/rentalserversmailin
test -f $DATADIR/dialuphttpout && /bin/rm -f $DATADIR/dialuphttpout
test -f $DATADIR/dialupftpout && /bin/rm -f $DATADIR/dialupftpout
test -f $DATADIR/dialupmailout && /bin/rm -f $DATADIR/dialupmailout
test -f $DATADIR/dialuphttpin && /bin/rm -f $DATADIR/dialuphttpin
test -f $DATADIR/dialupftpin && /bin/rm -f $DATADIR/dialupftpin
test -f $DATADIR/dialupmailin && /bin/rm -f $DATADIR/dialupmailin
test -f $DATADIR/adslhttpout && /bin/rm -f $DATADIR/adslhttpout
test -f $DATADIR/adslftpout && /bin/rm -f $DATADIR/adslftpout
test -f $DATADIR/adslmailout && /bin/rm -f $DATADIR/adslmailout
test -f $DATADIR/adslhttpin && /bin/rm -f $DATADIR/adslhttpin
test -f $DATADIR/adslftpin && /bin/rm -f $DATADIR/adslftpin
test -f $DATADIR/adslmailin && /bin/rm -f $DATADIR/adslmailin

/bin/touch $DATADIR/rentalserversout
/bin/touch $DATADIR/rentalserversin
/bin/touch $DATADIR/dialupout
/bin/touch $DATADIR/dialupin
/bin/touch $DATADIR/adslout
/bin/touch $DATADIR/adslin

/bin/touch $DATADIR/rentalservershttpout
/bin/touch $DATADIR/rentalserversftpout
/bin/touch $DATADIR/rentalserversmailout
/bin/touch $DATADIR/rentalservershttpin
/bin/touch $DATADIR/rentalserversftpin
/bin/touch $DATADIR/rentalserversmailin
/bin/touch $DATADIR/dialuphttpout
/bin/touch $DATADIR/dialupftpout
/bin/touch $DATADIR/dialupmailout
/bin/touch $DATADIR/dialuphttpin
/bin/touch $DATADIR/dialupftpin
/bin/touch $DATADIR/dialupmailin
/bin/touch $DATADIR/adslhttpout
/bin/touch $DATADIR/adslftpout
/bin/touch $DATADIR/adslmailout
/bin/touch $DATADIR/adslhttpin
/bin/touch $DATADIR/adslftpin
/bin/touch $DATADIR/adslmailin

if test -f $LOGDIR/ip_trafficdata; then
        RS1=`cat $CFGDIR/rentalservers`
        for RS2 in $RS1
                do
                cat $LOGDIR/ip_trafficdata | grep -w $RS2 > $DATADIR/rentalservers
                cat $DATADIR/rentalservers | awk '{if($2=RA) print $0}' RA=$RS2 >>
$DATADIR/rentalserversout
                cat $DATADIR/rentalservers | awk '{if($4=RA) print $0}' RA=$RS2 >>
$DATADIR/rentalserversin
                done
/bin/echo "done pulling out rs data"

        DU1=`cat $CFGDIR/dialup`
        for DU2 in $DU1
                do
                cat $LOGDIR/ip_trafficdata | grep -w $DU2 > $DATADIR/dialup
                cat $DATADIR/dialup | awk '{if($2=DA) print $0}' DA=$DU2 >>
$DATADIR/dialupout
                cat $DATADIR/dialup | awk '{if($4=DA) print $0}' DA=$DU2 >>
$DATADIR/dialupin
                done
/bin/echo "done pulling out du data"
        ADSL1=`cat $CFGDIR/adsl`
        for ADSL2 in $ADSL1
                do
                cat $LOGDIR/ip_trafficdata | grep -w $ADSL2 > $DATADIR/adsl
                cat $DATADIR/adsl | awk '{if($2=DSL) print $0}' DSL=$ADSL2 >>
$DATADIR/adslout
                cat $DATADIR/adsl | awk '{if($4=DSL) print $0}' DSL=$ADSL2 >>
$DATADIR/adslin
                done
/bin/echo "done pulling out adsl data"

                cat $DATADIR/rentalserversin | grep -w 20 >
$DATADIR/rentalserversftp
                cat $DATADIR/rentalserversftp | awk '{if($3=="20" || $5=="20") print
$0}' >> $DATADIR/rentalserversftpin
                cat $DATADIR/rentalserversout | grep -w 20 >
$DATADIR/rentalserversftp1
                cat $DATADIR/rentalserversftp1 | awk '{if($3=="20" || $5=="20")
print $0}' >> $DATADIR/rentalserversftpout
                cat $DATADIR/rentalserversin | grep -w 21 >
$DATADIR/rentalserversftp
                cat $DATADIR/rentalserversftp | awk '{if($3=="21" || $5=="21") print
$0}' >> $DATADIR/rentalserversftpin
                cat $DATADIR/rentalserversout | grep -w 21 >
$DATADIR/rentalserversftp1
                cat $DATADIR/rentalserversftp1 | awk '{if($3=="21" || $5=="21")
print $0}' >> $DATADIR/rentalserversftpout
                cat $DATADIR/rentalserversin | grep -w 25 >
$DATADIR/rentalserversmail
                cat $DATADIR/rentalserversmail | awk '{if($3=="25" || $5=="25")
print $0}' >> $DATADIR/rentalserversmailin
                cat $DATADIR/rentalserversout | grep -w 25 >
$DATADIR/rentalserversmail1
                cat $DATADIR/rentalserversmail1 | awk '{if($3=="25" || $5=="25")
print $0}' >> $DATADIR/rentalserversmailout
                cat $DATADIR/rentalserversin | grep -w 80 >
$DATADIR/rentalservershttp
                cat $DATADIR/rentalservershttp | awk '{if($3=="80" || $5=="80")
print $0}' >> $DATADIR/rentalservershttpin
                cat $DATADIR/rentalserversout | grep -w 80 >
$DATADIR/rentalservershttp1
                cat $DATADIR/rentalservershttp1 | awk '{if($3=="80" || $5=="80")
print $0}' >> $DATADIR/rentalservershttpout
                cat $DATADIR/rentalserversin | grep -w 110 >
$DATADIR/rentalserversmail
                cat $DATADIR/rentalserversmail | awk '{if($3=="110" || $5=="110")
print $0}' >> $DATADIR/rentalserversmailin
                cat $DATADIR/rentalserversout | grep -w 110 >
$DATADIR/rentalserversmail1
                cat $DATADIR/rentalserversmail1 | awk '{if($3=="110" || $5=="110")
print $0}' >> $DATADIR/rentalserversmailout
/bin/echo "done pulling out rs protocol data"

                cat $DATADIR/dialupin | grep -w 20 > $DATADIR/dialupftp
                cat $DATADIR/dialupftp | awk '{if($3=="20" || $5=="20") print $0}'

                cat $DATADIR/dialupout | grep -w 20 > $DATADIR/dialupftp1
                cat $DATADIR/dialupftp1 | awk '{if($3=="20" || $5=="20") print $0}'
                cat $DATADIR/dialupin | grep -w 21 > $DATADIR/dialupftp
                cat $DATADIR/dialupftp | awk '{if($3=="21" || $5=="21") print $0}'
                cat $DATADIR/dialupout | grep -w 21 > $DATADIR/dialupftp1
                cat $DATADIR/dialupftp1 | awk '{if($3=="21" || $5=="21") print $0}'
                cat $DATADIR/dialupin | grep -w 25 > $DATADIR/dialupmail
                cat $DATADIR/dialupmail | awk '{if($3=="25" || $5=="25") print $0}'
                cat $DATADIR/dialupout | grep -w 25 > $DATADIR/dialupmail1
                cat $DATADIR/dialupmail1 | awk '{if($3=="25" || $5=="25") print $0}'
                cat $DATADIR/dialupin | grep -w 80 > $DATADIR/dialuphttp
                cat $DATADIR/dialuphttp | awk '{if($3=="80" || $5=="80") print $0}'
                cat $DATADIR/dialupout | grep -w 80 > $DATADIR/dialuphttp1
                cat $DATADIR/dialuphttp1 | awk '{if($3=="80" || $5=="80") print $0}'
                cat $DATADIR/dialupin | grep -w 110 > $DATADIR/dialupmail
                cat $DATADIR/dialupmail | awk '{if($3=="110" || $5=="110") print
$0}' >> $DATADIR/dialupmailin
                cat $DATADIR/dialupout | grep -w 110 > $DATADIR/dialupmail1
                cat $DATADIR/dialupmail1 | awk '{if($3=="110" || $5=="110") print
$0}' >> $DATADIR/dialupmailout
/bin/echo "done pulling out du protocol data"

                cat $DATADIR/adslin | grep -w 20 > $DATADIR/adslftp
                cat $DATADIR/adslftp | awk '{if($3=="20" || $5=="20") print $0}' >>
$DATADIR/adslftpin
                cat $DATADIR/adslout | grep -w 20 > $DATADIR/adslftp1
                cat $DATADIR/adslftp1 | awk '{if($3=="20" || $5=="20") print $0}' >>
$DATADIR/adslftpout
                cat $DATADIR/adslin | grep -w 21 > $DATADIR/adslftp
                cat $DATADIR/adslftp | awk '{if($3=="21" || $5=="21") print $0}' >>
$DATADIR/adslftpin
                cat $DATADIR/adslout | grep -w 21 > $DATADIR/adslftp1
                cat $DATADIR/adslftp1 | awk '{if($3=="21" || $5=="21") print $0}' >>
$DATADIR/adslftpout
                cat $DATADIR/adslin | grep -w 25 > $DATADIR/adslmail
                cat $DATADIR/adslmail | awk '{if($3=="25" || $5=="25") print $0}' >>
$DATADIR/adslmailin
                cat $DATADIR/adslout | grep -w 25 > $DATADIR/adslmail1
                cat $DATADIR/adslmail1 | awk '{if($3=="25" || $5=="25") print $0}'

                cat $DATADIR/adslin | grep -w 80 > $DATADIR/adslhttp
                cat $DATADIR/adslhttp | awk '{if($3=="80" || $5=="80") print $0}' >>
$DATADIR/adslhttpin
                cat
...

read more »

    To answer the easy question first:

    The only way it would delete the info in file1, is if you
    redirect back to file1:

        awk something file1 > file1

    (cat is unnecessary; the same goes for almost all instances in the
     script below.)

Quote:

> ######################################!/bin/sh

       #!/bin/sh

    If you use the shebang, the first 2 bytes of the file must be #!

Quote:> CFGDIR=/usr/local/iptraf/config
> DATADIR=/usr/local/iptraf/data
> DBDIR=/usr/local/iptraf/db
> WEBDIR=/usr/local/iptraf/www
> LOGDIR=/var/log/iptraf
> DATE=`/bin/date`

> #make sure log is present
> if test -f $LOGDIR/ip_traffic.log; then
>       :
> else
>       exit
> fi

    Why not invert the logic you use in a later section of the script:

        test -f $LOGDIR/ip_traffic.log || exit

    Or:

        if [ ! -f $LOGDIR/ip_traffic.log ]
        then
             exit
        fi

Quote:> #create data file for tcp/udp services
> /bin/cat $LOGDIR/ip_traffic.log | /bin/sed -e "s/;//g" | /bin/sed -e
> "s/:/ /g" | awk '{print $10,$13,$14,$16,$17}' > $LOGDIR/ip_trafficdata

    cat and sed are unnecessary; it can all be done with awk.

        awk ' {
                gsub(";",""}; gsub(":"," ")
                print $10,$13,$14,$16,$17
              }' > $LOGDIR/ip_trafficdata

Quote:> test -f $DATADIR/rentalservers && /bin/rm -f $DATADIR/rentalserversin

   Test is unnecessary; so are multiple calls to rm:

        rm -f $DATADIR/rentalserversin $DATADIR/dialupin $DATADIR/adslin .....

    In fact, rm is unnecessary, since the redirection operator
    ">" will remove the file if it already exists.

  [snip]

Quote:> /bin/touch $DATADIR/rentalserversin

   If it were necessary to use touch (it's not), you could do it with
   a single call:

        touch $DATADIR/rentalserversin  $DATADIR/dialupout ....
[snip]

Quote:

> if test -f $LOGDIR/ip_trafficdata; then
>       RS1=`cat $CFGDIR/rentalservers`
>       for RS2 in $RS1
>               do
>               cat $LOGDIR/ip_trafficdata | grep -w $RS2 > $DATADIR/rentalservers

    Here, as in the rest of your script, cat is unnecessary:

        grep -w $RS2 $LOGDIR/ip_trafficdata > $DATADIR/rentalservers

Quote:>               cat $DATADIR/rentalservers | awk '{if($2=RA) print $0}' RA=$RS2 >> $DATADIR/rentalserversout
>               cat $DATADIR/rentalservers | awk '{if($4=RA) print $0}' RA=$RS2 >> $DATADIR/rentalserversin
>               done
> /bin/echo "done pulling out rs data"

   echo is built-in to the shell; if you give the full path, the shell
   will make an unnecessary (and much slower) call to an external
   program:

        echo "done pulling out rs data"

    I haven't gone over the rest of your script in detail, but it
    seems to me that you could do it with a single awk script (perhaps
    2 or 3).

[snip]

Quote:>               cat $DATADIR/rentalserversin | grep -w 20 > $DATADIR/rentalserversftp
>               cat $DATADIR/rentalserversftp | awk '{if($3=="20" || $5=="20") print $0}' >> $DATADIR/rentalserversftpin

    What's the grep for? Do the above two lines do anything that
    this wouldn't do:

        awk '{if($3=="20" || $5=="20") print $0}' $DATADIR/rentalserversin >> $DATADIR/rentalserversftpin

[snip] [snip] [snip]

Quote:> ####There more but why post it.

    I suggest that you work on perfecting the script for just one type
    of data; once that is working, use what you have learned to add
    the rest.

--

        =================================================================

Quote:> Howdy all!!  First off let me tell you that I'm new to shell
> scripting.  I have been trying to get the log file from IPTraff to
> give me such results as total bytes and bps sent from and to our
> dialup & adsl customers as well as our web hosting servers based
> on TCP protocols. I'm using the ip_traffic log produced from
> IPTraf that looks like this:

[..]

I would compile/run ntop (www.ntop.org), if possible, sound as if
it does what you want.

Michael Heiming
--
Remove the +SIGNS case mail bounces.

Chris - THANK YOU FOR ALL THE GREAT SUGGESTIONS.  Didn't know I could
do the stuff that you were suggesting.  Like I said I'm a newbie at
shell scripting.

Part of the reason for using grep in the script was to make it faster
in spitting out the data.  I was having a hard time getting it to
finish in a relatively short period of time.  However, I suspect that
it also may have corrupted some of the data since I compared the data
files that were produced and some of the remote IP addresses were
replaced with local ones.  Don't ask how that happened.  It didn't
happen all the time though.

If you have any recommendations for info on the web, it would be much
appreciated.

THANKS AGAIN.

Jim

Thnaks Michael -

I did play around with NTOP.  Only problem or solution I could find
let's say was how to keep the data.  Read the docs, but couldn't get
anything to work.

Quote:> I did play around with NTOP.  Only problem or solution I could
> find let's say was how to keep the data.  Read the docs, but
> couldn't get anything to work.

I wonder, for me the '-P' works, as 'man ntop' suggests?

Michael Heiming
--
Remove the +SIGNS case mail bounces.

Maybe I didn't read this right but the man page states:

       -P This allows to specify where  db-files  are  searched  or
        created  (default  "."). In addition DBPATH/html is added
        to the searchlist for the WEB-files

Does that mean that later on, say a month down the road, you can go
back and see what the situation was like today?  Anyways, going the
IPTraf and RRDTool route has really helped my shell scripting skills.

Thanks.
Jim