Removing multiple repeted lines...

Removing multiple repeted lines...

Post by Dean Brissing » Sat, 27 May 1995 04:00:00



   I've setup multiple skels so users can run a korn script and change
their account from one default setup to another--shell and all.  The log's
format is this:

$USER;<I/U>;<what>;`date`;
       ^^^
(Install/Uninstall)

What I'm doing is this:

grep \;I\;skel2\; inst.log | cut -f1 -d";" | uniq | wc -l

My problem is that uniq does not eliminate all of the duplications and I
don't understand why.  Can anyone help me?

I get duplications when the same user installs the same skel at two
different times.

--
Dean Brissinger

 
 
 

Removing multiple repeted lines...

Post by Tim Kous » Sat, 27 May 1995 04:00:00


[snip]
: grep \;I\;skel2\; inst.log | cut -f1 -d";" | uniq | wc -l

: My problem is that uniq does not eliminate all of the duplications and I
: don't understand why.  Can anyone help me?
[snip]

Try piping through sort before uniq, that way uniq works better. So:

grep \;I\;skel2\; inst.log | cut -f1 -d";" | sort | uniq | wc -l

Should work. Good Luck...

Tim

 
 
 

Removing multiple repeted lines...

Post by Stephen Pott » Sat, 27 May 1995 04:00:00


[Followups trimmed to comp.unix.questions.  This isn't, strictly speaking,
an admin question, it more of a scripting question.  Maybe it should go to
comp.unix.shell]


   [snip]
   : grep \;I\;skel2\; inst.log | cut -f1 -d";" | uniq | wc -l

   : My problem is that uniq does not eliminate all of the duplications and I
   : don't understand why.  Can anyone help me?
   [snip]

   Try piping through sort before uniq, that way uniq works better. So:

   grep \;I\;skel2\; inst.log | cut -f1 -d";" | sort | uniq | wc -l

1) Doesn't help if you don't want the output sorted.
2) You should use sort -u, not sort | uniq
3) Use perl and an associative array or two.

    #!/usr/local/perl
    open(FILE, "$ARGV[0]") || die "Can't open $ARGV[0]: $!\n";
    while (<FILE>) {
        next unless /;I;skel2;/;
        ($entry) = split(/;/, $_);
        $count++ unless $seen{$entry}++;
    }
    print $count;

or as a one-liner (split over two lines for legibility).

perl -ne '/;I;skel2;/ && do{($e)=split(/;/,$_);$c++ unless $s{$e}++};
            END{print $c}' inst.log  

Steve
--

2350 Commerce Park Drive, Suite 4                           Palm Bay, FL 32905
(407) 676-3222                                            CAD/CAM/CAE/Software

 
 
 

Removing multiple repeted lines...

Post by Matthew Crav » Sat, 27 May 1995 04:00:00




>grep \;I\;skel2\; inst.log | cut -f1 -d";" | uniq | wc -l
>My problem is that uniq does not eliminate all of the duplications and I
>don't understand why.  Can anyone help me?

Hmm...what OS are you using? I just tried this on my system (Solaris
2.4 Sparc), and it worked. Try taking out the wc -l and look at what
it prints -- make sure that it really is chopping the right fields,
and such.

FYI, I created a file like this:

Fred;I;skel2;xxxxx
Fred;I;skel2;xxxxxx
Joe;I;skel2;xxxxx
Amy;I;skel1;xxxxx

And when I ran your command, I got:

Fred
Joe

So I don't know why it is not working for you.

/Matthew

--
Matthew Cravit, System Manager     |  "After all is said and done,
Academic Technology Development    |   more is said than done."
DePaul University, Chicago IL      |        -- Anonymous

 
 
 

Removing multiple repeted lines...

Post by Brian A. Buch » Sun, 28 May 1995 04:00:00




:
: >grep \;I\;skel2\; inst.log | cut -f1 -d";" | uniq | wc -l
:
: >My problem is that uniq does not eliminate all of the duplications and I
: >don't understand why.  Can anyone help me?
:
: Hmm...what OS are you using? I just tried this on my system (Solaris
: 2.4 Sparc), and it worked. Try taking out the wc -l and look at what
: it prints -- make sure that it really is chopping the right fields,
: and such.
:
: FYI, I created a file like this:
:
: Fred;I;skel2;xxxxx
: Fred;I;skel2;xxxxxx
: Joe;I;skel2;xxxxx
: Amy;I;skel1;xxxxx
:
: And when I ran your command, I got:
:
: Fred
: Joe
:
: So I don't know why it is not working for you.

for uniq to remove identical lines, the lines have to be consecutive,
as are your lines above
a file that has
Fred;I;skel2
Joe;I;skel2
Fred;I;skel2
Amy;I;skel1
will not be effected.

Boo

--
Doom: 5% Health, 0% Armor, 2 Shotgun Shells, NO FEAR.

 
 
 

Removing multiple repeted lines...

Post by Roman Fiet » Tue, 30 May 1995 04:00:00



   [snip]
   : grep \;I\;skel2\; inst.log | cut -f1 -d";" | uniq | wc -l

   : My problem is that uniq does not eliminate all of the duplications and I
   : don't understand why.  Can anyone help me?
   [snip]

   Try piping through sort before uniq, that way uniq works better. So:

   grep \;I\;skel2\; inst.log | cut -f1 -d";" | sort | uniq | wc -l

   Should work. Good Luck...

sort has a -u switch that automagically does the same as "sort |
uniq".

Roman

--
Roman Fietze (Mail Code 5023)              Kodak AG Stuttgart/Germany