Very Computer

I know that this is a really stupid and basic question, but here goes:

I read somewhere how to remove similar lines from a file. So that is you
have say

pear
apple
pear
banana
apple

it can be sorted to become

apple
banana
pear

I can't for the life of me remember how to do this. Is this part of "sort"
or "diff" or yet another command?

Any help would be appreciated!

Greg

cat file | sort | uniq > newfile

uniq will get rid of adjacent identical lines...

Nick Russo

This Week's Useless Use of Cat Award goes to

Nick> cat file | sort | uniq > newfile

And of course, if you've been following along for a week or two, you know
that this (BING!) is a Useless Use of Cat!

Rememeber, nearly all cases where you have:

        cat file | some_command and its args ...

you can rewrite it as:

        <file some_command and its args ...

and in some cases, such as this one, you can move the filename
to the arglist as in:

        some_command and its args ... file

Of course, the real tragedy in this post is that sort also has a -u
switch, meaning this could have been simply:

        sort -u file >newfile

Oh well.

Just another Useless Use of Usenet,
--
Name: Randal L. Schwartz / Stonehenge Consulting Services (503)777-0095
Keywords: Perl training, UNIX[tm] consulting, video production, skiing, flying

Web: <A HREF="http://www.stonehenge.com/merlyn/">My Home Page!</A>
Quote: "I'm telling you, if I could have five lines in my .sig, I would!" -- me

As soon as I posted this, I realized that I didn't need cat. Then later,
reading another message, I realized sort had a u flag for unique. I now
realize there's a reason pine says "Posted message may go to thousands of
readers. Really post?" And now, I slink away into the background...

I'll go back to quietly reading more thought-out posts, at least until I
wake up...

Nick Russo

This will remove the dup's and sort the file for you.

--
Michael P. Di Fulvio         W3 Solutions Inc.
PO Box 451011                Sunrise, Florida 33345              

Unix/Internet Consultants    Web Hosting,Security,Integration
--

Philippe

--
i

:wq

Srikanth

2.  How about a further complication:

I have blank lines interspersed with the good data:

pear
apple
              <--- null
pear
banana
              <--- null
apple

"sort -u"   yields:

            <--- null
apple
banana
pear

I want to get rid of that null line.
I can pipe the whole thing through grep or awk:

   sort -u filename | grep -v '^$' >
   sort -u filename | awk '$0 !~ /^$/ {print$0}'

But that means every line in the file is checked for null,  when, in fact, only the first line will (might) be
null.   This seems to approximately triple processing time ( For long lists 45 vs. 15 seconds on my box).

What is a better way?
What if I did someting like
   "cat one.linefeed filename | sort -u | [some command to always remove 1st line]"

With gratitude in advance,
Charles Stevenson

                        sort -u filename

(-u for unique)

: > I read somewhere how to remove similar lines from a file. So that is you
: > have say
: >
: > pear
: > apple
: > pear
: > banana
: > apple
: >
: > it can be sorted to become
: >
: > apple
: > banana
: > pear
: >
: > I can't for the life of me remember how to do this. Is this part of "sort"
: > or "diff" or yet another command?

: try "uniq"

: Philippe

: --
: i

: :wq
--
=-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
al aab, seders moderator                                      sed u soon
               it is not zat we do not see the  s o l u t i o n          
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-+