Very Computer

by **hwk..** » Thu, 26 Aug 1999 04:00:00

Here is the input:

1|mary||1|2|4|12
2|bob|admin|2|2|3|107
3|dave|operator|4|3|4|7|8|14
4|ron|analyst|8|8|110

Here is what I want:

1|mary||1|2|4|12
2|bob|admin|2|3|107
3|dave|operator|4|3|7|8|14
4|ron|analyst|8|110

You may notice, the duplicate numbers in lines 2 3 and 4 were
eliminated.

I thought about using an array to dupe them but shells don't have
arrays, and for reasons I don't want to go into we can't use Perl.

Thanks in advance...

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.

by **Al Shark** » Thu, 26 Aug 1999 04:00:00

> Here is the input:

> 1|mary||1|2|4|12
> 2|bob|admin|2|2|3|107
> 3|dave|operator|4|3|4|7|8|14
> 4|ron|analyst|8|8|110

> Here is what I want:

> 1|mary||1|2|4|12
> 2|bob|admin|2|3|107
> 3|dave|operator|4|3|7|8|14
> 4|ron|analyst|8|110

> I thought about using an array to dupe them but shells don't have
> arrays, and for reasons I don't want to go into we can't use Perl.

I'm sure someone will say there's a more efficient way, but here goes:

awk -F"|" '{printf "%s|%s|%s|%s", $1, $2, $3, $4;
for (i = 5; i <= NF; i++) {
for (j = i-1; j >= 4; j--) {
if ($i == $j) break;
else if (j == 4) printf "|%s", $i;
}
}
printf "\n";
}' your_file

Print the first four fields without a newline.
Starting with the fifth field, compare the value to all the previous
fields (except the first three,) and stop looking if you find a match.
If you got all the way back to the fourth field, it didn't match, so
print it without a newline.

Do the same with each successive field on the line.
When you get to the end of the line, print a newline character.

by **Ken Pizzi** » Thu, 26 Aug 1999 04:00:00

>Here is the input:

>1|mary||1|2|4|12
>2|bob|admin|2|2|3|107
>3|dave|operator|4|3|4|7|8|14
>4|ron|analyst|8|8|110

>Here is what I want:

>1|mary||1|2|4|12
>2|bob|admin|2|3|107
>3|dave|operator|4|3|7|8|14
>4|ron|analyst|8|110

>You may notice, the duplicate numbers in lines 2 3 and 4 were
>eliminated.

>I thought about using an array to dupe them but shells don't have
>arrays, and for reasons I don't want to go into we can't use Perl.

Well, some shells lack arrays, but many do have them (ksh, bash,
zsh, rc, es; even csh and tcsh).

But, looking at your task, my first inclination is to suggest
using "awk".
awk -F\| '{printf "%s|%s|%s|", $1,$2,$3
delete a
for (i=4;i<=NF;++i) { if (!a[$i]++) printf "%s|", $i}
print ""}'

(The special-casing of $1,$2,$3 is because I note that you have
2|bob|admin|2|2|3|107
mapping to
2|bob|admin|2|3|107
instead of
2|bob|admin|3|107
.)

--Ken Pizzini

by **Neil Schemenau** » Thu, 26 Aug 1999 04:00:00

>I thought about using an array to dupe them but shells don't have
>arrays, and for reasons I don't want to go into we can't use Perl.

I hope you have awk. Note that I am not an awk export. Go easy
on me. This script works with mawk on my Debian system.

#!/usr/bin/awk -f
BEGIN { FS="|"}
{
for (i=4; i<=NF; i++) {
nums[$i] = ""
}
printf "%s|%s|%s", $1, $2, $3
for (num in nums) {
printf "|%s", num
delete nums[num]
}
printf "\n"

Quote:}

by **Raja** » Sun, 29 Aug 1999 04:00:00

use the associative arrays of awk or nawk..

> Here is the input:

> 1|mary||1|2|4|12
> 2|bob|admin|2|2|3|107
> 3|dave|operator|4|3|4|7|8|14
> 4|ron|analyst|8|8|110

> Here is what I want:

> 1|mary||1|2|4|12
> 2|bob|admin|2|3|107
> 3|dave|operator|4|3|7|8|14
> 4|ron|analyst|8|110

> You may notice, the duplicate numbers in lines 2 3 and 4 were
> eliminated.

> I thought about using an array to dupe them but shells don't have
> arrays, and for reasons I don't want to go into we can't use Perl.

> Thanks in advance...

> Sent via Deja.com http://www.deja.com/
> Share what you know. Learn what you don't.

Very Computer

Eliminating duplicate fields from a line

Eliminating duplicate fields from a line

Eliminating duplicate fields from a line

Eliminating duplicate fields from a line

Eliminating duplicate fields from a line

Eliminating duplicate fields from a line