sometimes awk works and sometimes /usr/xpg4/bin/awk works ..

sometimes awk works and sometimes /usr/xpg4/bin/awk works ..

Post by Dennis Clark » Wed, 13 Aug 2003 04:04:35



 just a rant ..

Sometimes it seems like I should just link /usr/bin/awk to /usr/xpg4/bin/awk

$ grep "1000A-10-ENC" foo.dat | awk 'BEGIN{FS=";"}{print $1 "\t" $4 }'
awk: record `1000A-10-ENC; ;2990....' has too many fields
 record number 1

$ grep "1000A-10-ENC" foo.dat | /usr/xpg4/bin/awk 'BEGIN{FS=";"}{print $1 "\t" $4 }'
1000A-10-ENC    4

Dennis

 
 
 

sometimes awk works and sometimes /usr/xpg4/bin/awk works ..

Post by Markus Gyg » Wed, 13 Aug 2003 05:19:54



> Sometimes it seems like I should just link /usr/bin/awk to /usr/xpg4/bin/awk

I'm using PATH=/usr/xpg4/bin:/usr/bin:... for many years (as
user and as root) and I think the only thing that didn't work
so far was the AnswerBook server startup script (because awk
complained)...

Markus

 
 
 

sometimes awk works and sometimes /usr/xpg4/bin/awk works ..

Post by Dennis Clark » Wed, 13 Aug 2003 06:19:02




>> Sometimes it seems like I should just link /usr/bin/awk to /usr/xpg4/bin/awk

>I'm using PATH=/usr/xpg4/bin:/usr/bin:... for many years (as
>user and as root) and I think the only thing that didn't work
>so far was the AnswerBook server startup script (because awk
>complained)...

 Isn't that interesting, I had the same thing for a while and never noticed an
issue.  I wonder why I changed back to /usr/sbin:/usr/bin first.  I'll have to
think about that.

 The man page for awk says to see nawk for info about /usr/xpg4/bin/awk.  OK
 The man page for nawk says that I can specify my FS with an option :

-F ERE
           Define the input field separator to  be  the  extended
           regular  expression ERE, before any input is read (can
           be a character).

 This is different from awk.  Perhaps I should just leave my awk scripts alone
and use /usr/xpg4/bin/awk until something breaks.  Seems to work better.

Dennis

 
 
 

sometimes awk works and sometimes /usr/xpg4/bin/awk works ..

Post by Richard L. Hamilt » Wed, 13 Aug 2003 08:38:18




Quote:

>  just a rant ..

> Sometimes it seems like I should just link /usr/bin/awk to /usr/xpg4/bin/awk

> $ grep "1000A-10-ENC" foo.dat | awk 'BEGIN{FS=";"}{print $1 "\t" $4 }'
> awk: record `1000A-10-ENC; ;2990....' has too many fields
>  record number 1

> $ grep "1000A-10-ENC" foo.dat | /usr/xpg4/bin/awk 'BEGIN{FS=";"}{print $1 "\t" $4 }'
> 1000A-10-ENC    4

Here's the results of some brute force testing of a line like

    perl -e 'print "x " x '"${f}"';' | ${awk} '{x=$'"${f}"'}'

for values of ${f} starting with 1 and different flavors of awk for ${awk}:

awk: trying to access field 100
oawk failed with 100 fields

nawk: trying to access field 500
 source line number 1
 context is
         >>> {x=$500 <<< }
nawk failed with 500 fields

/usr/xpg4/bin/awk: line 0 (NR=1): Too many fields (LIMIT: 4000)
/usr/xpg4/bin/awk failed with 4001 fields

/usr/xpg4/bin/awk does fairly well compared to the first two.  But with
gawk, I knew it would be hopeless to simply increment, so I went to doubling.
It handled 2097152 fields just fine, but took long enough on the next doubling
(4194304) that I didn't feel like waiting and killed it.  Probably VM was
the limitation; on a larger system with more than 1 1/8 GB RAM, it would've
probably kept going somewhat longer (with enough RAM, nominally until it hit the
limits of 32 bit (signed or even unsigned) numbers, but more realistically
until it maxed out a 32-bit address space; I haven't needed to explore the
possibility of building a 64-bit gawk executable).

So if you want a version of awk that for most practical purposes doesn't have
a maximum number of fields limitation, it's rather clear to me which one that
would be.

Recently I ran into a case where nawk was bombing during patchadd (too
darn many patches installed, I guess).  So I just moved it over and replaced
it with a symlink to gawk.  Thus far, nothing has broken, and I can once
again install patches without that particular problem.

Of course you do have to scrounge and build gawk yourself (unless it's on
the freeware CD, or you're willing to get it from one of the sites with
prebuilt binaries for Solaris), but if you need it, it's well worth having.

But I think the real answer for patching problems would be if Sun rewrote
the patching (and any other package related) scripts in perl, which could
also cut the number of child processes (since perl can do pretty much
everything that sh, awk, sed, etc. can do and then some) and considerably
speed up patch installation.  Since a stable (for a given version of the
OS) version of perl is pretty much a core part of Solaris >= 8, I can't see
any good reason (aside from the manpower needed) _not_ to.

--

 
 
 

sometimes awk works and sometimes /usr/xpg4/bin/awk works ..

Post by Dennis Clark » Wed, 13 Aug 2003 10:24:35





>>  just a rant ..

>> Sometimes it seems like I should just link /usr/bin/awk to /usr/xpg4/bin/awk

>> $ grep "1000A-10-ENC" foo.dat | awk 'BEGIN{FS=";"}{print $1 "\t" $4 }'
>> awk: record `1000A-10-ENC; ;2990....' has too many fields
>>  record number 1

>> $ grep "1000A-10-ENC" foo.dat | /usr/xpg4/bin/awk 'BEGIN{FS=";"}{print $1 "\t" $4 }'
>> 1000A-10-ENC    4

>Here's the results of some brute force testing of a line like

>    perl -e 'print "x " x '"${f}"';' | ${awk} '{x=$'"${f}"'}'

>for values of ${f} starting with 1 and different flavors of awk for ${awk}:

>awk: trying to access field 100
>oawk failed with 100 fields

 yep .. I get the same problem.  I have extracted data from a database and
there are 768 fields to a record.  kaboom.

Quote:

>nawk: trying to access field 500
> source line number 1
> context is
>         >>> {x=$500 <<< }
>nawk failed with 500 fields

I did not try anything else other than /usr/xpg4/bin/awk so I wouldn't know.
You seem to have nailed down the situation quite neatly though.

Quote:

>/usr/xpg4/bin/awk: line 0 (NR=1): Too many fields (LIMIT: 4000)
>/usr/xpg4/bin/awk failed with 4001 fields

That is enough for most demanding situations.

Quote:

>/usr/xpg4/bin/awk does fairly well compared to the first two.  But with
>gawk, I knew it would be hopeless to simply increment, so I went to doubling.
>It handled 2097152 fields just fine, but took long enough on the next doubling
>(4194304) that I didn't feel like waiting and killed it.

okay .. well it is reasonable to say that gawk would work for even those
situations where the record size is completely ridiculous.  I would guess that
it would work given enough RAM.  If you like I can test it on a V880 with 8Gb
of RAM.  Just to see if it breaks at some reasonable ( or unreasonable )
point.

Quote:> Probably VM was
>the limitation; on a larger system with more than 1 1/8 GB RAM, it would've
>probably kept going somewhat longer (with enough RAM, nominally until it hit the
>limits of 32 bit (signed or even unsigned) numbers, but more realistically
>until it maxed out a 32-bit address space; I haven't needed to explore the
>possibility of building a 64-bit gawk executable).

I'll check with blastwave.org to see what's up with a build of gawk for 64 bit
scenarios.   I don't know how relevant it will be though.  Seems like everyone
is going Intel these days and the 64-bit architecture is a great idea but not
needed.

Quote:

>So if you want a version of awk that for most practical purposes doesn't have
>a maximum number of fields limitation, it's rather clear to me which one that
>would be.

I agree.

Quote:

>Recently I ran into a case where nawk was bombing during patchadd (too
>darn many patches installed, I guess).  So I just moved it over and replaced
>it with a symlink to gawk.  Thus far, nothing has broken, and I can once
>again install patches without that particular problem.

More than 500 patches?

Quote:

>Of course you do have to scrounge and build gawk yourself (unless it's on
>the freeware CD, or you're willing to get it from one of the sites with
>prebuilt binaries for Solaris), but if you need it, it's well worth having.

Probably in the GNU textutils from blastwave.org site.

Quote:

>But I think the real answer for patching problems would be if Sun rewrote
>the patching (and any other package related) scripts in perl, which could
>also cut the number of child processes (since perl can do pretty much
>everything that sh, awk, sed, etc. can do and then some) and considerably
>speed up patch installation.  Since a stable (for a given version of the
>OS) version of perl is pretty much a core part of Solaris >= 8, I can't see
>any good reason (aside from the manpower needed) _not_ to.

The source to awk probably has not been touched since Solaris 2.5.1 days.

Dennis

 
 
 

sometimes awk works and sometimes /usr/xpg4/bin/awk works ..

Post by Richard L. Hamilt » Wed, 13 Aug 2003 20:29:26





[...]
>>Recently I ran into a case where nawk was bombing during patchadd (too
>>darn many patches installed, I guess).  So I just moved it over and replaced
>>it with a symlink to gawk.  Thus far, nothing has broken, and I can once
>>again install patches without that particular problem.

> More than 500 patches?

$ cd /var/sadm/patch
$ ls|wc -l                           # different patches and/or revisions
    1542
$ ls|sed -e 's/-.*//'|sort -u|wc -l  # different patches, regardless of rev
     511

[...]

--

 
 
 

sometimes awk works and sometimes /usr/xpg4/bin/awk works ..

Post by Anthony Mandi » Wed, 13 Aug 2003 22:49:36



> > More than 500 patches?

> $ cd /var/sadm/patch
> $ ls|wc -l                           # different patches and/or revisions
>     1542
> $ ls|sed -e 's/-.*//'|sort -u|wc -l  # different patches, regardless of rev
>      511

        It might be an idea to employ patchrm once in a while.
        Especially on the bigger patches.

-am     ? 2003

 
 
 

sometimes awk works and sometimes /usr/xpg4/bin/awk works ..

Post by Dennis Clark » Thu, 14 Aug 2003 01:14:07


Quote:>> More than 500 patches?
>$ cd /var/sadm/patch
>$ ls|wc -l                           # different patches and/or revisions
>    1542
>$ ls|sed -e 's/-.*//'|sort -u|wc -l  # different patches, regardless of rev
>     511

wow!

 what's the date-of-birth on this system?  Five years or more ago?

dc

 
 
 

sometimes awk works and sometimes /usr/xpg4/bin/awk works ..

Post by Richard L. Hamilt » Thu, 14 Aug 2003 09:42:44




Quote:

>>> More than 500 patches?

>>$ cd /var/sadm/patch
>>$ ls|wc -l                           # different patches and/or revisions
>>    1542
>>$ ls|sed -e 's/-.*//'|sort -u|wc -l  # different patches, regardless of rev
>>     511

> wow!

>  what's the date-of-birth on this system?  Five years or more ago?

It's a Sun Blade 100 purchased shortly after they came out.

I'm just a little compulsive about staying up to date, and only clean out
the larger old patches when I'm quite sure that the latest version is
stable.  Not too much incentive since space isn't all that tight.

--

 
 
 

1. /usr/xpg4/bin/tr bug (or don't use /usr/xpg4/bin)

Hi

I ran into some problems because I had /usr/xpg4/bin in my path
on Solaris 7 x86.

I was stung by a bug in tr:
$ printf "%s\n" debug | /usr/xpg4/bin/tr a-z A-Z
cdatf

The correct behaviour is:
$ printf "%s\n" debug | /usr/bin/tr a-z A-Z
DEBUG

My locale is:
$ locale
LANG=en_GB.ISO8859-15
LC_CTYPE=en_GB.ISO8859-15
LC_NUMERIC=en_GB.ISO8859-15
LC_TIME=en_GB.ISO8859-15
LC_COLLATE=en_GB.ISO8859-15
LC_MONETARY=en_GB.ISO8859-15
LC_MESSAGES=C
LC_ALL=

I hope this post will help others avoid my mistake!

Nick

2. Configuring Snort

3. Solaris /usr/xpg4/bin/awk patch

4. Gnome Desktop Status Bar Missing

5. Sometimes the Euro works, sometimes it doesn't.

6. Monitronix (Sun?) MX-200S monitor specs?

7. ErrorDocument directive sometimes works - sometimes not??

8. Recentralization Study

9. SCSI <-> 1.2GB MO Sometimes works, sometimes doesn't

10. PPPD & MacPPP problems, sometimes works, sometimes doesn't?

11. Telnet, FTP ... sometimes work, sometimes they don't!

12. YDL: Sometimes BootX works, sometimes not

13. why does kde sometimes install to /opt/kde, and sometimes to /usr/local/kde?