Help with large 'sort of' file on AIX 4.2.1

Help with large 'sort of' file on AIX 4.2.1

Post by Bob Mariot » Fri, 26 Feb 1999 04:00:00



We have a client with an F50 running 4.2.1
They have two files totalling 1.1GB in size.
Their /usr file system has 5.8GB free space based on df free blocks.
When we attempt to merge these new files, even as root with all soft
file sizes set to 0, it bombs out with a write failure error after
about 9,600,000 110 byte records or 1.06GB.

According to my calculations there is still about 4+GB free space
available.

Anyone have any clues????
TIA

 
 
 

Help with large 'sort of' file on AIX 4.2.1

Post by Paul-Marc Maro » Fri, 26 Feb 1999 04:00:00




Quote:>We have a client with an F50 running 4.2.1
>They have two files totalling 1.1GB in size.
>Their /usr file system has 5.8GB free space based on df free blocks.
>When we attempt to merge these new files, even as root with all soft
>file sizes set to 0, it bombs out with a write failure error after
>about 9,600,000 110 byte records or 1.06GB.

>According to my calculations there is still about 4+GB free space
>available.

>Anyone have any clues????
>TIA

As I can remember, sort places temporary files in /var/tmp, /usr/tmp if not
possible in /var/tmp, and then in /tmp

So, perhaps you should consider using -T option of sort command to specify
location of temporary files.

Sorry if it did'nt help

Paul-Marc
----------------------------------------------------
"When you have to kill a man, it costs nothing to be polite"

-- Winston Churchill, On formal declaration of war

 
 
 

Help with large 'sort of' file on AIX 4.2.1

Post by Werner Klause » Fri, 26 Feb 1999 04:00:00


Quote:> Could it be that you're not allowed to write larger files than about
> 1GB? What does ulimit say?
> When we attempt to merge these new files, even as root with all soft
> file sizes set to 0, it bombs out with a write failure error after
> about 9,600,000 110 byte records or 1.06GB.

 
 
 

Help with large 'sort of' file on AIX 4.2.1

Post by Kent Perrie » Fri, 26 Feb 1999 04:00:00



> We have a client with an F50 running 4.2.1
> They have two files totalling 1.1GB in size.
> Their /usr file system has 5.8GB free space based on df free blocks.
> When we attempt to merge these new files, even as root with all soft
> file sizes set to 0, it bombs out with a write failure error after
> about 9,600,000 110 byte records or 1.06GB.

Look at the filesize limits of the users (ulimit) and loot to see if
the filesystem was built with "large file support."  If the fs was not
built to support large files then the whole excerise is pointless
anyway.

Kent

 
 
 

Help with large 'sort of' file on AIX 4.2.1

Post by Norman Levi » Fri, 26 Feb 1999 04:00:00


Not sure I'm understanding your 'merge' but many
commands use /tmp for utility space and you may want
to check that file system.  For example, "sort" uses /tmp
to save the stdin data it sorts before it passes it toe
stdout in a pipeline.
--
Norman Levin
vm/dynAmIX inc.
 
 
 

Help with large 'sort of' file on AIX 4.2.1

Post by Bob Mariot » Sat, 27 Feb 1999 04:00:00


Thanks for the suggestions.  But here's what really happened...

The culprit was the ulimit -f  as it was set to 1.1GB which is where
the merged file reached before aborting.

Modifying the /etc/security/limits file the default fsize was set to
unlimited.  This SOLVED the problem.

Even after 9 years of AIX we still learn something new every day...



Quote:>Not sure I'm understanding your 'merge' but many
>commands use /tmp for utility space and you may want
>to check that file system.  For example, "sort" uses /tmp
>to save the stdin data it sorts before it passes it toe
>stdout in a pipeline.
>--
>Norman Levin
>vm/dynAmIX inc.

 
 
 

Help with large 'sort of' file on AIX 4.2.1

Post by David McConnel » Tue, 02 Mar 1999 04:00:00


One observation....
    I sure hope this is some sand-box machine you guys can afford to
reinstall and such.  Monkeying around with huge files in /usr is not exactly
what I would call sound system administration practices on a business
critical system.   Just food for thought....;}

--
My opinions are completely my own.  As for my heart and soul,
I'm not so sure....but my ass definately belongs to the IRS.


>We have a client with an F50 running 4.2.1
>They have two files totalling 1.1GB in size.
>Their /usr file system has 5.8GB free space based on df free blocks.
>When we attempt to merge these new files, even as root with all soft
>file sizes set to 0, it bombs out with a write failure error after
>about 9,600,000 110 byte records or 1.06GB.

>According to my calculations there is still about 4+GB free space
>available.

>Anyone have any clues????
>TIA

 
 
 

Help with large 'sort of' file on AIX 4.2.1

Post by Norman Levi » Tue, 02 Mar 1999 04:00:00


If you can get to this point, you may also want to unmount the file system (if
not a raw volume).

Quote:> One suggestion I would also submit is that, before you "freeze"
> the copy, you should stop all database/filesystem access, then run the
> chlvcopy to freeze the copy.  This would prevent the possibility of
> inconsistencies in the backup.

--
Norman Levin
vm/dynAmIX inc.
 
 
 

1. How to do i.e. 'sort <file >file'?

I would like to write commands which use the same file
for input and output like:
        sort <file >file

On the OS9 operating system I had a command "into" with
that I could write:
        sort <file | into file

Now I'm looking for a similar program for UNIX.

Thanks in advance

Martin

--
__________
Martin Kreutzer        | Fraunhofer Gesellschaft IIS | Phone: +49-9131-776-325

                       | D-91058 Erlangen            |

2. Problem with IDS-POP

3. default secondary sort line in 'sort'

4. Where to look at ?

5. using the 'join' or 'sort' command...

6. PCMCIA with DVI output?

7. Sorting output of 'ls' command by file extension?

8. auditd question?

9. Get 'ls -al' To Sort Dot-files Like Every Other Un*x

10. What sort of things should be defined in 'rc' files ?

11. How can I sort a file which delimited by '\t'

12. alpha sort: 'sed'/'awk'?

13. 'sort' question--checking for longest lines in a file?