Merging huge text files...

Merging huge text files...

Post by Ray Van Dols » Tue, 25 Apr 2000 04:00:00



I have two webserver log files each of size ~2GBs.  I want to merge them
together into one file, however, most of the solutions I've tried so far
can't handle text files that large.  (cat file1 >> file2 for example).  I
also tried a quick Python script to do the job, but it also couldn't handle
that large a text file.  Do I need to write something in C to accomplish
this or are their some utilities out there I could try?  

Thanks for any help/ideas.

Ray Van Dolson

 
 
 

Merging huge text files...

Post by Vilmos Sot » Tue, 25 Apr 2000 04:00:00



Quote:> I have two webserver log files each of size ~2GBs.  I want to merge them
> together into one file, however, most of the solutions I've tried so far
> can't handle text files that large.  (cat file1 >> file2 for example).  I
> also tried a quick Python script to do the job, but it also couldn't handle
> that large a text file.  Do I need to write something in C to accomplish
> this or are their some utilities out there I could try?  

I think the problem is that you are hitting the 2GB filesize limit
present on the Intel Platform.

Vilmos

 
 
 

Merging huge text files...

Post by pe.. » Tue, 25 Apr 2000 04:00:00




>> I have two webserver log files each of size ~2GBs.  I want to merge them
>> together into one file, however, most of the solutions I've tried so far
>> can't handle text files that large.  (cat file1 >> file2 for example).  I
>> also tried a quick Python script to do the job, but it also couldn't handle
>> that large a text file.  Do I need to write something in C to accomplish
>> this or are their some utilities out there I could try?  
> I think the problem is that you are hitting the 2GB filesize limit
> present on the Intel Platform.

Present on *some* Intel platforms.

*BSD will allow files up to Terabytes.

Quote:> Vilmos

--
--
Peter H?kanson        
        Manet Networking      (At the Riverside of Gothenburg, home of Volvo)
           Sorry about my e-mail address, but i'm trying to keep spam out.

 
 
 

Merging huge text files...

Post by Peter T. Breu » Tue, 25 Apr 2000 04:00:00



: I have two webserver log files each of size ~2GBs.  I want to merge them
: together into one file, however, most of the solutions I've tried so far

You can't. How would you address them? On a 32 bit system, you only
have 4GB of offset addressing available within th efile, and one
bit of that is for the sign.

You'll need to seek out 64bit patches for linux VFS on i86, and possibly
alter the c library slightly as well - I've never bothered to apply
the patches so I don't know.

The simplest answer is to do it on a 64bot machine, such as an alpha
or ultrasparc.

Peter

 
 
 

Merging huge text files...

Post by Leslie Mikese » Tue, 25 Apr 2000 04:00:00





>: I have two webserver log files each of size ~2GBs.  I want to merge them
>: together into one file, however, most of the solutions I've tried so far

>You can't. How would you address them? On a 32 bit system, you only
>have 4GB of offset addressing available within th efile, and one
>bit of that is for the sign.

This is irrelevant for appending or reading a file sequentially.

Quote:>You'll need to seek out 64bit patches for linux VFS on i86, and possibly
>alter the c library slightly as well - I've never bothered to apply
>the patches so I don't know.

>The simplest answer is to do it on a 64bot machine, such as an alpha
>or ultrasparc.

Or use one of the *bsd's on x86.  Or NT on NTFS.

 Les Mikesell

 
 
 

1. how to split a very huge text file into small files

Hi,

I would like to split my very huge text file into many very small text
files by using size information instead
of line count. Because working by using line count is not very useful, I
want to use size information
at the process of splitting.

How can I split this file into small files?

Is there any help?

Thanks in advance,
Zekeriya.

david >ll dmspl_updates.listing
-rw-r-----   1 zekeriya   RI1        440815129 Jun 26 14:54
dmspl_updates.listing

2. privateICE for Kernel2.4 - attn:Darklady

3. Urgent, need script or command to extract lines from a huge text file

4. Tamandua N-IDS Installation

5. ksh script to analyse data in a huge text file

6. Mars and Pipe

7. Looking for Text files comparing and merging utility

8. search & replace symlinks

9. merge text from 3 files?

10. ** Merging Two Text Files Parallel **

11. Merging text & decoding files on UNIX

12. text file contents merge and subtraction

13. Q: Script to merge together text files