Large file creation slowness

Large file creation slowness

Post by gee- » Sat, 05 May 2001 05:19:19



When creating files under 600mb new file creation is fast, over 600mb
the file creation slows to a crawl.  The system has 2gb of RAM,
8-200mhz pp cpus and is connected to EMC Symmetrix storage with 3gb of
cache.  Any ideas on how to increase to speed of creating large files?

Thanks,
Greg

 
 
 

Large file creation slowness

Post by ECSta » Sun, 06 May 2001 12:04:13


How are you creating the files?

On what type of file system?

Fragmentation issues?

What if you create the file on a fresh file system, perhaps a different type
file system?

What system bottlenecks are observed (including buffer cache)?

Any conflicts on the array ie systems sharing disks?

Indications of H/W problems? Bus contention?

Using SCSI? Fibre channel?

Regards,

Eric Stahl


>When creating files under 600mb new file creation is fast, over 600mb
>the file creation slows to a crawl.  The system has 2gb of RAM,
>8-200mhz pp cpus and is connected to EMC Symmetrix storage with 3gb of
>cache.  Any ideas on how to increase to speed of creating large files?

>Thanks,
>Greg


 
 
 

Large file creation slowness

Post by gee- » Thu, 10 May 2001 00:21:15



Quote:>How are you creating the files?

using simple cp from one filesystem to another, we observe the same
thing using cpio.

Quote:>On what type of file system?
Veritas - vxfs
>Fragmentation issues?

could be, I need to remake the filesystem
Quote:>What if you create the file on a fresh file system, perhaps a different type
>file system?

I am going to try larger block sizes

Quote:>What system bottlenecks are observed (including buffer cache)?
>Any conflicts on the array ie systems sharing disks?

No, this happens on all of our filesystems
Quote:>Indications of H/W problems? Bus contention?
I don't think so
>Using SCSI? Fibre channel?
SCSI

>Regards,

>Eric Stahl

Greg


>>When creating files under 600mb new file creation is fast, over 600mb
>>the file creation slows to a crawl.  The system has 2gb of RAM,
>>8-200mhz pp cpus and is connected to EMC Symmetrix storage with 3gb of
>>cache.  Any ideas on how to increase to speed of creating large files?

>>Thanks,
>>Greg

 
 
 

Large file creation slowness

Post by ECSta » Thu, 10 May 2001 13:03:13


Quote:>>On what type of file system?
>Veritas - vxfs
>>Fragmentation issues?
>could be, I need to remake the filesystem

Whether base or online Veritas JFS, fsadm should be able to report
fragmentation issues (-D and -E options). I'd still try a fresh file system
(both VxFS and UFS).

Quote:>>Indications of H/W problems? Bus contention?
>I don't think so

You're right. Doesn't sound like that sort of problem.

Eric Stahl



>>How are you creating the files?
>using simple cp from one filesystem to another, we observe the same
>thing using cpio.

>>On what type of file system?
>Veritas - vxfs
>>Fragmentation issues?
>could be, I need to remake the filesystem
>>What if you create the file on a fresh file system, perhaps a different type
>>file system?
>I am going to try larger block sizes
>>What system bottlenecks are observed (including buffer cache)?

>>Any conflicts on the array ie systems sharing disks?
>No, this happens on all of our filesystems
>>Indications of H/W problems? Bus contention?
>I don't think so
>>Using SCSI? Fibre channel?
>SCSI

>>Regards,

>>Eric Stahl
>Greg

 
 
 

1. Large file creation crashing

i
I have a Pentuim II 400 Mhz machine with 384 Mb of RAM and a
couple of 47 Gb SCSI Hard Drive running Slakcware 3.5 and kernel
2.0.36. A user on the machine has been usuccessful in modifying
a 3 Gb file using the following routine:
i_test=fwrite(&buffer[first],4,n_buf,fp_out)
if (i_test !=n_buf) {
        [Complain]
He moves the the data in 2^15 or 32768 blocks. When the output file
reaches 1073741824 or 2^30 the system simply stops writing to the
output file. ANy ideas? The drive is formated at 2048 bytes per Inode.

--
my signature file
Ultimate - Let it fly and I'll be there in the end zone.

2. how can I call a CGI-script from an Java-applet

3. Thread creation fails after large mallocs

4. Trying to get IPX over PPP working. All I get is "Received bad configure-ack"

5. Library management tools for large number of large data files.

6. Unix - Frequently Asked Questions (Contents) [Frequent posting]

7. Large Shared Memory, large files

8. DAT is my problem.

9. "Standard Journaled File System" vs "Large File Enabled Journaled File System"

10. File corruption accessing files on a large-file-enabled fs using RM-Cobol

11. Gvim is not giving a new file creation, when several files already opened..

12. Enable Large File Size in Aix file system

13. Which file system for large files?