Speeding up GNU Fileutils

Speeding up GNU Fileutils

Post by Michael Ritze » Thu, 09 Dec 1993 07:03:55



Recently there was a discussion in this group how the speed of cp and mv of the
GNU distribution
could be improved. I have written a quick mod which has a drastical effect
on the performance. It might be still improved --- e.g., allocate space
for buffers in power of two sized chunks. I don't provide diffs because i
have only very old sources of the fileutils available. So diffs to my sources
are of limited use. My changes are within the #ifdef atarist portions of the
code fragments below. I hope the parts of the code i supply suffice to make clear
how to implement this in the most recent version (just posted) of the fileutils.

cat should also be modified in a similar manner (i don't have the source at hand).

Here it goes, have fun.

M. Ritzert


########################################################################
cp.c:
########################################################################

/* Copy a regular file from SRC_PATH to DST_PATH.  Large blocks of zeroes,
   as well as holes in the source file, are made into holes in the
   target file.  (Holes are read as zeroes by the `read' system call.)
   Return 0 if successful, -1 if an error occurred. */

int
copy_reg (src_path, dst_path)
     char *src_path;
     char *dst_path;
{
  char *buf;
  int buf_size;
  int target_desc;
  int source_desc;
  int n_read;
  int n_written;
  struct stat sb;
  char *cp;
  int *ip;
  int return_val = 0;
  long n_read_total = 0;
  int last_write_made_hole = 0;
  int make_holes = 0;

  source_desc = open (src_path, O_RDONLY);
  if (source_desc < 0)
    {
      error (0, errno, "%s", src_path);
      return -1;
    }

  /* Create the new regular file with small permissions initially,
     to not create a security hole.  */

  target_desc = open (dst_path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
  if (target_desc < 0)
    {
      error (0, errno, "cannot create regular file `%s'", dst_path);
      return_val = -1;
      goto ret2;
    }

  /* Find out the optimal buffer size.  */

#ifdef atarist
  if (fstat (source_desc, &sb))
#else
  if (fstat (target_desc, &sb))
#endif
    {
      error (0, errno, "%s", dst_path);
      return_val = -1;
      goto ret;
    }
#ifdef atarist
  buf_size = sb.st_size * ST_BLKSIZE (sb);
#else
  buf_size = ST_BLKSIZE (sb);
#endif atarist

#ifndef ST_BLOCKS_MISSING
  if (S_ISREG (sb.st_mode))
    {
      /* Find out whether the file contains any sparse blocks. */

      if (fstat (source_desc, &sb))
        {
          error (0, errno, "%s", src_path);
          return_val = -1;
          goto ret;
        }

      /* If the file has fewer blocks than would normally
         be needed for a file of its size, then
         at least one of the blocks in the file is a hole. */
      if (S_ISREG (sb.st_mode) &&
          sb.st_size - (sb.st_blocks * DEV_BSIZE) >= DEV_BSIZE)
        make_holes = 1;
    }
#endif

  /* Make a buffer with space for a sentinel at the end.  */

#ifdef atarist
    buf = (char *) malloc (min(512000,buf_size + sizeof (int)));
    while ( buf == NULL && buf_size > 1024 ) {
        buf_size /= 2;
        buf = (char *) malloc (buf_size + sizeof (int));
    }
    if( buf == NULL ) {
        error (0, errno, "%s", "not enough memory to hold the file buffer");
        return_val=-1;
        goto ret;
    }
#else
  buf = (char *) alloca (buf_size + sizeof (int));
#endif atarist

  for (;;)
    {                                           /* mjr: main copy loop */

   ...

    }                                           /* mjr: end of main copy loop */
#ifdef atarist
        free(buf);
#endif

   ...

########################################################################
mv.c:
########################################################################

copy (from, to)
     char *from, *to;
{
  int ifd;
  int ofd;
#ifndef atarist
  char buf[1024 * 8];
#else
  struct stat sp;
  long buf_size;
  char * buf;   /* use dynamic memory allocation */
#endif
  int len;                      /* Number of bytes read into `buf'. */

  if (!S_ISREG (from_stats.st_mode))
    {
      error (0, 0, "cannot move `%s' across filesystems: Not a regular file",
             from);
      return 1;
    }

   ...

#ifdef atarist
    fstat(ifd, &sp);
    buf_size = sp.st_size * ST_BLKSIZE (sp);
    buf = (char *) malloc (min(512000L,buf_size + sizeof (long)));
    while ( buf == NULL && buf_size > 1024 ) {
        buf_size /= 2L;
        buf = (char *) malloc (buf_size + sizeof (long));
    }
    if( buf == NULL ) {
        error (0, errno, "%s", "not enough memory to hold the file buffer");
        goto ret;
    }
  while ((len = read (ifd, buf, buf_size)) > 0)
#else
  while ((len = read (ifd, buf, sizeof(buf))) > 0)
#endif atarist
    {                                            /* mjr: main copy loop */
      int wrote = 0;

   ...

      /* inside the main copy-loop */
   ...

          len -= wrote;
        } while (len > 0);
    }                                           /* mjr: end of main copy loop */

#ifdef atarist
        free(buf);
#endif atarist

   ...

 
 
 

1. Porting GNU fileutils to MiNT

I've put a few hours work into porting some of the GNU fileutils to
MiNT.  So far I've got a (more-or-less) working cat, chmod, cp, cut,
dir, du, head, ls, mkdir, mv, rm, rmdir, tail, and vdir.  I've also
got compiled cmp, dd, ginstall, and paste but I haven't tried them yet
(I don't even know what they're for...)  

Compiling them was easy except GCC kept crashing (Bus error or illegal
instruction, both GCC 1.37 and 1.39) and there seems to be a typo in
<types.h> (in struct utimbuf, axtime instead of actime, but I'm not
sure if that was on purpose or what since I don't know what's supposed
to be there...anyone know?)  The rest of the utilities will require
more work to port than I have time for right now.  If anyone wants to
pick this up from here I'll send you what I've got so far.  I'll
probably post them when I've cleaned them up a bit but if anyone wants
them now to work on them, just let me know...

Eric and Jwahar, thanks for all your effort which made this sort of
thing possible (are you guys getting sick of me thanking you yet?)

entropy

2. SSS

3. GNU fileutils 1.4

4. Question -- Script??

5. Query: TINY pictures and speed-ups for desktop publishing

6. initialize FileName error?

7. Fileutils on Atari ST (tos)

8. cable router & digital TV

9. Speeding up processing speed

10. Mint NAES lock-ups

11. Newsie Lock-ups

12. question about tape back ups

13. Shoot-em-ups.