Fastest way to transfer whole filesystems?

Fastest way to transfer whole filesystems?

Post by Andy Pippin / MIS Sun Admin. (Syste » Fri, 13 Nov 1992 06:55:56




>What's the fastest way to transfer an entire filesystem from one
>disk partition to another (different type disk, larger partition)?
>I thought that a tar would be much faster than a dump and restore
>from tape, but when I tried it, using a command like:

>    tar cf - . | ( cd /otherfs; tar xpf - )

>I got a surprising slow rate, something like 2 minutes per megabyte.  Am I
>using the wrong command, or using the right command the wrong way, or is this
>just an inevitably slow procedure?  What I'm moving is the news partition,
>and I've already learned that restoring it from dump tape is horribly slow.
>The system is a Sun SPARC machine.

>Thanks for any assistance.

        I don't use tar to copy filesystems - it cannot handle devices and
        the like.  I use the dump/restore pipe that is in the restore(8)
        man page.

                dump 0f - /dev/rxy0g | ( cd /mnt; restore xf - )

        It's not too bad for speed - but you're definitely I/O bound on this
        type of command.

    aBp.
---
Andy Pippin, Systems Administrator                An exit from one place,

 
 
 

Fastest way to transfer whole filesystems?

Post by Susan Co » Fri, 13 Nov 1992 08:55:15



>What's the fastest way to transfer an entire filesystem from one
>disk partition to another (different type disk, larger partition)?
>I thought that a tar would be much faster than a dump and restore
>from tape, but when I tried it, using a command like:

>    tar cf - . | ( cd /otherfs; tar xpf - )

>I got a surprising slow rate, something like 2 minutes per megabyte....  

Thanks for the many replies!  It appears that the majority recommends
using dump and restore instead of tar, as in (to quote one example):

  dump 0f - /filesystem1 | (cd /filesystem2; restore rf -)

Another recommendation was:

  find . -xdev -depth -print | cpio -pd /newdir

And more than one person pointed out that /new has to created so many inodes
that it's bound to be somewhat slow to restore it.

- Susan

 
 
 

Fastest way to transfer whole filesystems?

Post by Jurgen Bo » Fri, 13 Nov 1992 13:02:41



>    I don't use tar to copy filesystems - it cannot handle devices and
>    the like.

GNU tar can.
--


Mount Holyoke College        |      Voice: (US) 413-538-2375 (daytime)
South Hadley, MA, USA        | Snail Mail: J. Botz, 01075-0629
 
 
 

Fastest way to transfer whole filesystems?

Post by Marc Unang » Fri, 13 Nov 1992 14:28:19



>just an inevitably slow procedure?  What I'm moving is the news partition,
>and I've already learned that restoring it from dump tape is horribly slow.

Unfortunately, copying it with cpio -p or two tars in a pipe is going
to be slow for the same reason that restoring it from a dump tape is
slow -- you are restoring lots of little files, and there is a lot of
overhead involved in creating the directory entries for those little
files.  There really is no fast way of doing this, other than a
sector-by-sector copy of the entire disk {en masse}; unfortunately you
can't do this if you're going to a different size or type of disk.

--
Marc Unangst, N8VRH         | "There are two ways to solve this problem:

                            | with the hard way."
                            |   - W. Scheider, from a Physics lecture

 
 
 

Fastest way to transfer whole filesystems?

Post by Jess Alvara » Fri, 13 Nov 1992 22:49:33


Try using CPIO.

--
=======================================================================

| TRT COMMUNICATIONS, INC.       |         UUCP:    uunet!engx25!jess |
| 1352 NW. 40th Ave.             |         Voice:   (305) 797-3167    |

 
 
 

Fastest way to transfer whole filesystems?

Post by Ken L » Fri, 13 Nov 1992 01:19:49



>What's the fastest way to transfer an entire filesystem from one
>disk partition to another (different type disk, larger partition)?
>I thought that a tar would be much faster than a dump and restore
>from tape, but when I tried it, using a command like:
>    tar cf - . | ( cd /otherfs; tar xpf - )
>I got a surprising slow rate, something like 2 minutes per megabyte.  Am I
>using the wrong command, or using the right command the wrong way, or is this
>just an inevitably slow procedure?  What I'm moving is the news partition,
>and I've already learned that restoring it from dump tape is horribly slow.
>The system is a Sun SPARC machine.

I've had to copy entire partitions, I just used cp -R.  It doesn't work
correctly with links though.  I believe that this should be a bit faster
since, piping involves an extra step of creating a temporary file.

--
Ken Lam                                                     (803)-884-7615 (H)
System Administrator                                        (803)-792-4969 (W)
Senior Systems Consultant                                   (803)-792-5446(FAX)

MultiMedia Works & Video Productions ======= Teaching the world in better ways

 
 
 

Fastest way to transfer whole filesystems?

Post by Gary Hest » Sat, 14 Nov 1992 00:32:36



>What's the fastest way to transfer an entire filesystem from one
>disk partition to another (different type disk, larger partition)?

Probably cpio, with the p option:

        find . -print | cpio -pd /newdir

Adding a blocking factor might help; I've had acceptable performance
from this and haven't bothered with one.

--

The Chairman of the Board and the CFO speak for SCI. I'm neither.
"Data sheet: HSN-3000 Nuclear Event Detector. The [NED] senses the gamma
radiation pulse [from a] nuclear weapon." As if we wouldn't notice...

 
 
 

Fastest way to transfer whole filesystems?

Post by Randy Dav » Sat, 14 Nov 1992 08:49:27


|I've had to copy entire partitions, I just used cp -R.  It doesn't work
|correctly with links though.  I believe that this should be a bit faster
|since, piping involves an extra step of creating a temporary file.
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Not in any flavor of UNIX I've seen.  Perhaps a FIFO in some instances,
but no temporary data file.

  The dump | restore method is the fastest under SunOS - of course, I've only
been able to get this work acceptably when copying a COMPLETE partition to
another drive.  If you want to copy only PART of the partition, I've had
better results from cpio ("find . -depth -print | cpio -pdmuv /destination")
than from tar.  This, of course, makes sense, since this was what cpio was
designed for.

  Of course, if you are copying a filesystem with inodes greater than 65k and
plan on using cpio under SunOS 4.1, 4.1.1, or 4.1.2 (I don't know about 4.1.3),
make sure that your cpio has been patched (patch 100556).  There is a bug in
cpio under SunOS 4.1.[12] where the table keeping track of the hard links was
defined as a "short", when a short won't cut it when you exceed 65536 as an
inode value...  (We found this out the hard way, when moving some 1800 MB
partitions using cpio).



Megatek Corporation, San Diego, California    ucsd!megatek.uucp!randy

 
 
 

Fastest way to transfer whole filesystems?

Post by Rodney Campbe » Tue, 17 Nov 1992 15:44:02



>What's the fastest way to transfer an entire filesystem from one
>disk partition to another (different type disk, larger partition)?
>I thought that a tar would be much faster than a dump and restore
>from tape, but when I tried it, using a command like:
>    tar cf - . | ( cd /otherfs; tar xpf - )

If you are moving files on the same machine then you can use the simple
version of cpio. Try something like:

% cd sourcedir
% find . -depth -print | cpio -pdmv destdir

This will make a nice duplicate of the filesystem within sourcedir at the
destdir location, symlinks, permissions, ownerships and all.

                                                                Rodney...
--

   Advanced Network Products   |Snail : Level 1, 18-20 Orion Rd, Lane Cove West
 Customised Software Solutions |  -  PO Box A792, Sydney South 2000, Australia.
        Centre - Sydney        |Phone : +61 (0)2 911 3123   Fax: +61 2 911 3199

 
 
 

Fastest way to transfer whole filesystems?

Post by Colin Paniss » Thu, 19 Nov 1992 11:19:08




}>What's the fastest way to transfer an entire filesystem from one
}>disk partition to another (different type disk, larger partition)?
}>I thought that a tar would be much faster than a dump and restore
}>from tape, but when I tried it, using a command like:
}>   tar cf - . | ( cd /otherfs; tar xpf - )
}If you are moving files on the same machine then you can use the simple
}version of cpio. Try something like:
}% cd sourcedir
}% find . -depth -print | cpio -pdmv destdir
}This will make a nice duplicate of the filesystem within sourcedir at the
}destdir location, symlinks, permissions, ownerships and all.

You will find, however, that this creates a duplicate filesystem that actually
uses more space than did the original. This is due to the block allocation
methods used when tar and cpio both do their restore.

Instead, use dump & restore - you'll get an exact duplicate of the original,
inode for inode - and with the same degree of fragmentation as the original.

Try:

  # dump 0bf 5000 - /origfs | (cd /otherfs ; restore xf -)

"b 5000" gives you a blocking factor of 5000, which gives quite acceptable
performance for this sort of thing.

--

  -Colin Panisset *:^)                         | "CTBCPP (Clay Tablet By

                                     So There. |  defined in RFC-39127"

 
 
 

Fastest way to transfer whole filesystems?

Post by John R Ruckstuhl » Sun, 22 Nov 1992 19:20:26





> }>What's the fastest way to transfer an entire filesystem from one
> }>disk partition to another (different type disk, larger partition)?
> You will find, however, that this creates a duplicate filesystem that actually
> uses more space than did the original. This is due to the block allocation
> methods used when tar and cpio both do their restore.
> Instead, use dump & restore - you'll get an exact duplicate of the original,
> inode for inode - and with the same degree of fragmentation as the original.

NOT NECESSARILY ?

I noticed when that when I duplicated filesystems with dump & restore,
the duplicate was NOT exactly as the original --

I don't know the correct terminology to describe this, but, you know how
you make extra space in the lost+found directory structure for a
filesystem, so that fsck will have plenty of slots to put junk,

well the dump & restore didn't preserve that bloated lost+found.

So, you may want to go back and re-bloat.  :)

I'm using SunOS 4.1.1

Regards,
ruck
--



 
 
 

1. Copy whole filesystem to another filesystem over network

Here's my spec:

Server #1: (it's a Samba server)
/hda1 <-- "/"

/hdb1 <-- "/home"  it got plently of space...about 20G left

Server #2: (my linux workstation)

/hda1 <-- "/"
/hdb1 <-- "/home" a 40G 7200 HD.... only 5G are used while rest are
free....

I need to repartition /hdb (probably into 4 partitions 10G each).

I need to temporarily copy over the 5G worth of content on /hdb from
Server #2 to /hdb1 of Server #1. So I can repartition the /hdb and
then copy the content back from Server #1.

My question..what is the best way to do it?

rsync?  or just mount NFS and cp /home/* /home/server2?

want more efficient and safe and quicky way as possible without
stress...

thanks..

kellyboy

2. Shared FS with per user write

3. Some ways to get my whole lan on the net with one ip?

4. Help with sco.announce group.

5. Fastest transfer from file to socket?

6. "kudzu" option in /etc/fstab entries ???

7. Fastest possible disk data transfer?

8. ipfw rules help..

9. fastest file transfer protocol

10. Fastest File Transfer Possible?

11. Q: Is there an FTP program that can transfer a whole directory tree at once?

12. fastest filesystem for small files and symlinks

13. World's Fastest Video Card now on the World's Fastest PC