Very Computer

Hello!

I'm looking for the fastest way to transfer files between 2 SUN
boxes sitting on a 100MB full-duplex ethernet.  Data: ~100Gb of
binary files ~30Mb each.

Copying and moving disks around is not an option - don't have
physical access to the servers.

Files are not organized by filesystem and/or directories.  I.e. I
have to transfer a list of files spread all over the places.

Things I'm considering:

1.  cd /net/src; tar cf - list_of_files | (cd /net/dst; tar xf -)
2.  for i in list_of_files; do rsync -options /net/src/$i /net/dst/$i; done
3.  for i in list_of_files; do ftp each file; done
4.  for i in list_of_files; do cp each file; done

For all 4 options I can break the list_of_files into sublists and
do them in parallel.

- Approximately 5% of files are the same on src and dst servers;
other files may have the same names, but are different and not
even similar (chunks of a database)

- Files compress with gzip 1:2 - I could reduce the total amount
of data travelling over the link by 2 via "z" option in tar or
rsync, but I'm afraid the overhead of gziping/ungzipping them
will consume more time, than I save on the network transers.
Server details: SRC server - E6500, 22 400MHz cpu, 48Gb RAM;
DST server - E10K, 4 333MHz cpu, 4Gb RAM.

Any thoughts on the above 4 methods and any other ideas on moving
things will be appreciated.

Thanks,

--
Andrei

In my experience, FTP has always been faster than rcp/rsync or NFS
(I'm assuming it's over NFS that you proposed to use tar and cp).  I
would recommend moving everything via FTP and then using rsync to
regain permissions and ownerships--that combination should give you
the best results in the shortest time.  But you may want to test
various options with a small set of files first to verify.

Victor

In some tests I did a couple of months ago I've also found that FTP was the
fastest.
wget will help if you decide to go with FTP.

rsync / rdist can be very fast if the files are not changing a lot.

Alberto

If there's a chance that some of the files *won't* have changed every
time you're doing the copy, rsync's option to only copy updated files
might come in handy.  3,000 30Mb files is a good bit of data, and if
you only *need* to copy 2,000, doing so is a win.

But if you have to change their names in the process, I'm not sure how
well rsync would handle that.

Quote:> Server details: SRC server - E6500, 22 400MHz cpu, 48Gb RAM;
> DST server - E10K, 4 333MHz cpu, 4Gb RAM.

Nice toys.

-Dan

--
Dan Birchall - Palolo Valley - Honolulu HI - http://dan.scream.org/
My addresses expire... take out the hex stamp if your reply bounces