|
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
|
Re: [PLUG] Large data transfer
|
In the message dated: Sat, 29 Jul 2006 12:01:12 EDT,
The pithy ruminations from John Von Essen on
<Re: [PLUG] Large data transfer> were:
=> By all means rsync is the way to go. It will be faster then scp since
=> it doesn't have to encrypt the data. And unlike regular FTP, rsync
=> will ensure that you have an "exact" data copy.
Well, rsync may be doing encryption, depending on the transport mechanism
specified (ie., it can use ssh natively), and scp can be configured to use
blowfish (quite fast) and do compression. SCP doesn't have the overhead of
checking to see if files exist on the destination end, so they may be closer
than you think, particularly if the data consists of a relatively small number
of large files.
Some FTP clients don't support dynamic creation of directory trees, making that
a complete loss.
=>
=> I also dont recommend the USB drive thing. I have messed with this in
=> the past, and though it looks good on paper, in reality it always
=> ends up sucking. If any device in the mix is USB1 (the drive, the
=> port, etc.,.) your transfer rate will be really low, like 950KBps.
=> USB2 is fast, but alot of servers and desktops that arent that old
=> still have USB1 in them.
Absolutely!
=>
=> If you do rsync, your going to get data integrity, and the rate will
=> be very fast.
However, if the data is in a very wide or deep tree (or a very, very large
number of files), be aware that rsync can have some very large memory
requirements, and a there will be a significant delay until the first data is
transferred, as it builds a tree in-memory of everything to be sent.
You might want to consider using tar, as in:
tar cf - /source/directory/path | (cd /destination/directory/path ;
tar xf - )
The pipe shown above (for a locally mounted disk) could be replaced by a
network transport mechanism (ssh or nc) to move data over the network.
Depending on your flavor of tar, there are options for compression, verbose
logging, ec.
Of course, ideas of what constitutes a "large" data transfer vary...I'm
presently working out how to move 6~9TB from our lab to the new datacenter...
USB is not an option!
Mark
=>
=> -John
=>
=>
=> On Jul 28, 2006, at 3:19 PM, Art Alexion wrote:
=>
=> > On Friday 28 July 2006 13:39, Jeff Abrahamson wrote:
=> >> On Fri, Jul 28, 2006 at 01:25:00PM -0400, Art Alexion wrote:
=> >>> This is a one time thing. Am I understand that the main advantage
=> >>> of rsync over cp is in subsequent incremental backups? --
=> >>
=> >> No, it also does a checksum. You can get around this by computing
=> >> yourself a good checksum (e.g., sha1) of each file and diffing the
=> >> checksums.
=> >>
=> >> $ find . -exec sha1sum \{\} \; > /tmp/sha-1.txt
=> >> $ # Then do same on other machine or other dir,
=> >> $ # and copy file of checksums to /tmp/sha-2.txt
=> >> $ diff /tmp/sha-[12].txt
=> >
=> > No reason I didn't want to use rsync (other than never having used
=> > it before).
=> > Seems easier for me just to read rsync --help and learn how to use
=> > it rather
=> > than do the checksums manually.
=> > --
=> >
=> > _____________________________________________________________
=> > Art Alexion
=> > Arthur S. Alexion LLC
=> >
=> > PGP fingerprint: 52A4 B10C AA73 096F A661 92D2 3B65 8EAC ACC5 BA7A
=> > The attachment - signature.asc - is my electronic signature; no
=> > need for
=> > alarm.
=> > Info @
=> > http://mysite.verizon.net/art.alexion/encryption/
=> > signature.asc.what.html
-----
Mark Bergman Biker, Rock Climber, Unix mechanic, IATSE #1 Stagehand
http://wwwkeys.pgp.net:11371/pks/lookup?op=get&search=bergman%40merctech.com
I want a newsgroup with a infinite S/N ratio! Now taking CFV on:
rec.motorcycles.stagehands.pet-bird-owners.pinballers.unix-supporters
15+ So Far--Want to join? Check out: http://www.panix.com/~bergman
___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
|
|