Art Alexion on 30 Jul 2006 00:31:17 -0000


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] Large data transfer


On Saturday 29 July 2006 15:22, bergman@merctech.com wrote:
> Well, rsync may be doing encryption, depending on the transport mechanism
> specified (ie., it can use ssh natively), and scp can be configured to use
> blowfish (quite fast) and do compression. SCP doesn't have the overhead of
> checking to see if files exist on the destination end, so they may be
> closer than you think, particularly if the data consists of a relatively
> small number of large files.

This is a large number of files of size ranging from text files of 1kb or less 
per directory, lossy audio files of 3.5-12 MB, lossless audio files from 
20-100 MB, and a few graphics files in the 500 kb - 3 MB range. 10k-15k files 
total.


> =>
> => If you do rsync, your going to get data integrity, and the rate will  
> => be very fast.
>
> However, if the data is in a very wide or deep tree (or a very, very large
> number of files), be aware that rsync can have some very large memory
> requirements, and a there will be a significant delay until the first data
> is transferred, as it builds a tree in-memory of everything to be sent.

I assume this describes me.

>
> You might want to consider using tar, as in:
>
>         tar cf - /source/directory/path | (cd /destination/directory/path ;
>                 tar xf - )
>
> The pipe shown above (for a locally mounted disk) could be replaced by a
> network transport mechanism (ssh or nc) to move data over the network.
> Depending on your flavor of tar, there are options for compression, verbose
> logging, ec.

No problem mounting the share for the transfer.  Is this what you are 
recommending?

-- 

_____________________________________________________________
Art Alexion
Arthur S. Alexion LLC

PGP fingerprint: 52A4 B10C AA73 096F A661  92D2 3B65 8EAC ACC5 BA7A
The attachment - signature.asc - is my electronic signature; no need for 
alarm.
Info @ 
http://mysite.verizon.net/art.alexion/encryption/signature.asc.what.html
_____________________________________________________________

Attachment: pgpzx15PdRuSj.pgp
Description: PGP signature

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug