Re: [PLUG] Transfer REALLY large file(s) between Linux and OS X

["W. Chris Shank" <chris.shank@acetechgroup.com>]

I was able to transfer the 28G file from a 500Mhz Dual P3 machine to a
73Mhz G4 OS X Firewire attached drive using scp (from CLI) in about 1
1/2 hours. I started to use sftp via transmit - but it estimated the
transfer taking 53 hours.

My load on the servers was light - but I did still use the P3 box during
this time and didn't notice any problems - even as i was making the
tarball.
 

On Sat, 2003-10-18 at 23:48, gabriel rosenkoetter wrote:
> On Sat, Oct 18, 2003 at 09:35:41PM -0400, Mental Patient wrote:
> > Could you explain why? The encryption overhead is negligable on modern 
> > processors.
> 
> That's flatly not true. s{cp,ftp} a moderately sized file (10 GB or
> larger) and watch how much processor it chews up. And then be glad
> you're not trying to do something with the machines at either end.
> Now consider how long that'd go on if you were transfering something
> that's actually large (1 TB or greater).
> 
> Using a cipher that's quick on 32-bit processors (blowfish isn't
> bad, twofish is better, but I don't know if even OpenSSH supports
> that yet) helps, but you're still going to lose ~50% processor for
> the duration. 3DES will kill your cycles completely for the duration.
> 
> People sell (and make good money from selling) crypto accelerator
> cards for very good reasons.
> 
> > I regularly tunnel rsync over ssh. Whats the problem?
> 
> Probably not much, since rsync is precisely designed to transfer as
> little information at possible... at the cost of processor devoted
> to the checksum calculations at either end. (The checksums are
> probably cheaper than the streaming crypto, unless you're
> off-loading the crypto, in which case rsync will be a net loss
> unless time of transfer is the only relevant metric.)
> 
> I'd also guess that you probably aren't actually trying to *use* the
> endpoints of this transfer while it's in progress.
-- 
W. Chris Shank
ACE Technology Group, LLC
chris.shank@acetechgroup.com
http://www.acetechgroup.com

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug