Re: [PLUG] Large data transfer

[Dan Widyono <dan@widyono.net>]

> However, if the data is in a very wide or deep tree (or a very, very large
> number of files), be aware that rsync can have some very large memory
> requirements,

I wrote a wrapper for this issue.  It dives into the tree one level (I could
imagine making this variable, which can be set from command line) and rsync's
each subdirectory.  Otherwise rsync would keel over and die from the number
of inodes it needed to handle (and this is *without* hard link analysis
turned on!).  Might be simple way for your 6-9TB transfers if you know how
your trees are balanced (or if not, run a weekly analysis on the trees and
output to a file, do the analysis before rsync runs, and rsync
appropriately).

Dan W.
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug