Re: [PLUG] Transfer REALLY large file(s) between Linux and OS X

[Jeff Abrahamson <jeff@purple.com>]

I went to a talk once by a guy from AT&T or some related Bell company.
He worked on the following problem: so many calls are made every day
that it is a real challenge to move all the billing data around and
have the day's billing data in the right places and in the right
databases before the day was over. Because the next day was going to
come, so you can't take more than 24 hours.

I don't remember the numbers, but if you assumed even 100 bytes per
billing record, you quickly saw that oracle and even many file systems
were going to reach their limits on less than a day's worth of calls.

Anyway, he said they ran into some very low probability events because
of the amount of data they were moving. Things like every few billion
(trillion?) pages copied one page would be wrong, from somewhere else
in the file.

Their solution: checksum the source and destination files after each
copy: confirm that you succeeded, not just that your copy program
returned with status 0.

So if you really care about your data, and especially after the
unexpected problems you've already had, may I suggest you consider
checksumming the file on both ends to make sure you have what you
think you have.

-Jeff


On Fri, Oct 17, 2003 at 07:38:08PM -0400, W. Chris Shank wrote:
>   [46 lines, 333 words, 1975 characters]  Top characters: etoiasnl
> 
> I've had little success moving even a few hundred megs between linux and
> os x via samba - same files between win2k and linux (linux is the samba
> server in all these scenarios) work flawlessly.
> 
> Thank you everyone for your assistance. The questions regarding HFS and
> HFS+ got me to thinking about the volume format. The disk i'm moving it
> to was setup quite a long time ago - and i had forgotten that I wanted
> to be able to plug it into a linux box as well as an OS X box (actually
> was the same box - booted to linux or os x). Of course, I selected MSDOS
> format (DOOH!!) thinking it would be the easiest format to work with
> (HFS+ is not supported in linux - or at least it wasn't when i installed
> this disk). So I beleive that is the root of my problem. I should have
> checked the format first - I assumed it to be HFS+ and asked a mac guru
> friend of mine about this as well - so he will likely beat me in person
> - saving all of you them trouble :)
> 
> Thanks again for your help. I'll be sure to let you know if reformatting
> as HFS+ solves this issue.
> 
> 
> 
> 
> On Fri, 2003-10-17 at 19:03, Tobias DiPasquale wrote:
> > On Fri, 2003-10-17 at 18:58, Adam Turoff wrote:
> > > Hm.  Maybe publishing the Linux volume over NFS or Samba might help.
> > > The Finder should be 64-bit clean, so a drag copy should work.
> > > 
> > > But don't quote me on that.  ;-)
> > > 
> > > Z.
> > 
> > That depends on what version of OS X you are running. Check this link
> > for details on the bug in version prior to 10.2.2 that will not let that
> > succeed.
> > 
> > http://www.bresink.de/osx/DocsNFSManager/Notes.html
> -- 
> W. Chris Shank
> ACE Technology Group, LLC
> chris.shank@acetechgroup.com
> http://www.acetechgroup.com
> 
> ___________________________________________________________________________
> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

-- 
 Jeff

 Jeff Abrahamson  <http://www.purple.com/jeff/>
 GPG fingerprint: 1A1A BA95 D082 A558 A276  63C6 16BF 8C4C 0D1D AE4B

Attachment: pgpTNYg6xr4vG.pgp
Description: PGP signature