[PLUG] Compression in archives: was How to Store Video Files for 25 Years?

[Steve Litt <slitt@troubleshooters.com>]

On Fri, 7 Dec 2018 16:50:51 -0500
Rich Kulawiec <rsk@gsp.org> wrote:


> 7. Compression is tempting...but don't.
> 
> The cost of storage continues to plummet.  The cost of recovering
> from a compression error (if you even can) is enormous.

I've been making my personal and business backups for 30 years now.
These backups are intended to last a decade or longer, so hardware,
software and format are selected to stand the test of time. But not
once in that thirty years has the thought of not compressing popped
into my brain. It's an interesting thought.

Upon further thought it's obvious that the easiest format for recovery
would be verbatim copies of the files on my optical disc, probably
accompanied by an md5sum of each. Even if things go (partially) wrong,
such a protocol would be partially recoverable by a knowledgeable
person. From the point of recovery, it's ideal.

But some issues must be resolved...

* If you have tens of thousands of files, it's very possible to overrun
  the number of inodes or whatever the format's file id structure.

* The Linux definition of filenames might not match the optical disc
  format's definition. For instance, pure iso9660 does only 8.3. Even
  Rockridge and Joliet have, if I remember correctly, deviations such
  that some Linux filenames might not be copyable to the optical disc.

* It's possible that not all your Linux file metadata will be copyable
  to the disc.

* If one is willing to use tar to turn all files into one file, tar
  (without any compression) can solve the three preceding problems
  without making recoverability from minor corruption impossible.

* But the economics of the situation often dictate this not be done
  often. My compressed backups span more than one Blu-ray, and would
  span about 8 DVDs. Going unencrypted could almost double these
  figures. And although spinning rust drives have doubled in capacity
  every few years, optical and other non-tape backup formats haven't
  kept pace.

* With many archives and backups, you're in a heap of trouble if
  somebody else gains access to them, hence encryption. Once you
  encrypt the data on an optical disc, recoverability is slim anyway in
  the face of damage.


After reading your #7 suggestion, I'm thinking of making a very few
backups; maybe one a year, no-encrypt no-compress, kept safely in a
bank vault.

Thanks for the great idea.

SteveT

Steve Litt 
December 2018 featured book: Rapid Learning for the 21st Century
http://www.troubleshooters.com/rl21
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug