Re: [PLUG] How to Store Video Files for 25 Years?

[Rich Freeman <r-plug@thefreemanclan.net>]

On Fri, Dec 7, 2018 at 4:28 AM Casey Bralla <MailList@nerdworld.org> wrote:
>
> How do I store the files so that they are secure and readable in 25 years?

As Steve mentioned, I'd plan on having to do SOME kind of migration
along the way - perhaps more than once.  This sort of thing comes up
at work all the time.  Managers want to never throw anything away, and
you have to explain the true costs of actually keeping it accessible.
Too often I see half-hearted attempts that just result in spending
money and not actually providing useful access to the data down the
line.

Also, you need to consider the entirety of the problem.  If you have
the world's fanciest tape/DVD/whatever and it melts in a fire, it is
lost.  So, factor redundancy into your plan, and periodic checks to
ensure the data is STILL redundant.  I'd plan for integrity testing
along the way so that you know if a copy is actually good (vs just
being different).  That is built into some filesystems/etc
(zfs/btrfs), or you can easily just store files with hashes, or use
something like par2.

I'd optimize your strategy for data integrity, redundancy, and ease of
migration.  You probably WILL have to migrate the data, so not having
it scattered across 100 discrete devices will make that easier.

>
> My options:
>
> 1.  DVD.   Pros:  Media will survive intact in normal environments.
> Cons:  Bulky.  Will you be able to buy old DVD players at Goodwill in 25
> years?

DVD CAN survive quite a long time, but you need to choose media
specifically rated for this, or you're taking your chances.  Don't
just buy a random spindle for $20 and toss it in a closet.  Read the
storage conditions on the media you use - if it is intended for
archival it WILL specify these, as these are the conditions the
manufacturer did their own testing.

And of course you need a device to read them.  They'll be around for a
while.  They might not be around for 25 years, in which case you'll
need to migrate.

One advantage of DVDs is that they're read-only.  Maybe you have to
copy them to something else at some point, or rewrite one from a
backup when it fails an integrity test.  However, at all times when
you're doing integrity tests or retrieving data the existing copy
isn't going to accidentally change.  After you make a copy you can
make sure it is good before discarding the original.  You don't get
this with a lot of the other alternatives - with a hard drive maybe a
virus wipes out or silently encrypts your data while you're doing a
routine integrity check, and so on.

> 2.  USB Hard Disk:  Pros:  Media will probably survive.  Good size (not
> too big, but not too small to lose)   Cons:  Will USB 3.0 still exist?

A hard drive sitting in a closet will NOT survive 25 years.  You CAN
use hard drives for archival, but you need to plan for the certainty
of drive failure.  That means multiple copies, with periodic checks.
This sounds like work, but it might actually be one of your better
options, because capacity just keeps going up which means that even if
you don't start with only one drive per copy you'll end up there
quickly as you replace drives.

I actually use old hard drives for this purpose.  I just set them up
as ZFS mirrors and drop them in an enclosure.  Once in a while I plug
them in and run a scrub.  But, this isn't stuff I care THAT much
about.

Hard drives are definitely optimized for migration though.  Checking
and migrating the data is very easy, and if you're always keeping up
with it and turning over the media every few years anyway, then you'll
never worry about being faced with a 25 year old SATA interface that
only works in a museum.

> 3.  USB SSD:   Pros: Small size.  Physically robust for storage. Cons:
> Will SSD data last 25 years?   Will USB 3.0 still exist?

This might be an alternative to hard drives, but honestly I have no
idea how stable flash is at rest.  You might get away with less
frequent tests, or not.  Read up before you do this option, but I'd
otherwise treat them the same as hard drives.

> 4.  USB Thumb Drive:  Pros: Easy to mail to them.   Cons:  Small, so can
> get lost easily.  Will data last 25 years?  Will USB 3.0 exist?

No different from SSD really, except almost certainly manufactured at
a lower level of quality.

>
> Any suggestions?

Two more options: cloud and tape.

I'd strongly consider the cloud, and especially something like Amazon
Glacier.  That is basically doing one of the other strategies here but
outsourcing all the work.  A reputable cloud provider is going to be
storing multiple copies in appropriate media/conditions and refreshing
everything as needed.  For the sake of security I'd consider using two
different reputable providers and giving each a copy.  Try to see if
you can find one that provides some ability to check up on things
without incurring much cost.

Some specific thoughts regarding Glacier:  It costs nothing to upload,
which is the one operation you're definitely doing.  The storage costs
are pretty low.  It is specifically intended for long-term storage.
Where you pay more is on the retrieval side, but you can't be certain
it will even be retrieved, and if it is retrieved IN 25 YEARS I can
practically guarantee that the data transfer costs will be MUCH lower
than they charge today.

I'd also consider the cloud as a backup to your backup.  That is,
maybe you have DVDs or whatever.  Just upload an extra copy to the
cloud.  Maybe you'll never need it (and thus never pay retrieval
fees).  Maybe you'll be happy you did it.

One downside to the cloud is obviously security.  You can obviously
solve this with encryption, but key management will be critical.  If
you lose the key, you lose the data.  Also, you need to make sure the
decryption software is around in 25 years.  Don't assume that merely
uploading a copy of gpg will do it either - that might be about as fun
as dealing with a copy of gpg for DOS that you uploaded 25 years ago,
and I wouldn't assume x86 is a thing in 25 years either.  If your
family movies are of the mundane variety you might seriously just
accept the risk that some guy at Amazon might end up watching them.

Another option is tape.  IMO this isn't going to be worth it unless
you have a LOT of data to store, as the hardware to read/write it is
pretty expensive.  I'd consider it a high-volume alternative to DVDs,
but tape is definitely a standard solution to this kind of problem.
Again, you need good-quality media stored under the specified
conditions, and even then I would plan on having to refresh it.
Depending on the implementation tapes may also have read-only lockout
at the hardware level which provides some additional security while
manipulating it.  But, definitely read up on this because if the
implementation is on the software side you're not as safe.  Viruses
that encrypt and ransom tape libraries aren't a thing either, as far
as I'm aware, unlike ones that attack mounted hard drives.

Finally, a more somber note.  Do consider how likely it is that you'll
ever use it.  When I was archiving memories after my wife died I noted
how many things we had saved that we had never really looked at.
There is a good chance that they'll be sitting in a box that hasn't
been subsequently opened when I pass away.  Much of this stuff was not
of interest to the kids, though that could change and so I save the
more valuable stuff.  So, don't make this process too burdensome for
yourself.  One advantage of using hard drives is that you can keep
adding to them, which may help reduce the burden.  However, you still
need to be checking them and turning them over as you WILL have
failures.  One of the advantages of the Glacier approach is that most
of the cost is for retrieval, and if you never retrieve, then you
never pay it.

And do let your kids/family/etc know where you keep this stuff.
Otherwise they might never know it is there, especially with the
cloud.  If they have to retrieve it after you're gone that old museum
tape drive might as well be a paperweight unless you've arranged for a
backup of yourself to do it for them.

--
Rich
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug