Re: [PLUG] Backup solution

[Rich Freeman via plug <plug@lists.phillylinux.org>]

On Wed, Feb 3, 2021 at 5:02 PM Keith <kperry@daotechnologies.com> wrote:
>
> On 2/3/21 4:16 PM, Rich Freeman wrote:
> > LOL, my duplicity daily backups currently use 780GB of space on AWS.
> > That includes two full backup sets (a full plus a month or two of
> > incrementals in each set).  My permanent backups of stuff that doesn't
> > change (which includes photos older than a month) is up to 1.9TB.
> >
> > My AWS bill is a few dollars per month.
>
> Ok.  Fair enough on the price... a few dollars I'm guessing it is under
> $10 or maybe under $5.  You have the advantage on being on FIOS.  Not
> everyone is so pushing 780Gb daily might not be possible.

Under $5.  My daily backups are usually only a few MB.  A full backup
is usually under 50GB.  The 780GB figure is the sum of two full
backups and about 4 months of daily incremental backups.

That might not seem like it should add up, but a lot of this is
photos.  I take about 100GB of photos per month, and I rotate about
100GB of photos every month out of my daily backup set and into the
permanent archive.  So, the incrementals get hit with about 100GB per
month of churn and there are 4 months of those.  The files being
backed up at the end of that period aren't the same as the ones at the
beginning.

> You could make the argument that daily is not necessary. That said, backing up
> 780Gb locally will always be faster and costs nothing.  If you have
> enough space, do both.

You can lose a lot of stuff in a day actually.  The daily backups cost
me nearly nothing on the days with minimal churn (a few MB of traffic)
and they ensure stuff is protected very quickly.

> If you're not pulling down your data from AWS regularly to test a full
> rebuild, you are just assuming they are doing what they need to do to
> keep your data safe.

I periodically test my backup solution, though I generally do so with
test data sets.  I'll do tests that don't transfer the data on the
full set, like just reading the indexes.

However, I think it is pretty unlikely that Amazon Glacier is just
going to randomly lose my backups.  They're far less likely to do so
than anything I'd improvise.

> I would seriously recommend that anyone
> using any cloud or remote resource go through the entire process to make
> sure you understand it... especially the timeframe.  I can envision
> scenario's where waiting 24 hours might be too long.

Certainly you should test your recovery solution.  You don't want to
find out that the only copy of your backup's gpg key is stored inside
the backup.  When doing a restoration be sure to start out with
nothing - such as a blank VM.

> I'm using 10Tb mirrors now so 2Tb doesn't sound like to me :) (most of
> that is a 5Tb nilfs2 filesystem image that contains versions of my file
> system dumps).  I hear you though- your point is to get your data
> offsite and really that is the point.

Oh, I have 74TiB of space in LizardFS.  Most of that isn't backed up
to the cloud.  The cloud backup only covers my most important stuff.
I have onsite backups of everything that isn't completely expendible,
but those are more easily lost.

-- 
Rich
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug