Lee H. Marzke via plug on 4 Feb 2021 07:52:44 -0800


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] Backup solution


I've been using restic for my Ubuntu laptop for several years now.

It supports incremental backup, de-dup,  pruning old data by age.  Individual backups are snapshots
of the repository of chunks.

Note that breaks up data into small chunks identified by SHA1 sum or similar, and often
packs many of these chunks into a single container file to reduce number of files.
Because of indexing of chunks, and a local cache of the index, it's possible to efficiently
incrementally backup to non mounted FS such as S3.   Many other back-ends are supported such as
Backblaze, Azure, Google Cloud, SFTP, etc.

The splitting of files into chunks is done such that small changes to data ( like adding a few character to a file)
only affect the current chunk, and don't change the boundry of the next chunk,  preserving the
de-duplication.

Encryption of chunks on the fly to backend supported.

It supports GFS backups ( grandfather, father, son ) ,e.g. daily , weekly, monthly backups.

I wound up writing a bash front-end to restic , and call it by cron nightly. 

The 'forget' operation marks outdated chunks for removal,  and the 'prune' operation
which takes quite a while deletes the marked chunks , makes new containers and uploads
them to storage.


There are summary and verification options of your repository.

I'm finding that for my 600GB home partition on SSD, that typical backups take 5 to 10 minutes, 
to find and upload new chunks. I run it via Cron nightly.  A forget/prune operation run about
once a month takes about 45 min.

A local cache of index records is kept on your computer so some space is required.
The cache is ~/.cache/restic/____randomString_______/...
For my computer that is taking ~ 550MB

bash_completion supported by running. ( On Ubuntu )
sudo restic generate --bash-completion /etc/bash_completion.d/restic 

or for man pages
sudo restic generate --man /usr/share/man/man1


If those options sound interesting, suggest you try it out, YMMV.  Available in both snap and Apt
packages in Ubuntu, and 'Go' builds on restic.net

My PLUG 2018 talk on Restic is here:
https://www.youtube.com/watch?v=dGlP7xaOc2c

Lee

----- Original Message -----
> From: "Philadelphia Linux User's Group Discussion List" <plug@lists.phillylinux.org>
> To: "Robert" <mlists@zoominternet.net>
> Cc: "Philadelphia Linux User's Group Discussion List" <plug@lists.phillylinux.org>
> Sent: Wednesday, 3 February, 2021 19:06:26
> Subject: Re: [PLUG] Backup solution

> In the message dated: Tue, 02 Feb 2021 23:31:48 -0500,
> The pithy ruminations from Robert via plug on
> [[PLUG] Backup solution] were:
> => Hello,
> =>
> => I am looking for a good backup solution for my home system. I've been
> => running dd and making a complete backup of my system but this requires
> => that I stop the system and the boot from USB to get this done.
> =>
> => Now I'm looking for a better solution.  I prefer a bare bones backup if
> => at all possible but anything really close to this with minimal OS
> => install would be great.
> =>
> => Any suggestions?
> 
> 
> There are many possible backup solutions, but without knowing your needs &
> constraints
> any suggested backup system is just a guess.
> 
> For example, these are some of the questions that you might want to answer for
> yourself when thinking about how & why you do backups:
> 
>	how much data do you have to backup (total volume for a "full" backup)
> 
>	how much (volume) data changes in a given time (frequency)
> 
>	what's your comfort level with potential data loss -- do you need a full backup
>	every 4 hours or
>	every 4 months? Do you need an incremental backup nightly or weekly? Etc., etc.
> 
>	do you need multiple backups (ie., 4 fulls, spanning 1 year) or is a single copy
>	sufficient
> 
>	do you need to store backups off-site
> 
>	how much time can a backup take to complete (impact on system performance, files
>	changing during the backup, data loss window)
> 
>	how many machines are being backed up -- are they networked
> 
>	what OS[es] are being backed up
> 
>	is commercial backup software acceptable, or free only
> 
>	do the backups need to be encrypted
> 
>	how much effort can you expend in configuring the system, making backups, and
>	testing restores
> 
>	what's your budget
> 
>	what is the primary risk you're trying to mitigate via backups (for example:
>	accidental deletion of individual files,
>	silent file corruption, failure of primary hard drive, legal requirement for
>	audit trail)
> 
> Mark
> 
> =>
> => --
> =>
> => Regards,
> => Robert
> =>
> =>
> => Smile, it increases your face value.........
> =>
> => ___________________________________________________________________________
> => Philadelphia Linux Users Group         --        http://www.phillylinux.org
> => Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> => General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug
> =>
> 
> --
> Mark Bergman    Biker, Rock Climber, SCUBA Diver, Unix mechanic, IATSE #1
> Stagehand
> '94 Yamaha GTS1000A^1					      2015 Aprilia Caponord
>	  		https://www.flickr.com/photos/rmsppu
> 
> ___________________________________________________________________________
> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

-- 
"Between subtle shading and the absence of light lies the nuance of iqlusion..." - Kryptos 

Lee Marzke  http://marzke.net/lee/
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug