Robert via plug on 4 Feb 2021 09:33:31 -0800


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] Backup solution 

Hello Lee,
Thank you for this.  I'll be sure to look at your video as this looks like something that might work for now and possibly into the future. 


On 04/02/2021 10:52, Lee H. Marzke wrote:

I've been using restic for my Ubuntu laptop for several years now.

It supports incremental backup, de-dup,  pruning old data by age.  Individual backups are snapshots
of the repository of chunks.

Note that breaks up data into small chunks identified by SHA1 sum or similar, and often
packs many of these chunks into a single container file to reduce number of files.
Because of indexing of chunks, and a local cache of the index, it's possible to efficiently
incrementally backup to non mounted FS such as S3.   Many other back-ends are supported such as
Backblaze, Azure, Google Cloud, SFTP, etc.

The splitting of files into chunks is done such that small changes to data ( like adding a few character to a file)
only affect the current chunk, and don't change the boundry of the next chunk,  preserving the
de-duplication.

Encryption of chunks on the fly to backend supported.

It supports GFS backups ( grandfather, father, son ) ,e.g. daily , weekly, monthly backups.

I wound up writing a bash front-end to restic , and call it by cron nightly.

The 'forget' operation marks outdated chunks for removal,  and the 'prune' operation
which takes quite a while deletes the marked chunks , makes new containers and uploads
them to storage.


There are summary and verification options of your repository.

I'm finding that for my 600GB home partition on SSD, that typical backups take 5 to 10 minutes,
to find and upload new chunks. I run it via Cron nightly.  A forget/prune operation run about
once a month takes about 45 min.

A local cache of index records is kept on your computer so some space is required.
The cache is ~/.cache/restic/____randomString_______/...
For my computer that is taking ~ 550MB

bash_completion supported by running. ( On Ubuntu )
sudo restic generate --bash-completion /etc/bash_completion.d/restic

or for man pages
sudo restic generate --man /usr/share/man/man1


If those options sound interesting, suggest you try it out, YMMV.  Available in both snap and Apt
packages in Ubuntu, and 'Go' builds on restic.net

My PLUG 2018 talk on Restic is here:
https://www.youtube.com/watch?v=dGlP7xaOc2c

Lee

----- Original Message -----

From: "Philadelphia Linux User's Group Discussion List" <plug@lists.phillylinux.org>
To: "Robert" <mlists@zoominternet.net>
Cc: "Philadelphia Linux User's Group Discussion List" <plug@lists.phillylinux.org>
Sent: Wednesday, 3 February, 2021 19:06:26
Subject: Re: [PLUG] Backup solution
In the message dated: Tue, 02 Feb 2021 23:31:48 -0500,
The pithy ruminations from Robert via plug on
[[PLUG] Backup solution] were:
=> Hello,
=>
=> I am looking for a good backup solution for my home system. I've been
=> running dd and making a complete backup of my system but this requires
=> that I stop the system and the boot from USB to get this done.
=>
=> Now I'm looking for a better solution.  I prefer a bare bones backup if
=> at all possible but anything really close to this with minimal OS
=> install would be great.
=>
=> Any suggestions?


There are many possible backup solutions, but without knowing your needs &
constraints
any suggested backup system is just a guess.

For example, these are some of the questions that you might want to answer for
yourself when thinking about how & why you do backups:

	how much data do you have to backup (total volume for a "full" backup)

	how much (volume) data changes in a given time (frequency)

	what's your comfort level with potential data loss -- do you need a full backup
	every 4 hours or
	every 4 months? Do you need an incremental backup nightly or weekly? Etc., etc.

	do you need multiple backups (ie., 4 fulls, spanning 1 year) or is a single copy
	sufficient

	do you need to store backups off-site

	how much time can a backup take to complete (impact on system performance, files
	changing during the backup, data loss window)

	how many machines are being backed up -- are they networked

	what OS[es] are being backed up

	is commercial backup software acceptable, or free only

	do the backups need to be encrypted

	how much effort can you expend in configuring the system, making backups, and
	testing restores

	what's your budget

	what is the primary risk you're trying to mitigate via backups (for example:
	accidental deletion of individual files,
	silent file corruption, failure of primary hard drive, legal requirement for
	audit trail)

Mark

=>
=> --
=>
=> Regards,
=> Robert
=>
=>
=> Smile, it increases your face value.........
=>
=> ___________________________________________________________________________
=> Philadelphia Linux Users Group         --        http://www.phillylinux.org
=> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
=> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug
=>

--
Mark Bergman    Biker, Rock Climber, SCUBA Diver, Unix mechanic, IATSE #1
Stagehand
'94 Yamaha GTS1000A^1					      2015 Aprilia Caponord
	  		https://www.flickr.com/photos/rmsppu

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug


--

Regards,
Robert


Smile, it increases your face value.........

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug