| Keith C. Perry via plug on 13 Jul 2020 12:13:56 -0700 |
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
| Re: [PLUG] Best Solution for Multiple Volume Backups |
I would be interested in what you come up with because after dealing with back up technology for very long time and seeing them fail when most needed, I just stopped doing them and switched to snapshotting and keeping 2 and soon 3 copies (offsite or offline). I also only keep a rolling 2 to 3 years of data in the backup system so my system is probably smaller than what you are looking for. 10+TB is very large change set. Have you considered running nilfs2 for your backup system filesystem and dumping that data to an LTO tape set? What I currently do dump to an 8Tb filesystem container on my LizardFS net. I'm able to keep about 6 months of data (weekly snapshots). Then the most recent snapshot is rsynced to a drive that is stored when not in use. What is coming up this summer is that I will have another location coming online that will mirror that 8Tb container. This way I'll have 6 months of backups in two places plus the most recently copy on a local drive stored offline. For me this avoids have to keep up with LTO tapes and storage systems (which becomes a pita when you have to decide what to do with the tapes and equipment that pass the 3rd generations backwards). One of the things I have never investigated is being able to compare dataset to see what's been added or deleted. I certainly could mount two snapshots and diff a file system tree but that isn't as slick as what you are probably envisioning. If wanted to build my system out, I would store that data in a database so I could perform that sort of a analysis but in practice I've always been able to find what I need quickly so that would really be overkill for my needs. BTW, are you using LTO tapes? I'm way out of sync with tape technology so if you are using another type of tape technology let me know. ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ Keith C. Perry, MS E.E. Managing Member, DAO Technologies LLC (O) +1.215.525.4165 x2033 (M) +1.215.432.5167 www.daotechnologies.com ----- Original Message ----- From: "Rich Freeman via plug" <plug@lists.phillylinux.org> To: "Philadelphia Linux User's Group Discussion List" <plug@lists.phillylinux.org> Sent: Monday, July 13, 2020 2:09:46 PM Subject: [PLUG] Best Solution for Multiple Volume Backups I want to backup one or more paths potentially across multiple volumes. I want support for: * Include/Exclude * Incremental backups * Tracking deleted files * Ideally not too dependent on mtime * Backup solution is itself easy to backup/restore (might just do this by running it in a container and sticking a tarball on the backup drive) * If it uses indexes/etc these should be re-creatable if lost * Backups must be able to span multiple devices which may or may not all be mounted at once * 100% CLI. I don't mind if it has a nice X11 client but I need to be able to run from the CLI, or do things like hit the start button on X11, then kill the X server and the backup keeps running and I can just re-connect to check on it. * Reasonably atomic - full backups are going to take many hours so stuff is going to change while running. It might be able to work around this with snapshots I guess... Backup sizes are going to be large - 10+TB. It wouldn't hurt if this can be interrupted/resumed but if not there needs to be some reasonable way to restart a backup if interrupted (ie interrupted incremental shouldn't require a new full backup to recover). I'm pretty sure Bacula could do the job, but it is a bit heavyweight and I suspect all the media management might get cumbersome with disks. I don't want to be rotating $200 hard drives the way you might rotate $10 tapes. Are there other solutions that might make sense? I'll probably start looking at Bacula again - in the past my biggest concern was that it was hard to back up, but sticking it in a container and creating a tarball might solve that. -- Rich ___________________________________________________________________________ Philadelphia Linux Users Group -- http://www.phillylinux.org Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug ___________________________________________________________________________ Philadelphia Linux Users Group -- http://www.phillylinux.org Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug