| Next revision | Previous revision |
| rethinking_my_backup_strategy [01.01.2021 18:50] – created Pascal Suter | rethinking_my_backup_strategy [12.08.2021 17:42] (current) – [First POC - Burp + rsync] Pascal Suter |
|---|
| the main goal of Mobi was to be as portable and simple as possible and to provide incremental backups where i had full snapshots of each backup in a simple folder structure to facilitate restores and make browsing of backups easy. | the main goal of Mobi was to be as portable and simple as possible and to provide incremental backups where i had full snapshots of each backup in a simple folder structure to facilitate restores and make browsing of backups easy. |
| |
| ===== Design Goals for Mobi 2.0 ===== | ===== What i want from my new backup solution ===== |
| * No more root access should be necessary between any of the involved machines! | * No more root access should be necessary between any of the involved machines! |
| * untrusted client machines --> we have to assume, that the machine we are creating a backup for could be hacked, so we have to make sure, that not even root on a client machine could delete or tamper with previous benchmarks. We will assume however, that the retention time for the backups is longer than the time the client machine has been hacked, meaning, we don't need to detect if the client is hacked or not.. we will continue to make backups for as long as the hacker wants to and hope that the admin of the hacked client will notice the attack before his oldest backup is purged from the repository | * untrusted client machines --> we have to assume, that the machine we are creating a backup for could be hacked, so we have to make sure, that not even root on a client machine could delete or tamper with previous benchmarks. We will assume however, that the retention time for the backups is longer than the time the client machine has been hacked, meaning, we don't need to detect if the client is hacked or not.. we will continue to make backups for as long as the hacker wants to and hope that the admin of the hacked client will notice the attack before his oldest backup is purged from the repository |
| [[https://www.borgbackup.org/|Borg]] is a very smart and capable backup solution with lots and lots of features and most importantly, block level deduplication. It can do a lot more than what I need, **BUT** it sadly does not allow to create a user that can only write new backups but not delete old ones as well. While there is a ''append-only'' policy available, it is impractical to rely on it, as one has to either forget about automatic purging of old backups. That's the deal breaker for me. Read more about this in the [[https://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-protect-against-a-hacked-backup-client|FAQ]] and the [[https://borgbackup.readthedocs.io/en/stable/usage/notes.html#drawbacks|Drawbacks of append-only mode]] | [[https://www.borgbackup.org/|Borg]] is a very smart and capable backup solution with lots and lots of features and most importantly, block level deduplication. It can do a lot more than what I need, **BUT** it sadly does not allow to create a user that can only write new backups but not delete old ones as well. While there is a ''append-only'' policy available, it is impractical to rely on it, as one has to either forget about automatic purging of old backups. That's the deal breaker for me. Read more about this in the [[https://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-protect-against-a-hacked-backup-client|FAQ]] and the [[https://borgbackup.readthedocs.io/en/stable/usage/notes.html#drawbacks|Drawbacks of append-only mode]] |
| ==== Burp ==== | ==== Burp ==== |
| [[https://burp.grke.org/|Burp]] is the best backup tool i know to make backups of clients that aren't available 24/7 such as workstations or even notebooks. it also supports linux, windows and Mac OS. Burp actually does pretty much anything I want and a lot more out of the box. the only draw back that i found so far is, that it seems rather difficult to create secondary backups. there is an offsite-backup script but it says in the header of the script, that it doesn't quite work.. the main issue seems to be the fact, that burp moves the full-backup always along to the youngest backup and only keeps the previous versioins of changed or deleted files stored in the previous backup's data directory. this makes it very easy do purge old backups of course, but it is a little trickier to create secondary backups.. i haven't tested if this can be overcome by using hardlinked backups ( a config option in burp ) though. burp also supports client side encryption which breaks delta uploads of modified files. At the end of the day, burp provides no real advantage to me over using a combination of some other tools and some custom scripts to glue them together, but it adds complexity mainly to the offsit backup part instead and it adds potential security risks by setting wrong configuration options. Still, it is a complete and running software and would probably save some time on my end, and it offers so many more features which i currently don't use (like windows backups) but that might come in handy in the future. | [[https://burp.grke.org/|Burp]] is the best backup tool i know to make backups of clients that aren't available 24/7 such as workstations or even notebooks. it also supports linux, windows and Mac OS. Burp actually does pretty much anything I want and a lot more out of the box. the only draw back that i found so far is, that it seems rather difficult to create secondary backups. there is an offsite-backup script but it says in the header of the script, that it doesn't quite work.. the main issue seems to be the fact, that burp moves the full-backup always along to the youngest backup and only keeps the previous versioins of changed or deleted files stored in the previous backup's data directory. this makes it very easy do purge old backups of course, but it is a little trickier to create secondary backups.. i haven't tested if this can be overcome by using hardlinked backups ( a config option in burp ) though. burp also supports client side encryption which breaks delta uploads of modified files. Burp is a complete and running software and would probably save some time on my end versus a home-made solution made of some other tools glued together, and it offers so many more features which i currently don't use (like windows backups) but that might come in handy in the future. |
| So maybe my final solution could be writing a offsite-bakckup for burp to comlete the requried feature set for me :) | So maybe my final solution could be writing a offsite-bakckup for burp to complete the required feature set for me :) |
| ==== Restic ==== | ==== Restic ==== |
| [[https://github.com/restic/restic|Restic]] seems to be an awesome tool that does almost everything i want my new backup tool to do.. Most importantly, it creates client-side encrypted incremental backups of your servers and can then store it to a broad range of storages available including S3 compatible storages etc. This is all very nice, **BUT** it is run on the client side only, which menas, if a hacker gains control over your server and decides to encrypt or delete your date, he can simply delete all your backups and you are screwed.. so it sadly fails our security requirements.. but maybe it could be used as the client side of my backup solution, with a server that prevents deleting old backups from the client.. this will need some further research | [[https://github.com/restic/restic|Restic]] seems to be an awesome tool that does almost everything i want my new backup tool to do.. Most importantly, it creates client-side encrypted incremental backups of your servers and can then store it to a broad range of storages available including S3 compatible storages etc. This is all very nice, **BUT** Restic itself runs on the client side only. This means, if you just use it like a standalone solution, an attacker who has (root)shell access to your system might delete your backups and then encrypt your data. The keyword here is "append only" backups and it must be implemented on the storage side. This in turn means, that restic won't be able to rotate backups anymore, so that needs to be implemented on the storage side. Restic comes with its own solution. They provide a REST server which has the option to accept append-only backups and therefore secures your backup from other attackers. This last part was something i had missed at first but with this server in the pbackage, it makes for a good alternative to burp. |
| | ==== dar ==== |
| | [[http://dar.linux.free.fr/|dar]] is a tool i have to take a closer look at.. it is used in various other backup software as an alternative to rsync and provides backup images which means we can retain permissions in backups even with only unprivileged access to the target storage. this would allow to implement a backup script that creates a backup to a write-only share which is then made available read-only for future use, hence protecting it from a possible attacker with control over the backup target. the big question is, how do they handle incremental backups with regards to rotating old backups and restoring backups. there is a decremental backup method available too (where the full backup is always the latest and then decrements (file states before they where changed to the current state) are saved for older backups). decrementals obviously create a lot of load at the end of a backup and will probably not work when older backups are read-only.. so this might be tricky. |
| |
| | ==== SFTPgo ==== |
| | [[https://github.com/drakkan/sftpgo|SFTPgo]] would make a good server side for a backup solution. Basically it comes with lots of features that can be helpful to implement an append only server to store backups. It is, as the name says, an SFTP server for linux and some more OS's which at first sounds strange, as linux already has openssh which is broadly used for sftp as well because it is on almost every linux server out there to provide ssh access anyway. However, SFTPgo is much more than just a simple sftp server. it comes with its own user management, it provides more protocols such as webdav and it even accepts rsync connections. it can share more than just a plain linux filesystem too.. it can share S3 storage, provide DAR encryption and it supports setting very granular access rights on top of the posix layer. So with this a share could be configered where one would only have write but not overwrite or delete access. pretty nifty. |
| ===== possible solutions ===== | ===== possible solutions ===== |
| ==== Burp ==== | ==== Burp ==== |
| * use rsync daemon on the server to provide access to the backup repos for each of the clients. | * use rsync daemon on the server to provide access to the backup repos for each of the clients. |
| * use ''rsync --link-dest'' or ''cp -alx'' to first create a fully hard-linked copy to the last successful backup and then share this via rsync daemon for the client to then update the changed file in this repo.. this should probably result in a similar backup structure as my current moby script does, but with the added separation of client and server. | * use ''rsync --link-dest'' or ''cp -alx'' to first create a fully hard-linked copy to the last successful backup and then share this via rsync daemon for the client to then update the changed file in this repo.. this should probably result in a similar backup structure as my current moby script does, but with the added separation of client and server. |
| * provide a read-only share via rsync daemon where the client can access all its backups to restore files from. --> **this needs some more thinking / research**, as the backups will contain encrpyted file- and directory names as well as data.. so we would need some other means of sharing the backups in read-only mode but that will retain the orignal linux permissuons upon restore. the share should be mountable on the client, so that we can use again gocryptfs to decrypt the backup before restoring files. Maybe NFS piped through ssh or something similar might be a solution. | * provide a read-only share via rsync daemon where the client can access all its backups to restore files from. --> **this needs some more thinking / research**, as the backups will contain encrpyted file- and directory names as well as data.. so we would need some other means of sharing the backups in read-only mode but that will retain the orignal linux permissions upon restore. the share should be mountable on the client, so that we can use again gocryptfs to decrypt the backup before restoring files. Maybe NFS piped through ssh or something similar might be a solution. |
| * use the same set of tools again to create backups from the primary backup server to the secondary. | * use the same set of tools again to create backups from the primary backup server to the secondary. |
| |
| Unsolved issues of this solution: | Unsolved issues of this solution: |
| * **file ownership** is retained on all the files, so a file belonging to root on the client will belong to root on the backup server.. this brings some security issues, as for example a privilege escalation could be made possible by backing up a copy of bash belonging to root and with the suid bit set.. once the attacker gets unprivileged user access to the backup server, he could start this shell and become root. So it would be preferrable to change at least file ownership to a dedicated user and limit the possibilities for an attack | * **file ownership** is retained on all the files, so a file belonging to root on the client will belong to root on the backup server.. this brings some security issues, as for example a privilege escalation could be made possible by backing up a copy of bash belonging to root and with the suid bit set.. once the attacker gets unprivileged user access to the backup server, he could start this shell and become root. So it would be preferable to change at least file ownership to a dedicated user and limit the possibilities for an attack. [[https://github.com/linuxkit/linuxkit/tree/master/projects/shiftfs|shiftfs]] in combination with ''unshare'' to create a linux user namespace could be a solution here. |
| * **restoring files and browsing backups** needs to be simple. for example it should be possible to either use normal ''rsync -l'' or even better, to mount complete backups from the backup server onto the client server and then browse through them. however, this is currently not so simple because: | * **restoring files and browsing backups** needs to be simple. for example it should be possible to either use normal ''rsync -l'' or even better, to mount complete backups from the backup server onto the client server and then browse through them. however, this is currently not so simple because: |
| * backups are encrypted before rsync lays a hand on the file, so ''rsync -l'' will list encrypted file- and directory names and it will download encrypted files which will then need to be decrypted.. so finding the latest version of a file that contains a string X for example is very cumbresome | * backups are encrypted before rsync lays a hand on the file, so ''rsync -l'' will list encrypted file- and directory names and it will download encrypted files which will then need to be decrypted.. so finding the latest version of a file that contains a string X for example is very cumbresome |
| * i have found [[https://github.com/zaddach/fuse-rsync|fuse-rsync]] which allows mounting an rsync module via a fuse mount. however, this is merely a proof of concept that has not been developed any further in the past 7 years, so not really an option here. | * i have found [[https://github.com/zaddach/fuse-rsync|fuse-rsync]] which allows mounting an rsync module via a fuse mount. however, this is merely a proof of concept that has not been developed any further in the past 7 years, so not really an option here. |
| |
| | ===== First POC - Burp + rsync ===== |
| | **NOTICE** i had to give up on rsyncd as a rsync server to push the backup to, as rsyncd [[https://www.rapid7.com/blog/post/2018/12/21/rsunk-your-battleship-an-ocean-of-data-exposed-through-rsync/|seems to be unsuitable]] for sharing files over the internet. Instead i later went with SFTPgo (see above) on the burp server to share the latest backups via sftp and then have the offsite server pull the data from the burp server. I recon this is still pretty safe as authentication can be done with ssh keys, SFTPgo allows to make sure that the user gets read-only access and nothing else, the data the user can read is encrypted with a key that neither the burp server nor the offsite server know, so a data leak through this channel would be pretty worthless i'd recon (given a good encryption password of course) |
| | |
| | with all the arguments above considered, I decided to proceed a burp based solution and just add off-site capabilities to burp. Here is the targeted setup: |
| | * "Local" backup server running burp in server mode with the following key settings: |
| | * ''hardlinked_archive = 1'' |
| | * ''client_can_delete = 0'' |
| | * ''user=jdoe'' and ''group=jdoe'' where ''jdoe'' is some unprivileged non-root user |
| | * one needs to make sure that all the necessary paths mentioned in ''burp-server.conf'' and ''CA.cnf'' are writable and or readable by the unprivileged user who's running burp |
| | * "Remote" backup server, running a rsyncd service which shares a single directory i.e. ''/backups/current'' |
| | * clients run the burp client and use client-side encryption with a strong password. the following additional core settings are used: |
| | * ''server_can_restore = 0'' |
| | * ''server_can_override_includes = 0'' |
| | * a script on the burp server uses ''rsync -aAhHvXxR --numeric-ids --delete ...'' sync the latest backups stored in ''/var/spool/burp/<client name>/<backup timestamp>'' where ''backup timestamp'' is the target of the ''current'' symlink in the same directory. so a script which uses something along the lines of ''readlink'' should find the ''current'' backup which was just completed and then rsync this to the offsite server's ''current directory'' |
| | * on the offsite server, a script is called (somehow, haven't figured out yet how exactly this will be done) after the rsync from the burp server successfully finished. the script will use ''cp -alx /backups/current /backups/`date +%Y.%m.%d-%H%M`'' to create hardlinked copies of the current directory. by using this script, we can avoid to use the ''%%--%%link-dest'' option of rsync which in turn would make it necessary to at least include the latest completed backup also in the writable share. |
| | |
| | to try it all out i used a bunch of [[ubuntu test docker image | ubuntu test docker containers]] |
| | |
| | docker network create burp |
| | to create the custom network |
| | docker run --net burp --name burpsrv -ti ubuntu-test:latest |
| | to create the container for the burp server, and similar commands for the other servers. |
| | |
| | for testing i ran the burp server with this command line: |
| | burp -v -F -c /home/jdoe/burp/etc/burp-server.conf |
| | |
| | which outputs any logs directly to stdout and keeps the daemon in the foreground. |
| | |
| | ==== offsite backup file encryption ==== |
| | since all files are encrypted on the client side before they are sent to the backup server, we don't have to encrypt them again when uploading them from the backup server to the offsite backup. this basically removes all the challenges mentioned in the "self made" backup solution above. |
| | |
| | ==== restore from an offsite backup ==== |
| | here is how i have tested the restore-ability of an offsite backup in case we have completely lost the backup server in between. |
| | - set up a new burp backup server with the same client config |
| | - setup a new client or delete the certificates if the client is still there and should be re-used |
| | - using ''burp -a l'' execute the initial connection between the client and server and let burp create all the SSL keys. |
| | - using ''rsync -aAHhvXxR --numeric-ids'' copy the desired backup to the ''/var/spool/burp/<client name>/<backup name>'' folder where ''backup name'' is identical to the one stored on the offsite server. |
| | - now use ''burp -a l'' again and you should see the backup listed |
| | - restore using ''burp -a r -b 5 -d /'' or a similar command, depending on your situation to restore the specified backup |
| | |
| | this worked flawlessyl in my test, of coures as long as I still had the **encryption password** available from somewhere! .. needless to say, if you don't store your encryption password your backup is completely useless, so make sure your encryption password is saved somewhere where you will still have it, even if you loose the client, and of course, don't store it together with the backup on the backup servers :) |
| |