rethinking_my_backup_strategy

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
rethinking_my_backup_strategy [01.01.2021 23:23] – [First POC - Burp + rsync] Pascal Suterrethinking_my_backup_strategy [12.08.2021 17:42] (current) – [First POC - Burp + rsync] Pascal Suter
Line 33: Line 33:
 [[https://www.borgbackup.org/|Borg]] is a very smart and capable backup solution with lots and lots of features and most importantly, block level deduplication. It can do a lot more than what I need, **BUT** it sadly does not allow to create a user that can only write new backups but not delete old ones as well. While there is a ''append-only'' policy available, it is impractical to rely on it, as one has to either forget about automatic purging of old backups. That's the deal breaker for me. Read more about this in the [[https://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-protect-against-a-hacked-backup-client|FAQ]] and the [[https://borgbackup.readthedocs.io/en/stable/usage/notes.html#drawbacks|Drawbacks of append-only mode]] [[https://www.borgbackup.org/|Borg]] is a very smart and capable backup solution with lots and lots of features and most importantly, block level deduplication. It can do a lot more than what I need, **BUT** it sadly does not allow to create a user that can only write new backups but not delete old ones as well. While there is a ''append-only'' policy available, it is impractical to rely on it, as one has to either forget about automatic purging of old backups. That's the deal breaker for me. Read more about this in the [[https://borgbackup.readthedocs.io/en/stable/faq.html#how-can-i-protect-against-a-hacked-backup-client|FAQ]] and the [[https://borgbackup.readthedocs.io/en/stable/usage/notes.html#drawbacks|Drawbacks of append-only mode]]
 ==== Burp ==== ==== Burp ====
-[[https://burp.grke.org/|Burp]] is the best backup tool i know to make backups of clients that aren't available 24/7 such as workstations or even notebooks. it also supports linux, windows and Mac OS. Burp actually does pretty much anything I want and a lot more out of the box. the only draw back that i found so far is, that it seems rather difficult to create secondary backups. there is an offsite-backup script but it says in the header of the script, that it doesn't quite work.. the main issue seems to be the fact, that burp moves the full-backup always along to the youngest backup and only keeps the previous versioins of changed or deleted files stored in the previous backup's data directory. this makes it very easy do purge old backups of course, but it is a little trickier to create secondary backups.. i haven't tested if this can be overcome by using hardlinked backups ( a config option in burp ) though. burp also supports client side encryption which breaks delta uploads of modified files. At the end of the day, burp provides no real advantage to me over using a combination of some other tools and some custom scripts to glue them together, but it adds complexity mainly to the offsit backup part instead and it adds potential security risks by setting wrong configuration options. Still, it is a complete and running software and would probably save some time on my end, and it offers so many more features which i currently don't use (like windows backups) but that might come in handy in the future.  +[[https://burp.grke.org/|Burp]] is the best backup tool i know to make backups of clients that aren't available 24/7 such as workstations or even notebooks. it also supports linux, windows and Mac OS. Burp actually does pretty much anything I want and a lot more out of the box. the only draw back that i found so far is, that it seems rather difficult to create secondary backups. there is an offsite-backup script but it says in the header of the script, that it doesn't quite work.. the main issue seems to be the fact, that burp moves the full-backup always along to the youngest backup and only keeps the previous versioins of changed or deleted files stored in the previous backup's data directory. this makes it very easy do purge old backups of course, but it is a little trickier to create secondary backups.. i haven't tested if this can be overcome by using hardlinked backups ( a config option in burp ) though. burp also supports client side encryption which breaks delta uploads of modified files. Burp is a complete and running software and would probably save some time on my end versus a home-made solution made of some other tools glued together, and it offers so many more features which i currently don't use (like windows backups) but that might come in handy in the future.  
-So maybe my final solution could be writing a offsite-bakckup for burp to comlete the requried feature set  for me :) +So maybe my final solution could be writing a offsite-bakckup for burp to complete the required feature set for me :) 
 ==== Restic ==== ==== Restic ====
-[[https://github.com/restic/restic|Restic]] seems to be an awesome tool that does almost everything i want my new backup tool to do.. Most importantly, it creates client-side encrypted incremental backups of your servers and can then store it to a broad range of storages available including S3 compatible storages etc. This is all very nice, **BUT** it is run on the client side only, which menas, if a hacker gains control over your server and decides to encrypt or delete your date, he can simply delete all your backups and you are screwed.. so it sadly fails our security requirements.. but maybe it could be used as the client side of my backup solution, with a server that prevents deleting old backups from the client.. this will need some further research +[[https://github.com/restic/restic|Restic]] seems to be an awesome tool that does almost everything i want my new backup tool to do.. Most importantly, it creates client-side encrypted incremental backups of your servers and can then store it to a broad range of storages available including S3 compatible storages etc. This is all very nice, **BUT** Restic itself runs on the client side only. This means, if you just use it like standalone solution, an attacker who has (root)shell access to your system might delete your backups and then encrypt your dataThe keyword here is "append only" backups and it must be implemented on the storage sideThis in turn means, that restic won't be able to rotate backups anymore, so that needs to be implemented on the storage sideRestic comes with its own solutionThey provide a REST server which has the option to accept append-only backups and therefore secures your backup from other attackers. This last part was something i had missed at first but with this server in the pbackage, it makes for a good alternative to burp.  
- +==== dar ==== 
 +[[http://dar.linux.free.fr/|dar]] is a tool i have to take a closer look at.. it is used in various other backup software as an alternative to rsync and provides backup images which means we can retain permissions in backups even with only unprivileged access to the target storage. this would allow to implement backup script that creates a backup to a write-only share which is then made available read-only for future use, hence protecting it from a possible attacker with control over the backup target. the big question is, how do they handle incremental backups with regards to rotating old backups and restoring backups. there is a decremental backup method available too (where the full backup is always the latest and then decrements (file states before they where changed to the current state) are saved for older backups)decrementals obviously create a lot of load at the end of a backup and will probably not work when older backups are read-only.. so this might be tricky. 
  
 +==== SFTPgo ====
 +[[https://github.com/drakkan/sftpgo|SFTPgo]] would make a good server side for a backup solution. Basically it comes with lots of features that can be helpful to implement an append only server to store backups. It is, as the name says, an SFTP server for linux and some more OS's which at first sounds strange, as linux already has openssh which is broadly used for sftp as well because it is on almost every linux server out there to provide ssh access anyway. However, SFTPgo is much more than just a simple sftp server. it comes with its own user management, it provides more protocols such as webdav and it even accepts rsync connections. it can share more than just a plain linux filesystem too.. it can share S3 storage, provide DAR encryption and it supports setting very granular access rights on top of the posix layer. So with this a share could be configered where one would only have write but not overwrite or delete access. pretty nifty. 
 ===== possible solutions ===== ===== possible solutions =====
 ==== Burp ==== ==== Burp ====
Line 74: Line 76:
  
 ===== First POC - Burp + rsync ===== ===== First POC - Burp + rsync =====
 +**NOTICE** i had to give up on rsyncd as a rsync server to push the backup to, as rsyncd [[https://www.rapid7.com/blog/post/2018/12/21/rsunk-your-battleship-an-ocean-of-data-exposed-through-rsync/|seems to be unsuitable]] for sharing files over the internet. Instead i later went with SFTPgo (see above) on the burp server to share the latest backups via sftp and then have the offsite server pull the data from the burp server. I recon this is still pretty safe as authentication can be done with ssh keys, SFTPgo allows to make sure that the user gets read-only access and nothing else, the data the user can read is encrypted with a key that neither the burp server nor the offsite server know, so a data leak through this channel would be pretty worthless i'd recon (given a good encryption password of course) 
 +
 with all the arguments above considered, I decided to proceed a burp based solution and just add off-site capabilities to burp. Here is the targeted setup:  with all the arguments above considered, I decided to proceed a burp based solution and just add off-site capabilities to burp. Here is the targeted setup: 
   * "Local" backup server running burp in server mode with the following key settings:    * "Local" backup server running burp in server mode with the following key settings: 
Line 100: Line 104:
  
 ==== offsite backup file encryption ==== ==== offsite backup file encryption ====
-since all files are encrypted on the client side before they are sent to the backup server, we don't have to encrypt them again when uploading them from the backup server to the offsite backup. this basically removes all the challenges mentioned in the "self made" backup +since all files are encrypted on the client side before they are sent to the backup server, we don't have to encrypt them again when uploading them from the backup server to the offsite backup. this basically removes all the challenges mentioned in the "self made" backup solution above.  
 + 
 +==== restore from an offsite backup ==== 
 +here is how i have tested the restore-ability of an offsite backup in case we have completely lost the backup server in between.  
 +  - set up a new burp backup server with the same client config  
 +  - setup a new client or delete the certificates if the client is still there and should be re-used 
 +  - using ''burp -a l'' execute the initial connection between the client and server and let burp create all the SSL keys.  
 +  - using ''rsync -aAHhvXxR --numeric-ids'' copy the desired backup to the ''/var/spool/burp/<client name>/<backup name>'' folder where ''backup name'' is identical to the one stored on the offsite server.  
 +  - now use ''burp -a l'' again and you should see the backup listed  
 +  - restore using ''burp -a r -b 5 -d /'' or a similar command, depending on your situation to restore the specified backup 
 + 
 +this worked flawlessyl in my test, of coures as long as I still had the **encryption password** available from somewhere! .. needless to say, if you don't store your encryption password your backup is completely useless, so make sure your encryption password is saved somewhere where you will still have it, even if you loose the client, and of course, don't store it together with the backup on the backup servers :)  
  • rethinking_my_backup_strategy.1609539816.txt.gz
  • Last modified: 01.01.2021 23:23
  • by Pascal Suter