encrypted_backups_to_the_cloud

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
encrypted_backups_to_the_cloud [13.08.2017 12:50] – [gocryptfs installation] Pascal Suterencrypted_backups_to_the_cloud [15.08.2017 08:44] (current) – [the script] Pascal Suter
Line 30: Line 30:
 I decided to go with gocryptfs in reverse mode and rsync to backup my files to a remote server. As i can put a physical harddrive into the remote backup server to which the VPS will then have exclusive access, i will first run the backup locally from disk to disk and then introduce ssh as a transport tunnel to do remote incrementals in the future.  I decided to go with gocryptfs in reverse mode and rsync to backup my files to a remote server. As i can put a physical harddrive into the remote backup server to which the VPS will then have exclusive access, i will first run the backup locally from disk to disk and then introduce ssh as a transport tunnel to do remote incrementals in the future. 
  
 +now I use [[backup_with_rubi|rubi]] as a backup tool on my server and it creates a new directory for every backup containing a full backup (it basically does an incremental backup and hardlinks all unchanged files, so every backup directory contains a full backup in the end). Rubi has a file called ''lastdst'' which contains the full path to the last successful backup. 
 +In order to use this with the --reverse functionality of gocryptfs we need to create a directory that can be initialized with gocryptfs (where the config is stored) that then contains a sub directory to which we will bind-mount the latest backup before doing the rsync of the encrypted gocryptfs mount. with this method all hardlinked files will stay the same in the crypted version and only changed files will be transfered by rsync later on. once the backup is complete we will unmount the bind mount. 
 ===== the setup ===== ===== the setup =====
 ==== Prepare the Target ==== ==== Prepare the Target ====
Line 57: Line 59:
   mv gocryptfs.1 /usr/local/share/man/man1/   mv gocryptfs.1 /usr/local/share/man/man1/
   gocryptfs --version   gocryptfs --version
 +==== setup the mount points ====
 +''/backupHome'' contains my backups inside folders like ''2017.08.12-0300''
 +  mkdir -p /backupHome/ /offsiteBackup/plain/backup /offsiteBackup/crypted
 +now we can initialize ''/offsiteBackup/plain'' as our plain text directory we want to encrypt using gocryptfs --reverse: 
 +  gocryptfs --init --reverse /offsiteBackup/plain
 +enter your desired password when prompted. 
 +
 +now mount the crypted directory: 
 +  gocryptfs --reverse /offsiteBackup/plain/ /offsiteBackup/crypted/
 +you will be prompted for your password and it will show you your master key.. NOTE THAT KEY! it will be your only way to access your offsite Backup once your main server is gone! make sure you safe it somewhere where you still have access even when you lost all your data you are backing up here ;) 
 +
 +==== the script ====
 +now this is the script that i will run daily in a cron job. the script assumes that the gfscrypt directory will always be left mounted. this way there is no need to safe the password on the server, instead you will need to manually mount gocryptfs after a reboot of the server. if you forget that, the backup script will inform you by mail the next time it runs that it could not do the backup because the mount was not there. 
 +
 +in case you want to mount the gocryptfs mount automatically and unmount it after each backup you can do that by using the ''-extpass'' option as described in the manpage. this might be useful if you reboot your server or computer frequently, but on my server a reboot is a rare occasion so i rather keep my password safe instead. 
 +
 +<code bash offsiteBackup.sh>
 +#!/bin/bash 
 +
 +# (c) 2017 Pascal Suter, Version 0.10 Beta
 +# this script creates an enecrypted offsite backup of a locally kept backup. 
 +# ideally suited to work with rubi (http://www.0x1b.ch/misc/hacks/rubi)
 +# for a full description and setup instructions read 
 +# http://wiki.psuter.ch/doku.php?id=encrypted_backups_to_the_cloud
 +# uses gocryptfs (https://nuetzlich.net/gocryptfs/) with -reverse option. 
 +# you may use, modify and re-distribute this script AT YOUR OWN RISK free of charge.
 +
 +CRYPTED="/offsiteBackup/crypted" # folder where the encrypted files are at
 +TARGET="/mnt/offsiteBackup" # remote location: use server:/target/directory for ssh or rsync targets
 +LATEST=$(cat /backupHome/lastdst) # full path to the last successful backup to sync to offsite target
 +PLAINDIR="/offsiteBackup/plain" # directory where the plaintext version of the cryptfsmount is 
 +PLAINMOUNT="$PLAINDIR/backup" # mount point where the lastdest should be mounted to. this must be inside PLAINDIR!
 +RECIPIENTS="root" # email address of mail recipients, separate multiple addresses with space
 +LOCKFILE="/var/lock/offsiteBackup" # to make sure this script is not run twice at the same time
 +RSYNCOPTS="" # additional options to rsync.
 +#RSYNCOPTS='-e "ssh -p 2882"' # use a non-standard port for an ssh connection to the remote target
 +
 +function fail {
 +    echo "$1" | mail -s "offsiteBackup failed" "$RECIPIENTS"
 +    exit 1
 +}
 +
 +function success {
 +    ( echo "the offsite backup was successfully updated to backup version $LATEST"
 +    echo "here are the last lines of the rsync process:"
 +    tail -n 3 /tmp/offsiteBackup.log ) | mail -s "offsiteBackup successfully updated" "$RECIPIENTS"
 +    umount $PLAINMOUNT 2>/dev/null
 +    exit 0
 +}
 +
 +me=`basename "$0"`
 +
 +# get a lock and run me embedded 
 +if [ "$1" != "--embedded" ]; then
 +    echo "staring $0" 
 +    flock -E 66 -n ${LOCKFILE} $0 --embedded | tee /tmp/offsiteBackup.log 2>&1
 +    state=$?
 +    if [ $state -eq 66 ]; then
 +        fail "there was another offsiteBackup process still running, so we skipped this round"
 +    fi
 +    exit $state
 +fi
 +
 +# make sure our crypted directory is mounted
 +grep "$CRYPTED" /proc/mounts > /dev/null
 +if [ $? -gt 0 ]; then
 +    fail "$CRYPTED was not mounted, please login to your server and run # gocryptfs --reverse $PLAINDIR $CRYPTED"
 +fi
 +
 +# unmount any previous bind mounts to $PLAINMOUNT and check it is no longer mounted
 +umount $PLAINMOUNT 2>/dev/null 
 +grep "$PLAINMOUNT" /proc/mounts > /dev/null
 +if [ $? -eq 0 ]; then
 +    fail "There seems to be a stale mount on $PLAINMOUNT, please login to your server and unmount this directory manually"
 +fi
 +
 +# mount the latest backup: 
 +mount -B "$LATEST" "$PLAINMOUNT"
 +if [ $? -gt 0 ]; then 
 +    fail "there was a problem mounting the latest backup from $LATEST to $PLAIMOUNT" 
 +fi
 +
 +# rsync to offsite location
 +rsync -AaHvXx --delete $RSYNCOPTS "$CRYPTED/" "$TARGET" 2>&1
 +res=$?
 +if [ $res -gt 0 ]; then
 +    if [ $res -eq 24 ]; then 
 +        #some files vanished during the backup, that's not a failure of the backup, so send the success message 
 +        success
 +    else 
 +        fail "there was a problem with the offsite backup, check /tmp/offsiteBackup.log on the server"
 +    fi
 +else 
 +    success
 +fi
 +</code>
 +=== Known Issues ===
 +For some reason gocryptfs seems to generate some files (two in my case) like '' gocryptfs.longname.*.name'' that are there when rsync scans the directory but vanish before rsync can sync them. this leads to rsync complaining about vanished files. when i mount the backup with gocryptfs (forward) and then compare the unencrypted backup with the original through rsync --dry-run i find a single file to be missing in my encrypted backup. It has a very long filename: ''xxxx/xxxx/vlc-shares/public/images/fsthumbs/thumbs/MjovVFYvQWxhcm0gZnVyIENvYnJhIDExIC0gRGllIEF1dG9iYWhucG9saXplaSAtIFNlYXNvbiAxLTEyL0NvYnJhIDExIFN0YWZmZWwgMDEvQ29icmEgMTEgUzAxIEUwMDYgKERlciBBbHRlIHVuZCBkZXIgSnVuZ2UpLmF2aQ==.jpg''. the script accepts exit code 24 (vanished files) as successful ending of rsync for now until I have time to further investigate and either file a bug at gocryptfs or work around it somehow. 
 +
 +===== Restoring Files =====
 +to restore files you could use ''sshfs'' for example to mount the remote directory via ssh on your local server...
 +  sshfs user@remote.server:/offsiteDirectory /mnt/offsiteBackup
 +and now use gocryptfs to uncrypt the contents and restore some files: 
 +  gocryptfs /mnt/offsiteBackup /mnt/uncrypted 
 +now you should see all your files in /mnt/uncrypted
 +
 +unmount both mounts once you are done. 
  • encrypted_backups_to_the_cloud.1502621448.txt.gz
  • Last modified: 13.08.2017 12:50
  • by Pascal Suter