encrypted_backups_to_the_cloud

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
encrypted_backups_to_the_cloud [13.08.2017 13:48] – [setup the mount points] Pascal Suterencrypted_backups_to_the_cloud [15.08.2017 08:44] (current) – [the script] Pascal Suter
Line 70: Line 70:
 you will be prompted for your password and it will show you your master key.. NOTE THAT KEY! it will be your only way to access your offsite Backup once your main server is gone! make sure you safe it somewhere where you still have access even when you lost all your data you are backing up here ;)  you will be prompted for your password and it will show you your master key.. NOTE THAT KEY! it will be your only way to access your offsite Backup once your main server is gone! make sure you safe it somewhere where you still have access even when you lost all your data you are backing up here ;) 
  
-create a directory where the backup script will reside (i like /opt/ for that+==== the script ==== 
-  mkdir /opt/offsiteBackup+now this is the script that i will run daily in cron job. the script assumes that the gfscrypt directory will always be left mounted. this way there is no need to safe the password on the server, instead you will need to manually mount gocryptfs after a reboot of the server. if you forget that, the backup script will inform you by mail the next time it runs that it could not do the backup because the mount was not there. 
  
 +in case you want to mount the gocryptfs mount automatically and unmount it after each backup you can do that by using the ''-extpass'' option as described in the manpage. this might be useful if you reboot your server or computer frequently, but on my server a reboot is a rare occasion so i rather keep my password safe instead. 
 +
 +<code bash offsiteBackup.sh>
 +#!/bin/bash 
 +
 +# (c) 2017 Pascal Suter, Version 0.10 Beta
 +# this script creates an enecrypted offsite backup of a locally kept backup. 
 +# ideally suited to work with rubi (http://www.0x1b.ch/misc/hacks/rubi)
 +# for a full description and setup instructions read 
 +# http://wiki.psuter.ch/doku.php?id=encrypted_backups_to_the_cloud
 +# uses gocryptfs (https://nuetzlich.net/gocryptfs/) with -reverse option. 
 +# you may use, modify and re-distribute this script AT YOUR OWN RISK free of charge.
 +
 +CRYPTED="/offsiteBackup/crypted" # folder where the encrypted files are at
 +TARGET="/mnt/offsiteBackup" # remote location: use server:/target/directory for ssh or rsync targets
 +LATEST=$(cat /backupHome/lastdst) # full path to the last successful backup to sync to offsite target
 +PLAINDIR="/offsiteBackup/plain" # directory where the plaintext version of the cryptfsmount is 
 +PLAINMOUNT="$PLAINDIR/backup" # mount point where the lastdest should be mounted to. this must be inside PLAINDIR!
 +RECIPIENTS="root" # email address of mail recipients, separate multiple addresses with space
 +LOCKFILE="/var/lock/offsiteBackup" # to make sure this script is not run twice at the same time
 +RSYNCOPTS="" # additional options to rsync.
 +#RSYNCOPTS='-e "ssh -p 2882"' # use a non-standard port for an ssh connection to the remote target
 +
 +function fail {
 +    echo "$1" | mail -s "offsiteBackup failed" "$RECIPIENTS"
 +    exit 1
 +}
 +
 +function success {
 +    ( echo "the offsite backup was successfully updated to backup version $LATEST"
 +    echo "here are the last lines of the rsync process:"
 +    tail -n 3 /tmp/offsiteBackup.log ) | mail -s "offsiteBackup successfully updated" "$RECIPIENTS"
 +    umount $PLAINMOUNT 2>/dev/null
 +    exit 0
 +}
 +
 +me=`basename "$0"`
 +
 +# get a lock and run me embedded 
 +if [ "$1" != "--embedded" ]; then
 +    echo "staring $0" 
 +    flock -E 66 -n ${LOCKFILE} $0 --embedded | tee /tmp/offsiteBackup.log 2>&1
 +    state=$?
 +    if [ $state -eq 66 ]; then
 +        fail "there was another offsiteBackup process still running, so we skipped this round"
 +    fi
 +    exit $state
 +fi
 +
 +# make sure our crypted directory is mounted
 +grep "$CRYPTED" /proc/mounts > /dev/null
 +if [ $? -gt 0 ]; then
 +    fail "$CRYPTED was not mounted, please login to your server and run # gocryptfs --reverse $PLAINDIR $CRYPTED"
 +fi
 +
 +# unmount any previous bind mounts to $PLAINMOUNT and check it is no longer mounted
 +umount $PLAINMOUNT 2>/dev/null 
 +grep "$PLAINMOUNT" /proc/mounts > /dev/null
 +if [ $? -eq 0 ]; then
 +    fail "There seems to be a stale mount on $PLAINMOUNT, please login to your server and unmount this directory manually"
 +fi
 +
 +# mount the latest backup: 
 +mount -B "$LATEST" "$PLAINMOUNT"
 +if [ $? -gt 0 ]; then 
 +    fail "there was a problem mounting the latest backup from $LATEST to $PLAIMOUNT" 
 +fi
 +
 +# rsync to offsite location
 +rsync -AaHvXx --delete $RSYNCOPTS "$CRYPTED/" "$TARGET" 2>&1
 +res=$?
 +if [ $res -gt 0 ]; then
 +    if [ $res -eq 24 ]; then 
 +        #some files vanished during the backup, that's not a failure of the backup, so send the success message 
 +        success
 +    else 
 +        fail "there was a problem with the offsite backup, check /tmp/offsiteBackup.log on the server"
 +    fi
 +else 
 +    success
 +fi
 +</code>
 +=== Known Issues ===
 +For some reason gocryptfs seems to generate some files (two in my case) like '' gocryptfs.longname.*.name'' that are there when rsync scans the directory but vanish before rsync can sync them. this leads to rsync complaining about vanished files. when i mount the backup with gocryptfs (forward) and then compare the unencrypted backup with the original through rsync --dry-run i find a single file to be missing in my encrypted backup. It has a very long filename: ''xxxx/xxxx/vlc-shares/public/images/fsthumbs/thumbs/MjovVFYvQWxhcm0gZnVyIENvYnJhIDExIC0gRGllIEF1dG9iYWhucG9saXplaSAtIFNlYXNvbiAxLTEyL0NvYnJhIDExIFN0YWZmZWwgMDEvQ29icmEgMTEgUzAxIEUwMDYgKERlciBBbHRlIHVuZCBkZXIgSnVuZ2UpLmF2aQ==.jpg''. the script accepts exit code 24 (vanished files) as successful ending of rsync for now until I have time to further investigate and either file a bug at gocryptfs or work around it somehow. 
 +
 +===== Restoring Files =====
 +to restore files you could use ''sshfs'' for example to mount the remote directory via ssh on your local server...
 +  sshfs user@remote.server:/offsiteDirectory /mnt/offsiteBackup
 +and now use gocryptfs to uncrypt the contents and restore some files: 
 +  gocryptfs /mnt/offsiteBackup /mnt/uncrypted 
 +now you should see all your files in /mnt/uncrypted
 +
 +unmount both mounts once you are done. 
  • encrypted_backups_to_the_cloud.1502624880.txt.gz
  • Last modified: 13.08.2017 13:48
  • by Pascal Suter