Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
backup_with_rubi [04.12.2017 20:10]
Pascal Suter
backup_with_rubi [04.12.2017 21:21] (current)
Pascal Suter
Line 4: Line 4:
 RUBI is [[http://​www.0x1b.ch/​~beat/​|Beat Rubischons]] “Rubi’s Ultimat Backup Implementation” and it’s indeed quite a clever thing ;) RUBI is [[http://​www.0x1b.ch/​~beat/​|Beat Rubischons]] “Rubi’s Ultimat Backup Implementation” and it’s indeed quite a clever thing ;)
  
-here’s what it does, or what i think it does ;) +here’s what it does: 
-every time you run rubi, it copies files from a source to a destination. therefore it creates a new folder in the destination drive containing the current date and time in the directory name. it then copys all files from the source into that new directory. but if it did just regular copies, it would cause quite some disk usage, so rubi came up with a clever solution. all the unchanged files are hardlinked to the previously downloaded copies. this way, we only need the disk space once, but we still have all those directories where we have a whole snapshot of every day. No more messing around with incremental stuff :)+every time you run rubi, it copies files from a source to a destination. therefore it creates a new folder in the destination drive containing the current date and time in the directory name. it then copies ​all files from the source into that new directory. but if it did just regular copies, it would cause quite some disk usage, so rubi came up with a clever solution. all the unchanged files are hardlinked to the previously downloaded copies. this way, we only need the disk space once, but we still have all those directories where we have a whole snapshot of every day. No more messing around with incremental stuff :) 
 + 
 +if you know rsync, you might be familiar with the ''​link-dest''​ option. that is exactly what rubi does and since version 4 rubi is fully rsync based and uses this option to do all the hardlinking work :) 
  
 the only drawback i found so far.. there’s no documentation.. the only drawback i found so far.. there’s no documentation..
  
-so here’s how i got it up and running.. but be aware, im not really a linux expert ;)+so here’s how i got it up and running:
  
-first, create a new home for rubi. i suggest to install it into /opt so we don’t need to create a directory at all :) +first, create a new home for rubi. i suggest to install it into ''​/opt''​ 
-then download and unpack rubi. it can be downloaded from [[http://​www.0x1b.ch/​misc/​hacks/​rubi|the rubi section of beat’s webpage]] +then download and unpack rubi. it can be downloaded from [[http://​www.0x1b.ch/​misc/​hacks/​rubi|the rubi section of beat’s webpage]] always check for updates before copy/​pasting my lines! ​
-by the time i wrote this article, the most recent version was 3.1.0. ​always check for updates before copy/​pasting my lines! ​+
 <​code>​ <​code>​
 cd /opt cd /opt
-wget http://​www.0x1b.ch/​misc/​hacks/​rubi/​rubi-3.2.0.tar.gz +wget http://​www.0x1b.ch/​misc/​hacks/​rubi/​rubi-4.0.0.tar.gz 
-tar xzvf rubi-3.2.0.tar.gz+tar xzvf rubi-4.0.0.tar.gz
 rm rubi-* rm rubi-*
 </​code>​ </​code>​
-now we should also install some dependencies and find out where they are installed. the defaults will probably mostly work.+by the time I first wrote this article, the most recent version was 3.1.0. and rubi depended on many common tools that where included in most os installs. I recently updated this article and rubi is now at verison 4, where most of these dependencies ​have been dropped ​and all we need now besides bash and perl are rsync and find.
  
-rubi needs the folowing shell tools: +using ''​which'' ​or ''​whereis'' ​you can find out where these tools are located so we can tell rubi 
-cpio, find, mkfifo, rsync and ssh or rsh if you want to backup through a network rather than only from one harddisk to another on the same server. +  ​which rsync 
-these tools are mostly system essentials which you probably won’t need to install.. except for rsync.. on debian for example run  +for example will tell you where to find rsync
-  apt-get update  +
-  apt-get install rsync +
-using whereis you can find out where these tools are located so we can tell rubi +
-  ​whereis cpio +
-for example will tell you where to find cpio+
  
-note down all tose paths, we need them for the configuration of rubi+note down those paths for rsync and find, we need them for the configuration of rubi
  
-now since there is no documentation to rubi, beat has simply included ​many examples of how he did things, ​so we can just pick the one that fits our needs the most and modify it… +now since there is no documentation to rubi, beat has simply included ​a sample rubi run-script ​so you can copy that one and work from there. ​ 
-all examples are in ''​/​opt/​rubi/​bin/​rubi-xy''​ + 
-that’s ​the place for our final configuration ​of ruby to be too. +you find the scirpt ​in /​opt/​rubi/​bin ​which is also the place for our final run-script ​of rubi too. 
   cd /​opt/​rubi/​bin   cd /​opt/​rubi/​bin
-  cp rubi-test rubi  +  cp rubi-sample ​rubi  
-  ​vi rubi+  ​nano rubi
 now let’s configure it the way we want it now let’s configure it the way we want it
-first the global settings.. just make sure all the path’s ​are correct ​and uncomment ​the right remote shell tool for your system.+first the global settings.. just make sure all the paths are correct. usually ​the defaults work. make sure that ''​PREFIX''​ contains ​your rubi install directory
  
-you don’t need to change anything to the source shell functions section ​+you don’t need to change anything to the "source shell functions" ​section ​...
  
-and now the most important part.. our actual backup setups+and now the most important part.. our actual backup setups..
 here’s an example of what it looks like:  here’s an example of what it looks like: 
 <​code>​ <​code>​
 # ------ # ------
-# Backup of pssrv+# Backup of max
 # ------ # ------
-SRCHOST=192.168.168.43 ​                         ​# source host +SRCHOST=localhost ​                              # source host 
-SRCPART="/" ​                                    ​# source partitions +SRCPART="/ ​/boot /export /​export/​home /​export/​data /opt/cms" ​   # source partitions 
-BASEDST=/mnt/​suter/​psuter_backup/psbackup ​      # base Backup ​dir +BASEDST=/backup/max                             # base backup ​dir 
-DSTDIR=$BASEDST/​pssrv_`date +%Y.%m.%d-%H%M` ​    ​# destination+DSTDIR=$BASEDST/​`date +%Y.%m.%d-%H:%M`          # destination
 OLDDST="​`cat $BASEDST/​lastdst 2>/​dev/​null`" ​    # last destination OLDDST="​`cat $BASEDST/​lastdst 2>/​dev/​null`" ​    # last destination
-SRCCPIO=/​bin/​cpio ​                              remote GNU cpio +KEEPG=15                                        ​successful backups to keep 
-SRCRSYNC=/​usr/​bin/​rsync ​                        # remote rsync +KEEPD=15                                        # ​min age before delete
-KEEPD=10                                        # ​keep 10 days +
-KEEPS= ​                                         # keep string +
-UTIME=1 ​                                        # set utimes+
  
 doit | tee $DSTDIR.log doit | tee $DSTDIR.log
 </​code>​ </​code>​
-if you want to do several different backup sets right after each other, simply put several of these blocks at the end of your rubi file. Please make sure to use different BASEDST for each of these blocks (the reason why can be seen a little further down in the text at the BASEDST description).+if you want to do several different backup sets right after each other, simply put several of these blocks at the end of your rubi file. Make sure you always include ​the "​doit"​ line at the end of each block, otherwise your backup will not run
  
-SRCHOST is the source host name. if you do local backups, thats localhost, otherwhise it’s an ip or hostname of a remot computer and you will need rsh or ssh and rsync if you have larger files than 2gig+''​SRCHOST'' ​is the source host name. if you do local backups, thats localhost, otherwhise it’s an ip or hostname of a remot computer and you will need rsh or ssh and rsync if you have larger files than 2gig
  
-SRCPART is a list of directories you want to backup. rubi will descend in all subdirectories of these derecotires listed here… ​the single ​directories are separated with a spacer and it has to be inside quotes. so it could look like this: +''​SRCPART'' ​is a list of directories you want to backup. rubi will descend in all subdirectories of these direcotires as long as they are in the same filesystem. Since rubi won't cross filesystem borders you need to add the mount paths to your other filesystems if you want to back them up as well. Look at your ''/​etc/​fstab''​ or the output of the ''​mount''​ command to see what's mounted where. For example if you backup your root filesystem ''/''​ and you have a separate boot partition, you also need to specify ''/​boot''​ in your list of source directories. ​the directories are separated with a spacer and it has to be inside quotes. so it could look like this: 
   SRCPART="/​dir1 /dir2 /dir3"   SRCPART="/​dir1 /dir2 /dir3"
 this will create a backup of ''/​dir1'',​ ''/​dir2''​ and ''/​dir3''​ into the ''​BASEDST''​ directory this will create a backup of ''/​dir1'',​ ''/​dir2''​ and ''/​dir3''​ into the ''​BASEDST''​ directory
Line 75: Line 68:
 ''​DSTDIR''​ is the definition of how a timestamped directory name is being put together ''​DSTDIR''​ is the definition of how a timestamped directory name is being put together
  
-''​OLDDST''​ as far as i understood ​it, that’s ​the place where a timestamped directory goes before ​it is being erased after it expired+''​OLDDST'' ​usually this line can be left as it is in this example. this variable should contain the path to your last successful backup which is usually stored in the lastdst filehence the cat command in there. so leave this the way it is unless you know better ;)  
 + 
 +''​KEEPG''​ sets how many successful backups should be kept.
  
-then you can tell where cpi and rsync are to find on the source server (needed when it’s not in the same place on a remote server)+''​KEEPD''​ sets how old a backup should at least be before it can be deleted. so you could say that you want to keep at least 15 successful backups but you want your backups to reach a minimum age of 30 days. if you run a daily backup this would keep 30 backups. if more than half your backups fail, it will keep the last 15 backups which by then might be older than 30 days. 
  
-''​KEEPD''​ sets how many days a backup ​should be kept befor it is being erased again+don’t forget to put the doit line after each config block.. that's what starts the backup
  
-the other two opitons i don’t understand yet.+this "​config"​ file is simply a shell script and you can put any sort of shell commands in front of and after the doit command. by doing so, you can for example mount your backup target first, then run your backup and then unmount your target again, or you can create a lvm snapshot of your storage and then run your backup from the snapshot if data consistency is important to you
  
-don’t forget tho put the doit line after the config block.. that's what starts ​the backup+**some thaughts on local backups** 
 +if you run your backup locally, i.e. you backup your server to a separate harddisk attached to the same server, you should consider adding some extra commands around your doit line, to keep your backup safe:  
 +  * ''​rm -rf /''​ can erase your backup as well if it is mounted to say /backup because rm by default does not honor filesystem boundaries. in order to avoid losing your backup over a scripting mistake, you should unmount your backup ​after it's done! 
 +  * ransomware: ransomware encrypts all files it can find. the hackers behind it will then ask for money in order to give you the key to decrypt your data againif you have a backup you don't need to bother paying, you simply restore and you're goodUsually ransomware is a piece of software ​that looks for all kind of sources for files to encrypt, recently including cloud services and network shares that are mounted to the affected machine. To protect your backup ​from being encrypted when your data is encrypted it is best to unmount your backup target and to not write its path into your /etc/fstab. also don't share it permanetly via samba to windows machines :) 
  
-if you want to do anything else, check out the different examples from beat. 
  
 now to run your backup simply run  now to run your backup simply run 
Line 93: Line 90:
 or by editing the main crontab directly ​ or by editing the main crontab directly ​
   vi /​etc/​crontab   vi /​etc/​crontab
-**NOTE:** you need to get ssh autologin to work. so make sure your backup machine'​s public ssh key is in the authorized_keys file on each target machine, otherwhise your backup won't run automatically as the backup server won't be able to login. ​+**NOTE:** you need to get ssh autologin to work. so make sure your backup machine'​s public ssh key is in the authorized_keys file on each target machine, otherwhise your backup won't run automatically as the backup server won't be able to login. Also the user you use to connect to the remote machine should have full access to all files you want to backup of course. if you want to do a full system backup there is no way around using root for this
  
 **addon: backup from or to a qnap**\\ **addon: backup from or to a qnap**\\
-**NOTE** //the description ​that followes ​below was written before Qnap supported Docker containers. Nowadays I recommend to simply setup a docker container and run your linux stuff in there rather than hacking it into the qnap OS as the docker method will remain functional evena after firmware upgrades. I leave this here just for reference. //+**NOTE** //the description below was written before Qnap supported Docker containers. Nowadays I recommend to simply setup a docker container and run your linux stuff in there rather than hacking it into the qnap OS as the docker method will remain functional evena after firmware upgrades. I leave this here just for reference. //
  
 a little addon to install rubi on a qnap or to backup from a qnap (only on intel versions, as there seems to be no ipkg repo for the ARM based ones) a little addon to install rubi on a qnap or to backup from a qnap (only on intel versions, as there seems to be no ipkg repo for the ARM based ones)