Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
mobi_backup [26.09.2018 15:53]
Pascal Suter [MOBI Backup (rsync wrapper)]
mobi_backup [21.01.2019 13:12] (current)
Pascal Suter
Line 6: Line 6:
 In most cases, this proves to be simple but still efficient enough, rather than trying block-level incrementals. ​ In most cases, this proves to be simple but still efficient enough, rather than trying block-level incrementals. ​
  
-at the end of a successful backupa rotation is made and old backups ​are being deleted where appropriatealso a summary email is sent to the admin+One specialty ​of MOBI, and the main reason for writing this script in the first place, is that it runs multiple ​backups ​in parallelIt has sort of its own queue manager ​to do that. It will take all Job Definitions and put them in the queue and it will then run multiple in parallel. The number of parallel backup processes can be defined by setting the ''​PARALLELPROCS=8''​ variable in the script. Default is ''​8''​
  
-so here is the script.. use it at your own risk and let me know if you find bugs or have contributions ​to makesimply send me an email to contact at psuter dot ch+The advantage of running multiple backup jobs at once is, that you can usually reach a much higher overall throughput with multiple rsyncs running in parallel than running them one after the other because rsync is single threaded and the overhead for ssh and file checking etcis hugeSo it usually makes no sense to wait for one host to complete before backing up a second host\\ 
 +You could also define multiple backup jobs for the same host but different directories on the host, to increase the speed of large backups
  
 +if you are looking for a solution to speed up an rsync copy process with parallel rsync invocations,​ take a look at my [[parallel_rsync]] article. Sadly the function I wrote there does not help in speeding up incremental backups at all (in contrary, it adds more overhead and hence makes an incremental backup even slower!). ​
 +
 +at the end of a successful backup, a rotation is made and old backups are being deleted where appropriate. also a summary email is sent to the admin. ​
 +===== Configuration =====
 to configure, simply edit the lines or add more blocks after the to configure, simply edit the lines or add more blocks after the
 <​code>##############################################################​ <​code>##############################################################​
Line 18: Line 23:
  
 the script will write a hidden file named .lastdst to the backup base directory for each backup job. this file always contains the folder name of the sub directory of the last successful backup. ​ the script will write a hidden file named .lastdst to the backup base directory for each backup job. this file always contains the folder name of the sub directory of the last successful backup. ​
 +
 +===== Logs and Debugging =====
 +the script writes multiple log files. First of all it writes a new log file for every invocation to /​tmp/<​date-time>​.log. this is a general log which contains information printed by the "queue manager"​. It then writes two log files for each backup job it runs. these can be found in the same directory where these backup jobs are stored (defined by ''​BASEDST''​) again with the start date and time of the job. one ends on ''​log''​ the other one ends on ''​.err''​. the ''​.log''​ file contains the standard-output of the process and the ''​err''​ file contains the standard-error output. so if copying a file failed for example it should be listed in the ''​err''​ file. 
 +
 +===== the script =====
 +so here is the script.. use it at your own risk and let me know if you find bugs or have contributions to make. simply send me an email to contact at psuter dot ch. 
  
 <code bash mobi.sh> <code bash mobi.sh>