Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
mobi_backup [11.06.2018 16:25] – Pascal Suter | mobi_backup [15.03.2022 16:23] (current) – [Error due to old flock version] Pascal Suter | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== MOBI Backup (rsync wrapper) ====== | ====== MOBI Backup (rsync wrapper) ====== | ||
- | like probably every Linux admin, I eventually came to the point where I felt like it was time to write **M**y **O**wn **B**ackup **I**mplementation as an rsync wrapper to do some backups. This script is in its funcitonality very similar to what rubi does: it creates a new sub-directory with the date of the backup as directory each time the backup is run. every backup directory contains a full backup of the source, but only the difference since the last backup does actually need to be synced. when i say the difference i mean files that have changed.. yes, files, not blocks! so if your 2GB log file gets a new line, 2GB will have to be downloaded. but if your file does not change, it will be hard-linked to the previous backup and therefore nothing needs to be downloaded. | + | like probably every Linux admin, I eventually came to the point where I felt like it was time to write **M**y **O**wn **B**ackup **I**mplementation as an rsync wrapper to do some backups. This script is in its funcitonality very similar to what [[backup_with_rubi|rubi]] does: it creates a new sub-directory with the date of the backup as directory each time the backup is run. every backup directory contains a full backup of the source, but only the difference since the last backup does actually need to be synced. when i say the difference i mean files that have changed.. yes, files, not blocks! so if your 2GB log file gets a new line, 2GB will have to be downloaded. but if your file does not change, it will be hard-linked to the previous backup and therefore nothing needs to be downloaded. |
to achieve this, I use rsync' | to achieve this, I use rsync' | ||
Line 6: | Line 6: | ||
In most cases, this proves to be simple but still efficient enough, rather than trying block-level incrementals. | In most cases, this proves to be simple but still efficient enough, rather than trying block-level incrementals. | ||
- | at the end of a successful backup, a rotation is made and old backups | + | One specialty |
- | so here is the script.. use it at your own risk and let me know if you find bugs or have contributions | + | The advantage of running multiple backup jobs at once is, that you can usually reach a much higher overall throughput with multiple rsyncs running in parallel than running them one after the other because rsync is single threaded and the overhead for ssh and file checking etc. is huge. So it usually makes no sense to wait for one host to complete before backing up a second host. \\ |
+ | You could also define multiple backup jobs for the same host but different directories on the host, to increase the speed of large backups. | ||
+ | if you are looking for a solution to speed up an rsync copy process with parallel rsync invocations, | ||
+ | |||
+ | at the end of a successful backup, a rotation is made and old backups are being deleted where appropriate. also a summary email is sent to the admin. | ||
+ | ===== Configuration ===== | ||
to configure, simply edit the lines or add more blocks after the | to configure, simply edit the lines or add more blocks after the | ||
< | < | ||
### Backup Job Definitions start here ####################### | ### Backup Job Definitions start here ####################### | ||
############################################################## | ############################################################## | ||
- | >/code> | + | </code> |
comment in the script. some lines above the comment you can find different config options mixed with some code that should not be changed ;).. i know.. usability and such wasn't the main focus here but instead i wanted to keep everything in a file and as simple as possible to read the code and modify it to your own needs. | comment in the script. some lines above the comment you can find different config options mixed with some code that should not be changed ;).. i know.. usability and such wasn't the main focus here but instead i wanted to keep everything in a file and as simple as possible to read the code and modify it to your own needs. | ||
the script will write a hidden file named .lastdst to the backup base directory for each backup job. this file always contains the folder name of the sub directory of the last successful backup. | the script will write a hidden file named .lastdst to the backup base directory for each backup job. this file always contains the folder name of the sub directory of the last successful backup. | ||
+ | |||
+ | ===== Logs and Debugging ===== | ||
+ | the script writes multiple log files. First of all it writes a new log file for every invocation to / | ||
+ | |||
+ | ===== Known Issues ===== | ||
+ | On systems with old rsync versions (i.e. 3.0.6) and if your data contains extended Attributes or ACL's you may get lots of '' | ||
+ | ===== the script ===== | ||
+ | so here is the script.. use it at your own risk and let me know if you find bugs or have contributions to make. simply send me an email to contact at psuter dot ch. | ||
<code bash mobi.sh> | <code bash mobi.sh> | ||
#!/bin/bash | #!/bin/bash | ||
- | # (c) Pascal Suter (contact at psuter dot ch) 2017. Use at your own Risk | + | # (c) Pascal Suter, DALCO AG, 2018. Use at your own risk |
- | # licensed under GPL v3 | + | # no use without prior permission from DALCO AG |
- | # | + | # pascal.suter@dalco.ch |
- | # Change Log: | + | # the latest version and some documentation for this script can be found on http:// |
# version 1.1 | # version 1.1 | ||
# replaced ps aux | grep rsync style locking with flock locking to allow this script to run on servers that use rsync for other stuff as well :) | # replaced ps aux | grep rsync style locking with flock locking to allow this script to run on servers that use rsync for other stuff as well :) | ||
# version 1.2 | # version 1.2 | ||
# added eval in front of rsync call to properly evaluate the $OPTIONS variable contents when running the command | # added eval in front of rsync call to properly evaluate the $OPTIONS variable contents when running the command | ||
+ | # version 1.3 | ||
+ | # moved log from rsyncs stderr to a separate .err file which makes finding the relevant error messages in the rsync output alot easier | ||
report() { | report() { | ||
Line 222: | Line 237: | ||
if [ $go -eq 1 ]; then | if [ $go -eq 1 ]; then | ||
- | echo "flock -E 66 -n ${LOCKDIR}/ | + | echo "flock -E 66 -n ${LOCKDIR}/ |
- | eval flock -E 66 -n ${LOCKDIR}/ | + | eval flock -E 66 -n ${LOCKDIR}/ |
ret=$? | ret=$? | ||
else | else | ||
Line 245: | Line 260: | ||
exit 1; | exit 1; | ||
else | else | ||
- | echo " | + | echo " |
fi | fi | ||
echo -n " | echo -n " | ||
exit | exit | ||
- | |||
} | } | ||
Line 327: | Line 341: | ||
hosts=`lsof ${LOCKDIR}/ | hosts=`lsof ${LOCKDIR}/ | ||
echo "still have $children processes, currently backing up $hosts" | echo "still have $children processes, currently backing up $hosts" | ||
- | #ps aux | grep " | ||
children=`ps aux | grep " | children=`ps aux | grep " | ||
done | done | ||
+ | echo " | ||
report $MASTERLOG > / | report $MASTERLOG > / | ||
cat / | cat / | ||
rm -f / | rm -f / | ||
</ | </ | ||
+ | ===== run daily ===== | ||
+ | in order to run the backup daily, run '' | ||
+ | 00 1 * * * / | ||
+ | using the redirects of both stdout and stderr to ''/ | ||
+ | |||
+ | ===== Error due to old flock version ===== | ||
+ | when this script is run on an older linux distribution such as CentOS 6.5 for example, the provided version of flock is too old to know the '' | ||
+ | sed -i 's/-E 66 //' mobi.sh | ||
+ | this will make the script work on those systems. however, since now the exit code of flock is '' | ||
+ | |||
+ | ===== Migration from RUBI ===== | ||
+ | Since i was using [[backup_with_rubi|rubi]] before on many systems (private and customer systems) I'll provide a quick migration guide for those who want to migrate from rubi to mobi: | ||
+ | |||
+ | - download the above script and save it to ''/ | ||
+ | - edit the script | ||
+ | - adjust the '' | ||
+ | - under the '' | ||
+ | - '' | ||
+ | - '' | ||
+ | - '' | ||
+ | - '' | ||
+ | - to go the backup directory for each host and run this command < | ||
+ | - remove the ":" | ||
+ | - cleanup old log files and '' | ||
+ | - remember to come back and delete the old backups when it's time. old RUBI backups won't be rotated using mobi, this needs to be done manually. mobi will only include backups into the rotation for which it finds a log file of a successful backup job. |