- rsyncrsnapshot
- Jonathan Haack
- Haack’s Networking
- netcmnd@jonathanhaack.com
Just a reminder that the blog posts do get out of date, but I maintain changes/upgrades to this this workflow on my Wiki: rsyncrsnapshot
As many folks know, I teach full time at both the college and high school levels, but I also run a side IT business that focuses on educators, small businesses, and power residential users. My company aggressively leverages Free Software principles along with self-hosting principles, so accordingly I built my own virtualization stack with pure Debian, virsh, and qemu/kvm. The virtualization stack runs on a beefy new host up at Brown Rice data center and has VMs for my work infrastructure, stuff for my teaching/students, gaming projects, and production VPSs for IT clients of mine. Prior to this, I was using Digital Ocean and good old fashioned rsync (along with rsnapshot on the host that rsync ran on) in order to manage these services and keep backups and version control. It was getting cumbersome and since I upgraded the old server, I decided to use the old one off-site as a backup server at my office. Below, I will step you through the simple steps I took to keep snapshots of all these production environments.
Since I am backing up multiple remote hosts, I chose to create separate configs for each remote host instead of putting those destinations in the ”/etc/rsnapshot.conf” together with the localhost/backup server. I have the config set to retain 90 days worth of backups called “alpha” and it is set to sync first. The localhost configuration is as follows:
#base config
config_version 1.2
snapshot_root /mnt/backups/rsnapback/localhost.domain/
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_logger /usr/bin/logger
verbose 2
loglevel 3
logfile /var/log/rsnapshot.log
sync_first 1
use_lazy_deletes 1
retain alpha 90
#localhost
backup /home/ home/
backup /etc/ etc/
backup /usr/local/bin/ bin/
backup /root/ root/
backup /var/www/ www/
Then, to run this, I use a simple script I created. Again, this script is just for running the localhost, so it can also be used for someone’s single desktop:
#!/bin/bash
service="/usr/bin/rsnapshot"
logfile="/home/logs/tempa.log"
START1="$(date +%s)"
touch $logfile
$service -V sync > $logfile
END1="$(date +%s)"
DURATION1=$[ ${END1} - ${START1} ]
MINUTES=$[ ${DURATION1} / 60 ]
sed -i "1s/^/Jonathan, at $(date), the rsnapshot sync took exactly ${DURATION1} seconds which is approximately ${MINUTES} minutes to complete.\n/" $logfile
if
tail -n -5 $logfile | grep "completed"
then
START1="$(date +%s)"
echo "Jonathan, at $(date), the sync has completed and I am now running alpha."
$service -V alpha >> $logfile
END1="$(date +%s)"
DURATION1=$[ ${END1} - ${START1} ]
MINUTES=$[ ${DURATION1} / 60 ]
sed -i "1s/^/Jonathan, at $(date), the rsnapshot alpha took exactly ${DURATION1} seconds which is approximately ${MINUTES} minutes to complete.\n/" $logfile
mail -s "[$(hostname -f)]-rsnap-success-$(date)" alerts@alerts.com < $logfile
rm $logfile
chown user:user -R /mnt/backups/rsnapback/localhost.domain/
chmod 770 -R /mnt/backups/rsnapback/localhost.domain/
else
echo "Jonathan, at $(date), the sync has failed and I am now notifying you."
mail -s "[$(hostname -f)]-rsnap-failure-$(date)" alerts@alerts.com < $logfile
rm $logfile
fi
Then, in ”/etc/cron.d/rsnapshot”, I set up a cronjob to run this script on a schedule. For my particular use case, twice a day is sufficient, but this can obviously be adjusted to the user’s needs:
0 0,14 * * * root /bin/bash /usr/local/bin/rsnap-alpha-netback.jonathanhaack.com.sh > /dev/null 2>&1
Now that the localhost/backup server is backing up it’s own essential boot volume files to the backup zpool, it’s time to create the configs and scripts for the remote hosts. In my case, I have over 15 remote hosts set up this way, so I will highlight the setup for one in order to demonstrate what I’ve done. First, create a directory for the configs such as ”/etc/rsnapshot/” and then create a config file within it such as ”/etc/rsnapshot/rsnapshot-domain.com.conf”. The config for the remote host looks very similar to the localhost config, but it has syntax for connecting to the remote host. Of course, make sure that you have properly exchanged ssh keys and installed rsync on the remote host before proceeding:
#base config
config_version 1.2
snapshot_root /mnt/backups/rsnapback/domain.com/
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_logger /usr/bin/logger
verbose 2
loglevel 3
logfile /var/log/rsnapshot.log
sync_first 1
use_lazy_deletes 1
retain alpha 90
#directories
backup root@domain.com:/etc/ etc/
backup root@domain.com:/usr/local/bin/ bin/
backup root@domain.com:/var/www/ www/
backup root@domain.com:/home/ home/
backup root@domain.com:/root/ root/
Now that the config is setup, you use a script very similar to the localhost script above, but note the syntax changes for sync and alpha that will specify to rsnapshot to use this particular configuration file. Additionally, just like the localhost, I specified for rsnapshot to retain a snapshot called alpha for 90 days, which suits my use case. Here’s the script for the remote hosts:
#!/bin/bash
service="/usr/bin/rsnapshot"
logfile="/home/sexa/logs/domain.com.log"
host="domain.com"
START1="$(date +%s)"
touch $logfile
$service -c /etc/rsnapshot/rsnapshot-$host.conf -V sync > $logfile
END1="$(date +%s)"
DURATION1=$[ ${END1} - ${START1} ]
MINUTES=$[ ${DURATION1} / 60 ]
sed -i "1s/^/Jonathan, at $(date), the rsnapshot sync took exactly ${DURATION1} seconds which is approximately ${MINUTES} minutes to complete.\n/" $logfile
if
tail -n -5 $logfile | grep "completed"
then
START1="$(date +%s)"
echo "Jonathan, at $(date), $(hostname -f) ran a sync that completed so I am now running alpha."
$service -c /etc/rsnapshot/rsnapshot-$host.conf -V alpha >> $logfile
END1="$(date +%s)"
DURATION1=$[ ${END1} - ${START1} ]
MINUTES=$[ ${DURATION1} / 60 ]
sed -i "1s/^/Jonathan, at $(date), the rsnapshot alpha took exactly ${DURATION1} seconds which is approximately ${MINUTES} minutes to complete.\n/" $logfile
#echo "Jonathan, at $(date), the rsnapshot alpha took exactly ${DURATION1} seconds which is approximately ${MINUTES} minutes to complete." | tee -a $logfile
mail -s "[${host}]-rsnap-success-$(date)" alerts@alerts.com < $logfile
rm $logfile
chown user:user -R /mnt/backups/rsnapback/$host/
chmod 770 -R /mnt/backups/rsnapback/$host/
else
echo "Jonathan, at $(date), $(hostname -f) ran a sync that failed and I am now notifying you."
mail -s "[${host}]-rsnap-failure-$(date)" alerts@alerts.com < $logfile
rm $logfile
fi
Similarly, I need to set up a cronjob to execute this script in ”/etc/cron.d/rsnapshot” for which I once again chose twice a day:
0 00,14 * * * root /bin/bash /usr/local/bin/rsnap-alpha-comain.com.sh > /dev/null 2>&1
You would think the retain 90 would delete the old ones, but it does not seem to work for whatever reason, and so I setup a small script to delete anything past 90 which runs daily at 8am, well after the last snapshot completed. It’s also a loop which will make sure to prune all the snapshots between 90-100. If the script fails, it’s possible that some snapshots could run past 90, hence including a ten day buffer. Here’s the backup scrip, and trigger warning, it’s a kludge:
#!/bin/bash
date=date +"%Y%m%d-%H:%M:%S"
log="/home/logs/rsnap-delete.log"
img="domain.com domain.org domain.net"
for i in $img;
do
touch $log
START0="$(date +%s)"
rm -rf /mnt/backups/rsnapback/$i/alpha.90
rm -rf /mnt/backups/rsnapback/$i/alpha.91
rm -rf /mnt/backups/rsnapback/$i/alpha.92
rm -rf /mnt/backups/rsnapback/$i/alpha.93
rm -rf /mnt/backups/rsnapback/$i/alpha.94
rm -rf /mnt/backups/rsnapback/$i/alpha.95
rm -rf /mnt/backups/rsnapback/$i/alpha.96
rm -rf /mnt/backups/rsnapback/$i/alpha.97
rm -rf /mnt/backups/rsnapback/$i/alpha.98
rm -rf /mnt/backups/rsnapback/$i/alpha.99
END0="$(date +%s)"
DURATION0=$[ ${END0} - ${START0} ]
MINUTES0=$[ ${DURATION0} / 60 ]
echo "Jonathan, at $(date), the rsnapshot purge took exactly ${DURATION0} seconds which is approximately
${MINUTES0} minutes to complete." | tee -a $log
mail -s "[rsnap-delete-$i]-$(hostname -f)-$(date)" alerts@alerts.com < $log
rm $log
done
So far, everything is working great. In the future, I would like to debug why rsnapshot retains more than the 90 specified in alpha and/or polish up the delete script some more. I would also like to improve the rsnap-alpha script to check for a process id rather than checking the sync log for the conditional upon which it executes the script. I already have a colleague who has a good template for doing that and it’s in progress. In the mean time, however, I’ve already used this for 120 days or so and had to use it to restore some versions I accidentally deleted. Very pleased for now and, more importantly, satisfied that I have a more manageable system with which to protect the client, community, and student data that I manage.
If you would like to support my projects or communities, please subscribe to my Patreon: https://www.patreon.com/haacknet
Happy hacking folks!
-oemb1905