Howto: automate remote backups using rdiff-backup and Perl

Introduction:

Continuing with my backup articles (part two of my Amanda series coming soon ...), I thought I wowould tell you about how I do my remote backups. the program I use is rdiff-backup, with a Perl script to sort out e-mail Notification and logfile generation. I will take you through my script and show you how to enable SSH passwordless access using public and private keys, so no interaction is required for full automatic backups.

What is it?

Rdiff-Backup backs up one directory to another, possibly over a network. the target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. the idea is to combine the best features of a mirror and an incremental backup. rdiff-Backup also preserves subdirectories, hard links, Dev files, permissions, UID/GID ownership, and modification times. also, rdiff-Backup can operate in a bandwidth efficient manner over a pipe, like rsync. thus you can use rdiff-backup and SSH to securely back a hard drive up to a remote location, and only the differences will be transmitted. finally, rdiff-backup is easy to use and settings have sensical defaults.

Rdiff-backup, as you read, uses the rsync library, so why am I not using rsync it's self? Well, I couldn't be bothered processing ing it, as I just wanted to run one command that handled everything, including incremental backups, So let's begin.

Planning:

As with all backups, you need to plan what you wowould like to back up. for this howto, I have chosen only one directory, using the simplest rdiff-backup method. basically, I want to backup my Perl scripts directory for safe keeping. I will use the least possible configuration parameters, but if you wowould like to see more options, as usual, TypeMan rdiff-BackupOr visit the examples page.

Next, you shoshould decide on what machine you will be sending the backups. I will be using my home machine as the backup, which will receive the files from my work machine. lastly, make sure you have SSH access to both machines, as we will be usingSCP.

Implementing:

First, we need to install rdiff-backup on both machines. If you have Dag wieersYumOrAptSources configured, then all you need to do is:

[ghenry@work myscripts] apt-get install rdiff-backup

Or

[ghenry@work myscripts] yum install rdiff-backup

If not, grab the RPMs from rdiff-backup, half way down the front page. install them the usual way:

[ghenry@work myscripts] rpm -Uvh rdiff-backup-0.12.7-1.i386.rpm

Now we need to setup passwordless SSH access. If we call my home machine,HomeAnd my work machineWork(Clever eh ?), We need to tell myHomeMachine that myWorkMachine is allowed to login via passkeys. The pass key pair needs to be generated on the work machine only. (See Daniels SSH article for more info ).

First, generate a public/private DSA key pair onWork.

[ghenry@work myscripts]$ ssh-keygen -t dsa -f ~/.ssh/id_dsa

When you are asked for a passphrase, leave it empty, this gets rid of the password. now send the public keyHome

[ghenry@work myscripts] cd .ssh[ghenry@work myscripts] scp id_dsa.pub ghenry@home:~/.ssh

Next, log inHomeAnd add the public key to the list of authorized keys.

[ghenry@work myscripts] ssh ghenry@home[ghenry@home backup] cd .ssh[ghenry@home backup] cat id_dsa.pub >> authorized_keys2[ghenry@home backup] chmod 640 authorized_keys2[ghenry@home backup] rm -f id_dsa.pub

Note that the filename is authorized_keys2, not authorized_keys. That's it; you're ready to SSH fromWorkToHomeWithout having to enter a password (as long as you have not previusly messed/Etc/sshd_config).

Now that's all done, we can move on to rdiff-backup and the Perl script.

For those of you unfamiliar with Perl, I wocould suggest buying the O 'Reilly learning Perl book, as it is the best.

Here is my script called rdiff-script (which is always being updated and is available from my Perl site ). this is actually my first ever Perl script, so gurus be very kind, and I know it can be shorter and cleaner, but I am getting there. I am still only on chapter 8 of learning Perl you know ;-):

#!/usr/bin/perluse strict;use warnings;use Mail::Sendmail;use POSIX qw(strftime);##################################################  program:rdiff-script                    ##  license:GPL                             ##  author:Gavin Henry                     ##  company:Suretec Systems Ltd.            ##  url:http://www.suretecsystems.com   ##  version:v1.0                            ##                                               ##  first draft : 30-08-04                       ##  last update : 03-09-04################################################### Global variablesmy $rdiff       = '/usr/bin/rdiff-backup';my $option1     = '-v5';my $option2     = '--print-statistics';my $localdir    = '/your/localdir';my $userhost = 'you@home';my $remotedir = '/your/remotedir';my @args        = ( $option1, $option2, $localdir, "$userhost/:/:$remotedir" );my $to          = 'you@you.com;my $from        = 'them@them.com';my $sep         =  '-' x 76 . "/n";my $time        = localtime;  my $datestamp   =  strftime '%d.%m.%y.%T', localtime;# Messagesprint "/n", $sep, "Brought to you by Your Name./n", $sep;print "/n", $sep, "Initialising remote backup synchronsation on $time./n", $sep;# getting exit code and program outputmy $bdata = `$rdiff @args`;my $backup = $?;# Send e-mail with a few details for success and failures# Successif ($backup == 0) {my %mails = (     To      => "$to",    From    => "$from",    Subject => "Remote backup complete from $ENV{HOSTNAME} on $time",    Message => "The remote backup has been completed on $ENV{HOSTNAME}"                 . " on $time with the command:/n/n $rdiff @args/n"                 . " The commands output was /n/n$bdata/n/n");sendmail(%mails);# Success finish messageprint "/n", $sep, "Remote backup complete on $time. E-mail sent with details./n", $sep;# Create a success logfileopen LOG, ">>$datestamp-rdiff-backup-success.log"  or die "Cannot create logfile: $!";print LOG "Remote backup completed on $time, with the command:/n/n$rdiff @args/n/nOutput:/n/n $bdata/n/nAn e-mail has been sent./n";close LOG;print "Logfile created on $time./n/n";# Failure} else {my %mailf = (     To      => "$to",    From    => "$from",    Subject => "Remote backup failed from $ENV{HOSTNAME} on $time",    Message => "The remote backup has failed on $ENV{HOSTNAME}"                 . " on $time with the command:/n/n$rdiff @args/n/n"                 . " The commands output was /n/n$bdata/n/n");sendmail(%mailf);# Failure finish messageprint "/n", $sep, "Remote backup failed on $time. E-mail sent with details./n", $sep;# Create a failure logfileopen LOG, ">>$datestamp-rdiff-backup-failed.log"  or die "Cannot create logfile: $!";print LOG "Remote backup failed on $time, with the command:/n/n$rdiff @args/n/nOutput:/n/n $bdata/n/nAn e-mail has been sent./n";close LOG;print "Logfile created on $time./n/n";die "Backup exited funny: $?" unless $backup == 0;}# Program complete

For the above to work, you need to change a few variables, namely the e-mail address and the directories. It's pretty easy to see what's what.

You also needMail: SendmailModule, which can be installed whilst being root as follows:

[ghenry@work myscripts] perl -MCPAN -e shell[ghenry@work myscripts] 
 
   install Mail::Sendmail
 

Once, you have changed those, you can run the script. it will first create a full backup, as there are no files on the remote machine and from then on, it will create incremental backups (seen below ), or rather the difference between what is on the locale machine and remote and only send them. the script wiil give you an e-mail which will be sent on the success or failure of the backup and A logfile generated, which will be removed in the body of the E-mail and also saved in the directory from which the script was called.

[ghenry@database myscripts]$ ./rdiff-script                                                                                ----------------------------------------------------------------------------Brought to you by Your Name.----------------------------------------------------------------------------                                                                                ----------------------------------------------------------------------------Initialising remote backup synchronsation on Fri Sep  3 11:05:08 2004.----------------------------------------------------------------------------Processing changed file .Incrementing mirror file /home/ghenry/backup/perlProcessing changed file myscriptsIncrementing mirror file /home/ghenry/backup/myscriptsProcessing changed file myscripts/02.09.04.16:10:59-rdiff-backup-success.logIncrementing mirror file /home/ghenry/backup/perl/myscripts/02.09.04.16:10:59-rdiff-backup-success.logProcessing changed file myscripts/rdiff-scriptIncrementing mirror file /home/ghenry/backup/perl/myscripts/rdiff-scriptProcessing changed file myscripts/rdiff-script~Incrementing mirror file /home/ghenry/backup/perl/myscripts/rdiff-script~ ----------------------------------------------------------------------------Remote backup complete on Fri Sep  3 11:05:08 2004. E-mail sent with details.----------------------------------------------------------------------------Logfile created on Fri Sep  3 11:05:08 2004.

Now if it fails, you will get something like:

----------------------------------------------------------------------------Brought to you by Your Name.---------------------------------------------------------------------------- ----------------------------------------------------------------------------Initialising remote backup synchronsation on Fri Sep  3 11:33:29 2004.----------------------------------------------------------------------------ssh: fakedomain: Name or service not knownFatal Error: Truncated header string (problem probably originated remotely) Couldn't start up the remote connection by executing     ssh -C ghenry@fakedomain rdiff-backup --server Remember that, under the default settings, rdiff-backup must beinstalled in the PATH on the remote system.  See the man page for moreinformation on this.  This message may also be displayed if the remoteversion of rdiff-backup is quite different from the local version (0.12.7). ----------------------------------------------------------------------------Remote backup failed on Fri Sep  3 11:33:29 2004. E-mail sent with details.----------------------------------------------------------------------------Logfile created on Fri Sep  3 11:33:29 2004. Backup exited funny: 256 at ./rdiff-script line 89.

And the contents of the logfile and E-mail are something like and the same as abve for failures:

Remote backup completed on Fri Sep  3 11:28:26 2004, with the command:                                                                                /usr/bin/rdiff-backup -v5 --print-statistics /home/ghenry/perl ghenry@home::/home/ghenry/backups/perl                                                                                Output: Executing ssh -C ghenry@home rdiff-backup --server--------------[ Session statistics ]--------------StartTime 1094210228.00 (Fri Sep  3 12:17:08 2004)EndTime 1094210305.33 (Fri Sep  3 12:18:25 2004)ElapsedTime 77.33 (1 minute 17.33 seconds)SourceFiles 10719SourceFileSize 61807347 (58.9 MB)MirrorFiles 10719MirrorFileSize 61807331 (58.9 MB)NewFiles 0NewFileSize 0 (0 bytes)DeletedFiles 0DeletedFileSize 0 (0 bytes)ChangedFiles 3ChangedSourceSize 3139 (3.07 KB)ChangedMirrorSize 3123 (3.05 KB)IncrementFiles 3IncrementFileSize 608 (608 bytes)TotalDestinationSizeChange 624 (624 bytes)Errors 0--------------------------------------------------                                                                                                                                                                An e-mail has been sent.

This can all be automatedCronUsing the following settings, which will run the script once a day, every day, at 2 AM (shows all steps ):

[ghenry@work myscripts] crontab -e 0 2 * * * "./home/ghenry/scripts/rdiff-script"~~~~:wq[ghenry@work myscripts]$crontab: installing new crontab

Of course, you can run it more then once and at anytime time you like, seeMan crontabFor more info.

Summary:

I have only touched upon the settings of rdiff-backup, but you can see how easy it is. I have taken you through installing it, login ing SSH passkeys, testing and automation. it can be even easier than this if you don't want the Perl script. just enter the crontab entry to call rdiff-Backup directly. of course you won't get the nice e-mails and tailored logfiles

Well, that's it for now. For any comments or corrections, please E-mail me.