Installation and configuration of duplicity for encrypted SFTP remote backup

Note: This howto is old and may not work anymore at the time you read this (04 June 2013)

In this how-to, I’d like to show how to install and configure duplicity to perform remote backups. This is presented as a walk-through, and it represents the way I installed and configured duplicity on my own server.

In the following I assume that Debian is used. Very few should differ if using another distribution, with the exception of the installation of duplicity. You should then use the most common way to install software on your server.

Overview of the process:

• Generation of RSA keys and installation for key-based SFTP authentication
• Generation of GPG keys for duplicity’s encryption feature
• Installation of duplicity
• Configuration of duplicity

Detailed walk-through:

RSA keys generation and installation for key-based SFTP authentication

First step is to generate a pair of RSA keys that will be used for SFTP authentication.
Go to your home directory and use the ssh-keygen tool with the options given hereafter. Do NOT enter any passphrase when prompted (because duplicity is meant to be run on background and won’t be able to supply a password each time it will authenticate on the remote server). It is up to you to protect the private key from unauthorized access.

root@myhost:~# ssh-keygen -t rsa -b 2048
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): [press enter]
Enter passphrase (empty for no passphrase): [press enter]
Enter same passphrase again: [press enter]
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx root@myhost

Make sure the permissions of the public and private keys are set respectively to 644 and 600 :

root@myhost:~# ls -l .ssh/
-rw------- 1 root root 1,7K 3 mai 20:50 id_rsa
-rw-r--r-- 1 root root 395 3 mai 20:50 id_rsa.pub

Copy the content of .ssh/id_rsa.pub on your local server and add it to .ssh/authorized_keys on the remote server by any means (usually through SSH shell access). In case you don’t have SSH access to the remote server you can fetch the file, edit it locally and then re-upload it via (S)FTP.

root@myhost:~# cat .ssh/id_rsa.pub
ssh-rsa AAAAB4NzaC7yc2EAAAADAQABABBBAQCxQTp1vYmCoPOJ5gre5SGD98gFlUk
ZYmSAHbKcg98gl0j9LJm2h0fezqfEfsKy2K/bSfmnM+s917A9FUhgYFiJjnVRFV3rih
2hfrFMU/BMp/9FUl9/4T+22zueppdNhETcmQRCCOFLart0beL+CI8G8wyZ/rNLzWIq2
L598Ag/Am/zgh/WbPH2bNPXzza+MP2rNp+eE/u2cIDDdaffLoez992K+CphhMVxZXz0
3f2+v3Yp9BVUpefezh9sa9RkSsRpoHXEktgQB04PUHm2wmzSkNdhNOwtJQlKIKfqvhK
njiSddsGRJIiha1ukZBP+haVnM1aa3AeD88dRdbLlTUNt root@myhost

root@myhost:~# sftp remoteuser@remoteserver.example.com
remoteuser@remoteserver.example.com's password:
Connected to remoteserver.example.com.
sftp> get .ssh/authorized_keys
Fetching /.ssh/authorized_keys to authorized_keys
/.ssh/authorized_keys 100% 398 0.4KB/s 00:00
sftp> ^D

root@myhost:~# cat .ssh/id_rsa.pub >> authorized_keys

root@myhost:~# sftp remoteuser@remoteserver.example.com
remoteuser@remoteserver.example.com's password:
Connected to remoteserver.example.com.
sftp> cd .ssh
sftp> put authorized_keys
Uploading authorized_keys to /.ssh/authorized_keys
authorized_keys 100% 399 0.4KB/s 00:00
sftp> ^D 

root@myhost:~# rm authorized_keys

If you did it right, then logging to the remote SFTP server shouldn’t require a password anymore.

root@myhost:~# sftp remoteuser@remoteserver.example.com
Connected to remoteserver.example.com.
sftp>

While you’re connected to the remote server, it’s a good time to prepare the remote directory structure for the backup process. Hereafter is an example where I create the directory structure /backup/myhost/etc-only
myhost is the name of the server I want to backup, and etc-only will be one of the backup set I will configure later, where only the /etc directory will be backuped.
Again this is only an example, and you’re free to design another directory structure that suits best your needs.

sftp> ls
dev etc home lib lib64 public usr
sftp> mkdir backup
sftp> cd backup/
sftp> mkdir myhost
sftp> cd myhost/
sftp> mkdir etc-only
sftp> ^D

First part is done! Let’s go for the second part:

GPG keys generation for duplicity’s encryption feature

If you want your backup encrypted before being uploaded to the remote server, duplicity will need a pair of GPG keys. I strongly suggest to create a dedicated pair of keys for duplicity to use and NOT to use your own existing keys, because of the hereafter mentioned limitation in the confidentiality of the stored passphrase.

The identity of the key (Name/email) doesn’t matter very much. However you have to be fully aware that the passphrase will appear in cleartext somewhere in duplicity’s configuration. So do NOT use one of your personal passphrases!

root@myhost:~# gpg --gen-key
gpg (GnuPG) 1.4.10; Copyright (C) 2008 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
 (1) RSA and RSA (default)
 (2) DSA and Elgamal
 (3) DSA (sign only)
 (4) RSA (sign only)
Your selection? [press enter]
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) [press enter]
Requested keysize is 2048 bits
Please specify how long the key should be valid.
 0 = key does not expire
 <n> = key expires in n days
 <n>w = key expires in n weeks
 <n>m = key expires in n months
 <n>y = key expires in n years
Key is valid for? (0) [press enter]
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
 "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: John Doe
Email address: john.doe@example.com
Comment:
You selected this USER-ID:
 "John Doe <john.doe@example.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.
Set a passphrase, this will be the one that protects the confidentiality of your data on the remote server.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

Not enough random bytes available. Please do some other work to give
the OS a chance to collect more entropy! (Need 108 more bytes)
...............+++++
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key D59A550A marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
pub 2048R/D59A550A 2012-05-03
 Key fingerprint = 382D A19B 931E D37E C2FF 9EBE AEE1 68B0 D59A 550A
uid John Doe <john.doe@example.com>
sub 2048R/D56116BE 2012-05-03

Note the ID of your key! For example here it’s D59A550A

Now that our RSA and GPG keys are ready, let’s install duplicity and configure it to run periodically in background.

Installation of duplicity

For Debian, duplicity is packaged.

root@myhost:~# aptitude install duplicity

There may be some dependencies.

Configuration of duplicity

Here I describe the “simple” way to configure duplicity in order to backup automatically thanks to the cron deamon. At the end of this page you’ll find another way to call duplicity through an helper script that I recommend.

For the example, we’ll ask duplicity to backup only /etc with encryption and all other parameters at their defaults. If you want to tune duplicity’s parameters, please look at its manpage.

If the installation went well, let’s test our setup so far:

root@myhost:~# duplicity --encrypt-key D59A550A /etc scp://remoteuser@remoteserver.com/backup/myhost/etc-only

If everything went well, you should see such the following output:

Synchronizing remote metadata to local cache...
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1336076161.79 (Thu May 3 22:16:01 2012)
EndTime 1336076163.46 (Thu May 3 22:16:03 2012)
ElapsedTime 1.67 (1.67 seconds)
SourceFiles 1803
SourceFileSize 3024131 (2.88 MB)
NewFiles 1803
NewFileSize 3024131 (2.88 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1803
RawDeltaSize 2239652 (2.14 MB)
TotalDestinationSizeChange 703554 (687 KB)
Errors 0
-------------------------------------------------

If you try to launch this command twice, you’ll notice that it doesn’t seem to work. The error message you get is related to the fact that duplicity needs to decrypt remote content and it doesn’t know the passphrase and doesn’t ask for it interactively either.

You can check if further executions of duplicity work by typing:

root@myhost:~# PASSPHRASE="the_chosen_passphrase" duplicity --encrypt-key D59A550A /etc scp://remoteuser@remoteserver.example.com/backup/myhost/etc-only

Don’t forget to wipe out your command history if needed, so that the passphrase doesn’t get logged in it.

If you went so far and the two last commands exited without error, congratulations! Your first backup is done.

The only task left is to automatize the backup process.

Let’s create a little shell script to launch duplicity once a day with the cron deamon:

root@myhost:~# vim /etc/cron.daily/duplicity

and copy this:

#!/bin/sh

test -x /usr/bin/duplicity || exit 0
. /root/.duplicity_pass
export PASSPHRASE
/usr/bin/duplicity --encrypt-key D59A550A /etc scp://remoteuser@remotehost.example.com/backup/myhost/etc-only

As we said before, for duplicity to be able to encrypt the backup with the GPG key, it must have some way to know the passphrase.

One of the “less ugly” solution I found was to store the passphrase in a file that only root could read (you can choose the location as you want, but don’t forget to adapt the above script consequently):

root@myhost:~# vim /root/.duplicity_pass

Write the following in it:

PASSPHRASE="replace_with_the_chosen_passphrase_for_the_gpg_key"

And then chmod it to 700:

root@myhost:~# chmod 700 /root/.duplicity.pass

Eventually, make the cron script executable:

root@myhost:~# chmod +x /etc/cron.daily/duplicity

Test the script to check if everything is working properly:

root@myhost:~# /etc/cron.daily/duplicity
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu May 3 22:15:59 2012
--------------[ Backup Statistics ]--------------
StartTime 1336080076.59 (Thu May 3 23:21:16 2012)
EndTime 1336080077.08 (Thu May 3 23:21:17 2012)
ElapsedTime 0.50 (0.50 seconds)
SourceFiles 1804
SourceFileSize 3024328 (2.88 MB)
NewFiles 3
NewFileSize 8389 (8.19 KB)
DeletedFiles 0
ChangedFiles 1
ChangedFileSize 232 (232 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 4
RawDeltaSize 436 (436 bytes)
TotalDestinationSizeChange 866 (866 bytes)
Errors 0
-------------------------------------------------

If you don’t see any error, then you’re done, your backup will launch automatically once a day.

Don’t forget to learn how to recover your data (before needing to do so) and to test your backups regularly.

Hereafter I describe how to use a more advanced script to handle duplicity backup configuration.

Using duplicity-backup.sh to do the job

I maintain a wrapper script for duplicity called duplicity-backup.sh and originally developed by Damon Timm.

This wrapper script has many functionality to help you administrate your backup sets (backup, recover, check remote files state…) and fine-tune the configuration (include/exclude files and folders…).

Once duplicity is properly installed and able to backup, you can install and use duplicity-backup.sh to wrap the job.

You can put duplicity-backup at the location of your choice, then configure it (see duplicity-backup page for details) and eventually the only thing to change is the script launched by the cron deamon.

root@myhost:~# vim /etc/cron.daily/duplicity

For example I use this one (replace or comment any preexistent content):

#!/bin/sh
test -x /root/scripts/duplicity-backup/duplicity-backup.sh || exit 0
cd /root/scripts/duplicity-backup && ./duplicity-backup.sh -c /etc/duplicity-backup.conf --backup

(I installed duplicity-backup at /root/scripts/duplicity-backup/duplicity-backup.sh and the config file is at /etc/duplicity-backup.conf)

Here you are, through this how-to you should have been able to install and configure duplicity as your fire-and-forget remote backup solution.