Note: This howto will be rewritten as soon as possible to be more generic and clear (04 June 2013)
In this how-to, I’d like to show how to install and configure duplicity to perform remote backups. This is presented as a walk-through, and it represents the way I installed and configured duplicity on my own server.
In the following I assume that Debian is used. Very few should differ if using another distribution, with the exception of the installation of duplicity. You should then use the most common way to install software on your server.
Overview of the process:
- Generation of RSA keys and installation for key-based SFTP authentication
- Generation of GPG keys for duplicity’s encryption feature
- Installation of duplicity
- Configuration of duplicity
RSA keys generation and installation for key-based SFTP authentication
First step is to generate a pair of RSA keys that will be used for SFTP authentication.
Go to your home directory and use the ssh-keygen tool with the options given hereafter. Do NOT enter any passphrase when prompted (because duplicity is meant to be run on background and won’t be able to supply a password each time it will authenticate on the remote server). It is up to you to protect the private key from unauthorized access.
root@myhost:~# ssh-keygen -t rsa -b 2048 Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): [press enter] Enter passphrase (empty for no passphrase): [press enter] Enter same passphrase again: [press enter] Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx root@myhost
Make sure the permissions of the public and private keys are set respectively to 644 and 600 :
root@myhost:~# ls -l .ssh/ -rw------- 1 root root 1,7K 3 mai 20:50 id_rsa -rw-r--r-- 1 root root 395 3 mai 20:50 id_rsa.pub
Copy the content of
.ssh/id_rsa.pub on your local server and add it to
.ssh/authorized_keys on the remote server by any means (usually through SSH shell access). In case you don’t have SSH access to the remote server you can fetch the file, edit it locally and then re-upload it via (S)FTP.
root@myhost:~# cat .ssh/id_rsa.pub ssh-rsa AAAAB4NzaC7yc2EAAAADAQABABBBAQCxQTp1vYmCoPOJ5gre5SGD98gFlUk ZYmSAHbKcg98gl0j9LJm2h0fezqfEfsKy2K/bSfmnM+s917A9FUhgYFiJjnVRFV3rih 2hfrFMU/BMp/9FUl9/4T+22zueppdNhETcmQRCCOFLart0beL+CI8G8wyZ/rNLzWIq2 L598Ag/Am/zgh/WbPH2bNPXzza+MP2rNp+eE/u2cIDDdaffLoez992K+CphhMVxZXz0 3f2+v3Yp9BVUpefezh9sa9RkSsRpoHXEktgQB04PUHm2wmzSkNdhNOwtJQlKIKfqvhK njiSddsGRJIiha1ukZBP+haVnM1aa3AeD88dRdbLlTUNt root@myhost root@myhost:~# sftp email@example.com firstname.lastname@example.org's password: Connected to remoteserver.example.com. sftp> get .ssh/authorized_keys Fetching /.ssh/authorized_keys to authorized_keys /.ssh/authorized_keys 100% 398 0.4KB/s 00:00 sftp> ^D root@myhost:~# cat .ssh/id_rsa.pub >> authorized_keys root@myhost:~# sftp email@example.com firstname.lastname@example.org's password: Connected to remoteserver.example.com. sftp> cd .ssh sftp> put authorized_keys Uploading authorized_keys to /.ssh/authorized_keys authorized_keys 100% 399 0.4KB/s 00:00 sftp> ^D root@myhost:~# rm authorized_keys
If you did it right, then logging to the remote SFTP server shouldn’t require a password anymore.
root@myhost:~# sftp email@example.com Connected to remoteserver.example.com. sftp>
While you’re connected to the remote server, it’s a good time to prepare the remote directory structure for the backup process. Hereafter is an example where I create the directory structure
myhost is the name of the server I want to backup, and
etc-only will be one of the backup set I will configure later, where only the
/etc directory will be backuped.
Again this is only an example, and you’re free to design another directory structure that suits best your needs.
sftp> ls dev etc home lib lib64 public usr sftp> mkdir backup sftp> cd backup/ sftp> mkdir myhost sftp> cd myhost/ sftp> mkdir etc-only sftp> ^D
First part is done! Let’s go for the second part:
GPG keys generation for duplicity’s encryption feature
If you want your backup encrypted before being uploaded to the remote server, duplicity will need a pair of GPG keys. I strongly suggest to create a dedicated pair of keys for duplicity to use and NOT to use your own existing keys, because of the hereafter mentioned limitation in the confidentiality of the stored passphrase.
The identity of the key (Name/email) doesn’t matter very much. However you have to be fully aware that the passphrase will appear in cleartext somewhere in duplicity’s configuration. So do NOT use one of your personal passphrases!
root@myhost:~# gpg --gen-key gpg (GnuPG) 1.4.10; Copyright (C) 2008 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? [press enter] RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) [press enter] Requested keysize is 2048 bits Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) [press enter] Key does not expire at all Is this correct? (y/N) y You need a user ID to identify your key; the software constructs the user ID from the Real Name, Comment and Email Address in this form: "Heinrich Heine (Der Dichter) <firstname.lastname@example.org>" Real name: John Doe Email address: email@example.com Comment: You selected this USER-ID: "John Doe <firstname.lastname@example.org>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O You need a Passphrase to protect your secret key. Set a passphrase, this will be the one that protects the confidentiality of your data on the remote server. We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. Not enough random bytes available. Please do some other work to give the OS a chance to collect more entropy! (Need 108 more bytes) ...............+++++ gpg: /root/.gnupg/trustdb.gpg: trustdb created gpg: key D59A550A marked as ultimately trusted public and secret key created and signed. gpg: checking the trustdb gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u pub 2048R/D59A550A 2012-05-03 Key fingerprint = 382D A19B 931E D37E C2FF 9EBE AEE1 68B0 D59A 550A uid John Doe <email@example.com> sub 2048R/D56116BE 2012-05-03
Note the ID of your key! For example here it’s
Now that our RSA and GPG keys are ready, let’s install duplicity and configure it to run periodically in background.
Installation of duplicity
For Debian, duplicity is packaged.
root@myhost:~# aptitude install duplicity
There may be some dependencies.
Configuration of duplicity
Here I describe the “simple” way to configure duplicity in order to backup automatically thanks to the cron deamon. At the end of this page you’ll find another way to call duplicity through an helper script that I recommend.
For the example, we’ll ask duplicity to backup only
/etc with encryption and all other parameters at their defaults. If you want to tune duplicity’s parameters, please look at its manpage.
If the installation went well, let’s test our setup so far:
root@myhost:~# duplicity --encrypt-key D59A550A /etc scp://firstname.lastname@example.org/backup/myhost/etc-only
If everything went well, you should see such the following output:
Synchronizing remote metadata to local cache... Last full backup date: none No signatures found, switching to full backup. --------------[ Backup Statistics ]-------------- StartTime 1336076161.79 (Thu May 3 22:16:01 2012) EndTime 1336076163.46 (Thu May 3 22:16:03 2012) ElapsedTime 1.67 (1.67 seconds) SourceFiles 1803 SourceFileSize 3024131 (2.88 MB) NewFiles 1803 NewFileSize 3024131 (2.88 MB) DeletedFiles 0 ChangedFiles 0 ChangedFileSize 0 (0 bytes) ChangedDeltaSize 0 (0 bytes) DeltaEntries 1803 RawDeltaSize 2239652 (2.14 MB) TotalDestinationSizeChange 703554 (687 KB) Errors 0 -------------------------------------------------
If you try to launch this command twice, you’ll notice that it doesn’t seem to work. The error message you get is related to the fact that duplicity needs to decrypt remote content and it doesn’t know the passphrase and doesn’t ask for it interactively either.
You can check if further executions of duplicity work by typing:
root@myhost:~# PASSPHRASE="the_chosen_passphrase" duplicity --encrypt-key D59A550A /etc scp://email@example.com/backup/myhost/etc-only
Don’t forget to wipe out your command history if needed, so that the passphrase doesn’t get logged in it.
If you went so far and the two last commands exited without error, congratulations! Your first backup is done.
The only task left is to automatize the backup process.
Let’s create a little shell script to launch duplicity once a day with the cron deamon:
root@myhost:~# vim /etc/cron.daily/duplicity
and copy this:
#!/bin/sh test -x /usr/bin/duplicity || exit 0 . /root/.duplicity_pass export PASSPHRASE /usr/bin/duplicity --encrypt-key D59A550A /etc scp://firstname.lastname@example.org/backup/myhost/etc-only
As we said before, for duplicity to be able to encrypt the backup with the GPG key, it must have some way to know the passphrase.
One of the “less ugly” solution I found was to store the passphrase in a file that only root could read (you can choose the location as you want, but don’t forget to adapt the above script consequently):
root@myhost:~# vim /root/.duplicity_pass
Write the following in it:
And then chmod it to 700:
root@myhost:~# chmod 700 /root/.duplicity.pass
Eventually, make the cron script executable:
root@myhost:~# chmod +x /etc/cron.daily/duplicity
Test the script to check if everything is working properly:
root@myhost:~# /etc/cron.daily/duplicity Local and Remote metadata are synchronized, no sync needed. Last full backup date: Thu May 3 22:15:59 2012 --------------[ Backup Statistics ]-------------- StartTime 1336080076.59 (Thu May 3 23:21:16 2012) EndTime 1336080077.08 (Thu May 3 23:21:17 2012) ElapsedTime 0.50 (0.50 seconds) SourceFiles 1804 SourceFileSize 3024328 (2.88 MB) NewFiles 3 NewFileSize 8389 (8.19 KB) DeletedFiles 0 ChangedFiles 1 ChangedFileSize 232 (232 bytes) ChangedDeltaSize 0 (0 bytes) DeltaEntries 4 RawDeltaSize 436 (436 bytes) TotalDestinationSizeChange 866 (866 bytes) Errors 0 -------------------------------------------------
If you don’t see any error, then you’re done, your backup will launch automatically once a day.
Don’t forget to learn how to recover your data (before needing to do so) and to test your backups regularly.
Hereafter I describe how to use a more advanced script to handle duplicity backup configuration.
Using duplicity-backup to do the job
I maintain a wrapper script for duplicity called duplicity-backup.sh and originally developed by Damon Timm.
This wrapper script has many functionality to help you administrate your backup sets (backup, recover, check remote files state…) and fine-tune the configuration (include/exclude files and folders…).
Once duplicity is properly installed and able to backup, you can install and use duplicity-backup.sh to wrap the job.
You can put duplicity-backup at the location of your choice, then configure it (see duplicity-backup page for details) and eventually the only thing to change is the script launched by the cron deamon.
root@myhost:~# vim /etc/cron.daily/duplicity
For example I use this one (replace or comment any preexistent content):
#!/bin/sh test -x /root/scripts/duplicity-backup/duplicity-backup.sh || exit 0 cd /root/scripts/duplicity-backup && ./duplicity-backup.sh -c /etc/duplicity-backup.conf --backup
(I installed duplicity-backup at
/root/scripts/duplicity-backup/duplicity-backup.sh and the config file is at
Here you are, through this how-to you should have been able to install and configure duplicity as your fire-and-forget remote backup solution.