duplicity-backup.sh

Foreword

Thanks to Damon Timm who created this script at the first place. Every thanks I receive should be shared with him too, and with all the contributors of this fork on Github!

Description

duplicity-backup.sh is a wrapper script derived from the dt-s3-backup.sh script of Damon Timm (http://damontimm.com/code/dt-s3-backup) and designed to automate and simplify the remote backup process of duplicity on Amazon S3 instances or other backup destinations (ftp, rsync, sftp, local file…).

See the README file hereafter for more information.

duplicity-backup.sh IS NOT duplicity

It is only a wrapper script for duplicity written in bash!

This means the following:

  • You need to install and configure duplicity BEFORE using duplicity-backup.sh
  • The official documentation of duplicity is relevant to duplicity-backup.sh too. Virtually any option supported by duplicity can be specified in the config file of duplicity-backup.sh. See the STATIC_OPTIONS, CLEAN_UP_TYPE and CLEAN_UP_VARIABLE parameters in particular.
  • Before asking something about duplicity-backup.sh, ensure that your question isn’t actually concerning duplicity ;) First, make sure you can perform a backup with duplicity without using this script. If you can’t make the backup work with duplicity alone, the problem is probably concerning duplicity and not this script. If you manage to make a backup with duplicity alone but not with this script, then there is probably a problem with duplicity-backup.sh.

Project page / Bug reports / Download

The code is hosted on Github : http://github.com/zertrin/duplicity-backup (Last update: 27 February 2018)

Feedback and pull requests are welcome! However please always use individual feature branches for each pull request. I may not accept a pull request from a master branch.

Please use Github issues for questions and bug reports.

Until I finally find time to write a real how-to for duplicity-backup, here is the README:

README


34 thoughts on “duplicity-backup.sh”

  1. Thank you for maintaining this script.

    If you’re using a variant of RHEL with Postfix, the following command may work better for you:

    On line 257:
    ${MAILCMD} -s """${EMAIL_SUBJECT}""" -r ${EMAIL_FROM} ${EMAIL_TO} < ${LOGFILE}

    The original mail command had a tendency of creating a few invalid 'To' addresses.
    ie:
    To: SERVER3-backup@domain.com, -f@localdomain.local,
    --@localdomain.local, THESPECIFIED@EMAIL.com,
    SERVER3-backup@someotherdomain.com

  2. After looking into this more, this may only be needed if the script does not detect the presence and use of mailx properly.

  3. Thanks for the script! Is it possible to restore a directory to a specific directory ? for example, restore /var/www to /tmp ?

    1. Hi Joeri,

      Unfortunately this functionality is still waiting in the wishlist for a boring weekend where I could find time to implement it directly in the script…

      It is possible to do it using directly duplicity, so it should be possible to implement it the wrapper too.

      If someone feels like implementing it, please don’t hesitate!

    2. It seems it already works in fact. See Ville’s comment on 25 July 2013 at 19:31 for more details. :)

  4. Hi,
    With this script is possible to generate a custom output as defined by owner?
    Example: --output-log /home/user/logs/backup.log

    Thanks.

    1. Not yet directly from the command line, but you can still have different config files for each user, and set up custom output directory for each of them.

      The addition of a command line option to override the log directory specification in the config file would be very easy to add. But I’m not sure if it would really be useful…

  5. Me again.

    Your script is very amazing! Nice, but I have a last quetion, duplicity supports FTP_PASSWORD variable, is possible to set this on a conf file?

    1. EDIT:
      Of course, this can go into the STATIC_OPTIONS variable of the config file

      1. I misunderstood your comment, the FTP_PASSWORD variable doesn’t go in STATIC_OPTIONS of course.
        Duplicity’s official documentation states “Duplicity can also access a repository via ftp. If a user name is given, the environment variable FTP_PASSWORD is read to determine the password”.

        In order to do so a simple FTP_PASSWORD="thepassword" followed by export FTP_PASSWORD in your personal config file should do the trick.

  6. Why is “–restore-dir” option on the wish list (as mentioned on the duplicity-backup front page)? How would it be different from:

    duplicity-backup –config /path/to/duplicity-backup.conf –restore-file rel/path/startdir /target/restorepath

    That seems to restore everything below the mentioned “rel/dir/path”. Would it perhaps include the defined top level directory (i.e. “startdir” in /rel/path/startdir), rather than restoring everything below it?

    1. Did you try it? I didn’t try myself yet. If you have positive (or negative) results, let me know ;)
      I’m quite busy currently and don’t have much time to check this myself.

  7. Yep, the command above seems to work OK to restore either a file or a directory. It restores everything down from the given directory (hence it’s a good idea to make sure that the target is either empty, or otherwise suitable location for all the files found in the directory defined to be restored :)). It seems then that the functionality for “–restore-dir” is already there, and it can, perhaps, be added as an alias to “–restore-file”). Suppose “–restore-file” could alternatively be renamed to “–restore-file-or-dir”, but that might break cron jobs and scripts that rely on the existing arguments.

    1. Ok great, I’ll try to document it soon then and remove it from the todo-list ;)

  8. Hi again Zertrin!

    I have one question, I saw that CLEAN_UP_VARIABLE variable have number 31 and I set this on my script.

    My question is, when duplicity removes old revisions of backup, it’s remove only incremental or duplicity create a new full backup and remove all backup before this?

    1. Short answer :

      it depends of the value of CLEAN_UP_TYPE and REMOVE_INCREMENTALS_OLDER_THAN variables in the config file.

      Long answer:

      The backup process and the cleaning process are distinct:

      • first a backup is done. The type of backup (full or incremental) depends on the status of the current chain of backup and of the backup parameters in the script (in particular --full-if-older-than in STATIC_OPTIONS)
      • then a cleanup is done in two steps :
        • first a cleanup done according to the cleanup parameters in duplicity-backup.conf (CLEAN_UP_TYPE and CLEAN_UP_VARIABLE) that are directly given to duplicity.
        • second, if and only if the REMOVE_INCREMENTALS_OLDER_THAN parameter in the config file is set, a cleanup of type remove-all-inc-of-but-n-full is done

      All backup and cleanup parameters and types are in the end managed by duplicity, so to check what really happens see duplicity’s documentation (http://duplicity.nongnu.org/duplicity.1.html)

    2. In your case you seem to be using the remove-older-than cleanup type.

      Duplicity’s documentation states :

      remove-older-than time
      Delete all backup sets older than the given time. Old backup sets will not be deleted if backup sets newer than time depend on them.

  9. And hi again.

    Is possible to use duplicity in reverse mode? From s3/ftp/ssh to local folder/file?

    1. I’m not sure what is exactly the use case you’re thinking about, but according to duplicity’s documentation if you swap the remote URL and the local path “Duplicity enters restore mode because the URL comes before the local directory.

      Note that I can’t help you more than citing duplicity’s documentation, duplicity-backup.sh being only a wrapper for duplicity, all the limitations of duplicity apply to the script…

  10. Hey there, I’m trying to use this script (very nice and easy to use, thanks for sharing!) as an anacron job. I’m using GPG keys for signing and encrypting the backup, but the script does not run correctly as root since it can’t see the user’s gpg-agent. If I try to run as root (sudo su; ./run-script), I get this output on stdout. Any idea how to work around this and keep using GPG? It works flawlessly if I run the cron job as myself instead of root.

    GPGError: GPG Failed, see log below:
    ===== Begin GnuPG log =====
    gpg: encrypted with RSA key, ID XXXXXXXX
    gpg: decryption failed: No secret key
    ===== End GnuPG log =====

    GPGError: GPG Failed, see log below:
    ===== Begin GnuPG log =====
    gpg: encrypted with RSA key, ID XXXXXXXX
    gpg: decryption failed: No secret key
    ===== End GnuPG log =====

    1. Hi Andrew,

      Well I have no idea what your script is actually doing (the “run-script”) so I have no clues here. I have no idea about your GPG keys setup either…

      It looks like you have a common issue that consist of letting “cron” scripts access the gpg-agent keyring… But I have no experience with gpg-agent at all… so I’m sorry not being really able to help here.

      If your problem is related to a bug in the duplicity-backup.sh script or if your problem can be solved by improving the script somehow I would be happy to hear about it and/or help to implement it :)

    1. Hi,

      The script supports anything that duplicity supports (it’s only a wrapper).
      I don’t know if duplicity support Google Drive though, never needed it before…

      1. Thanks, I will test here, if works, I post here and fork project to include google drive as example

  11. Hi!
    My script send three emails at one alert for three different adresess – one realy and two:

    --@7thodroid
    -f@7thodroid

    7thodroid – is hostname.

    My conf:

    HOSTNAME="7thOdroid"

    #EMAIL="real@mail.ru"
    EMAIL_TO="real@mail.ru"
    #EMAIL_FROM=
    #EMAIL_SUBJECT=""
    EMAIL_FAILURE_ONLY="yes"

    # command to use to send mail
    MAIL="mail"

    What I can do with it?
    As I understand it from this line of script, but I don’t know what to do with it:
    cat ${LOGFILE} | ${MAILCMD} -s """${EMAIL_SUBJECT}""" $EMAIL_FROM ${EMAIL_TO} -- -f ${EMAIL_FROM}

    1. Hmm. I just delete -- -f and all alright.
      New line:
      cat ${LOGFILE} | ${MAILCMD} -s “””${EMAIL_SUBJECT}””” $EMAIL_FROM ${EMAIL_TO} ${EMAIL_FROM}

      1. Hi, thanks for reporting your problem. What is your system as reported by the uname -a command?

        Could you also provide the result of this command dpkg -S $(readlink -e $(which mailx)) (if your system uses APT packages, otherwise I don’t know how to do the same with other package systems)

        1. Hi, same problem on debian,

          dpkg -S $(readlink -e $(which mailx))
          bsd-mailx: /usr/bin/bsd-mailx

  12. Would be nice if it has a feature to select storage type (Infrequent access, RRS etc)

Comments are closed.