How-to: FTP-Backup a Linux server with Duply

My Linux root server’s hosting price plan includes a 50 GB backup storage option. The backup server can only be reached using FTP, unsecured. Even though the backup server is only visible from within my server provider’s local network, I still don’t want to expose all my server settings, accounts and databases in clear text to a man in the middle. Or, I don’t want an untrustworthy subject with access to the backup server’s file system to be able to read my backup. Also, I don’t want to send my entire site to the backup storage every day as this would eat up my 50 GB within days.

This is where Duply (formerly known as FTPlicity) comes in handy. Duply claims to make your incremental encrypted backups on non-trusted spaces a child’s play. It manages backup job settings in profiles and allows to batch execute commands. It supports symmetric and asymmetric encryption using GPG.

Duply is not limited to FTP, you can also send your backups to a IMAP, WebDAV or ssh destination (and even more!). I haven’t tried the IMAP method but in theory you could send your encrypted backup to a free Google mail account as long as you don’t exceed the 7 GB provided by Gmail. However, in this sample, I’m using a FTP scenario with asymmetric encryption.

Installation & backup profile configuration

So let’s set up Duply from scratch and schedule incremental encrypted backups including the server’s /etc and /home directories and a dump of the MySQL databases.

  1. Install Duplicity. Duply is a wrapper for Duplicity. If you’re on a Debian based system simply type
    apt-get install duplicity
  2. Download Duply and put the duply script into /usr/bin. Make sure it has execute permissions for the user you plan to run your backups with (e.g. root).
  3. Create the directory /etc/duply. If it doesn’t exist, Duply will write the profile into your current user’s $HOME.
  4. Create a new profile. I’m gonna name it “offsite” with
    duply offsite create
  5. Make sure Duply created a conf file in /etc/duply/offsite
  6. Create a GPG key with
    gpg --gen-key

    Use the defaults for encryption method (DSA and Elgamal), key size (2048) and expiration (never), enter some personal information and then wisely choose a secure passphrase which you’re still able to remember in at least 10 years from now :-)
    You should see an output similar to this:

    gpg: key --> D4B41D3E <-- marked as ultimately trusted
    public and secret key created and signed.
  7. The public and private keys are now added to your GPG keystore in $HOME/.gnupg. To show a list of all your current keys in the GPG keystore just type
    gpg --list-keys
  8. Edit /etc/duply/offsite/conf and set the key identifier (in this sample it’s D4B41D3E, see GPG output above) and the passphrase you used to generate the keys like this:
    # gpg key data (for symmetric encryption comment out GPG_KEY)
    GPG_KEY='D4B41D3E'
    GPG_PW='I _could_ tell you BUT then I'd have 2 kill u'
  9. Set the destination server address and credentials. Duply is not limited to FTP but in this sample I’m using a FTP destination. Make sure the FTP user has write permission in the destination path specified.
    TARGET='ftp://ftpuser:ftppasswd@ftp.mybackup987.com/backupdir'
  10. Choose the source path of your backup. You might use something like /home but for my sample I’m using the root directory. As this would generate a lot of files that are not backup-worthy I’m using an exclusion list as well. To exclude directories I don’t want to backup I create a file named exclude in /etc/duply/offsite. The content for a Debian web server could look like this:
    /bin
    /boot
    /dev
    /lib
    /lost+found
    /media
    /mnt
    /opt
    /proc
    /sbin
    /srv
    /sys
    /tmp
    /usr
    /var
    /home/my-web-site/logs

    You may have to adjust the directories for your own needs. With this exclusion list I can safely set the source path in conf to:

    # base directory to backup
    SOURCE='/'

    Now, important directories like /etc and /home are automatically included.

  11. Duply provides hooks prior and after every backup to run custom commands. Just put these commands in files named pre and post in the /etc/duply/offsite directory. I’m using MySQL on the server so I want to include a SQL dump of all databases to be included in my backup. The MySQL dump has to be created before the actual backup runs, so I have to include this line in a file named pre in /etc/duply/offsite:
    /usr/bin/mysqldump --all-databases -u root -pmysqlpassword > /home/my-web-site/tmp/db.sql

    The database dump will be written to a directory that is included in my “offsite” Duply profile and therefore included in every incremental backup as a whole. I know this kind of a poor man’s database backup method because the database is still running while the dump is performed but this is still good enough for most non-mission-critical systems. Make sure the pre and post files have execute permission set or they won’t run.

  12. To run the first backup for my “offsite” profile enter
    /usr/bin/duply offsite backup

    Depending on the amount of data to backup and connection speed, this will take quite some time.

  13. To clean up obsolete backup sets monthly and to create daily incremental backups I use this crontab:
    27 4 1 * * root /usr/bin/duply offsite backup_verify_purge --force
    23 3 * * * root /usr/bin/duply offsite backup

To sum it up, my “offsite” Duply profile directory /etc/duply/offsite includes the following files:

drwx------ 2 root root 4096 2009-05-17 08:54 .
drwxr-xr-x 3 root root 4096 2010-01-01 10:01 ..
-rw------- 1 root root 2668 2009-05-17 08:50 conf
-rw------- 1 root root  316 2008-12-25 10:39 exclude
-rwx------ 1 root root   75 2008-03-23 09:31 pre
root@myserver:/etc/duply/offsite#

Remember, you can always add more Duply profiles. For instance, I’m using another profile to backup to my Mac at home so I always have another backup at hand just in case something really goes awry. I’m using a static DynDNS alias and the DynDNS Updater app to reach my Mac at home so I don’t have to look up the IP address every time I run this profile. You may have to configure the router for incoming FTP connections and if you’re using a software firewall, add rules for incoming FTP traffic and of course enable the FTP server on the Mac itself.

Restore

To list the current backup sets on your remote backup path use the following command:

duply offsite status

To restore the whole “offsite” backup to /tmp I’d use this command:

duply offsite restore /tmp/restore

To restore the lastest version of a file from my “offsite” backup to /tmp I’d use this command:

duply offsite fetch etc/myfile /tmp/restore

To restore the version from 4 days ago of a file from my “offsite” backup to /tmp I’d use this command:

duply offsite fetch etc/myfile /tmp/restore 4D

The combination of Duplicity and Duply is a very powerful solution for backups on non-trusted spaces. The setup is a piece of cake compared to other Linux based backup solutions.

Neat tricks

Wondering how much free space is left on your offsite FTP backup server without manually logging into that FTP account?

echo du . | lftp -u ftpuser,ftppass ftp.mybackup987.com | awk -v LIMIT="50" '$1~/[0-9]+/ {QUOTA=LIMIT*1024*1024; print (QUOTA-$1)/1024/1024 " GB backup space left"}'

Or use a cron job to write the free space into your system’s logging facility. Put these lines into an executable script file in the cron.daily directory:

#!/bin/sh
MAILTO=spamtrap@trick77.com
echo du . | lftp -u ftpuser,ftppass ftp.mybackup987.com | awk -v LIMIT="50" '$1~/[0-9]+/ {QUOTA=LIMIT*1024*1024; print (QUOTA-$1)/1024/1024 " GB backup space left"}' | logger

20 thoughts on “How-to: FTP-Backup a Linux server with Duply

  1. Thank you for the nice post.

    One correction however when it comes to restore:

    duply offsite restore /etc/myfile /tmp/restore will not work and you have to enter duply offsite fetch /etc/myfile /tmp/restore instead. If wished with an age like fetch []

    Great work!

  2. Another small mistake: The file or the folder to be restored should be without the leading /
    I did write it wrong in my own comment, sorry for that. The right syntax is:
    duply offsite fetch etc/myfile /tmp/restore

  3. Thanks for the great post! There’s not that much documentation about duply on the web, and this tutorial makes the process really easy!

    Now my backups are secured. Let’s hope I don’t lose the secret key ;)

  4. Hi there,

    First, thanks a lot for this great info. I installed duply after reading your post and I use it since a little while now. It works great but the other day I needed to restore a file and I faced a problem: there are spaces in the path pointing to the file I need.

    I tried these commands:

    duply offsite fetch path/containing spaces/pointing to/myfile /tmp/restore 4D

    duply offsite fetch “path/containing spaces/pointing to/myfile” /tmp/restore 4D

    duply offsite fetch path/containing\ spaces/pointing\ to/myfile /tmp/restore 4D

    But none of them worked. I ended up restoring the whole “path” part of my example (meaning I wrote the path up to but not including the first directory with a space in its name). Not really efficient…

    Do you know how to restore a file when the path contains one or more spaces?

    Thanks a lot!

  5. Thanks for your reply. I just tried and I still get the same error:
    [...] not found in archive, no files restored.
    13:47:51.000 Task ‘FETCH’ failed with exit code ’19′.

    No luck so far

    chris

  6. In case others are wondering, it was actually a bug in duply. I contacted the project admin on sourceforge and he fixed it (thanks ede!). The changes are included in the latest dev snapshot.

    Cheers,

    Christian

  7. Duply keeps asking me for the passphrase when running the backup. The correct passphrase is in the conf file, but still I need to input it!

    Any ideas?

  8. Just wondering,

    If I want to use my public key to sign (thus without having to use a password for the private key) how does one go about with that? I know how to do it in duplicity but I didn’t find the option yet in duply.

  9. Christos Prassas

    Hi,

    Firstly thatnk you for the tutorial!

    One think I want to do is to send email with each backup log file to my mail….
    How can I do that to the post file?

    Thank you

  10. Christos, probably the most straight forward thing to do is to install an MTA and cron will automatically send you the email after the backup.

    Cheers,
    Jan

  11. Hello i’m trying to restore a full backup that i made in a VM. Do you know how to do this? thanks in advance

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>