Duplicity – secure incremental backup

Overview:

Duplicity is a tool to create GPG-encrypted (this way you can store your backups at remote servers without having to worry about who has access to your data) incremental backups to remote servers. Its a quite handy and secure method.

Installation

The steps to install duplicity is as follows

wget http://code.launchpad.net/duplicity/trunk/0.6.02/+download/duplicity-0.6.02.tar.gz

tar -xvf duplicity-0.6.02.tar.gz

cd duplicity-0.6.02.tar.gz

python setup.py install

If you come across any  librsync.so errors . You can resolve it by using the following steps

wget http://sourceforge.net/projects/librsync/files/librsync/0.9.7/librsync-0.9.7.tar.gz/download

tar -xzvf librsync-0.9.7.tar.gz
cd librsync-0.9.7

./configure

make

make install

Now we got duplicity installed 

Create a GPG key

In order to be able to encrypt your backups, you have to create a GPG key.  Open a second shell and run the following command (this generates some “randomness” on your system, which will be useful to create a secure key). Kill the command with CTRL+C when you are done with key generation.

while /bin/true; do cat /var/log/messages > ~/temp.txt; sleep 1; done;

On your other shell, create your GPG key. Be sure to use a secure passphrase and to copy/write down the key ID which is displayed at the end of the generation process (we’ll need it for ftplicity). Also, make sure to backup the key to a secure location outside your server. As all your backups will be encrypted, they will be worthless if your server crashes and you lose the key.

gpg –gen-key

Default options should be fine. This will create your key in ~/.gnupg/. Once its done you can verify the existence of your key using the command

gpg –list-keys

The next step is to prepare an off-site location to receive the backup files.
The software supports different protocols like FTP,RSYNC,SCpP.
I am restricting myself with SCP here

Simple unEncrypted Backup over SCP

Setup ssh keys on the backup server allowing root to seamlessly login to the backup server.

duplicity /home/me scp://uname@other.host/usr/backup

If the above command is run repeatedly, the first session will be a full backup, and subsequent ones will be incremental.

The full option can be used to force a full backup. The next command also excludes the /tmp directory.

duplicity full –exclude /tmp /home/me scp://uname@other.host/usr/backup

Basic restore command—restore the /home/me directory backed up with scp above to directory restored_dir:

duplicity scp://uid@other.host//usr/backup restored_dir

To enable verbose mode use the option -v
Specify verbosity level (0 is total silent, 4 is the default, and 9 is noisiest)
The command would look like

duplicity  -v5 /home/me scp://uid@other.host/some_dir

Encrypted Backup over SCP

Here we use the GPG key generated earlier
The format would be look like this

duplicity
    --encrypt-key=${GPG_KEY} \
    --sign-key=${GPG_KEY} \
    --include=/boot \
    --include=/etc \
    --include=/home \
    --include=/root \
    --include=/var/lib/mysql \
    --exclude=/** \
    ${SOURCE} ${DEST}

Needless to say the include and exclude options are for specifying the backup criteria.

duplicity –encrypt-key=”FFF7730B” –sign-key=”FFF7730B” -v5 /home/me scp://uid@other.host/some_dir

you will be asked for a GnuPG passphrase. You can type in any password you like; this has to be done everytime you run duplicity. The backup will be encrypted with the help of GnuPG. Permissions and ownerships will be preserved in the backup.

To avoid this issue , you can simply set the passphrase as  environment variable using the command

export PASSPHRASE=gpgpassphrase

Backup Format & Explanation

Once it is executed ,  you can see the backup in the server and it would look like the following way

duplicity-full-signatures.2005-11-27T01:00:01-05:00.sigtar.gpg
duplicity-full.2005-11-27T01:00:01-05:00.manifest.gpg
duplicity-full.2005-11-27T01:00:01-05:00.vol1.difftar.gpg
duplicity-full.2005-11-27T01:00:01-05:00.vol2.difftar.gpg

The signatures file contains, signatures of each file that is backed up so that Duplicity can figure out which part of a file has changed. With that information it can upload only the missing part to complete a new backup set.

The manifest file contains a listing of all the files in the backup set and a SHA1 hash of each file, probably so Duplicity can tell very quickly whether a file has been changed or not since the last backup.

The volume files (vol1 and vol2) contain the actual file data. It appears that Duplicity volumes are at most 5MB. That’s helpful during restores so the entire backup set does’t not need to be downloaded to retrieve a single file. Duplicity will only download the volume containing that file.

Common Options:

Depending on the parameters and order of the parameters in the duplicity command, different functions can be performed. For example, an archive can be verified to see if a complete backup was made and what files, if any, have changed since the last backup.

duplicity verify [options] source_url target_directory

duplicity verify -v4 scp://user@bakuphost/etc /etc

Listfiles

It’s sometimes handy to check which files are in the latest backup set.

duplicity list-current-files [options] target_url

The command would look like

duplicity list-current-files –archive-dir /root/test/ scp://user@backupserver/some_dir

Restore

The main purpose of backup is to restore data which has been lost.  The following is the common format for restoring the data from the latest backup

duplicity scp://uid@other.host/some_dir  /home/me

Duplicity enters restore mode because the URL comes before the local directory. If we wanted to restore just the file “Mail/article” in /home/me as it was three days ago into /home/me/restored_file:

duplicity -t 3D –file-to-restore Mail/article scp://uid@other.host/some_dir /home/me/restored_file

The following command compares the files we backed up, so see what has changed since then:

duplicity verify scp://uid@other.host/some_dir /home/me

The following command can be used to retrieve a  single file from backup

duplicity –encrypt-key “” –sign-key “” –file-to-restore home/sburke/file.txt  scp://user@server.com/backup/  /var/tmp/file.txt

1. The path to the file that is to be restored is relative to the directory on which the backup set is based. So in the command above,  home/sburke/file.txt plus the directory on which we based our backup (/backup) equals /backup/home/sburke/file.txt/. It would not work to put /backup/home/sburke/file.txt as the source path because the backup will not recognize /backup as a valid path. The last portion in the above command is the location where the file will be restored.

To delete old backups, we can use the following  command

 duplicity --full --remove-older-than 1Y /media/data/backup scp://uid@server/personal

To automate the tasks, you can write a shell script