manbytesgnu_site

20220111_backup_rsync_duplicity.rst (6064B)

---

 Combining duplicity and rsync
 #############################
 
 :date: 2022-01-15 16:57
 :category: Archiving
 :author: Louis Holbrook
 :tags: backup,rsync,duplicity,bash
 :slug: backup-rsync-duplicity
 :summary: An exercise in combining plain and encrypted backups on local and remote hosts
 :series: Organizing backups
 :seriesprefix: organizing-backups
 :seriespart: 1
 :lang: en
 :status: published
 
 
 There are two awesome, weathered tools out there that are all you really need for your personal backups. [1]_ One is the `rsync cli`_, the other is duplicity_.
 
 The former should need no introduction.
 
 The latter operates more like tar. But it still works over ssh like rsync. In fact, it's based on librsync_ which implements the `rsync protocol`_. The special sauce however is, of course, *encryption*.
 
 
 Backup categories
 =================
 
 Let's for the sake of argument say that our personal backups can be divided in three categories:
 
 
 Stuff that can be public
 ------------------------
 
 Code snippets, git repositories, public data store states (e.g. blockchain ledgers), copies of OS packages and any other assets assets without redistribution issues.
 
 For this we will use rsync_.
 
 
 Sensitive stuff
 ---------------
 
 Passwords, keys, contacts, calendars, contracts, invoices, task lists, databases, system configurations, application data.
 
 For this we will use duplicity_.
 
 Secret stuff
 ------------
 
 Long-lived keys, password- and volume decryption keys, cryptocurrency keys and meta-information about the backups themselves.
 
 This will not be addressed now.
 
 
 Why not just one or the other?
 ==============================
 
 Duplicity_ stores everything in an archive file format. That means that you must first authenticate, decrypt and unpack the archive in order to even browse the files inside.
 
 If there is no reason to keep the files from prying eyes, then it's much more practical to be able to browse the files where they lie, with the regular filesystem tools. In such a case, rsync_ will scratch your itch.
 
 For the **sensitive** and **secret stuff**, there would be no real need to use duplicity_ if you were only operating on your local host. You'd just use an encrypted volume [2]_ and rsync_ everything in there.
 
 But half the point here is to keep remote copies aswell as your local ones. You know, in case of fire, hardware-eating locust swarms or some totalitarian minions nabbing all your electronics. Unless "remote" here means some box hidden in some moated leisure castle of yours, you'll want to encrypt everything *before* you ship it off. And that's where duplicity_ comes in.
 
 
 Vive la difference
 ==================
 
 Of course, it would be too much to hope for that duplicity_ and `rsync cli`_ have aligned the ways they parse their invocation parameters.
 
 Here are some examples [3]_ of how they do *not* match:
 
 
 local to local
 --------------
 
 .. code-block:: bash
 
         $ rsync -a src/ /path/to/dst/
 
         $ duplicity src/ file:///path/to/dst/
 
 
 local to remote, relative path
 ------------------------------
 
 .. code-block:: bash
 
         $ rsync -a src/ user@remotehost:path/to/dst/
 
         $ duplicity src/ scp://user@remotehost/path/to/dst
 
 
 toggle dotfiles from current path
 ---------------------------------
 
 .. code-block:: bash
 
         # include only .foo/foo.txt given the current structure:
         $ tree src/ -a
         src/
         ├── .bar
         ├── baz
         └── .foo
             └── foo.txt
         
         $ rsync --exclude=".b*" --include=".*/***" --exclude="*" ./ ../dst/
 
         $ duplicity --exclude="./.b*" --include="./.*/***" --exclude="*" ./ file:///home/lash/tmp/dst/
 
 logging
 -------
 
 .. code-block:: bash
 
         # spill the beans
         $ rsync -vv ...
 
         $ duplicity -v debug
 
 
 
 Batchin'
 ========
 
 Since you will want to select up front which tool to use for which sensititivy category, you'll be writing the includes and excludes specifically for the tool anyway.
 
 So the only real issue with the above is the way remote host is specified.
 
 Let's say we choose to stick to the `rsync cli`_ host format. That means we need to make the following translations:
 
 .. list-table::
         :widths: 50 50
         :header-rows: 1
 
         * - rsync
           - duplicity
         * - ``foo/bar``
           - ``file://foo/bar``
         * - ``/foo/bar``
           - ``file:///foo/bar``
         * - ``user@host:foo/bar``
           - ``scp://user@host/foo/bar``
         * - ``user@host:/foo/bar``
           - ``scp://user@host//foo/bar``
 
 
 Expressed in ``bash`` that could look like this:
 
 .. include:: code/backup-rsync-duplicity/translate.sh
         :code: bash
 
 
 Let's behave and test our code:
 
 .. include:: code/backup-rsync-duplicity/translate_test.sh
         :code: bash
 
 .. code-block:: bash
 
         # 0 == good!
         $ BAK_TEST=1 bash remote.sh && echo $?
         0
 
 
 Now we can use the `rsync cli`_ path input, and use that same input to a batch of single backup steps, each which may use `rsync cli`_ or duplicity_
 
 .. code-block:: bash
 
         to_duplicity_remote localhost:/foo/bar
 
         rsync -avzP pub/ $remote_base:src/
 
         duplicity -v info secret/ $remote_duplicity_base:secret/
 
 
 See also
 ========
 
 * https://git.defalsify.org/rsync-duplicity-backups
 
 
 ..
 
         .. [1] Ok, I know, I assuming that you are using ``git`` in daily life, too.
 
         .. [2] Provided, of course, that it's an encrypted volume that you don't keep unlocked all the time.
 
         .. [3] Duplicity needs at a minimum a password for symmetric encryption, and will prompt for it unless it's set in the environment. Simply ``export PASSPHRASE=test`` for these examples to relieve you of the annoyance.
 
 ..
 
         .. _duplicity: https://duplicity.gitlab.io/duplicity-web/
 
         .. _Duplicity: https://duplicity.gitlab.io/duplicity-web/
 
         .. _`rsync cli`: https://rsync.samba.org/
 
         .. _rsync: https://rsync.samba.org/
 
         .. _librsync: http://librsync.sourcefrog.net/
 
         .. _`rsync protocol`: https://rsync.samba.org/tech_report/