commit 08bdbcc1c21b5425007d0b142ac0010f9cd71578
parent fc4bee71d694b895f44671c6cabe88690f68d058
Author: lash <dev@holbrook.no>
Date: Sat, 1 Oct 2022 13:21:59 +0000
Add previously missing files
Diffstat:
6 files changed, 371 insertions(+), 0 deletions(-)
diff --git a/content/20220111_backup_rsync_duplicity.rst b/content/20220111_backup_rsync_duplicity.rst
@@ -0,0 +1,202 @@
+Combining duplicity and rsync
+#############################
+
+:date: 2022-01-15 16:57
+:category: Archiving
+:author: Louis Holbrook
+:tags: backup,rsync,duplicity,bash
+:slug: backup-rsync-duplicity
+:summary: An exercise in combining plain and encrypted backups on local and remote hosts
+:series: Organizing backups
+:seriesprefix: organizing-backups
+:seriespart: 1
+:lang: en
+:status: published
+
+
+There are two awesome, weathered tools out there that are all you really need for your personal backups. [1]_ One is the `rsync cli`_, the other is duplicity_.
+
+The former should need no introduction.
+
+The latter operates more like tar. But it still works over ssh like rsync. In fact, it's based on librsync_ which implements the `rsync protocol`_. The special sauce however is, of course, *encryption*.
+
+
+Backup categories
+=================
+
+Let's for the sake of argument say that our personal backups can be divided in three categories:
+
+
+Stuff that can be public
+------------------------
+
+Code snippets, git repositories, public data store states (e.g. blockchain ledgers), copies of OS packages and any other assets assets without redistribution issues.
+
+For this we will use rsync_.
+
+
+Sensitive stuff
+---------------
+
+Passwords, keys, contacts, calendars, contracts, invoices, task lists, databases, system configurations, application data.
+
+For this we will use duplicity_.
+
+Secret stuff
+------------
+
+Long-lived keys, password- and volume decryption keys, cryptocurrency keys and meta-information about the backups themselves.
+
+This will not be addressed now.
+
+
+Why not just one or the other?
+==============================
+
+Duplicity_ stores everything in an archive file format. That means that you must first authenticate, decrypt and unpack the archive in order to even browse the files inside.
+
+If there is no reason to keep the files from prying eyes, then it's much more practical to be able to browse the files where they lie, with the regular filesystem tools. In such a case, rsync_ will scratch your itch.
+
+For the **sensitive** and **secret stuff**, there would be no real need to use duplicity_ if you were only operating on your local host. You'd just use an encrypted volume [2]_ and rsync_ everything in there.
+
+But half the point here is to keep remote copies aswell as your local ones. You know, in case of fire, hardware-eating locust swarms or some totalitarian minions nabbing all your electronics. Unless "remote" here means some box hidden in some moated leisure castle of yours, you'll want to encrypt everything *before* you ship it off. And that's where duplicity_ comes in.
+
+
+Vive la difference
+==================
+
+Of course, it would be too much to hope for that duplicity_ and `rsync cli`_ have aligned the ways they parse their invocation parameters.
+
+Here are some examples [3]_ of how they do *not* match:
+
+
+local to local
+--------------
+
+.. code-block:: bash
+
+ $ rsync -a src/ /path/to/dst/
+
+ $ duplicity src/ file:///path/to/dst/
+
+
+local to remote, relative path
+------------------------------
+
+.. code-block:: bash
+
+ $ rsync -a src/ user@remotehost:path/to/dst/
+
+ $ duplicity src/ scp://user@remotehost/path/to/dst
+
+
+toggle dotfiles from current path
+---------------------------------
+
+.. code-block:: bash
+
+ # include only .foo/foo.txt given the current structure:
+ $ tree src/ -a
+ src/
+ ├── .bar
+ ├── baz
+ └── .foo
+ └── foo.txt
+
+ $ rsync --exclude=".b*" --include=".*/***" --exclude="*" ./ ../dst/
+
+ $ duplicity --exclude="./.b*" --include="./.*/***" --exclude="*" ./ file:///home/lash/tmp/dst/
+
+logging
+-------
+
+.. code-block:: bash
+
+ # spill the beans
+ $ rsync -vv ...
+
+ $ duplicity -v debug
+
+
+
+Batchin'
+========
+
+Since you will want to select up front which tool to use for which sensititivy category, you'll be writing the includes and excludes specifically for the tool anyway.
+
+So the only real issue with the above is the way remote host is specified.
+
+Let's say we choose to stick to the `rsync cli`_ host format. That means we need to make the following translations:
+
+.. list-table::
+ :widths: 50 50
+ :header-rows: 1
+
+ * - rsync
+ - duplicity
+ * - ``foo/bar``
+ - ``file://foo/bar``
+ * - ``/foo/bar``
+ - ``file:///foo/bar``
+ * - ``user@host:foo/bar``
+ - ``scp://user@host/foo/bar``
+ * - ``user@host:/foo/bar``
+ - ``scp://user@host//foo/bar``
+
+
+Expressed in ``bash`` that could look like this:
+
+.. include:: code/backup-rsync-duplicity/translate.sh
+ :code: bash
+
+
+Let's behave and test our code:
+
+.. include:: code/backup-rsync-duplicity/translate_test.sh
+ :code: bash
+
+.. code-block:: bash
+
+ # 0 == good!
+ $ BAK_TEST=1 bash remote.sh && echo $?
+ 0
+
+
+Now we can use the `rsync cli`_ path input, and use that same input to a batch of single backup steps, each which may use `rsync cli`_ or duplicity_
+
+.. code-block:: bash
+
+ to_duplicity_remote localhost:/foo/bar
+
+ rsync -avzP pub/ $remote_base:src/
+
+ duplicity -v info secret/ $remote_duplicity_base:secret/
+
+
+See also
+========
+
+* https://git.defalsify.org/rsync-duplicity-backups
+
+
+..
+
+ .. [1] Ok, I know, I assuming that you are using ``git`` in daily life, too.
+
+ .. [2] Provided, of course, that it's an encrypted volume that you don't keep unlocked all the time.
+
+ .. [3] Duplicity needs at a minimum a password for symmetric encryption, and will prompt for it unless it's set in the environment. Simply ``export PASSPHRASE=test`` for these examples to relieve you of the annoyance.
+
+..
+
+ .. _duplicity: https://duplicity.gitlab.io/duplicity-web/
+
+ .. _Duplicity: https://duplicity.gitlab.io/duplicity-web/
+
+ .. _`rsync cli`: https://rsync.samba.org/
+
+ .. _rsync: https://rsync.samba.org/
+
+ .. _librsync: http://librsync.sourcefrog.net/
+
+ .. _`rsync protocol`: https://rsync.samba.org/tech_report/
diff --git a/content/20220112_clortho.rst b/content/20220112_clortho.rst
@@ -0,0 +1,33 @@
+Clortho
+#######
+
+:date: 2022-01-12 15:03
+:modified: 2022-01-12 15:03
+:category: Code
+:author: Louis Holbrook
+:tags: crypto,hash,sha512,bash,cli
+:slug: clortho
+:summary: A key value store at your fingertips
+:lang: en
+:status: draft
+
+
+Ever since I started using the pass_ CLI as my password manager, I've found myself putting all sorts of stuff in there; usernames, email, urls, crypto addresses, api keys, you name it.
+
+It makes total sense that some of these items are in there. For example, I store the url to a service together with the password, usually accompanied by the username and the email used [1]_. No password recoveries needed.
+
+However, I've also started putting in things like crypto addresses, or even token smart conrtact addresses in there, it seems less of a good fit. One thing is that it spams the password directory. But another more sinister issues is that it's pretty clear for anyone reading the directory what items you are storing data for.
+
+
+Hiding the key
+==============
+
+So what if I want to store key/value pairs, and at the same time I want to hide what I am storing?
+
+..
+
+ .. [1] I use a different email for each service I sign up to, and for every other context I have to leave my email for something.
+
+..
+
+ .. _pass: https://www.passwordstore.org/
diff --git a/content/20220124_clique_openethereum.rst b/content/20220124_clique_openethereum.rst
@@ -0,0 +1,43 @@
+The clique extra data secret
+############################
+
+:date: 2022-01-24 20:05
+:category: code
+:author: Louis Holbrook
+:tags: rust,blockchain,openethereum,ethereum
+:slug: openethereum-clique-extradata
+:summary: Unlocking the extra data in the clique configuration
+:lang: en
+:status: draft
+
+
+..
+
+ extradata: 0x536172616675204b61726962752053616e6100000000000000000000000000005C5aB0D602EEF41f82B6fc087A24e61383589C398c6f7b75d90c3b32bdc9b4fcfbb4ad43c853446a2c0c9d3a9590a3349c8e45ce50446bd276cbb99c1f38132cd7e520d6be0ccba78acc59ab50a82f711ff7cb9d00
+
+ 32 bytes vanity
+ 20 bytes address
+ 65 bytes signature
+
+
+.. code-block:: rust
+
+ ethcore/src/engines/clique/mod.rs
+
+ // Protocol constants
+ /// Fixed number of extra-data prefix bytes reserved for signer vanity
+ pub const VANITY_LENGTH: usize = 32;
+ /// Fixed number of extra-data suffix bytes reserved for signer signature
+ pub const SIGNATURE_LENGTH: usize = 65;
+
+
+.. code-block:: rust
+
+ ethcore/src/engines/clique/util.rs
+
+ pub fn extract_signers(header: &Header) -> Result<BTreeSet<Address>, Error> {
+
+ ie data.len() <= VANITY_LENGTH + SIGNATURE_LENGTH {
+ /// Nonce value for DROP vote | Err(EngineError::CliqueCheckpointNoSigner)?
+ pub const NONCE_DROP_VOTE: H64 = H64([0; 8]); | }
+
diff --git a/content/code/backup-rsync-duplicity/translate.sh b/content/code/backup-rsync-duplicity/translate.sh
@@ -0,0 +1,65 @@
+to_duplicity_remote() {
+
+ # remote host is defined in rsync format
+ # ... and we will only support scp
+ # $remote_base is the path we want to parse
+ remote_duplicity_base=
+ remote_base=$1
+
+ # substring up until the first slash
+ s_firstslash=${remote_base%%/*}
+
+ # substring up until the first colon
+ s_firstcolon=${remote_base%%:*}
+
+ # string index of the first slash
+ i_firstslash=$((${#s_firstslash}))
+
+ # string index of the first colon
+ i_firstcolon=$((${#s_firstcolon}))
+
+ # if colon is before first slash that most likely means we have a remote host
+ # (Exception is if first directory of path happens to have ":" in it. Seriously, don't use ":" in filenames)
+ if [ "$i_firstcolon" -gt "0" ]; then
+
+ if [ "$i_firstcolon" -lt "$i_firstslash" ]; then
+
+ # pexpect addition due to lack of implicit private key fetch, without pexpect works only with key wo pwd
+ # https://serverfault.com/questions/982591/duplicity-backup-fails-private-key-file-is-encrypted
+ remote_duplicity_base="pexpect+scp://"
+
+ # trim url so that colon after hostname is removed
+ # (no support here for setting an alternate port number)
+ remote_duplicity_base=${remote_duplicity_base}${remote_base:0:$i_firstcolon}
+ remote_duplicity_base=${remote_duplicity_base}/${remote_base:(($i_firstcolon+1))}
+
+ # indicate that we have a remote host
+ remote=1
+ fi
+ fi
+
+ # If it's not a remote host, treat it as a file
+ if [ -z $remote_duplicity_base ]; then
+ remote_duplicity_base="file://${remote_base}"
+ fi
+}
+
+if [ ! -z "$BAK_TEST" ]; then
+ src=(/foo/bar foo/bar localhost:foo/bar localhost:/foo/bar)
+ res=(file:///foo/bar file://foo/bar pexpect+scp://localhost/foo/bar pexpect+scp://localhost//foo/bar)
+
+ i=0
+ for case_src in ${src[@]}; do
+ case_res=${res[$i]}
+
+ to_duplicity_remote $case_src
+ if [ "$remote_duplicity_base" != "$case_res" ]; then
+ >&2 echo "expected $case_res got $remote_duplicity_base from $case_src"
+ exit 1
+ elif [ "$remote_base" != "$case_src" ]; then
+ >&2 echo "$case_src got mangled into $remote_base"
+ exit 1
+ fi
+ i=$((i+1))
+ done
+fi
diff --git a/content/code/backup-rsync-duplicity/translate_test.sh b/content/code/backup-rsync-duplicity/translate_test.sh
@@ -0,0 +1,19 @@
+if [ ! -z "$BAK_TEST" ]; then
+ src=(/foo/bar foo/bar localhost:foo/bar localhost:/foo/bar)
+ res=(file:///foo/bar file://foo/bar pexpect+scp://localhost/foo/bar pexpect+scp://localhost//foo/bar)
+
+ i=0
+ for case_src in ${src[@]}; do
+ case_res=${res[$i]}
+
+ to_duplicity_remote $case_src
+ if [ "$remote_duplicity_base" != "$case_res" ]; then
+ >&2 echo "expected $case_res got $remote_duplicity_base from $case_src"
+ exit 1
+ elif [ "$remote_base" != "$case_src" ]; then
+ >&2 echo "$case_src got mangled into $remote_base"
+ exit 1
+ fi
+ i=$((i+1))
+ done
+fi
diff --git a/content/code/openethereum-clique-extradata/extra.sh b/content/code/openethereum-clique-extradata/extra.sh
@@ -0,0 +1,9 @@
+VANITY_STRING=${VANITY_STRING:-"Kitabu Sarafu Karibu Sana"}
+echo -n "$VANITY_STRING" > .vanity
+truncate .vanity -s 32 && \
+hexdump -v -n 32 -e '1/1 "%02x"' .vanity > .extra
+
+WALLET_PASSPHRASE=test eth-keyfile -0 > keyfile_validator.json
+WALLET_PASSPHRASE=test eth-sign-msg -0 -f keyfile_validator.json `cat .extra` > .sig
+WALLET_PASSPHRASE=test eth-keyfile -0 -d keyfile_validator.json >> .extra
+cat .sig >> .extra
