commit 3b75de040cc623fe6a53fa134068e92720918860 parent 2f052574447f02fbf6e9a7810b813b67847f8b5b Author: lash <dev@holbrook.no> Date: Wed, 5 Oct 2022 14:45:29 +0000 Add wala post Diffstat:
| A | content/20221005_wala.rst | | | 179 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
1 file changed, 179 insertions(+), 0 deletions(-)
diff --git a/content/20221005_wala.rst b/content/20221005_wala.rst @@ -0,0 +1,179 @@ +Introducing Wala +################ + +:date: 2022-10-05 14:39 +:modified: 2022-10-05 14:39 +:category: Code +:author: Louis Holbrook +:tags: wala,pgp,crypto,storage,http,rust +:slug: wala +:summary: A simple HTTP content-addressed storage with cryptographic identity pointers. +:lang: en +:status: published + + +This little project is heavily inspired by the "Single-Owner Chunk" (previously "Mutable Resource") concept from the `Swarm project <https://ethswarm.org>`_ [1]_. + +Dubbed `wala <https://git.defalsify.org/wala>`_ [2]_, the rust application provides two features: + +1. Store any uploaded data under the SHA256 digest of the content itself. We call this "content addressed" storage. + +2. Create a link alias to the content using a keyword and a cryptographic identity. + +First, let's briefly outline what the first item on the list is. + + +Curl up and digest +================== + +Say an instance of wala is listnening on :code:`localhost:8000`. + +A simple demonstration of the content addressed storage could be: + +.. code:: console + + $ curl -X PUT http://localhost:8000 --data "foo" + 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae + $ curl -X GET http://localhost:8000/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae + foo + $ echo -n "foo" | sha256sum + 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae - + +In short, anything uploaded to the endpoint will be stored under and can be retrieved by its hash. + +That also means that if the content changes, so does the location. + + +Get to the pointer +================== + +Now let's say you want a permanent link to content that is changing. + +If you are the only user of the instance, this would be straightforward thing. We could just use a keyword in the url. + +But if you are not, then authentication is needed to make sure that one user doesn't overwrite the another user's permanent link. + +This is usually accomplished with a username and password stored and controlled server-side. + +But we can also do this by simply using public key cryptography instead. + + +Claiming what is yours +---------------------- + +Since I have a weakness for PGP, the first implementation of authentication in wala has been implemented using the `"sequoia openpgp" <https://sequoia-pgp.org>`_ library. + +The procedure is straightforward enough: + +1. Sign the content with your pgp key. +2. Add an `Authorization` header [3]_ with your public key and signature. +3. Upload the content with an arbitrary keyword. [4]_ + +If the signature over the content matches the public key, then a symbolic link to the content will be created on the server. That symbolic link will be a digest of the *key fingerprint* and the *keyword*. + +The content can then be accessed *both* using the content address *and* the symbolic link. + +In wala, the symbolic link is referred to as a *mutable reference*. We will use this term from now on. + + +It ain't pretty, but it works +----------------------------- + +Let's demonstrate creating such an entry in wala using some commonly available tools. [5]_ + +We will use a fictional gnupg private key with the fingerprint :code:`F3FAF668E82EF5124D5187BAEF26F4682343F692` and the keyword :code:`foo` to create the mutable reference. + +.. code:: console + + $ sig=`echo -n "bar" | gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 -w 0` + $ pub=`gpg --export F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 - w 0` + $ curl -v -X PUT http://localhost:8000/foo -H "Authorization: PUBSIG:$pub:$sig" --data bar + * Trying 127.0.0.1:8000... + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 8000 (#0) + > PUT /foo HTTP/1.1 + > Host: localhost:8000 + > User-Agent: curl/7.85.0 + > Accept: */* + > Authorization: PUBSIG pgp:mQGNBF+hSOgBDACpkPQEjADjnQtjmAsdPYpx5N+OMJBYj1DAoIYsDtV6vbcBJQt94Om3xl7RBhv9m2oLgzPsiRwjCEFRWyNSu0BUp5CFjcXfm0S4K2egx4erFnTnSSC9S6tmVNrVNEXvScE6sKAnmJ7JNX1ExJuEiWPbUDRWJ1hoI9+AR+8EONeJRLo/j0Np+S4IFDn0PsxdT+SB0GY0z2cEgjvjoPr4lW9IAb8Ft9TDYp+mOzejn1Fg7CuIrlBRSAv+sj7bVQw15dh1SpbwtS5xxubCa8ExEGI4ByXmeXdR0KZJ+EA5ksO0iSsQ/6ipSOdSg+i0niOClFNm1P/OhbUsYAxCUfiX654FMn2zoxVBEjJ3e7l0pH7ktodaxEctPofQLBA9LSDUIejqJsU0npw/DHDD2uvxG+/A6lgV9L8ETlvgp8RzeOCf2bHuiKYYz87txvkFwsXgU1+TZxbk+mtCBbngsVPLNarY/KGkVJL+yhcHRD0Pl4wXUd6auQuY6vQ9AuKiCT1We2sAEQEAAbQeTWVyIE1hbiA8bWVybWFuQGdyZXlza3VsbC5jb20+iQHUBBMBCAA+FiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAl+hSOgCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ7yb0aCND9pLwiwwAhFJbAyUK05TJKfDz81757N472STtB8sfr0auwmRr8Zs1utHRVM0b/jkjTuo4uJNr7YVVKTKgE7+rJ+pwhm3wlTQ44LVLjByWAi/7NWg3E9b2elm+qkfgm/RfFt3vkuOxGSyZyIFFh+/twv6iABPvr6w7MZwrFaS0UP3g1VGa5TFqg6KNxod9H/gPLxv45lutXf3VvBZTJpr1pxn7aLHlFzEyIgNZbP/N1QF44GSrN/k0DfL631sZjauUXaZXbi5xGsKKCYwJ1g3q587pi6mTdTV3n0hKgVuipO8hGy5++YeOv+hXsCxDwyZ+Shv+qavd/SapxYgCdEueuwONIFfsIsWCd3SCcjKXicTTEFMu8nvBmf7xuo2hv6vEOxoijlXV+4LkGrskdB8ZMg8PywEx6DLmDokgnAhTLrTc1ShbkOtQ3yNjjyFK7BDpqobsJal6d8SpbhccUJLepaSmsk0CgJsTjhAl6EwX0EYgTo3kP5fScqrbD8VwQaT8CcE4rCV4uQGNBF+hSOgBDADHtpTT1k4x+6FN5OeURpKAaIsoPHghkJ2lb6yWmESCa+DaR6GXAKlbd0L9UMcXLqnaCn4SpZvbf8hP4fJRgWdRl5uVN/rmyVbZLUVjM8NcVdFRIrTsNyu4mLBmydc3iA/90sCTEOj9e7DSvxLmmLFjpwM5xXLd6z0l6+9G+woNmARXVS3V/RryFntyKC3ATCqVlJoQBG45Tj2gMIunpadTJXWmdioooeGW3sLeUv5MM98mSB4SjKRlJqGPNjx5lO6MmJbZeXZ/L/aO6EsXUQD2h82Wphll4rpGYWPiHTCYqZYiqNYr6E3xUpzcvWVp3uCYVJWP6Ds117p7BoyKVz00yxC9ledF3eppktZWqFVowCMihQE3676L3DDTZsnJf1/8xKUh5U2Mj3lBvjlvCECKi00qo8b1mn/OklQjJ5T4WzTrH6X+/zpez8ZkmtcOayHdUKD/64roZ9dXbXG/hp5A+UWj8oSVYKg2QNAwAnZ+aiZ2KVRE/Y61DCgFg6Ccx/cAEQEAAYkBvAQYAQgAJhYhBPP69mjoLvUSTVGHuu8m9GgjQ/aSBQJfoUjoAhsMBQkDwmcAAAoJEO8m9GgjQ/aSIPcL/3jqL2A2SmC+s0BO4vMPEfCpa2gZ/vo1azzjUieZu5WhIxb5ik0V6T75EW5F0OeZj9qXI06gW+IM8+C6ImUgaR3l47UjBiBPq+uKO9QuT/nOtbSs2dXoTNCLMQN7MlrdUBix+lnqZZGSDgh6n/uVyAYw8Sh4c3/3thHUiR7xzVKGxAKDT8LoVjhHshTzYuQq8MqlfvwVI4eESLaryQ+Y+j5+VLDzSLgPAnnIqF/ui2JQjefJxm/VLoYNaPAGdqoz/u/R0Tmz94bZUfLjgQaDoUpnxYywK2JGlf3mPZ3PNWjxJzuQTF5Ge5bz/TylnRYIyBT7KD7oaKHO62fhDbYPJ4f94iZN4B6nnTAeP34zFDlkUbX4AHudXU7bvxT5OUk9x9c2tj7xwxQHaEhq2+JsYW0EVw27RLhbymnBfLjVVUktNF0nQGvU2TEocw4pr2ZkDHQkSnlbNa4kujlL7VzbpnEgyOmi5er9GaIuVSVADovBu+pz/Ov1y/3jUe8hZ/KleQ==:iQGzBAABCAAdFiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAmM9guIACgkQ7yb0aCND9pIEdwwAiLLqFlrKu0UsQebfuUP07cvGbYy9LfbCMsQj/3/pG/zl7q2mSl2YdXOalbaYD2uyGU/sm7J/+qQZXGyIjmDA7F53sNVAXTuYnrcKmYIzAmzW4lUAzfWA7yL55MtbR/eNUE1rqp/Gu/ejj1OedLyxi+tGFcXUHU0q8EnjQnzfHCJVzOa3PGMIX10NiXPjrF2pafAyE7q2ogwkKZdjJi+8tyAw0tviu4CRGOVlsNZlF+yxePZh55XdRZLCEt4n6mnJrccu0C22rM9R2dEReqGLAj8t/WhACI+UyNXtL+hICnu9y6wjk4spoMr0s0pqTQ76SMwfmRFzk11uZ+ge846hArcUxE27+AeBf9Q1IwT5Ypsc0Efm9ZPoJvA2ggcJv1Yyb58Ggfmd02xPW4EQ8MOEMLA/ZoAhOm3t3wATPNFG1ucm/o+NFNDpF7HNby+Savqv2NrbNwDMlWvFzRmER2+AIO0CIG2HVJScMEn7UkjF8jIm+ba3BIAXbz2FUZ3dytFF + > Content-Length: 3 + > Content-Type: application/x-www-form-urlencoded + > + } [3 bytes data] + * Mark bundle as not supporting multiuse + < HTTP/1.1 200 OK + < Server: tiny-http (Rust) + < Date: Wed, 5 Oct 2022 13:15:37 GMT + < Content-Type: text/plain; charset=UTF-8 + < Access-Control-Allow-Origin: * + < Access-Control-Allow-Methods: OPTIONS, PUT, GET + < Access-Control-Allow-Headers: Content-Type,Authorization,X-Filename + < Content-Length: 64 + < + { [64 bytes data] + 100 67 100 64 100 3 3548 166 --:--:-- --:--:-- --:--:-- 3722100 67 100 64 100 3 3534 165 --:--:-- --:--:-- --:--:-- 3722 + * Connection #0 to host localhost left intact + 32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 + $ curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 + bar + $ echo -n bar | sha256sum + fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9 + $ curl -X GET http://localhost:8000/fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9 + bar + +Now, by changing the data and signature, the data under the mutable resource changes: + +.. code:: console + + $ sig=`echo -n "xyzzy" | gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 -w 0` + $ curl -X PUT http://localhost:8000/foo -H "Authorization: PUBSIG:$pub:$sig" --data xyzzy + 32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 + $ curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 + xyzzy + + +Building your identity +====================== + +We can also recreate the symbolic link hash using local tools: + +.. code:: console + + $ t=`mktemp` + $ d=`echo -n foo | sha256sum | awk '{ printf "%s",$1; }' | sed -e 's/../\\\\x&/g'` + $ echo -ne $d > $t + $ k=`echo -n F3FAF668E82EF5124D5187BAEF26F4682343F692 | sed -e 's/../\\\\x&/g'` + $ echo -ne $k >> $t + $ cat $t | sha256sum + 32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 - + +In other words, the mutable reference is constructed using the following recipe. + +1. Calculate binary value :code:`R` of the SHA256 digest of the keyword (:code:`sha256(foo) -> 0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AE`) +2. Calculate binary value :code:`K` of the key fingerprint (:code:`0xF3FAF668E82EF5124D5187BAEF26F4682343F692`). +3. Calculate binary value of SHA256 digest of :code:`R | K` (:code:`sha256(0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AEF3FAF668E82EF5124D5187BAEF26F4682343F692) -> 0x32EA47ECC5A3EE2576AAB00C2F30EAABC2592D56E19F5C82FE4F7CF5874632B2`) + +That means that anyone who knows the keyword and the public key of the uploader can calculate the mutable reference themselves, and retrieve the data behind it. + + + +.. + + .. [1] Refer to `section 4.3 of The Book of Swarm <https://www.ethswarm.org/The-Book-of-Swarm.pdf>`_ for a description of "feeds" and "single owner chunks." Swarm uses signatures to allow propagation of the data in the network. :code:`wala` similarly uses signatures to accept update of resources "owned" by the holder of the private keys of an identity. + +.. + + .. [2] At the time of writing :code:`wala` is at version :code:`0.1.1` + +.. + + .. [3] A custom authorization scheme :codE:`PUBSIG` has been invented for this purpose. The format is :code:`Authorization: PUBSIG pgp:key-fingerprint-in-base64>:<signature-in-base64>`. Currently this is strictly just authentication, as there is no feature in wala (yet) to use access control lists to determine which public keys to allow PUTs from. + +.. + + .. [4] If you do not specify a keyword, the keyword value in the mutable resource reference will be the sha256 hash of an empty value (:code:`0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`) + +.. + + .. [5] :code:`wala` comes with a binary tool :code:`wala_send` that lets you specify a key fingerprint in your default pgp keyring and a keyword as arguments when you upload data. A lot less messy, but a lot less educational, too.