20221005_wala.rst (11238B)
1 Introducing Wala 2 ################ 3 4 :date: 2022-10-05 14:39 5 :modified: 2022-10-05 14:39 6 :category: Code 7 :author: Louis Holbrook 8 :tags: wala,pgp,crypto,storage,http,rust 9 :slug: wala 10 :summary: A simple HTTP content-addressed storage with cryptographic identity pointers. 11 :lang: en 12 :status: published 13 14 15 This little project is heavily inspired by the "Single-Owner Chunk" (previously "Mutable Resource") concept from the `Swarm project <https://ethswarm.org>`_ [1]_. 16 17 Dubbed `wala <https://git.defalsify.org/wala>`_ [2]_, the rust application provides two features: 18 19 1. Store any uploaded data under the SHA256 digest of the content itself. We call this "content addressed" storage. 20 21 2. Create a link alias to the content using a keyword and a cryptographic identity. 22 23 First, let's briefly outline what the first item on the list is. 24 25 26 Curl up and digest 27 ================== 28 29 Say an instance of wala is listnening on :code:`localhost:8000`. 30 31 A simple demonstration of the content addressed storage could be: 32 33 .. code:: console 34 35 $ curl -X PUT http://localhost:8000 --data "foo" 36 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae 37 $ curl -X GET http://localhost:8000/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae 38 foo 39 $ echo -n "foo" | sha256sum 40 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae - 41 42 In short, anything uploaded to the endpoint will be stored under and can be retrieved by its hash. 43 44 That also means that if the content changes, so does the location. 45 46 47 Get to the pointer 48 ================== 49 50 Now let's say you want a permanent link to content that is changing. 51 52 If you are the only user of the instance, this would be straightforward thing. We could just use a keyword in the url. 53 54 But if you are not, then authentication is needed to make sure that one user doesn't overwrite the another user's permanent link. 55 56 This is usually accomplished with a username and password stored and controlled server-side. 57 58 But we can also do this by simply using public key cryptography instead. 59 60 61 Claiming what is yours 62 ---------------------- 63 64 Since I have a weakness for PGP, the first implementation of authentication in wala has been implemented using the `"sequoia openpgp" <https://sequoia-pgp.org>`_ library. 65 66 The procedure is straightforward enough: 67 68 1. Sign the content with your pgp key. 69 2. Add an `Authorization` header [3]_ with your public key and signature. 70 3. Upload the content with an arbitrary keyword. [4]_ 71 72 If the signature over the content matches the public key, then a symbolic link to the content will be created on the server. That symbolic link will be a digest of the *key fingerprint* and the *keyword*. 73 74 The content can then be accessed *both* using the content address *and* the symbolic link. 75 76 In wala, the symbolic link is referred to as a *mutable reference*. We will use this term from now on. 77 78 79 It ain't pretty, but it works 80 ----------------------------- 81 82 Let's demonstrate creating such an entry in wala using some commonly available tools. [5]_ 83 84 We will use a fictional gnupg private key with the fingerprint :code:`F3FAF668E82EF5124D5187BAEF26F4682343F692` and the keyword :code:`foo` to create the mutable reference. 85 86 .. code:: console 87 88 $ sig=`echo -n "bar" | gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 -w 0` 89 $ pub=`gpg --export F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 - w 0` 90 $ curl -v -X PUT http://localhost:8000/foo -H "Authorization: PUBSIG:$pub:$sig" --data bar 91 * Trying 127.0.0.1:8000... 92 % Total % Received % Xferd Average Speed Time Time Time Current 93 Dload Upload Total Spent Left Speed 94 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 8000 (#0) 95 > PUT /foo HTTP/1.1 96 > Host: localhost:8000 97 > User-Agent: curl/7.85.0 98 > Accept: */* 99 > Authorization: PUBSIG pgp:mQGNBF+hSOgBDACpkPQEjADjnQtjmAsdPYpx5N+OMJBYj1DAoIYsDtV6vbcBJQt94Om3xl7RBhv9m2oLgzPsiRwjCEFRWyNSu0BUp5CFjcXfm0S4K2egx4erFnTnSSC9S6tmVNrVNEXvScE6sKAnmJ7JNX1ExJuEiWPbUDRWJ1hoI9+AR+8EONeJRLo/j0Np+S4IFDn0PsxdT+SB0GY0z2cEgjvjoPr4lW9IAb8Ft9TDYp+mOzejn1Fg7CuIrlBRSAv+sj7bVQw15dh1SpbwtS5xxubCa8ExEGI4ByXmeXdR0KZJ+EA5ksO0iSsQ/6ipSOdSg+i0niOClFNm1P/OhbUsYAxCUfiX654FMn2zoxVBEjJ3e7l0pH7ktodaxEctPofQLBA9LSDUIejqJsU0npw/DHDD2uvxG+/A6lgV9L8ETlvgp8RzeOCf2bHuiKYYz87txvkFwsXgU1+TZxbk+mtCBbngsVPLNarY/KGkVJL+yhcHRD0Pl4wXUd6auQuY6vQ9AuKiCT1We2sAEQEAAbQeTWVyIE1hbiA8bWVybWFuQGdyZXlza3VsbC5jb20+iQHUBBMBCAA+FiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAl+hSOgCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ7yb0aCND9pLwiwwAhFJbAyUK05TJKfDz81757N472STtB8sfr0auwmRr8Zs1utHRVM0b/jkjTuo4uJNr7YVVKTKgE7+rJ+pwhm3wlTQ44LVLjByWAi/7NWg3E9b2elm+qkfgm/RfFt3vkuOxGSyZyIFFh+/twv6iABPvr6w7MZwrFaS0UP3g1VGa5TFqg6KNxod9H/gPLxv45lutXf3VvBZTJpr1pxn7aLHlFzEyIgNZbP/N1QF44GSrN/k0DfL631sZjauUXaZXbi5xGsKKCYwJ1g3q587pi6mTdTV3n0hKgVuipO8hGy5++YeOv+hXsCxDwyZ+Shv+qavd/SapxYgCdEueuwONIFfsIsWCd3SCcjKXicTTEFMu8nvBmf7xuo2hv6vEOxoijlXV+4LkGrskdB8ZMg8PywEx6DLmDokgnAhTLrTc1ShbkOtQ3yNjjyFK7BDpqobsJal6d8SpbhccUJLepaSmsk0CgJsTjhAl6EwX0EYgTo3kP5fScqrbD8VwQaT8CcE4rCV4uQGNBF+hSOgBDADHtpTT1k4x+6FN5OeURpKAaIsoPHghkJ2lb6yWmESCa+DaR6GXAKlbd0L9UMcXLqnaCn4SpZvbf8hP4fJRgWdRl5uVN/rmyVbZLUVjM8NcVdFRIrTsNyu4mLBmydc3iA/90sCTEOj9e7DSvxLmmLFjpwM5xXLd6z0l6+9G+woNmARXVS3V/RryFntyKC3ATCqVlJoQBG45Tj2gMIunpadTJXWmdioooeGW3sLeUv5MM98mSB4SjKRlJqGPNjx5lO6MmJbZeXZ/L/aO6EsXUQD2h82Wphll4rpGYWPiHTCYqZYiqNYr6E3xUpzcvWVp3uCYVJWP6Ds117p7BoyKVz00yxC9ledF3eppktZWqFVowCMihQE3676L3DDTZsnJf1/8xKUh5U2Mj3lBvjlvCECKi00qo8b1mn/OklQjJ5T4WzTrH6X+/zpez8ZkmtcOayHdUKD/64roZ9dXbXG/hp5A+UWj8oSVYKg2QNAwAnZ+aiZ2KVRE/Y61DCgFg6Ccx/cAEQEAAYkBvAQYAQgAJhYhBPP69mjoLvUSTVGHuu8m9GgjQ/aSBQJfoUjoAhsMBQkDwmcAAAoJEO8m9GgjQ/aSIPcL/3jqL2A2SmC+s0BO4vMPEfCpa2gZ/vo1azzjUieZu5WhIxb5ik0V6T75EW5F0OeZj9qXI06gW+IM8+C6ImUgaR3l47UjBiBPq+uKO9QuT/nOtbSs2dXoTNCLMQN7MlrdUBix+lnqZZGSDgh6n/uVyAYw8Sh4c3/3thHUiR7xzVKGxAKDT8LoVjhHshTzYuQq8MqlfvwVI4eESLaryQ+Y+j5+VLDzSLgPAnnIqF/ui2JQjefJxm/VLoYNaPAGdqoz/u/R0Tmz94bZUfLjgQaDoUpnxYywK2JGlf3mPZ3PNWjxJzuQTF5Ge5bz/TylnRYIyBT7KD7oaKHO62fhDbYPJ4f94iZN4B6nnTAeP34zFDlkUbX4AHudXU7bvxT5OUk9x9c2tj7xwxQHaEhq2+JsYW0EVw27RLhbymnBfLjVVUktNF0nQGvU2TEocw4pr2ZkDHQkSnlbNa4kujlL7VzbpnEgyOmi5er9GaIuVSVADovBu+pz/Ov1y/3jUe8hZ/KleQ==:iQGzBAABCAAdFiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAmM9guIACgkQ7yb0aCND9pIEdwwAiLLqFlrKu0UsQebfuUP07cvGbYy9LfbCMsQj/3/pG/zl7q2mSl2YdXOalbaYD2uyGU/sm7J/+qQZXGyIjmDA7F53sNVAXTuYnrcKmYIzAmzW4lUAzfWA7yL55MtbR/eNUE1rqp/Gu/ejj1OedLyxi+tGFcXUHU0q8EnjQnzfHCJVzOa3PGMIX10NiXPjrF2pafAyE7q2ogwkKZdjJi+8tyAw0tviu4CRGOVlsNZlF+yxePZh55XdRZLCEt4n6mnJrccu0C22rM9R2dEReqGLAj8t/WhACI+UyNXtL+hICnu9y6wjk4spoMr0s0pqTQ76SMwfmRFzk11uZ+ge846hArcUxE27+AeBf9Q1IwT5Ypsc0Efm9ZPoJvA2ggcJv1Yyb58Ggfmd02xPW4EQ8MOEMLA/ZoAhOm3t3wATPNFG1ucm/o+NFNDpF7HNby+Savqv2NrbNwDMlWvFzRmER2+AIO0CIG2HVJScMEn7UkjF8jIm+ba3BIAXbz2FUZ3dytFF 100 > Content-Length: 3 101 > Content-Type: application/x-www-form-urlencoded 102 > 103 } [3 bytes data] 104 * Mark bundle as not supporting multiuse 105 < HTTP/1.1 200 OK 106 < Server: tiny-http (Rust) 107 < Date: Wed, 5 Oct 2022 13:15:37 GMT 108 < Content-Type: text/plain; charset=UTF-8 109 < Access-Control-Allow-Origin: * 110 < Access-Control-Allow-Methods: OPTIONS, PUT, GET 111 < Access-Control-Allow-Headers: Content-Type,Authorization,X-Filename 112 < Content-Length: 64 113 < 114 { [64 bytes data] 115 100 67 100 64 100 3 3548 166 --:--:-- --:--:-- --:--:-- 3722100 67 100 64 100 3 3534 165 --:--:-- --:--:-- --:--:-- 3722 116 * Connection #0 to host localhost left intact 117 32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 118 $ curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 119 bar 120 $ echo -n bar | sha256sum 121 fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9 122 $ curl -X GET http://localhost:8000/fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9 123 bar 124 125 Now, by changing the data and signature, the data under the mutable resource changes: 126 127 .. code:: console 128 129 $ sig=`echo -n "xyzzy" | gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 -w 0` 130 $ curl -X PUT http://localhost:8000/foo -H "Authorization: PUBSIG:$pub:$sig" --data xyzzy 131 32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 132 $ curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 133 xyzzy 134 135 136 Building your identity 137 ====================== 138 139 We can also recreate the symbolic link hash using local tools: 140 141 .. code:: console 142 143 $ t=`mktemp` 144 $ d=`echo -n foo | sha256sum | awk '{ printf "%s",$1; }' | sed -e 's/../\\\\x&/g'` 145 $ echo -ne $d > $t 146 $ k=`echo -n F3FAF668E82EF5124D5187BAEF26F4682343F692 | sed -e 's/../\\\\x&/g'` 147 $ echo -ne $k >> $t 148 $ cat $t | sha256sum 149 32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 - 150 151 In other words, the mutable reference is constructed using the following recipe. 152 153 1. Calculate binary value :code:`R` of the SHA256 digest of the keyword (:code:`sha256(foo) -> 0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AE`) 154 2. Calculate binary value :code:`K` of the key fingerprint (:code:`0xF3FAF668E82EF5124D5187BAEF26F4682343F692`). 155 3. Calculate binary value of SHA256 digest of :code:`R | K` (:code:`sha256(0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AEF3FAF668E82EF5124D5187BAEF26F4682343F692) -> 0x32EA47ECC5A3EE2576AAB00C2F30EAABC2592D56E19F5C82FE4F7CF5874632B2`) 156 157 That means that anyone who knows the keyword and the public key of the uploader can calculate the mutable reference themselves, and retrieve the data behind it. 158 159 160 161 .. 162 163 .. [1] Refer to `section 4.3 of The Book of Swarm <https://www.ethswarm.org/The-Book-of-Swarm.pdf>`_ for a description of "feeds" and "single owner chunks." Swarm uses signatures to allow propagation of the data in the network. :code:`wala` similarly uses signatures to accept update of resources "owned" by the holder of the private keys of an identity. 164 165 .. 166 167 .. [2] At the time of writing :code:`wala` is at version :code:`0.1.1` 168 169 .. 170 171 .. [3] A custom authorization scheme :codE:`PUBSIG` has been invented for this purpose. The format is :code:`Authorization: PUBSIG pgp:key-fingerprint-in-base64>:<signature-in-base64>`. Currently this is strictly just authentication, as there is no feature in wala (yet) to use access control lists to determine which public keys to allow PUTs from. 172 173 .. 174 175 .. [4] If you do not specify a keyword, the keyword value in the mutable resource reference will be the sha256 hash of an empty value (:code:`0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`) 176 177 .. 178 179 .. [5] :code:`wala` comes with a binary tool :code:`wala_send` that lets you specify a key fingerprint in your default pgp keyring and a keyword as arguments when you upload data. A lot less messy, but a lot less educational, too.