manbytesgnu_site

Source files for manbytesgnu.org
git clone git://holbrook.no/manbytesgnu_site.git
Info | Log | Files | Refs

20221005_wala.rst (11238B)


      1 Introducing Wala
      2 ################
      3 
      4 :date: 2022-10-05 14:39
      5 :modified: 2022-10-05 14:39
      6 :category: Code
      7 :author: Louis Holbrook
      8 :tags: wala,pgp,crypto,storage,http,rust
      9 :slug: wala
     10 :summary: A simple HTTP content-addressed storage with cryptographic identity pointers.
     11 :lang: en
     12 :status: published
     13 
     14 
     15 This little project is heavily inspired by the "Single-Owner Chunk" (previously "Mutable Resource") concept from the `Swarm project <https://ethswarm.org>`_ [1]_.
     16 
     17 Dubbed `wala <https://git.defalsify.org/wala>`_ [2]_, the rust application provides two features:
     18 
     19 1. Store any uploaded data under the SHA256 digest of the content itself. We call this "content addressed" storage.
     20 
     21 2. Create a link alias to the content using a keyword and a cryptographic identity.
     22 
     23 First, let's briefly outline what the first item on the list is.
     24 
     25 
     26 Curl up and digest
     27 ==================
     28 
     29 Say an instance of wala is listnening on :code:`localhost:8000`.
     30 
     31 A simple demonstration of the content addressed storage could be:
     32 
     33 .. code:: console
     34 
     35         $ curl -X PUT http://localhost:8000 --data "foo"
     36         2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae
     37         $ curl -X GET http://localhost:8000/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae
     38         foo
     39         $ echo -n "foo" | sha256sum
     40         2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae  -
     41 
     42 In short, anything uploaded to the endpoint will be stored under and can be retrieved by its hash.
     43 
     44 That also means that if the content changes, so does the location.
     45 
     46 
     47 Get to the pointer
     48 ==================
     49 
     50 Now let's say you want a permanent link to content that is changing.
     51 
     52 If you are the only user of the instance, this would be straightforward thing. We could just use a keyword in the url.
     53 
     54 But if you are not, then authentication is needed to make sure that one user doesn't overwrite the another user's permanent link.
     55 
     56 This is usually accomplished with a username and password stored and controlled server-side.
     57 
     58 But we can also do this by simply using public key cryptography instead.
     59 
     60 
     61 Claiming what is yours
     62 ----------------------
     63 
     64 Since I have a weakness for PGP, the first implementation of authentication in wala has been implemented using the `"sequoia openpgp" <https://sequoia-pgp.org>`_ library.
     65 
     66 The procedure is straightforward enough:
     67 
     68 1. Sign the content with your pgp key.
     69 2. Add an `Authorization` header [3]_ with your public key and signature.
     70 3. Upload the content with an arbitrary  keyword. [4]_
     71 
     72 If the signature over the content matches the public key, then a symbolic link to the content will be created on the server. That symbolic link will be a digest of the *key fingerprint* and the *keyword*.
     73 
     74 The content can then be accessed *both* using the content address *and* the symbolic link.
     75 
     76 In wala, the symbolic link is referred to as a *mutable reference*. We will use this term from now on.
     77 
     78 
     79 It ain't pretty, but it works
     80 -----------------------------
     81 
     82 Let's demonstrate creating such an entry in wala using some commonly available tools. [5]_
     83 
     84 We will use a fictional gnupg private key with the fingerprint :code:`F3FAF668E82EF5124D5187BAEF26F4682343F692` and the keyword :code:`foo` to create the mutable reference.
     85 
     86 .. code:: console
     87 
     88         $ sig=`echo -n "bar" | gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 -w 0`
     89         $ pub=`gpg --export F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 - w 0`
     90         $ curl -v -X PUT http://localhost:8000/foo -H "Authorization: PUBSIG:$pub:$sig" --data bar
     91         *   Trying 127.0.0.1:8000...
     92           % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
     93                                          Dload  Upload   Total   Spent    Left  Speed
     94           0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to localhost (127.0.0.1) port 8000 (#0)
     95         > PUT /foo HTTP/1.1
     96         > Host: localhost:8000
     97         > User-Agent: curl/7.85.0
     98         > Accept: */*
     99         > Authorization: PUBSIG pgp:mQGNBF+hSOgBDACpkPQEjADjnQtjmAsdPYpx5N+OMJBYj1DAoIYsDtV6vbcBJQt94Om3xl7RBhv9m2oLgzPsiRwjCEFRWyNSu0BUp5CFjcXfm0S4K2egx4erFnTnSSC9S6tmVNrVNEXvScE6sKAnmJ7JNX1ExJuEiWPbUDRWJ1hoI9+AR+8EONeJRLo/j0Np+S4IFDn0PsxdT+SB0GY0z2cEgjvjoPr4lW9IAb8Ft9TDYp+mOzejn1Fg7CuIrlBRSAv+sj7bVQw15dh1SpbwtS5xxubCa8ExEGI4ByXmeXdR0KZJ+EA5ksO0iSsQ/6ipSOdSg+i0niOClFNm1P/OhbUsYAxCUfiX654FMn2zoxVBEjJ3e7l0pH7ktodaxEctPofQLBA9LSDUIejqJsU0npw/DHDD2uvxG+/A6lgV9L8ETlvgp8RzeOCf2bHuiKYYz87txvkFwsXgU1+TZxbk+mtCBbngsVPLNarY/KGkVJL+yhcHRD0Pl4wXUd6auQuY6vQ9AuKiCT1We2sAEQEAAbQeTWVyIE1hbiA8bWVybWFuQGdyZXlza3VsbC5jb20+iQHUBBMBCAA+FiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAl+hSOgCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ7yb0aCND9pLwiwwAhFJbAyUK05TJKfDz81757N472STtB8sfr0auwmRr8Zs1utHRVM0b/jkjTuo4uJNr7YVVKTKgE7+rJ+pwhm3wlTQ44LVLjByWAi/7NWg3E9b2elm+qkfgm/RfFt3vkuOxGSyZyIFFh+/twv6iABPvr6w7MZwrFaS0UP3g1VGa5TFqg6KNxod9H/gPLxv45lutXf3VvBZTJpr1pxn7aLHlFzEyIgNZbP/N1QF44GSrN/k0DfL631sZjauUXaZXbi5xGsKKCYwJ1g3q587pi6mTdTV3n0hKgVuipO8hGy5++YeOv+hXsCxDwyZ+Shv+qavd/SapxYgCdEueuwONIFfsIsWCd3SCcjKXicTTEFMu8nvBmf7xuo2hv6vEOxoijlXV+4LkGrskdB8ZMg8PywEx6DLmDokgnAhTLrTc1ShbkOtQ3yNjjyFK7BDpqobsJal6d8SpbhccUJLepaSmsk0CgJsTjhAl6EwX0EYgTo3kP5fScqrbD8VwQaT8CcE4rCV4uQGNBF+hSOgBDADHtpTT1k4x+6FN5OeURpKAaIsoPHghkJ2lb6yWmESCa+DaR6GXAKlbd0L9UMcXLqnaCn4SpZvbf8hP4fJRgWdRl5uVN/rmyVbZLUVjM8NcVdFRIrTsNyu4mLBmydc3iA/90sCTEOj9e7DSvxLmmLFjpwM5xXLd6z0l6+9G+woNmARXVS3V/RryFntyKC3ATCqVlJoQBG45Tj2gMIunpadTJXWmdioooeGW3sLeUv5MM98mSB4SjKRlJqGPNjx5lO6MmJbZeXZ/L/aO6EsXUQD2h82Wphll4rpGYWPiHTCYqZYiqNYr6E3xUpzcvWVp3uCYVJWP6Ds117p7BoyKVz00yxC9ledF3eppktZWqFVowCMihQE3676L3DDTZsnJf1/8xKUh5U2Mj3lBvjlvCECKi00qo8b1mn/OklQjJ5T4WzTrH6X+/zpez8ZkmtcOayHdUKD/64roZ9dXbXG/hp5A+UWj8oSVYKg2QNAwAnZ+aiZ2KVRE/Y61DCgFg6Ccx/cAEQEAAYkBvAQYAQgAJhYhBPP69mjoLvUSTVGHuu8m9GgjQ/aSBQJfoUjoAhsMBQkDwmcAAAoJEO8m9GgjQ/aSIPcL/3jqL2A2SmC+s0BO4vMPEfCpa2gZ/vo1azzjUieZu5WhIxb5ik0V6T75EW5F0OeZj9qXI06gW+IM8+C6ImUgaR3l47UjBiBPq+uKO9QuT/nOtbSs2dXoTNCLMQN7MlrdUBix+lnqZZGSDgh6n/uVyAYw8Sh4c3/3thHUiR7xzVKGxAKDT8LoVjhHshTzYuQq8MqlfvwVI4eESLaryQ+Y+j5+VLDzSLgPAnnIqF/ui2JQjefJxm/VLoYNaPAGdqoz/u/R0Tmz94bZUfLjgQaDoUpnxYywK2JGlf3mPZ3PNWjxJzuQTF5Ge5bz/TylnRYIyBT7KD7oaKHO62fhDbYPJ4f94iZN4B6nnTAeP34zFDlkUbX4AHudXU7bvxT5OUk9x9c2tj7xwxQHaEhq2+JsYW0EVw27RLhbymnBfLjVVUktNF0nQGvU2TEocw4pr2ZkDHQkSnlbNa4kujlL7VzbpnEgyOmi5er9GaIuVSVADovBu+pz/Ov1y/3jUe8hZ/KleQ==:iQGzBAABCAAdFiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAmM9guIACgkQ7yb0aCND9pIEdwwAiLLqFlrKu0UsQebfuUP07cvGbYy9LfbCMsQj/3/pG/zl7q2mSl2YdXOalbaYD2uyGU/sm7J/+qQZXGyIjmDA7F53sNVAXTuYnrcKmYIzAmzW4lUAzfWA7yL55MtbR/eNUE1rqp/Gu/ejj1OedLyxi+tGFcXUHU0q8EnjQnzfHCJVzOa3PGMIX10NiXPjrF2pafAyE7q2ogwkKZdjJi+8tyAw0tviu4CRGOVlsNZlF+yxePZh55XdRZLCEt4n6mnJrccu0C22rM9R2dEReqGLAj8t/WhACI+UyNXtL+hICnu9y6wjk4spoMr0s0pqTQ76SMwfmRFzk11uZ+ge846hArcUxE27+AeBf9Q1IwT5Ypsc0Efm9ZPoJvA2ggcJv1Yyb58Ggfmd02xPW4EQ8MOEMLA/ZoAhOm3t3wATPNFG1ucm/o+NFNDpF7HNby+Savqv2NrbNwDMlWvFzRmER2+AIO0CIG2HVJScMEn7UkjF8jIm+ba3BIAXbz2FUZ3dytFF
    100         > Content-Length: 3
    101         > Content-Type: application/x-www-form-urlencoded
    102         > 
    103         } [3 bytes data]
    104         * Mark bundle as not supporting multiuse
    105         < HTTP/1.1 200 OK
    106         < Server: tiny-http (Rust)
    107         < Date: Wed,  5 Oct 2022 13:15:37 GMT
    108         < Content-Type: text/plain; charset=UTF-8
    109         < Access-Control-Allow-Origin: *
    110         < Access-Control-Allow-Methods: OPTIONS, PUT, GET
    111         < Access-Control-Allow-Headers: Content-Type,Authorization,X-Filename
    112         < Content-Length: 64
    113         < 
    114         { [64 bytes data]
    115         100    67  100    64  100     3   3548    166 --:--:-- --:--:-- --:--:--  3722100    67  100    64  100     3   3534    165 --:--:-- --:--:-- --:--:--  3722
    116         * Connection #0 to host localhost left intact
    117         32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2
    118         $ curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2
    119         bar
    120         $ echo -n bar | sha256sum
    121         fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9
    122         $ curl -X GET http://localhost:8000/fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9
    123         bar
    124 
    125 Now, by changing the data and signature, the data under the mutable resource changes:
    126 
    127 .. code:: console
    128 
    129         $ sig=`echo -n "xyzzy" | gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 -w 0`
    130         $ curl -X PUT http://localhost:8000/foo -H "Authorization: PUBSIG:$pub:$sig" --data xyzzy
    131         32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2
    132         $ curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2
    133         xyzzy
    134 
    135 
    136 Building your identity
    137 ======================
    138 
    139 We can also recreate the symbolic link hash using local tools:
    140 
    141 .. code:: console
    142 
    143         $ t=`mktemp`
    144         $ d=`echo -n foo | sha256sum | awk '{ printf "%s",$1; }' | sed -e 's/../\\\\x&/g'`
    145         $ echo -ne $d > $t
    146         $ k=`echo -n F3FAF668E82EF5124D5187BAEF26F4682343F692 | sed -e 's/../\\\\x&/g'`
    147         $ echo -ne $k >> $t
    148         $ cat $t | sha256sum 
    149         32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2  -
    150 
    151 In other words, the mutable reference is constructed using the following recipe.
    152 
    153 1. Calculate binary value :code:`R` of the SHA256 digest of the keyword (:code:`sha256(foo) -> 0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AE`)
    154 2. Calculate binary value :code:`K` of the key fingerprint (:code:`0xF3FAF668E82EF5124D5187BAEF26F4682343F692`).
    155 3. Calculate binary value of SHA256 digest of :code:`R | K` (:code:`sha256(0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AEF3FAF668E82EF5124D5187BAEF26F4682343F692) -> 0x32EA47ECC5A3EE2576AAB00C2F30EAABC2592D56E19F5C82FE4F7CF5874632B2`)
    156   
    157 That means that anyone who knows the keyword and the public key of the uploader can calculate the mutable reference themselves, and retrieve the data behind it.
    158 
    159 
    160 
    161 ..
    162 
    163         .. [1] Refer to `section 4.3 of The Book of Swarm <https://www.ethswarm.org/The-Book-of-Swarm.pdf>`_ for a description of "feeds" and "single owner chunks." Swarm uses signatures to allow propagation of the data in the network. :code:`wala` similarly uses signatures to accept update of resources "owned" by the holder of the private keys of an identity.
    164 
    165 ..
    166 
    167         .. [2] At the time of writing :code:`wala` is at version :code:`0.1.1`
    168 
    169 ..
    170 
    171         .. [3] A custom authorization scheme :codE:`PUBSIG` has been invented for this purpose. The format is :code:`Authorization: PUBSIG pgp:key-fingerprint-in-base64>:<signature-in-base64>`. Currently this is strictly just authentication, as there is no feature in wala (yet) to use access control lists to determine which public keys to allow PUTs from.
    172 
    173 ..
    174 
    175         .. [4] If you do not specify a keyword, the keyword value in the mutable resource reference will be the sha256 hash of an empty value (:code:`0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`)
    176 
    177 ..
    178 
    179         .. [5] :code:`wala` comes with a binary tool :code:`wala_send` that lets you specify a key fingerprint in your default pgp keyring and a keyword as arguments when you upload data. A lot less messy, but a lot less educational, too.