commit 36e7350f3e02ea69785320485cb1a1e728676c3c parent 9c5a606a498763fb226b3378ebbea9607c64c3d7 Author: lash <dev@holbrook.no> Date: Wed, 5 Oct 2022 14:41:06 +0000 Add wala introduction Diffstat:
35 files changed, 1125 insertions(+), 353 deletions(-)
diff --git a/09c60de8503f5115c11d7059476de0a930bdeaf0511609d24ed93d15aadef47b.asc b/09c60de8503f5115c11d7059476de0a930bdeaf0511609d24ed93d15aadef47b.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBYACgkQzcvSTdHQ -4AFW0Qf+KBuzE+eT48GOVBo99JZJHmVExVDEc3QfhbiljgFqmW9riVRKFosBXCCY -DpQZ4fwkCgbPIzE4MRvaTJMb/a/SisR4+gfswZktIYJRoauMoDvgzM8dHNbURLK7 -g5+eesbUqU9IZ79Rpz5EaIHe5fl7Qv/YgtXi1swU8IUCxeLbT2hILhGLI+PFGrv+ -rMIenT+hd5Q7zJQu4U32tu1D/l/OYoqUfIIm5cUV197z1c8Pd6LiuORGQMvi+74A -nTnRo3VeCQt4tiBLybGsgzbwReTLLaL6FmY4QozHzLfmjLDjm4D5vWrNME4URGsq -nLrC4pAZwK+Bp9Jhp1xRn6Ta4dLdNA== -=F5qt +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AEdIAgAqbDu/ip8RfF6r9b33Syh6GzqR8/s9t1b0ycb/DyfmqH/VDdCEC/i34l6 +5hmoeUpIwRcxvWqOnop8ucAMTb3SqIYW9zpJ1QZl46wX2vWEA3Tfoz6f6LN/I4El +2ee5AhKB1+GrUxEfzBKEfyL1tlwqlGio7kpoQXoD+FQZfQEEl4RHewflAMvzOwV9 +IR5Cmx2M6/KZzjDFJ/GRdEojwms88GIKZlbwRIQ3XLLSdW3DcZ/6jYRcIOcNxmuA +NoBUCFFcCrDEGriCEmzj3eRvzjjnaAXeMGP4bE7OL4l97cgar8bg74yCc6sG9INy +00QF0WPf5WLgS0y6ksFAU1NgEWHl6Q== +=gIjE -----END PGP SIGNATURE----- diff --git a/1147e630f5d17d981407c49cfc72bb0b340e2b672044f7df387490f23c8700c0.asc b/1147e630f5d17d981407c49cfc72bb0b340e2b672044f7df387490f23c8700c0.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBYACgkQzcvSTdHQ -4AGsFAgAg85b7GdNy7TV4leildZ3ycpBnYStnfoBcZnZoCJcQpSdKRSBmYja3+dt -mcXPDhf1saD2QnRT4U8wurtdrIx9gGoAoAIpvAPmgjhBIq+U+F63BJ6zG1mJIdmt -0S02oWJFVui5IwCkoXbe53Q9XK6VEwICDbJWSCnfs3icFZ9IFEucsRsor6lK0SzW -PnSLSk3BnbjKqpEZ5xoLIeFZZE0bTDFXcH5r07AwiieAo+1BcMTc07srkvPcneIb -IBzAumXgdBIT22fDYy32imr6SpU9EnKKZZ9+HsZPKQ+cSCoXoZnPxeG0YgBDqgaC -FpP1fvzLEwMT4uQb2g84nOZNpzF8mg== -=/enN +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AFoswgAh+kU/Qv5bxof6wtbjjG2aQizAQAcE2IOZbS0fQtFmtYgjzSVeIoBlXgz +0GQrs2NaA8d6lzCqg2xU/qX3iY5bcaAPa5QuTj37uVxis6mjCVhzch4zGUcoFwFd +yindnAIBy8mPzMX6ZrhJJsqy1u/b+JY3iSNN0TvQK9TvVXOXwUZ5h6KW096Qdslp +uYuxXBB/JpuZtwmR7269L85O9VwoXSgpebVJH5S4hGq5a19UhDLUBw15+LtAem2d +U931wv3h0w/SIDkR4UB4rLqUtG1hjs2jqeNUgn4i01t/TWQoMIzwsQ12vfDZDzWm ++aexMiPOWxcOhBmUTfP9mddxy6h9yg== +=yO3B -----END PGP SIGNATURE----- diff --git a/1e676d28d8bb972218b427f8f01ba31ced64f4f2390249f337f1d1e7c2035825 b/1e676d28d8bb972218b427f8f01ba31ced64f4f2390249f337f1d1e7c2035825 @@ -0,0 +1,253 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - A portable book metadata exercise</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li class="active"><a href="./category/archiving.html">Archiving</a></li> + <li><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./portable-book-metadata.html" rel="bookmark" + title="Permalink to A portable book metadata exercise">A portable book metadata exercise</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2022-10-01T12:40:00+02:00"> + Sat 01 October 2022 + </time> +in <a class="category" href="./category/archiving.html">archiving</a> + <a href="./tag/hash.html">hash</a> + <a href="./tag/kitab.html">kitab</a> + <a href="./tag/literature.html">literature</a> + <a href="./tag/metadata.html">metadata</a> + <a href="./tag/dublincore.html">dublincore</a> + <a href="./tag/libgen.html">libgen</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./backup-rsync-duplicity.html"> + Combining duplicity and rsync + </a> + </li> + | + + <li> + Next: <a href="./wala.html"> + Introducing Wala + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>One of the things I have been working on the last few weeks is a rust application I have dubbed <a class="reference external" href="https://git.defalsify.net/kitab">kitab</a> <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p> +<p>In short, the application makes it easy to extract literary metadata to a separate file structure.</p> +<p>The metadata can in turn be applied as <em>extended attributes</em> recursively on a directory for files that match.</p> +<p>The way it's accomplished it simple: The file name of the metadata is the hex representation of the digest of the file. The same digest is used to match files to metadata when applying it back to the file.</p> +<p>There are two advantages to this:</p> +<ol class="arabic simple"> +<li>The digest of the media file need not be affected by the metadata, i.e. by embedding metadata in the file itself.</li> +<li>You do not need to use the file name to keep record of what a file is.</li> +</ol> +<div class="section" id="yarr-ye-matey-data"> +<h2>Yarr, ye matey-data</h2> +<p>Let's demonstrate with an example.</p> +<p>The fabulous <a class="reference external" href="https://libgen.rs">Library Genesis</a> project has made available an endpoint to retrieve <tt class="docutils literal">bibtex</tt> entries based on the <tt class="docutils literal">md5</tt> hash of the book media file.</p> +<p>A version of the <a class="reference external" href="https://libgen.rs/book/index.php?md5=BCD99F1AB4155F2A2A362E5B7938A852">Bitcoin White Paper</a>, under the <code>md5</code> hash <code>bcd99f1ab4155f2a2a362e5b7938a852</code>, can be found there.</p> +<p>If you download this file using a synchronous download link, the browser will provide you with a filename to go with the download.</p> +<p>However, if you use the torrent alternative, the filename will be the <tt class="docutils literal">md5</tt> hash itself. If you are torrenting a bunch of those files, it quickly becomes a nuisance to distinguish them.</p> +<p>And, of course: In either case there is no guarantee the any metadata comes with the file.</p> +<div class="section" id="inside-the-book"> +<h3>Inside the book</h3> +<p>Kitab (v0.0.2) is able to read metadata from both a bibtex source and xattr entries on a file, as well as its native <a class="reference external" href="https://www.w3.org/TR/turtle/">rdf-turtle</a> format.</p> +<p>In kitab's data store, every media file entity in rdf-turtle is keyed with a <a class="reference external" href="https://www.rfc-editor.org/info/rfc8141">URN</a> specifying a digest for the file.</p> +<p>To see exactly what that looks like, let's download and import the bibtex metadata for the paper <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>:</p> +<pre class="code bash literal-block"> +<span class="nv">bibtex_file</span><span class="o">=</span><span class="sb">`</span>mktemp<span class="sb">`</span> +<span class="nv">kitab_dir</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> +curl -s -X GET https://libgen.rs/book/bibtex.php?md5<span class="o">=</span>BCD99F1AB4155F2A2A362E5B7938A852 -o <span class="nv">$bibtex_file</span> +kitab --store <span class="nv">$kitab_dir</span> import --digest md5:BCD99F1AB4155F2A2A362E5B7938A852 <span class="nv">$bibtex_file</span> +cat <span class="nv">$kitab_dir</span>/* +</pre> +<p>The output of the above should be:</p> +<pre class="code turtle literal-block"> +<span class="nv"><URN:md5:bcd99f1ab4155f2a2a362e5b7938a852></span> <span class="nv"><https://purl.org/dc/terms/title></span> <span class="s">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> +<span class="nv"><https://purl.org/dc/terms/creator></span> <span class="s">"Satoshi Nakamoto"</span> <span class="p">;</span> +<span class="nv"><https://purl.org/dc/terms/type></span> <span class="s">"book"</span> <span class="p">.</span> +</pre> +<p>Now let's say the media file itself has been downloaded to <tt class="docutils literal"><span class="pre">~/.local/share/transmission</span></tt>. We can apply this metadata as extended attributes.</p> +<p>This time we turn on logging to see what's going on:</p> +<pre class="code console literal-block"> +<span class="gp">$ </span><span class="nv">RUST_LOG</span><span class="o">=</span>info kitab --store <span class="nv">$kitab_dir</span> apply --digest md5 ~/.local/share/transmission +<span class="go">[2022-10-01T11:14:59Z INFO kitab] have index directory "/tmp/tmp.r0jBm6q4hW" +[2022-10-01T11:14:59Z INFO kitab] using digest type md5 +[2022-10-01T11:14:59Z INFO kitab] apply from path "/home/lash/.local/share/transmission/" +[2022-10-01T11:14:59Z INFO kitab] apply DirEntry("/home/lash/.local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852") -> title "Bitcoin: A Peer-to-Peer Electronic Cash System" author "Satoshi Nakamoto" digest md5:bcd99f1ab4155f2a2a362e5b7938a852 + +</span><span class="gp">$ </span>find ~/.local/share/transmission -type f -regextype sed -regex <span class="s2">".*/[a-f0-9]\{32\}</span>$<span class="s2">"</span> -exec getfattr -d <span class="o">{}</span> <span class="se">\;</span> +<span class="gp"># </span>file: .local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852 +<span class="go">user.dcterms:creator="Satoshi Nakamoto" +user.dcterms:title="Bitcoin: A Peer-to-Peer Electronic Cash System" +user.dcterms:type="book"</span> +</pre> +</div> +<div class="section" id="let-the-right-one-in"> +<h3>Let the right one in</h3> +<p>Conversely, the metadata can be re-imported directly from the extended attributes. And this time, let's store it both under the <tt class="docutils literal">md5</tt> and the <tt class="docutils literal">sha512</tt> hash:</p> +<pre class="code bash literal-block"> +$ <span class="nv">kitab_dir_new</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> +$ kitab --store <span class="nv">$kitab_dir_new</span> import --digest md5 --digest sha512 .local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852 +$ find <span class="nv">$kitab_dir_new</span> -type f -exec cat <span class="o">{}</span> <span class="se">\;</span> +/tmp/tmp.B6j41YMmEM/493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792 +<URN:sha512:493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792> <https://purl.org/dc/terms/title> <span class="s2">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> + <https://purl.org/dc/terms/creator> <span class="s2">"Satoshi Nakamoto"</span> <span class="p">;</span> + <https://purl.org/dc/terms/type> <span class="s2">"book"</span> <span class="p">;</span> + <https://purl.org/dc/terms/MediaType> <span class="s2">"application/epub+zip"</span> . +/tmp/tmp.B6j41YMmEM/bcd99f1ab4155f2a2a362e5b7938a852 +<URN:md5:bcd99f1ab4155f2a2a362e5b7938a852> <https://purl.org/dc/terms/title> <span class="s2">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> + <https://purl.org/dc/terms/creator> <span class="s2">"Satoshi Nakamoto"</span> <span class="p">;</span> + <https://purl.org/dc/terms/type> <span class="s2">"book"</span> <span class="p">;</span> + <https://purl.org/dc/terms/MediaType> <span class="s2">"application/epub+zip"</span> . +</pre> +</div> +</div> +<div class="section" id="level-up"> +<h2>Level up</h2> +<p>Finally, a bash script <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a> example that lets you retrieve and apply metadata for a batch of files found in the directory given as the <em>first positional arg</em>.</p> +<p>This script even renames the files according to the metadata applied.</p> +<pre class="code bash literal-block"> +<span class="ln"> 0 </span><span class="c1"># NOTE! this will only work if your fs supports xattr. +</span><span class="ln"> 1 </span><span class="c1"># That's why we cannot use tmpfs (mktemp) here; tmpfs does not support xattr. +</span><span class="ln"> 2 </span><span class="c1"></span> +<span class="ln"> 3 </span><span class="c1"># directory to copy media files to +</span><span class="ln"> 4 </span><span class="c1"></span><span class="nv">outdir</span><span class="o">=</span>./<span class="k">$(</span>uuidgen<span class="k">)</span> +<span class="ln"> 5 </span>mkdir -vp <span class="nv">$outdir</span> +<span class="ln"> 6 </span> +<span class="ln"> 7 </span><span class="c1"># Input dir is the first positional arg. +</span><span class="ln"> 8 </span><span class="c1"></span><span class="nv">indir</span><span class="o">=</span><span class="nv">$1</span> +<span class="ln"> 9 </span> +<span class="ln">10 </span><span class="nv">IFS</span><span class="o">=</span><span class="s1">$'\n'</span> +<span class="ln">11 </span> +<span class="ln">12 </span><span class="c1"># Retrieve metadata for each file and import it into the kitab store. +</span><span class="ln">13 </span><span class="c1"># Also copy the media file to the separate output directory. +</span><span class="ln">14 </span><span class="c1"></span><span class="k">for</span> f <span class="k">in</span> <span class="k">$(</span>find <span class="nv">$indir</span> -type f<span class="k">)</span><span class="p">;</span> <span class="k">do</span> +<span class="ln">15 </span> <span class="nv">sum</span><span class="o">=</span><span class="k">$(</span>md5sum <span class="nv">$f</span> <span class="p">|</span> awk <span class="s1">'{print $1;}'</span><span class="k">)</span> +<span class="ln">16 </span> <span class="nb">echo</span> <span class="s2">"downloading metadata for </span><span class="nv">$indir</span><span class="s2">/</span><span class="nv">$f</span><span class="s2">"</span> +<span class="ln">17 </span> <span class="nv">srct</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span> +<span class="ln">18 </span> curl -s -X GET https://libgen.rs/book/bibtex.php?md5<span class="o">=</span><span class="nv">$sum</span> -o <span class="nv">$srct</span> +<span class="ln">19 </span> <span class="nv">dstt</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span> +<span class="ln">20 </span> xmllint --html --xpath <span class="s1">'string(/html/body/textarea[@id="bibtext"])'</span> <span class="nv">$srct</span> > <span class="nv">$dstt</span> +<span class="ln">21 </span> kitab import --digest md5:<span class="nv">$sum</span> <span class="nv">$dstt</span> +<span class="ln">22 </span> cp <span class="nv">$f</span> <span class="nv">$outdir</span>/ +<span class="ln">23 </span><span class="k">done</span> +<span class="ln">24 </span> +<span class="ln">25 </span><span class="c1"># Apply metadata imported from bibtex as xattr for the media files. +</span><span class="ln">26 </span><span class="c1"></span><span class="nv">RUST_LOG</span><span class="o">=</span>info kitab apply --digest md5 <span class="nv">$outdir</span>/ +<span class="ln">27 </span> +<span class="ln">28 </span><span class="c1"># Rename the files according to the metadata title and media type. +</span><span class="ln">29 </span><span class="c1"></span><span class="k">for</span> f <span class="k">in</span> <span class="k">$(</span>ls <span class="nv">$outdir</span><span class="k">)</span><span class="p">;</span> <span class="k">do</span> +<span class="ln">30 </span> <span class="nv">title</span><span class="o">=</span><span class="k">$(</span>getfattr --only-values -n user.dcterms:title <span class="nv">$outdir</span>/<span class="nv">$f</span><span class="k">)</span> +<span class="ln">31 </span> +<span class="ln">32 </span> <span class="nv">f_typ</span><span class="o">=</span><span class="k">$(</span>file -b --mime-type <span class="nv">$outdir</span>/<span class="nv">$f</span><span class="k">)</span> +<span class="ln">33 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">""</span> +<span class="ln">34 </span> <span class="k">case</span> <span class="s2">"</span><span class="nv">$f_typ</span><span class="s2">"</span> <span class="k">in</span> +<span class="ln">35 </span> <span class="s2">"application/pdf"</span><span class="o">)</span> +<span class="ln">36 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".pdf"</span> +<span class="ln">37 </span> <span class="p">;;</span> +<span class="ln">38 </span> <span class="s2">"application/epub+zip"</span><span class="o">)</span> +<span class="ln">39 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".epub"</span> +<span class="ln">40 </span> <span class="p">;;</span> +<span class="ln">41 </span> <span class="s2">"application/x-mobipocket-ebook"</span><span class="o">)</span> +<span class="ln">42 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".mobi"</span> +<span class="ln">43 </span> <span class="p">;;</span> +<span class="ln">44 </span> <span class="s2">"text/plain"</span><span class="o">)</span> +<span class="ln">45 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".txt"</span> +<span class="ln">46 </span> <span class="p">;;</span> +<span class="ln">47 </span> <span class="s2">"text/html"</span><span class="o">)</span> +<span class="ln">48 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".html"</span> +<span class="ln">49 </span> <span class="p">;;</span> +<span class="ln">50 </span> *<span class="o">)</span> +<span class="ln">51 </span> ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> unhandled mime <span class="nb">type</span> <span class="nv">$f_typ</span> +<span class="ln">52 </span> <span class="nb">exit</span> <span class="m">1</span> +<span class="ln">53 </span> <span class="k">esac</span> +<span class="ln">54 </span> mv -v <span class="nv">$outdir</span>/<span class="nv">$f</span> <span class="nv">$outdir</span>/<span class="si">${</span><span class="nv">title</span><span class="si">}${</span><span class="nv">f_ext</span><span class="si">}</span> +<span class="ln">55 </span><span class="k">done</span> +</pre> +<p>This last example will result in:</p> +<ul class="simple"> +<li>A media file named <tt class="docutils literal">$outdir/Bitcoin: A <span class="pre">Peer-to-Peer</span> Electronic Cash System.epub</tt></li> +<li>... with metadata applied as extended attributes</li> +<li>An rdf-turtle metadata entry in <tt class="docutils literal"><span class="pre">~/.local/share/kitab/idx/bcd99f1ab4155f2a2a362e5b7938a852</span></tt></li> +</ul> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>The relevant documentation for <tt class="docutils literal">kitab</tt> at the time of writing is <a class="reference external" href="https://defalsify.org/doc/crates/kitab/0.0.2/kitab/">here</a>. To build kitab, simply <em>clone</em> the repository and build with <code>cargo build --all-features</code>.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>The <code>kitab</code> command in the script assumes you have built the <em>kitab binary</em> and made it available in your path.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>the script uses <code>xmllint</code> which on archlinux is provided by the <tt class="docutils literal">libxml2</tt> package.</td></tr> +</tbody> +</table> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/1e676d28d8bb972218b427f8f01ba31ced64f4f2390249f337f1d1e7c2035825.asc b/1e676d28d8bb972218b427f8f01ba31ced64f4f2390249f337f1d1e7c2035825.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AHweAf+JuTA2ZBOmOKwoaEGaKY6w+NJmsWgAlHEUxLBaTAdD0Z0ydWUm9y97tsN +a8Pyl55IM5mWbe/AM1claC7kY+7uh9u+VOvzCNW/oZH1aKLfLkcquVJkcC4YhBCP +vHGZ4RVHZSQ1TeQEeR4HvTHgBAaGqxwc9LcVMfaQWOs9ho5FEPAJw6GKaJVVLvNS +LpM/n/Q41L7WLWvMbAk/xhVANFtKaFOlKvmJeWeFlNgFU/38JXZt9sn2kXxbwr/y +8MJS1p0oW48zczGWjD27+GOjYSM4vpERsFdBY+EHiknPC57d1xntq91PTDnvsxP1 +/T5a+TUCUPoS3dyvHZtYzU4Knq/vqQ== +=aeOl +-----END PGP SIGNATURE----- diff --git a/24345ef2cf2edbf32d9ec3e55a6e0801a5a73f8337bb81bc1a7395ebbe3bf7e0.asc b/24345ef2cf2edbf32d9ec3e55a6e0801a5a73f8337bb81bc1a7395ebbe3bf7e0.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBYACgkQzcvSTdHQ -4AE4DwgAhMJoZbwt2N1FuCWUfoxaZzOeCgzI3Tz7ddhg+CLJ6cFTii54N9yRYPyX -N6rw3FAXOv3BFFuoJuhA78iGiPN15USMIhEoLKwKOAcocmX5teNMk+Ggg0v3vhPb -chBsQZbfoqO4ZHAvnO8Xc1VC7zRH2FX1p3N/qHM/kWiiNXitjYGZRIFcP6NifU64 -r5b/QNvvq8gwQqrz1TEr2rucKNTUljoDJEb5mr8Vy4uw9AOIe0OO4SaDOiBdNC17 -UazyOdHQYqG9kZHJt7547O7kIX5Xgykk4ztGCjwJ9n3Ywb0lb5A1ydF0dMPTNXFM -rm/04TMWY7Ok/z3gEG1rhGrcNNpm8Q== -=mTR3 +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AH7LQgArNp12BSIx+Uko1fA9uZU0ejYRQNuUOzd9N8aOcuEi5laiHgPt7Q8ImAm +7RfuBysskGVyGb6//9imvOF+UbFnQ57DX3kZTVVnIzT3KViB9vPrZp5lPia+cr/y +cHNA8bpUjcNujEbOEcsXRqfKahaHL71J9WDKNNrQgLdc6R32pVKQAbysY0mqpcGj +wOrDpgZb8NCIsToUB9ULghTvQ12dRfoAsGfKCFdo3x0O1D30/Jo1g0yfe7wfjeXd +OoOnFru9HgwifF0B442eG4r6GBxQOLTGFW0mxR3DMFMmBRsiQYLGVCj7YZiVuMV6 +Onjenk+qSjIArVm7sAil7OtZ1HNfjA== +=3NI/ -----END PGP SIGNATURE----- diff --git a/406a858d431883e0f1e8e05ea87fae94b152a03a408a0d239e6868cef369f198 b/406a858d431883e0f1e8e05ea87fae94b152a03a408a0d239e6868cef369f198 @@ -0,0 +1,238 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - Introducing Wala</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="./category/archiving.html">Archiving</a></li> + <li class="active"><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./wala.html" rel="bookmark" + title="Permalink to Introducing Wala">Introducing Wala</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2022-10-05T14:39:00+02:00"> + Wed 05 October 2022 + </time> +in <a class="category" href="./category/code.html">code</a> + <a href="./tag/wala.html">wala</a> + <a href="./tag/pgp.html">pgp</a> + <a href="./tag/crypto.html">crypto</a> + <a href="./tag/storage.html">storage</a> + <a href="./tag/http.html">http</a> + <a href="./tag/rust.html">rust</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./portable-book-metadata.html"> + A portable book metadata exercise + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>This little project is heavily inspired by the "Single-Owner Chunk" (previously "Mutable Resource") concept from the <a class="reference external" href="https://ethswarm.org">Swarm project</a> <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p> +<p>Dubbed <a class="reference external" href="https://git.defalsify.org/wala">wala</a> <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>, the rust application provides two features:</p> +<ol class="arabic simple"> +<li>Store any uploaded data under the SHA256 digest of the content itself. We call this "content addressed" storage.</li> +<li>Create a link alias to the content using a keyword and a cryptographic identity.</li> +</ol> +<p>First, let's briefly outline what the first item on the list is.</p> +<div class="section" id="curl-up-and-digest"> +<h2>Curl up and digest</h2> +<p>Say an instance of wala is listnening on <code>localhost:8000</code>.</p> +<p>A simple demonstration of the content addressed storage could be:</p> +<pre class="code console literal-block"> +<span class="gp">$ </span>curl -X PUT http://localhost:8000 --data <span class="s2">"foo"</span> +<span class="go">2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae +</span><span class="gp">$ </span>curl -X GET http://localhost:8000/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae +<span class="go">foo +</span><span class="gp">$ </span><span class="nb">echo</span> -n <span class="s2">"foo"</span> <span class="p">|</span> sha256sum +<span class="go">2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae -</span> +</pre> +<p>In short, anything uploaded to the endpoint will be stored under and can be retrieved by its hash.</p> +<p>That also means that if the content changes, so does the location.</p> +</div> +<div class="section" id="get-to-the-pointer"> +<h2>Get to the pointer</h2> +<p>Now let's say you want a permanent link to content that is changing.</p> +<p>If you are the only user of the instance, this would be straightforward thing. We could just use a keyword in the url.</p> +<p>But if you are not, then authentication is needed to make sure that one user doesn't overwrite the another user's permanent link.</p> +<p>This is usually accomplished with a username and password stored and controlled server-side.</p> +<p>But we can also do this by simply using public key cryptography instead.</p> +<div class="section" id="claiming-what-is-yours"> +<h3>Claiming what is yours</h3> +<p>Since I have a weakness for PGP, the first implementation of authentication in wala has been implemented using the <a class="reference external" href="https://sequoia-pgp.org">"sequoia openpgp"</a> library.</p> +<p>The procedure is straightforward enough:</p> +<ol class="arabic simple"> +<li>Sign the content with your pgp key.</li> +<li>Add an <cite>Authorization</cite> header <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a> with your public key and signature.</li> +<li>Upload the content with an arbitrary keyword. <a class="footnote-reference" href="#footnote-4" id="footnote-reference-4">[4]</a></li> +</ol> +<p>If the signature over the content matches the public key, then a symbolic link to the content will be created on the server. That symbolic link will be a digest of the <em>key fingerprint</em> and the <em>keyword</em>.</p> +<p>The content can then be accessed <em>both</em> using the content address <em>and</em> the symbolic link.</p> +<p>In wala, the symbolic link is referred to as a <em>mutable reference</em>. We will use this term from now on.</p> +</div> +<div class="section" id="it-ain-t-pretty-but-it-works"> +<h3>It ain't pretty, but it works</h3> +<p>Let's demonstrate creating such an entry in wala using some commonly available tools. <a class="footnote-reference" href="#footnote-5" id="footnote-reference-5">[5]</a></p> +<p>We will use a fictional gnupg private key with the fingerprint <code>F3FAF668E82EF5124D5187BAEF26F4682343F692</code> and the keyword <code>foo</code> to create the mutable reference.</p> +<pre class="code console literal-block"> +<span class="gp">$ </span><span class="nv">sig</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -n <span class="s2">"bar"</span> <span class="p">|</span> gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 <span class="p">|</span> base64 -w <span class="m">0</span><span class="sb">`</span> +<span class="gp">$ </span><span class="nv">pub</span><span class="o">=</span><span class="sb">`</span>gpg --export F3FAF668E82EF5124D5187BAEF26F4682343F692 <span class="p">|</span> base64 - w <span class="m">0</span><span class="sb">`</span> +<span class="gp">$ </span>curl -v -X PUT http://localhost:8000/foo -H <span class="s2">"Authorization: PUBSIG:</span><span class="nv">$pub</span><span class="s2">:</span><span class="nv">$sig</span><span class="s2">"</span> --data bar +<span class="go">* Trying 127.0.0.1:8000... +</span><span class="gp"> % </span>Total % Received % Xferd Average Speed Time Time Time Current +<span class="go"> Dload Upload Total Spent Left Speed + 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 8000 (#0) +> PUT /foo HTTP/1.1 +> Host: localhost:8000 +> User-Agent: curl/7.85.0 +> Accept: */* +> Authorization: PUBSIG pgp:mQGNBF+hSOgBDACpkPQEjADjnQtjmAsdPYpx5N+OMJBYj1DAoIYsDtV6vbcBJQt94Om3xl7RBhv9m2oLgzPsiRwjCEFRWyNSu0BUp5CFjcXfm0S4K2egx4erFnTnSSC9S6tmVNrVNEXvScE6sKAnmJ7JNX1ExJuEiWPbUDRWJ1hoI9+AR+8EONeJRLo/j0Np+S4IFDn0PsxdT+SB0GY0z2cEgjvjoPr4lW9IAb8Ft9TDYp+mOzejn1Fg7CuIrlBRSAv+sj7bVQw15dh1SpbwtS5xxubCa8ExEGI4ByXmeXdR0KZJ+EA5ksO0iSsQ/6ipSOdSg+i0niOClFNm1P/OhbUsYAxCUfiX654FMn2zoxVBEjJ3e7l0pH7ktodaxEctPofQLBA9LSDUIejqJsU0npw/DHDD2uvxG+/A6lgV9L8ETlvgp8RzeOCf2bHuiKYYz87txvkFwsXgU1+TZxbk+mtCBbngsVPLNarY/KGkVJL+yhcHRD0Pl4wXUd6auQuY6vQ9AuKiCT1We2sAEQEAAbQeTWVyIE1hbiA8bWVybWFuQGdyZXlza3VsbC5jb20+iQHUBBMBCAA+FiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAl+hSOgCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ7yb0aCND9pLwiwwAhFJbAyUK05TJKfDz81757N472STtB8sfr0auwmRr8Zs1utHRVM0b/jkjTuo4uJNr7YVVKTKgE7+rJ+pwhm3wlTQ44LVLjByWAi/7NWg3E9b2elm+qkfgm/RfFt3vkuOxGSyZyIFFh+/twv6iABPvr6w7MZwrFaS0UP3g1VGa5TFqg6KNxod9H/gPLxv45lutXf3VvBZTJpr1pxn7aLHlFzEyIgNZbP/N1QF44GSrN/k0DfL631sZjauUXaZXbi5xGsKKCYwJ1g3q587pi6mTdTV3n0hKgVuipO8hGy5++YeOv+hXsCxDwyZ+Shv+qavd/SapxYgCdEueuwONIFfsIsWCd3SCcjKXicTTEFMu8nvBmf7xuo2hv6vEOxoijlXV+4LkGrskdB8ZMg8PywEx6DLmDokgnAhTLrTc1ShbkOtQ3yNjjyFK7BDpqobsJal6d8SpbhccUJLepaSmsk0CgJsTjhAl6EwX0EYgTo3kP5fScqrbD8VwQaT8CcE4rCV4uQGNBF+hSOgBDADHtpTT1k4x+6FN5OeURpKAaIsoPHghkJ2lb6yWmESCa+DaR6GXAKlbd0L9UMcXLqnaCn4SpZvbf8hP4fJRgWdRl5uVN/rmyVbZLUVjM8NcVdFRIrTsNyu4mLBmydc3iA/90sCTEOj9e7DSvxLmmLFjpwM5xXLd6z0l6+9G+woNmARXVS3V/RryFntyKC3ATCqVlJoQBG45Tj2gMIunpadTJXWmdioooeGW3sLeUv5MM98mSB4SjKRlJqGPNjx5lO6MmJbZeXZ/L/aO6EsXUQD2h82Wphll4rpGYWPiHTCYqZYiqNYr6E3xUpzcvWVp3uCYVJWP6Ds117p7BoyKVz00yxC9ledF3eppktZWqFVowCMihQE3676L3DDTZsnJf1/8xKUh5U2Mj3lBvjlvCECKi00qo8b1mn/OklQjJ5T4WzTrH6X+/zpez8ZkmtcOayHdUKD/64roZ9dXbXG/hp5A+UWj8oSVYKg2QNAwAnZ+aiZ2KVRE/Y61DCgFg6Ccx/cAEQEAAYkBvAQYAQgAJhYhBPP69mjoLvUSTVGHuu8m9GgjQ/aSBQJfoUjoAhsMBQkDwmcAAAoJEO8m9GgjQ/aSIPcL/3jqL2A2SmC+s0BO4vMPEfCpa2gZ/vo1azzjUieZu5WhIxb5ik0V6T75EW5F0OeZj9qXI06gW+IM8+C6ImUgaR3l47UjBiBPq+uKO9QuT/nOtbSs2dXoTNCLMQN7MlrdUBix+lnqZZGSDgh6n/uVyAYw8Sh4c3/3thHUiR7xzVKGxAKDT8LoVjhHshTzYuQq8MqlfvwVI4eESLaryQ+Y+j5+VLDzSLgPAnnIqF/ui2JQjefJxm/VLoYNaPAGdqoz/u/R0Tmz94bZUfLjgQaDoUpnxYywK2JGlf3mPZ3PNWjxJzuQTF5Ge5bz/TylnRYIyBT7KD7oaKHO62fhDbYPJ4f94iZN4B6nnTAeP34zFDlkUbX4AHudXU7bvxT5OUk9x9c2tj7xwxQHaEhq2+JsYW0EVw27RLhbymnBfLjVVUktNF0nQGvU2TEocw4pr2ZkDHQkSnlbNa4kujlL7VzbpnEgyOmi5er9GaIuVSVADovBu+pz/Ov1y/3jUe8hZ/KleQ==:iQGzBAABCAAdFiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAmM9guIACgkQ7yb0aCND9pIEdwwAiLLqFlrKu0UsQebfuUP07cvGbYy9LfbCMsQj/3/pG/zl7q2mSl2YdXOalbaYD2uyGU/sm7J/+qQZXGyIjmDA7F53sNVAXTuYnrcKmYIzAmzW4lUAzfWA7yL55MtbR/eNUE1rqp/Gu/ejj1OedLyxi+tGFcXUHU0q8EnjQnzfHCJVzOa3PGMIX10NiXPjrF2pafAyE7q2ogwkKZdjJi+8tyAw0tviu4CRGOVlsNZlF+yxePZh55XdRZLCEt4n6mnJrccu0C22rM9R2dEReqGLAj8t/WhACI+UyNXtL+hICnu9y6wjk4spoMr0s0pqTQ76SMwfmRFzk11uZ+ge846hArcUxE27+AeBf9Q1IwT5Ypsc0Efm9ZPoJvA2ggcJv1Yyb58Ggfmd02xPW4EQ8MOEMLA/ZoAhOm3t3wATPNFG1ucm/o+NFNDpF7HNby+Savqv2NrbNwDMlWvFzRmER2+AIO0CIG2HVJScMEn7UkjF8jIm+ba3BIAXbz2FUZ3dytFF +> Content-Length: 3 +> Content-Type: application/x-www-form-urlencoded +> +} [3 bytes data] +* Mark bundle as not supporting multiuse +< HTTP/1.1 200 OK +< Server: tiny-http (Rust) +< Date: Wed, 5 Oct 2022 13:15:37 GMT +< Content-Type: text/plain; charset=UTF-8 +< Access-Control-Allow-Origin: * +< Access-Control-Allow-Methods: OPTIONS, PUT, GET +< Access-Control-Allow-Headers: Content-Type,Authorization,X-Filename +< Content-Length: 64 +< +{ [64 bytes data] +100 67 100 64 100 3 3548 166 --:--:-- --:--:-- --:--:-- 3722100 67 100 64 100 3 3534 165 --:--:-- --:--:-- --:--:-- 3722 +* Connection #0 to host localhost left intact +32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 +</span><span class="gp">$ </span>curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 +<span class="go">bar +</span><span class="gp">$ </span><span class="nb">echo</span> -n bar <span class="p">|</span> sha256sum +<span class="go">fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9 +</span><span class="gp">$ </span>curl -X GET http://localhost:8000/fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9 +<span class="go">bar</span> +</pre> +<p>Now, by changing the data and signature, the data under the mutable resource changes:</p> +<pre class="code console literal-block"> +<span class="gp">$ </span><span class="nv">sig</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -n <span class="s2">"xyzzy"</span> <span class="p">|</span> gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 <span class="p">|</span> base64 -w <span class="m">0</span><span class="sb">`</span> +<span class="gp">$ </span>curl -X PUT http://localhost:8000/foo -H <span class="s2">"Authorization: PUBSIG:</span><span class="nv">$pub</span><span class="s2">:</span><span class="nv">$sig</span><span class="s2">"</span> --data xyzzy +<span class="go">32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 +</span><span class="gp">$ </span>curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 +<span class="go">xyzzy</span> +</pre> +</div> +</div> +<div class="section" id="building-your-identity"> +<h2>Building your identity</h2> +<p>We can also recreate the symbolic link hash using local tools:</p> +<pre class="code console literal-block"> +<span class="gp">$ </span><span class="nv">t</span><span class="o">=</span><span class="sb">`</span>mktemp<span class="sb">`</span> +<span class="gp">$ </span><span class="nv">d</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -n foo <span class="p">|</span> sha256sum <span class="p">|</span> awk <span class="s1">'{ printf "%s",$1; }'</span> <span class="p">|</span> sed -e <span class="s1">'s/../\\\\x&/g'</span><span class="sb">`</span> +<span class="gp">$ </span><span class="nb">echo</span> -ne <span class="nv">$d</span> > <span class="nv">$t</span> +<span class="gp">$ </span><span class="nv">k</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -n F3FAF668E82EF5124D5187BAEF26F4682343F692 <span class="p">|</span> sed -e <span class="s1">'s/../\\\\x&/g'</span><span class="sb">`</span> +<span class="gp">$ </span><span class="nb">echo</span> -ne <span class="nv">$k</span> >> <span class="nv">$t</span> +<span class="gp">$ </span>cat <span class="nv">$t</span> <span class="p">|</span> sha256sum +<span class="go">32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 -</span> +</pre> +<p>In other words, the mutable reference is constructed using the following recipe.</p> +<ol class="arabic simple"> +<li>Calculate binary value <code>R</code> of the SHA256 digest of the keyword (<code>sha256(foo) -> 0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AE</code>)</li> +<li>Calculate binary value <code>K</code> of the key fingerprint (<code>0xF3FAF668E82EF5124D5187BAEF26F4682343F692</code>).</li> +<li>Calculate binary value of SHA256 digest of <code>R | K</code> (<code>sha256(0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AEF3FAF668E82EF5124D5187BAEF26F4682343F692) -> 0x32EA47ECC5A3EE2576AAB00C2F30EAABC2592D56E19F5C82FE4F7CF5874632B2</code>)</li> +</ol> +<p>That means that anyone who knows the keyword and the public key of the uploader can calculate the mutable reference themselves, and retrieve the data behind it.</p> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Refer to <a class="reference external" href="https://www.ethswarm.org/The-Book-of-Swarm.pdf">section 4.3 of The Book of Swarm</a> for a description of "feeds" and "single owner chunks." Swarm uses signatures to allow propagation of the data in the network. <code>wala</code> similarly uses signatures to accept update of resources "owned" by the holder of the private keys of an identity.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>At the time of writing <code>wala</code> is at version <code>0.1.1</code></td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>A custom authorization scheme <code>PUBSIG</code> has been invented for this purpose. The format is <code>Authorization: PUBSIG pgp:key-fingerprint-in-base64>:<signature-in-base64></code>. Currently this is strictly just authentication, as there is no feature in wala (yet) to use access control lists to determine which public keys to allow PUTs from.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-4" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-4">[4]</a></td><td>If you do not specify a keyword, the keyword value in the mutable resource reference will be the sha256 hash of an empty value (<code>0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</code>)</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-5" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-5">[5]</a></td><td><code>wala</code> comes with a binary tool <code>wala_send</code> that lets you specify a key fingerprint in your default pgp keyring and a keyword as arguments when you upload data. A lot less messy, but a lot less educational, too.</td></tr> +</tbody> +</table> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/406a858d431883e0f1e8e05ea87fae94b152a03a408a0d239e6868cef369f198.asc b/406a858d431883e0f1e8e05ea87fae94b152a03a408a0d239e6868cef369f198.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l30ACgkQzcvSTdHQ +4AGmCQgAl7JBjD2ha2ZDP8lhfCl13YPXfve7HblEaEKWsL/EbvvyvWLNpjHuPUY6 +AyLJiB44jxJ/qu8eQGKpeYh+nREhvxPXAwUz6lczitkKtqhjqydfR7h6BTJVMVzN +OUR3HaWVncesr/kRua+iyVpekLvthkDz7odq8emlA7LUsdNet4kgIw5eRSGb3AKX +opa8/eSx/lpDLsru/kKDyJg3Iz3YY6PxczL/rViAc5ykBc3OEhg56ID5COM3PDcB +FLi8q0eL9x1odK5b6hXYpDPMRoVP2uhrQZ7F9OZRH3jHrcQXDI3KeQUrzuy954GG +Wtvln5tV0ufWdCdLM+/uCjS9NmWJ5A== +=tc+T +-----END PGP SIGNATURE----- diff --git a/617878ecd470e80c0f5fadc0cf7e669ceea086a54343e56a35aeaf8131038d6c.asc b/617878ecd470e80c0f5fadc0cf7e669ceea086a54343e56a35aeaf8131038d6c.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBYACgkQzcvSTdHQ -4AH4LQf/QpPXHSeFij04B+m8/Msdzr6d3qNy94aGip1lceLPhxmbBi4/97R6KbtY -Ps3XjVw1+LGQQxs2VbWjSbyAIxXGxfcHaTkn6BVEY3d49TCMMeOc1Ac8W1ba7Ly5 -QXAJ5k8TJXiRSiAvEhpwjtIbov6YMVjFrj/n7lcF96p/aL6j7CO/f40gZEwkzP9s -Ey3TOj3cmS+8Ungl7qoUoGySWN21DrHLZ51oOa1HD1PegljrJu6kxn4GwVnc3U+X -rIXdeXFtynhNXr8hVPfIqLDUu8kdXA9I9hicrAa9J+1LqkzJZNKKT1MUEusp66Om -Mhx3kmyrqQ8xgnlCsOu5TAlU/La4Ng== -=afNx +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AElNAf/Y/vSHFCOOfL9wW8//IM7dsppsnrfGBvZMY/cpOEdHy/nK8Mxq/uuRkCw +XBsmSbBz4ENEDdCwVJCoTrho5jmD6O70b3sT25S/qRplwf7RGEcWypn+HHAfH/RV +d1dzdBqpC+vytwkjPKJ4C4Z5ymKBH+29XFqFPzWCgp4wuuy0vfrLcIo1V4OvdrYS +GDLZsV5fZCvwJA2gzxM5YWvOcMFghqvl0aH4fV0EnCyMGz0G91JCXD2leLIV0r5D +AkJfb2yI9PCEh309J1r1yuANLU13jZw/PpOUGARQrZGgVVoPzhYrn0W3EjMAcpb5 +Se48iYl/HmDhB73+GsYDUjKwhimvpA== +=syD4 -----END PGP SIGNATURE----- diff --git a/788f1df1d56d3aeb5a7374de267f9ebd41865fefabacd46810270f1596856a71.asc b/788f1df1d56d3aeb5a7374de267f9ebd41865fefabacd46810270f1596856a71.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBYACgkQzcvSTdHQ -4AGOegf/VrVUSyijWhQOI5H31/m/4zru8ry/xng/84b1Jr/T1hB96FrCyJHNXbqq -RaStZ5g2Pn2COPFlP1bL494YsFb4/9DbNhjyn3a6IAfSC4e/PolRNPNc86ymPOdG -82TQ0+LI40ZCmHTDbeLdxsUyC8L3zFOrkkg5M5pZHRERZ8L8+FNs6OEkTEeZE2FW -xGcQMu+RYzakJ+33CywfrEUDu1wXVji/6S6hulT0VKTc6i8jADSuZXirrwj7phk2 -nT+P0NP0X2Ybsz+H/Roz0qnXlO5V7ZIy7S7qpNHOja7NOyEKtHx/NLJGntysplMl -sirpAhmlQ/aEs0Fr82gKo7zB16VIpQ== -=tNZt +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AGUWQf/XPQwDHQDB787DRjGwZrrUf7Gl5IMwmHkBnwGRqL+r58I2QzM8gFJ/2Z5 +zUz66ADYpocd6R/ppbULB+brWHQKc+Qqq+zrJbSxEympb09Mvxq82yRTQ5z4UD/W +UvlTfPoDtq/+Pa4Y5ZiZCi3OhG7UPPtOEqmnAQGUgn2O+eDu6gCe1TmN9qSFHvY7 +xYIZscb5TVByRUWv+PTJHyohoinEXBDwPUlzrojH81PTWGF5T6OTxb8qydP9EGCr +WyRTYrvoN+tY3wqsz5UDNoSyxnIbww50u6xB7EFmapZhXF/tcGp7NYgg1WY8I8vD +BkGGyakddW+Lzu5eSU5cXYpwqyypTw== +=XXUN -----END PGP SIGNATURE----- diff --git a/80d5137e80648103735073b716f744b6e2210092a4c2ea556d87a931a3878c53.asc b/80d5137e80648103735073b716f744b6e2210092a4c2ea556d87a931a3878c53.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBYACgkQzcvSTdHQ -4AHTvgf/TnusNmm5xdYt5n9wv7D3C27bruLoKNl9ytXPFr3eqrTdXrHsjAjie5ue -4Iwe/g1KbEdH98i7amCGnSgp5UARxkDNHnX+5fbOsAve7AgFbMevkiLyF1ppR4oF -LJit4FXJKaZbHbG7UT3Jdrlz00SSFHHARfi/rYmV7xZgBuiM555hAGIKe9WG8PKD -TGVHKDjsY46JghiLydK++WDXQTcPDI0TR7hu26BfYeIfA8SRXvqxprjiwlA07T7K -tofxeXPphmBXsvH7m75uBaBzyxSqd1UAdYud6AW0YFwKYyuOj5QgTbOY5Xcy9bwW -aWfaOnCctDoq7jFMH67wayrPtj0LBw== -=QRPW +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AEcPAf9FJwZjK1ZquQlpuBIyRPlu2Fl6+3luZqw3dB+Tt+1A1/fk60kwmttP4i3 +nkw2zE6wrmq0TyIfVwvn5ghCteyQECs1ZywxYl+nlFZxCR3IbHR/LwKVOI0lKu4n +Q3c/6qDlpEt9gH6FgpdhhtUFpgAId9zDl7X6z6A6686Tqp07gNRSEgpCQJkCt+Cw +kwkNRQRnBw08bCiYTPYxPBreN7+bbxHyYoZW4lulAHXkAOTXv8athjwMXunWUmu9 +e7ImXnPhhoRl08xlSQMk8pwTaYe+/5k9EH4rW08c9lSpqZmecxMQj60GzeQXH478 +cNFURNgS2h3e2j2ehV9PxbFCp0RWng== +=8Q6x -----END PGP SIGNATURE----- diff --git a/92eb83e7074b2760e6001d40b17eecffb874b23f036abd5b6d2328f28f49bac8.asc b/92eb83e7074b2760e6001d40b17eecffb874b23f036abd5b6d2328f28f49bac8.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBYACgkQzcvSTdHQ -4AFLMwgAg7UzICQj5qPvbTaWb+rLLmIYWUDIb2yn9Ci251noBs3PtJP/66x05n3t -OhwjFhJhKu5B7qgT+i/TxPZAlUkwwXtokmByEF0ZFO69yMDLyr4boWOa0NeL8lUb -6vjB9+seiG9FFZ2aF+Xgm8qXT0mpxVHAguWpZUiYSoCXznuQVUpjuMFtnAz0S0y9 -7CTisrfZvdIyYsuObBOcOO2+l2uHv6Yd1AGbwf6nikYP5zF6sWcNr4IMuFkIAKRP -AtOUbccxBTtXo3Ww3QFe82KbnQea4LCat9MFwCGJ4K8BcHotMppYB1JHy7t75gl/ -B6CoWDqjLNikWYtoymBCUFAOdEovkg== -=gm4q +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AHGdwf/YlkaxnBmldzj7EAFsKv7rtDbQU3/PJ1TYYqKslS5d4u9Lxp6ZPzDqq5T +ljMRJZKYjwRZPhlrbhE6IOMZTNhTbt+fUBkn+/+sXUjOI1g7jiPwzBYUJd6YB5M4 +zQaOBw6Tei4mzOyl5uK9B2zecQSidbJ9hNPF15qQ4fAmdQaDq3qn8aedyP4k6e+7 +SxkuOowuOd13KlmkgX2Lbc4z3O5Y+gDmqrq/RHfx01WiLwkd8lJEjDsypKkw1JNA +fCnJhdBw8JoRJlhR7Xy0Q3guMImZFDaV7e0iePvChwonIwV6hUSwVwIsj+/9s+1D +0J6EXFHOWKM5T93EdJ/iWq75z5WyZQ== +=8U0u -----END PGP SIGNATURE----- diff --git a/archives.html b/archives.html @@ -29,6 +29,8 @@ <h1>Archives for man bytes gnu</h1> <dl> + <dt>Wed 05 October 2022</dt> + <dd><a href="./wala.html">Introducing Wala</a></dd> <dt>Sat 01 October 2022</dt> <dd><a href="./portable-book-metadata.html">A portable book metadata exercise</a></dd> <dt>Sat 15 January 2022</dt> diff --git a/author/louis-holbrook.html b/author/louis-holbrook.html @@ -31,6 +31,11 @@ <ol id="post-list"> <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../wala.html" rel="bookmark" title="Permalink to Introducing Wala">Introducing Wala</a></h3> </header> + <p><p class="first last">A simple HTTP content-addressed storage with cryptographic identity pointers.</p> +</p> + </article></li> + <li><article class="hentry"> <header> <h3 class="entry-title"><a href="../portable-book-metadata.html" rel="bookmark" title="Permalink to A portable book metadata exercise">A portable book metadata exercise</a></h3> </header> <p><p class="first last">Structured approach to generate portable metadata files for bibliographies and literature files using cryptographic hash mapping.</p> </p> @@ -75,11 +80,6 @@ <p><p class="first last">Generating proof of a web resource when you read and share</p> </p> </article></li> - <li><article class="hentry"> - <header> <h3 class="entry-title"><a href="../docker-offline-2-python.html" rel="bookmark" title="Permalink to Local python repository"> Offline Docker: Local python repository</a></h3> </header> - <p><p class="first last">Feeding python packages to your offline Docker setup</p> -</p> - </article></li> </ol><!-- /#posts-list --> <p class="paginator"> Page 1 / 2 diff --git a/author/louis-holbrook2.html b/author/louis-holbrook2.html @@ -31,6 +31,11 @@ <ol id="post-list"> <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../docker-offline-2-python.html" rel="bookmark" title="Permalink to Local python repository"> Offline Docker: Local python repository</a></h3> </header> + <p><p class="first last">Feeding python packages to your offline Docker setup</p> +</p> + </article></li> + <li><article class="hentry"> <header> <h3 class="entry-title"><a href="../docker-offline-1-routing.html" rel="bookmark" title="Permalink to The routing to freedom"> Offline Docker: The routing to freedom</a></h3> </header> <p><p class="first last">How to not be forced being online when forced to use docker</p> </p> diff --git a/authors.html b/authors.html @@ -28,7 +28,7 @@ </nav><!-- /#menu --> <h1>Authors on man bytes gnu</h1> <ul> - <li><a href="./author/louis-holbrook.html">Louis Holbrook</a> (11)</li> + <li><a href="./author/louis-holbrook.html">Louis Holbrook</a> (12)</li> </ul> <footer id="contentinfo" class="body"> <hr/> diff --git a/c38ce3a51eeaa98f5524cf9902e84ee8ba5d179d6fbe18c49f9707554c896b6e.asc b/c38ce3a51eeaa98f5524cf9902e84ee8ba5d179d6fbe18c49f9707554c896b6e.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBYACgkQzcvSTdHQ -4AGNKAf+MW45UwKWtcLlsRbm6Vdx9HpUB0ys7GLDnJQOZrmCxYkKF2BXoEfyiw32 -v6zQqdzPbE+GGdF5Cd5biEWvcJ+NzxLzNrlesg90tZ4WPlCK/jrsUsk/2PCRnkXy -l7XCko+IzEwOmDVULn7RU+kwcihRMvaU1PUpBMmJd659Ejdvapp1C6i/onhyshWf -NhXEvDjvXbepPTJGn6mcOgFIWCT16seBej4hPzpa74f64ojcRX2khlOlCCsava1X -EL+ip6e6TqOg22Eqb29k51xx/ZQSttyIj1Jj1JvPF9QM+r5wLNhVuONY4qFSLwK8 -+VrqFVTM7Xq+gORpTV+ZHACImw/u6g== -=FqT6 +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AFgiggAniLCxjPu5gJrTQpU5LlHp9/8AuymMWtb+7j2ZBZWwAjbIe93Vml7yQJF +Nclo/y9trkehOjCNQfvZiMIVg3MnKLpVml+VKX2hIgANWoCC+iXcAIApN+oJb7NQ +PSf+jcZPd857YYU5IrdXSuSK6y5+bHZDhDxGmVAfOwxwA4ZesK5zviMEPmWojyQ4 +3r3XOBCZMid2qBma2/xYLljFOwzlzERlMl00t8vRyPvbN2UpGz6ibcQ58g1xEa/1 +hJpBOG3dsDB+sQsiNI4buz+i3bf7EXdBcyv0Reo8QFu0a+/wDASecAYPZ1nblIPq +/dxzi3ozb6pbZI+oJ/vexEhlwHJQHA== +=wx/E -----END PGP SIGNATURE----- diff --git a/categories.html b/categories.html @@ -29,7 +29,7 @@ <h1>Categories on man bytes gnu</h1> <ul> <li><a href="./category/archiving.html">Archiving</a> (3)</li> - <li><a href="./category/code.html">Code</a> (3)</li> + <li><a href="./category/code.html">Code</a> (4)</li> <li><a href="./category/hygiene.html">Hygiene</a> (2)</li> <li><a href="./category/offlining.html">Offlining</a> (3)</li> </ul> diff --git a/category/code.html b/category/code.html @@ -31,6 +31,11 @@ <ol id="post-list"> <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../wala.html" rel="bookmark" title="Permalink to Introducing Wala">Introducing Wala</a></h3> </header> + <p><p class="first last">A simple HTTP content-addressed storage with cryptographic identity pointers.</p> +</p> + </article></li> + <li><article class="hentry"> <header> <h3 class="entry-title"><a href="../eth-log-bloom.html" rel="bookmark" title="Permalink to Making sense of Ethereum log bloom filters">Making sense of Ethereum log bloom filters</a></h3> </header> <p><p class="first last">How to generate values to match the log bloom filters in Ethereum</p> </p> diff --git a/cd80d0ad6005957cb7d4d6f35f4db78435728bef1fc3a2c3476697c343f7683b b/cd80d0ad6005957cb7d4d6f35f4db78435728bef1fc3a2c3476697c343f7683b @@ -1,246 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> - <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> - <title>man bytes gnu - A portable book metadata exercise</title> - <meta charset="utf-8" /> - <meta name="generator" content="Pelican" /> - -</head> - -<body id="index" class="home"> - - <header id="banner" class="body"> - <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> - </header> - <nav id="menu"> - <ul> - <li class="active"><a href="./category/archiving.html">Archiving</a></li> - <li><a href="./category/code.html">Code</a></li> - <li><a href="./category/hygiene.html">Hygiene</a></li> - <li><a href="./category/offlining.html">Offlining</a></li> - <li><a href="./pages/identities.html">Identities</a></li> - <li><a href="./pages/shares.html">Shares</a></li> - - <li><a href="/tags.html">tags</a></li> - </ul> - <hr/> - </nav><!-- /#menu --> - <header> - <h1 class="entry-title top-body-title"> - <a href="./portable-book-metadata.html" rel="bookmark" - title="Permalink to A portable book metadata exercise">A portable book metadata exercise</a></h1> - - - <div class="category meta"> - Posted -<time class="published" datetime="2022-10-01T12:40:00+02:00"> - Sat 01 October 2022 - </time> -in <a class="category" href="./category/archiving.html">archiving</a> - <a href="./tag/hash.html">hash</a> - <a href="./tag/kitab.html">kitab</a> - <a href="./tag/literature.html">literature</a> - <a href="./tag/metadata.html">metadata</a> - <a href="./tag/dublincore.html">dublincore</a> - <a href="./tag/libgen.html">libgen</a> - </div> - - <div class="neighbors meta"> - <ul> - <li> - Previous: <a href="./backup-rsync-duplicity.html"> - Combining duplicity and rsync - </a> - </li> - </ul> - </div> - <div class="meta"> -<hr/> - </div> -</header> - -<section id="content" class="body"> - <div class="entry-content"> - <p>One of the things I have been working on the last few weeks is a rust application I have dubbed <a class="reference external" href="https://git.defalsify.net/kitab">kitab</a> <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p> -<p>In short, the application makes it easy to extract literary metadata to a separate file structure.</p> -<p>The metadata can in turn be applied as <em>extended attributes</em> recursively on a directory for files that match.</p> -<p>The way it's accomplished it simple: The file name of the metadata is the hex representation of the digest of the file. The same digest is used to match files to metadata when applying it back to the file.</p> -<p>There are two advantages to this:</p> -<ol class="arabic simple"> -<li>The digest of the media file need not be affected by the metadata, i.e. by embedding metadata in the file itself.</li> -<li>You do not need to use the file name to keep record of what a file is.</li> -</ol> -<div class="section" id="yarr-ye-matey-data"> -<h2>Yarr, ye matey-data</h2> -<p>Let's demonstrate with an example.</p> -<p>The fabulous <a class="reference external" href="https://libgen.rs">Library Genesis</a> project has made available an endpoint to retrieve <tt class="docutils literal">bibtex</tt> entries based on the <tt class="docutils literal">md5</tt> hash of the book media file.</p> -<p>A version of the <a class="reference external" href="https://libgen.rs/book/index.php?md5=BCD99F1AB4155F2A2A362E5B7938A852">Bitcoin White Paper</a>, under the <code>md5</code> hash <code>bcd99f1ab4155f2a2a362e5b7938a852</code>, can be found there.</p> -<p>If you download this file using a synchronous download link, the browser will provide you with a filename to go with the download.</p> -<p>However, if you use the torrent alternative, the filename will be the <tt class="docutils literal">md5</tt> hash itself. If you are torrenting a bunch of those files, it quickly becomes a nuisance to distinguish them.</p> -<p>And, of course: In either case there is no guarantee the any metadata comes with the file.</p> -<div class="section" id="inside-the-book"> -<h3>Inside the book</h3> -<p>Kitab (v0.0.2) is able to read metadata from both a bibtex source and xattr entries on a file, as well as its native <a class="reference external" href="https://www.w3.org/TR/turtle/">rdf-turtle</a> format.</p> -<p>In kitab's data store, every media file entity in rdf-turtle is keyed with a <a class="reference external" href="https://www.rfc-editor.org/info/rfc8141">URN</a> specifying a digest for the file.</p> -<p>To see exactly what that looks like, let's download and import the bibtex metadata for the paper <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>:</p> -<pre class="code bash literal-block"> -<span class="nv">bibtex_file</span><span class="o">=</span><span class="sb">`</span>mktemp<span class="sb">`</span> -<span class="nv">kitab_dir</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> -curl -s -X GET https://libgen.rs/book/bibtex.php?md5<span class="o">=</span>BCD99F1AB4155F2A2A362E5B7938A852 -o <span class="nv">$bibtex_file</span> -kitab --store <span class="nv">$kitab_dir</span> import --digest md5:BCD99F1AB4155F2A2A362E5B7938A852 <span class="nv">$bibtex_file</span> -cat <span class="nv">$kitab_dir</span>/* -</pre> -<p>The output of the above should be:</p> -<pre class="code turtle literal-block"> -<span class="nv"><URN:md5:bcd99f1ab4155f2a2a362e5b7938a852></span> <span class="nv"><https://purl.org/dc/terms/title></span> <span class="s">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> -<span class="nv"><https://purl.org/dc/terms/creator></span> <span class="s">"Satoshi Nakamoto"</span> <span class="p">;</span> -<span class="nv"><https://purl.org/dc/terms/type></span> <span class="s">"book"</span> <span class="p">.</span> -</pre> -<p>Now let's say the media file itself has been downloaded to <tt class="docutils literal"><span class="pre">~/.local/share/transmission</span></tt>. We can apply this metadata as extended attributes.</p> -<p>This time we turn on logging to see what's going on:</p> -<pre class="code console literal-block"> -<span class="gp">$ </span><span class="nv">RUST_LOG</span><span class="o">=</span>info kitab --store <span class="nv">$kitab_dir</span> apply --digest md5 ~/.local/share/transmission -<span class="go">[2022-10-01T11:14:59Z INFO kitab] have index directory "/tmp/tmp.r0jBm6q4hW" -[2022-10-01T11:14:59Z INFO kitab] using digest type md5 -[2022-10-01T11:14:59Z INFO kitab] apply from path "/home/lash/.local/share/transmission/" -[2022-10-01T11:14:59Z INFO kitab] apply DirEntry("/home/lash/.local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852") -> title "Bitcoin: A Peer-to-Peer Electronic Cash System" author "Satoshi Nakamoto" digest md5:bcd99f1ab4155f2a2a362e5b7938a852 - -</span><span class="gp">$ </span>find ~/.local/share/transmission -type f -regextype sed -regex <span class="s2">".*/[a-f0-9]\{32\}</span>$<span class="s2">"</span> -exec getfattr -d <span class="o">{}</span> <span class="se">\;</span> -<span class="gp"># </span>file: .local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852 -<span class="go">user.dcterms:creator="Satoshi Nakamoto" -user.dcterms:title="Bitcoin: A Peer-to-Peer Electronic Cash System" -user.dcterms:type="book"</span> -</pre> -</div> -<div class="section" id="let-the-right-one-in"> -<h3>Let the right one in</h3> -<p>Conversely, the metadata can be re-imported directly from the extended attributes. And this time, let's store it both under the <tt class="docutils literal">md5</tt> and the <tt class="docutils literal">sha512</tt> hash:</p> -<pre class="code bash literal-block"> -$ <span class="nv">kitab_dir_new</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> -$ kitab --store <span class="nv">$kitab_dir_new</span> import --digest md5 --digest sha512 .local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852 -$ find <span class="nv">$kitab_dir_new</span> -type f -exec cat <span class="o">{}</span> <span class="se">\;</span> -/tmp/tmp.B6j41YMmEM/493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792 -<URN:sha512:493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792> <https://purl.org/dc/terms/title> <span class="s2">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> - <https://purl.org/dc/terms/creator> <span class="s2">"Satoshi Nakamoto"</span> <span class="p">;</span> - <https://purl.org/dc/terms/type> <span class="s2">"book"</span> <span class="p">;</span> - <https://purl.org/dc/terms/MediaType> <span class="s2">"application/epub+zip"</span> . -/tmp/tmp.B6j41YMmEM/bcd99f1ab4155f2a2a362e5b7938a852 -<URN:md5:bcd99f1ab4155f2a2a362e5b7938a852> <https://purl.org/dc/terms/title> <span class="s2">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> - <https://purl.org/dc/terms/creator> <span class="s2">"Satoshi Nakamoto"</span> <span class="p">;</span> - <https://purl.org/dc/terms/type> <span class="s2">"book"</span> <span class="p">;</span> - <https://purl.org/dc/terms/MediaType> <span class="s2">"application/epub+zip"</span> . -</pre> -</div> -</div> -<div class="section" id="level-up"> -<h2>Level up</h2> -<p>Finally, a bash script <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a> example that lets you retrieve and apply metadata for a batch of files found in the directory given as the <em>first positional arg</em>.</p> -<p>This script even renames the files according to the metadata applied.</p> -<pre class="code bash literal-block"> -<span class="ln"> 0 </span><span class="c1"># NOTE! this will only work if your fs supports xattr. -</span><span class="ln"> 1 </span><span class="c1"># That's why we cannot use tmpfs (mktemp) here; tmpfs does not support xattr. -</span><span class="ln"> 2 </span><span class="c1"></span> -<span class="ln"> 3 </span><span class="c1"># directory to copy media files to -</span><span class="ln"> 4 </span><span class="c1"></span><span class="nv">outdir</span><span class="o">=</span>./<span class="k">$(</span>uuidgen<span class="k">)</span> -<span class="ln"> 5 </span>mkdir -vp <span class="nv">$outdir</span> -<span class="ln"> 6 </span> -<span class="ln"> 7 </span><span class="c1"># Input dir is the first positional arg. -</span><span class="ln"> 8 </span><span class="c1"></span><span class="nv">indir</span><span class="o">=</span><span class="nv">$1</span> -<span class="ln"> 9 </span> -<span class="ln">10 </span><span class="nv">IFS</span><span class="o">=</span><span class="s1">$'\n'</span> -<span class="ln">11 </span> -<span class="ln">12 </span><span class="c1"># Retrieve metadata for each file and import it into the kitab store. -</span><span class="ln">13 </span><span class="c1"># Also copy the media file to the separate output directory. -</span><span class="ln">14 </span><span class="c1"></span><span class="k">for</span> f <span class="k">in</span> <span class="k">$(</span>find <span class="nv">$indir</span> -type f<span class="k">)</span><span class="p">;</span> <span class="k">do</span> -<span class="ln">15 </span> <span class="nv">sum</span><span class="o">=</span><span class="k">$(</span>md5sum <span class="nv">$f</span> <span class="p">|</span> awk <span class="s1">'{print $1;}'</span><span class="k">)</span> -<span class="ln">16 </span> <span class="nb">echo</span> <span class="s2">"downloading metadata for </span><span class="nv">$indir</span><span class="s2">/</span><span class="nv">$f</span><span class="s2">"</span> -<span class="ln">17 </span> <span class="nv">srct</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span> -<span class="ln">18 </span> curl -s -X GET https://libgen.rs/book/bibtex.php?md5<span class="o">=</span><span class="nv">$sum</span> -o <span class="nv">$srct</span> -<span class="ln">19 </span> <span class="nv">dstt</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span> -<span class="ln">20 </span> xmllint --html --xpath <span class="s1">'string(/html/body/textarea[@id="bibtext"])'</span> <span class="nv">$srct</span> > <span class="nv">$dstt</span> -<span class="ln">21 </span> kitab import --digest md5:<span class="nv">$sum</span> <span class="nv">$dstt</span> -<span class="ln">22 </span> cp <span class="nv">$f</span> <span class="nv">$outdir</span>/ -<span class="ln">23 </span><span class="k">done</span> -<span class="ln">24 </span> -<span class="ln">25 </span><span class="c1"># Apply metadata imported from bibtex as xattr for the media files. -</span><span class="ln">26 </span><span class="c1"></span><span class="nv">RUST_LOG</span><span class="o">=</span>info kitab apply --digest md5 <span class="nv">$outdir</span>/ -<span class="ln">27 </span> -<span class="ln">28 </span><span class="c1"># Rename the files according to the metadata title and media type. -</span><span class="ln">29 </span><span class="c1"></span><span class="k">for</span> f <span class="k">in</span> <span class="k">$(</span>ls <span class="nv">$outdir</span><span class="k">)</span><span class="p">;</span> <span class="k">do</span> -<span class="ln">30 </span> <span class="nv">title</span><span class="o">=</span><span class="k">$(</span>getfattr --only-values -n user.dcterms:title <span class="nv">$outdir</span>/<span class="nv">$f</span><span class="k">)</span> -<span class="ln">31 </span> -<span class="ln">32 </span> <span class="nv">f_typ</span><span class="o">=</span><span class="k">$(</span>file -b --mime-type <span class="nv">$outdir</span>/<span class="nv">$f</span><span class="k">)</span> -<span class="ln">33 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">""</span> -<span class="ln">34 </span> <span class="k">case</span> <span class="s2">"</span><span class="nv">$f_typ</span><span class="s2">"</span> <span class="k">in</span> -<span class="ln">35 </span> <span class="s2">"application/pdf"</span><span class="o">)</span> -<span class="ln">36 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".pdf"</span> -<span class="ln">37 </span> <span class="p">;;</span> -<span class="ln">38 </span> <span class="s2">"application/epub+zip"</span><span class="o">)</span> -<span class="ln">39 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".epub"</span> -<span class="ln">40 </span> <span class="p">;;</span> -<span class="ln">41 </span> <span class="s2">"application/x-mobipocket-ebook"</span><span class="o">)</span> -<span class="ln">42 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".mobi"</span> -<span class="ln">43 </span> <span class="p">;;</span> -<span class="ln">44 </span> <span class="s2">"text/plain"</span><span class="o">)</span> -<span class="ln">45 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".txt"</span> -<span class="ln">46 </span> <span class="p">;;</span> -<span class="ln">47 </span> <span class="s2">"text/html"</span><span class="o">)</span> -<span class="ln">48 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".html"</span> -<span class="ln">49 </span> <span class="p">;;</span> -<span class="ln">50 </span> *<span class="o">)</span> -<span class="ln">51 </span> ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> unhandled mime <span class="nb">type</span> <span class="nv">$f_typ</span> -<span class="ln">52 </span> <span class="nb">exit</span> <span class="m">1</span> -<span class="ln">53 </span> <span class="k">esac</span> -<span class="ln">54 </span> mv -v <span class="nv">$outdir</span>/<span class="nv">$f</span> <span class="nv">$outdir</span>/<span class="si">${</span><span class="nv">title</span><span class="si">}${</span><span class="nv">f_ext</span><span class="si">}</span> -<span class="ln">55 </span><span class="k">done</span> -</pre> -<p>This last example will result in:</p> -<ul class="simple"> -<li>A media file named <tt class="docutils literal">$outdir/Bitcoin: A <span class="pre">Peer-to-Peer</span> Electronic Cash System.epub</tt></li> -<li>... with metadata applied as extended attributes</li> -<li>An rdf-turtle metadata entry in <tt class="docutils literal"><span class="pre">~/.local/share/kitab/idx/bcd99f1ab4155f2a2a362e5b7938a852</span></tt></li> -</ul> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>The relevant documentation for <tt class="docutils literal">kitab</tt> at the time of writing is <a class="reference external" href="https://defalsify.org/doc/crates/kitab/0.0.2/kitab/">here</a>. To build kitab, simply <em>clone</em> the repository and build with <code>cargo build --all-features</code>.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>The <code>kitab</code> command in the script assumes you have built the <em>kitab binary</em> and made it available in your path.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>the script uses <code>xmllint</code> which on archlinux is provided by the <tt class="docutils literal">libxml2</tt> package.</td></tr> -</tbody> -</table> -</blockquote> -</div> - - </div><!-- /.entry-content --> - <footer class="meta"> - </footer> - -</section> - <footer id="contentinfo" class="body"> - <hr/> - <address id="about" class="vcard body"> - Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> - </address><!-- /#about --> - </footer><!-- /#contentinfo --> -</body> -</html> -\ No newline at end of file diff --git a/cd80d0ad6005957cb7d4d6f35f4db78435728bef1fc3a2c3476697c343f7683b.asc b/cd80d0ad6005957cb7d4d6f35f4db78435728bef1fc3a2c3476697c343f7683b.asc @@ -1,11 +0,0 @@ ------BEGIN PGP SIGNATURE----- - -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBEACgkQzcvSTdHQ -4AF87gf/fIP9XeRpslIvWfTS2Fy4jmdWsnW4uXb3A5Q1R5ONosj2sklnV3RaHbF4 -HhV0qzk9p3HZWGbtHDyj/aA/zJphqhkAhbkTvzAoWBapVqFuLQhb8aAKwU+vbqsA -uNc/NtunLzlUlR8HHGQPDsE/PzmYKyXVAoaiIc6EoDSKVGILEjJMRrApIY5+0pc3 -eRg/WvhaouJfhkK7rQDjFMEWUSz2xsFZM7GOxRA5lg3FPIVV1B+z+i6J3V2HsmM3 -S+5qMDManFCFHMsfRLVgJ012S5XzMbZE4ZshhpahA4FlD77JAsv3d0HC6cKUZ4Tq -HU2NlVCwXNonx6oSIP1HhntVjOLiRw== -=ha7U ------END PGP SIGNATURE----- diff --git a/dab643d2a647cfd037878b4fd6bd447457b8b568c3992f411350fc88977d4d88.asc b/dab643d2a647cfd037878b4fd6bd447457b8b568c3992f411350fc88977d4d88.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBYACgkQzcvSTdHQ -4AFUmAgAp0g2dUiky9TZIr6Ll/w2jG+6s3Ysf+3iw0QhmPjm+YC2uyTHIx/aQXne -T0MnwxvXnSyg5uQSrDP8mfp0EXxEAF+Apyk6PSeTYL3Yvyq2RPG9WP+74X9Ex9BG -L66prbz9BoFxMHP6pJgC6ThS1hvUkbHPjpP/p4qZNrAPhiGQd6a0AFiGUHqURs9E -cnmuChyXNWmpPsh1ba91aLP6uYEHN9RPl49hdz98CbBS1RuDnTqlHeHasFfK8iXN -7qNZPmeM4ZlGdB6vzajUYNJxpEBm/akNkg1J5EqFL5BHtKWaRbENaonXDMgZkF58 -eSIhGDsdSRSsbuUu89quUI6I8flG/w== -=vmYR +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AEEEQf/RiWLqHxYmr7qSKfch2qkKjKOWPc2sGNsc74DVBx/8IRJ+ZCq85lJlY1V +BOBQP00P+gsgwBMM49ZM2FCmt/e5QVpfBBmXTjYMrdrnBjgZPUsgl2XMqes8yku8 +uQ9aFGdeycQwFaN+008bXaRKTBMPE78G35jO8KsfBhqPTh/P8hdxnJbvwT3yCBDS +Fin5iPADaOKGeQYpSoLmt4zSGvicagS+Z5+zW6O+mLxunRk3AG/vfuW65Zjton8k +OWp3Pp/eIzdTLH5WSK6lKFZ538RYV94XHbQEk5c12jxCWrBAPdMtOqrhIM8UPD6e +ZXdv+Y4DK12TTEAJ7QjMBDdh8W19/g== +=9HWD -----END PGP SIGNATURE----- diff --git a/f72f88a69eb494ae7f8cbf441683bf156bebf44039b25e723d03d5ed44e0c3e4.asc b/f72f88a69eb494ae7f8cbf441683bf156bebf44039b25e723d03d5ed44e0c3e4.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4RBYACgkQzcvSTdHQ -4AHmUwf/RO4BbdSG61qBeGtopARvlMIpDpzYFV5ZcPjKnseCK+VDpL8F5o2VvWqw -b48gUenSyTySAqM2gAlYdfeSuf9eiIII97GmisK/mCIBsWejePrzRUSJNq3GpuUq -C9KKvP6zn+5eZcRHb3c8wK3M6bW1llIR1/jU9pQelCRuFU19KliorbkrjoE746Qu -a/GMRKVEDax9cHoJlTwhQ9f2JllSNo6GueKkcMOF9U6bRj3855PJ80JUFUxO1ntD -akFq6XBsFZhB9qMQM1yDYBd2PrDWKJ0wuofbCZfTB9GRWf6x0cdwXSOl/9B1sXA4 -MUs/UCiOZQU883CHRWvbd0OU4f2d6Q== -=ugtt +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM9l4EACgkQzcvSTdHQ +4AH4nQf9FThWQS5e+FFZJh05cQt0xdvqiP8Mg0DD3bdnRydva6PgOED9E5dZ4kpU +nIlUBZMN/UfO7/i78FLHBPQucDpz90ZkY+BPIqSoXsytlc0+xFneeV/PU5fY7Bfy +y7D35uVJnPGAcTwxJ3bvtr80ROtLuYtFvZ0/Qxar6ba0X8NA1if8A+QyXyGt2kEc +gu6T91hFTSwb5s+xjPcVFrAMd/IRHd++I1GB4aYDc5zPD0OoSEr7PDy5+Mh04fAQ +aFKjZEC6in6F9/ldMezbZpNbiJkiw/bkoCnFu/CP6ErMU4EvXp6X3uVnxMtnqmU0 +nNujs3acmevC08lJxsXTgd7dvoBEQg== +=vnfl -----END PGP SIGNATURE----- diff --git a/index.html b/index.html @@ -31,6 +31,11 @@ <ol id="post-list"> <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="./wala.html" rel="bookmark" title="Permalink to Introducing Wala">Introducing Wala</a></h3> </header> + <p><p class="first last">A simple HTTP content-addressed storage with cryptographic identity pointers.</p> +</p> + </article></li> + <li><article class="hentry"> <header> <h3 class="entry-title"><a href="./portable-book-metadata.html" rel="bookmark" title="Permalink to A portable book metadata exercise">A portable book metadata exercise</a></h3> </header> <p><p class="first last">Structured approach to generate portable metadata files for bibliographies and literature files using cryptographic hash mapping.</p> </p> @@ -75,11 +80,6 @@ <p><p class="first last">Generating proof of a web resource when you read and share</p> </p> </article></li> - <li><article class="hentry"> - <header> <h3 class="entry-title"><a href="./docker-offline-2-python.html" rel="bookmark" title="Permalink to Local python repository"> Offline Docker: Local python repository</a></h3> </header> - <p><p class="first last">Feeding python packages to your offline Docker setup</p> -</p> - </article></li> </ol><!-- /#posts-list --> <p class="paginator"> Page 1 / 2 diff --git a/index2.html b/index2.html @@ -31,6 +31,11 @@ <ol id="post-list"> <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="./docker-offline-2-python.html" rel="bookmark" title="Permalink to Local python repository"> Offline Docker: Local python repository</a></h3> </header> + <p><p class="first last">Feeding python packages to your offline Docker setup</p> +</p> + </article></li> + <li><article class="hentry"> <header> <h3 class="entry-title"><a href="./docker-offline-1-routing.html" rel="bookmark" title="Permalink to The routing to freedom"> Offline Docker: The routing to freedom</a></h3> </header> <p><p class="first last">How to not be forced being online when forced to use docker</p> </p> diff --git a/portable-book-metadata.html b/portable-book-metadata.html @@ -54,6 +54,13 @@ in <a class="category" href="./category/archiving.html">archiving</a> Combining duplicity and rsync </a> </li> + | + + <li> + Next: <a href="./wala.html"> + Introducing Wala + </a> + </li> </ul> </div> <div class="meta"> diff --git a/portable-book-metadata.sha256 b/portable-book-metadata.sha256 @@ -1 +1 @@ -cd80d0ad6005957cb7d4d6f35f4db78435728bef1fc3a2c3476697c343f7683b portable-book-metadata.html -\ No newline at end of file +1e676d28d8bb972218b427f8f01ba31ced64f4f2390249f337f1d1e7c2035825 portable-book-metadata.html +\ No newline at end of file diff --git a/tag/crypto.html b/tag/crypto.html @@ -31,6 +31,11 @@ <ol id="post-list"> <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../wala.html" rel="bookmark" title="Permalink to Introducing Wala">Introducing Wala</a></h3> </header> + <p><p class="first last">A simple HTTP content-addressed storage with cryptographic identity pointers.</p> +</p> + </article></li> + <li><article class="hentry"> <header> <h3 class="entry-title"><a href="../eth-log-bloom.html" rel="bookmark" title="Permalink to Making sense of Ethereum log bloom filters">Making sense of Ethereum log bloom filters</a></h3> </header> <p><p class="first last">How to generate values to match the log bloom filters in Ethereum</p> </p> diff --git a/tag/http.html b/tag/http.html @@ -0,0 +1,47 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="../theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="../theme/css/syntax.css" /> + <title>man bytes gnu - http tag</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="../">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="../category/archiving.html">Archiving</a></li> + <li><a href="../category/code.html">Code</a></li> + <li><a href="../category/hygiene.html">Hygiene</a></li> + <li><a href="../category/offlining.html">Offlining</a></li> + <li><a href="../pages/identities.html">Identities</a></li> + <li><a href="../pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> +<section id="content"> +<h1 class="top-body-title">Tag: http</h1> + +<ol id="post-list"> + <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../wala.html" rel="bookmark" title="Permalink to Introducing Wala">Introducing Wala</a></h3> </header> + <p><p class="first last">A simple HTTP content-addressed storage with cryptographic identity pointers.</p> +</p> + </article></li> +</ol><!-- /#posts-list --> +</section><!-- /#content --> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/tag/pgp.html b/tag/pgp.html @@ -0,0 +1,47 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="../theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="../theme/css/syntax.css" /> + <title>man bytes gnu - pgp tag</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="../">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="../category/archiving.html">Archiving</a></li> + <li><a href="../category/code.html">Code</a></li> + <li><a href="../category/hygiene.html">Hygiene</a></li> + <li><a href="../category/offlining.html">Offlining</a></li> + <li><a href="../pages/identities.html">Identities</a></li> + <li><a href="../pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> +<section id="content"> +<h1 class="top-body-title">Tag: pgp</h1> + +<ol id="post-list"> + <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../wala.html" rel="bookmark" title="Permalink to Introducing Wala">Introducing Wala</a></h3> </header> + <p><p class="first last">A simple HTTP content-addressed storage with cryptographic identity pointers.</p> +</p> + </article></li> +</ol><!-- /#posts-list --> +</section><!-- /#content --> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/tag/rust.html b/tag/rust.html @@ -0,0 +1,47 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="../theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="../theme/css/syntax.css" /> + <title>man bytes gnu - rust tag</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="../">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="../category/archiving.html">Archiving</a></li> + <li><a href="../category/code.html">Code</a></li> + <li><a href="../category/hygiene.html">Hygiene</a></li> + <li><a href="../category/offlining.html">Offlining</a></li> + <li><a href="../pages/identities.html">Identities</a></li> + <li><a href="../pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> +<section id="content"> +<h1 class="top-body-title">Tag: rust</h1> + +<ol id="post-list"> + <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../wala.html" rel="bookmark" title="Permalink to Introducing Wala">Introducing Wala</a></h3> </header> + <p><p class="first last">A simple HTTP content-addressed storage with cryptographic identity pointers.</p> +</p> + </article></li> +</ol><!-- /#posts-list --> +</section><!-- /#content --> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/tag/storage.html b/tag/storage.html @@ -0,0 +1,47 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="../theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="../theme/css/syntax.css" /> + <title>man bytes gnu - storage tag</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="../">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="../category/archiving.html">Archiving</a></li> + <li><a href="../category/code.html">Code</a></li> + <li><a href="../category/hygiene.html">Hygiene</a></li> + <li><a href="../category/offlining.html">Offlining</a></li> + <li><a href="../pages/identities.html">Identities</a></li> + <li><a href="../pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> +<section id="content"> +<h1 class="top-body-title">Tag: storage</h1> + +<ol id="post-list"> + <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../wala.html" rel="bookmark" title="Permalink to Introducing Wala">Introducing Wala</a></h3> </header> + <p><p class="first last">A simple HTTP content-addressed storage with cryptographic identity pointers.</p> +</p> + </article></li> +</ol><!-- /#posts-list --> +</section><!-- /#content --> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/tag/wala.html b/tag/wala.html @@ -0,0 +1,47 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="../theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="../theme/css/syntax.css" /> + <title>man bytes gnu - wala tag</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="../">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="../category/archiving.html">Archiving</a></li> + <li><a href="../category/code.html">Code</a></li> + <li><a href="../category/hygiene.html">Hygiene</a></li> + <li><a href="../category/offlining.html">Offlining</a></li> + <li><a href="../pages/identities.html">Identities</a></li> + <li><a href="../pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> +<section id="content"> +<h1 class="top-body-title">Tag: wala</h1> + +<ol id="post-list"> + <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../wala.html" rel="bookmark" title="Permalink to Introducing Wala">Introducing Wala</a></h3> </header> + <p><p class="first last">A simple HTTP content-addressed storage with cryptographic identity pointers.</p> +</p> + </article></li> +</ol><!-- /#posts-list --> +</section><!-- /#content --> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/tags.html b/tags.html @@ -36,7 +36,7 @@ <li><a href="./tag/clang.html">clang</a> (1)</li> <li><a href="./tag/community.html">community</a> (1)</li> <li><a href="./tag/crm.html">crm</a> (1)</li> - <li><a href="./tag/crypto.html">crypto</a> (1)</li> + <li><a href="./tag/crypto.html">crypto</a> (2)</li> <li><a href="./tag/decentralized-storage.html">decentralized storage</a> (1)</li> <li><a href="./tag/devops.html">devops</a> (2)</li> <li><a href="./tag/docker.html">docker</a> (3)</li> @@ -46,6 +46,7 @@ <li><a href="./tag/evm.html">evm</a> (1)</li> <li><a href="./tag/f2f.html">f2f</a> (1)</li> <li><a href="./tag/hash.html">hash</a> (3)</li> + <li><a href="./tag/http.html">http</a> (1)</li> <li><a href="./tag/i3wm.html">i3wm</a> (1)</li> <li><a href="./tag/iproute.html">iproute</a> (1)</li> <li><a href="./tag/iptables.html">iptables</a> (1)</li> @@ -63,12 +64,16 @@ <li><a href="./tag/npm.html">npm</a> (1)</li> <li><a href="./tag/openvpn.html">openvpn</a> (1)</li> <li><a href="./tag/p2p.html">p2p</a> (1)</li> + <li><a href="./tag/pgp.html">pgp</a> (1)</li> <li><a href="./tag/privacy.html">privacy</a> (1)</li> <li><a href="./tag/protocol.html">protocol</a> (1)</li> <li><a href="./tag/python.html">python</a> (1)</li> <li><a href="./tag/rsync.html">rsync</a> (1)</li> + <li><a href="./tag/rust.html">rust</a> (1)</li> + <li><a href="./tag/storage.html">storage</a> (1)</li> <li><a href="./tag/systemd.html">systemd</a> (1)</li> <li><a href="./tag/tmpfs.html">tmpfs</a> (1)</li> + <li><a href="./tag/wala.html">wala</a> (1)</li> <li><a href="./tag/wasm.html">wasm</a> (1)</li> <li><a href="./tag/web.html">web</a> (1)</li> </ul> diff --git a/wala.html b/wala.html @@ -0,0 +1,238 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - Introducing Wala</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="./category/archiving.html">Archiving</a></li> + <li class="active"><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./wala.html" rel="bookmark" + title="Permalink to Introducing Wala">Introducing Wala</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2022-10-05T14:39:00+02:00"> + Wed 05 October 2022 + </time> +in <a class="category" href="./category/code.html">code</a> + <a href="./tag/wala.html">wala</a> + <a href="./tag/pgp.html">pgp</a> + <a href="./tag/crypto.html">crypto</a> + <a href="./tag/storage.html">storage</a> + <a href="./tag/http.html">http</a> + <a href="./tag/rust.html">rust</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./portable-book-metadata.html"> + A portable book metadata exercise + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>This little project is heavily inspired by the "Single-Owner Chunk" (previously "Mutable Resource") concept from the <a class="reference external" href="https://ethswarm.org">Swarm project</a> <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p> +<p>Dubbed <a class="reference external" href="https://git.defalsify.org/wala">wala</a> <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>, the rust application provides two features:</p> +<ol class="arabic simple"> +<li>Store any uploaded data under the SHA256 digest of the content itself. We call this "content addressed" storage.</li> +<li>Create a link alias to the content using a keyword and a cryptographic identity.</li> +</ol> +<p>First, let's briefly outline what the first item on the list is.</p> +<div class="section" id="curl-up-and-digest"> +<h2>Curl up and digest</h2> +<p>Say an instance of wala is listnening on <code>localhost:8000</code>.</p> +<p>A simple demonstration of the content addressed storage could be:</p> +<pre class="code console literal-block"> +<span class="gp">$ </span>curl -X PUT http://localhost:8000 --data <span class="s2">"foo"</span> +<span class="go">2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae +</span><span class="gp">$ </span>curl -X GET http://localhost:8000/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae +<span class="go">foo +</span><span class="gp">$ </span><span class="nb">echo</span> -n <span class="s2">"foo"</span> <span class="p">|</span> sha256sum +<span class="go">2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae -</span> +</pre> +<p>In short, anything uploaded to the endpoint will be stored under and can be retrieved by its hash.</p> +<p>That also means that if the content changes, so does the location.</p> +</div> +<div class="section" id="get-to-the-pointer"> +<h2>Get to the pointer</h2> +<p>Now let's say you want a permanent link to content that is changing.</p> +<p>If you are the only user of the instance, this would be straightforward thing. We could just use a keyword in the url.</p> +<p>But if you are not, then authentication is needed to make sure that one user doesn't overwrite the another user's permanent link.</p> +<p>This is usually accomplished with a username and password stored and controlled server-side.</p> +<p>But we can also do this by simply using public key cryptography instead.</p> +<div class="section" id="claiming-what-is-yours"> +<h3>Claiming what is yours</h3> +<p>Since I have a weakness for PGP, the first implementation of authentication in wala has been implemented using the <a class="reference external" href="https://sequoia-pgp.org">"sequoia openpgp"</a> library.</p> +<p>The procedure is straightforward enough:</p> +<ol class="arabic simple"> +<li>Sign the content with your pgp key.</li> +<li>Add an <cite>Authorization</cite> header <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a> with your public key and signature.</li> +<li>Upload the content with an arbitrary keyword. <a class="footnote-reference" href="#footnote-4" id="footnote-reference-4">[4]</a></li> +</ol> +<p>If the signature over the content matches the public key, then a symbolic link to the content will be created on the server. That symbolic link will be a digest of the <em>key fingerprint</em> and the <em>keyword</em>.</p> +<p>The content can then be accessed <em>both</em> using the content address <em>and</em> the symbolic link.</p> +<p>In wala, the symbolic link is referred to as a <em>mutable reference</em>. We will use this term from now on.</p> +</div> +<div class="section" id="it-ain-t-pretty-but-it-works"> +<h3>It ain't pretty, but it works</h3> +<p>Let's demonstrate creating such an entry in wala using some commonly available tools. <a class="footnote-reference" href="#footnote-5" id="footnote-reference-5">[5]</a></p> +<p>We will use a fictional gnupg private key with the fingerprint <code>F3FAF668E82EF5124D5187BAEF26F4682343F692</code> and the keyword <code>foo</code> to create the mutable reference.</p> +<pre class="code console literal-block"> +<span class="gp">$ </span><span class="nv">sig</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -n <span class="s2">"bar"</span> <span class="p">|</span> gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 <span class="p">|</span> base64 -w <span class="m">0</span><span class="sb">`</span> +<span class="gp">$ </span><span class="nv">pub</span><span class="o">=</span><span class="sb">`</span>gpg --export F3FAF668E82EF5124D5187BAEF26F4682343F692 <span class="p">|</span> base64 - w <span class="m">0</span><span class="sb">`</span> +<span class="gp">$ </span>curl -v -X PUT http://localhost:8000/foo -H <span class="s2">"Authorization: PUBSIG:</span><span class="nv">$pub</span><span class="s2">:</span><span class="nv">$sig</span><span class="s2">"</span> --data bar +<span class="go">* Trying 127.0.0.1:8000... +</span><span class="gp"> % </span>Total % Received % Xferd Average Speed Time Time Time Current +<span class="go"> Dload Upload Total Spent Left Speed + 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 8000 (#0) +> PUT /foo HTTP/1.1 +> Host: localhost:8000 +> User-Agent: curl/7.85.0 +> Accept: */* +> Authorization: PUBSIG pgp:mQGNBF+hSOgBDACpkPQEjADjnQtjmAsdPYpx5N+OMJBYj1DAoIYsDtV6vbcBJQt94Om3xl7RBhv9m2oLgzPsiRwjCEFRWyNSu0BUp5CFjcXfm0S4K2egx4erFnTnSSC9S6tmVNrVNEXvScE6sKAnmJ7JNX1ExJuEiWPbUDRWJ1hoI9+AR+8EONeJRLo/j0Np+S4IFDn0PsxdT+SB0GY0z2cEgjvjoPr4lW9IAb8Ft9TDYp+mOzejn1Fg7CuIrlBRSAv+sj7bVQw15dh1SpbwtS5xxubCa8ExEGI4ByXmeXdR0KZJ+EA5ksO0iSsQ/6ipSOdSg+i0niOClFNm1P/OhbUsYAxCUfiX654FMn2zoxVBEjJ3e7l0pH7ktodaxEctPofQLBA9LSDUIejqJsU0npw/DHDD2uvxG+/A6lgV9L8ETlvgp8RzeOCf2bHuiKYYz87txvkFwsXgU1+TZxbk+mtCBbngsVPLNarY/KGkVJL+yhcHRD0Pl4wXUd6auQuY6vQ9AuKiCT1We2sAEQEAAbQeTWVyIE1hbiA8bWVybWFuQGdyZXlza3VsbC5jb20+iQHUBBMBCAA+FiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAl+hSOgCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ7yb0aCND9pLwiwwAhFJbAyUK05TJKfDz81757N472STtB8sfr0auwmRr8Zs1utHRVM0b/jkjTuo4uJNr7YVVKTKgE7+rJ+pwhm3wlTQ44LVLjByWAi/7NWg3E9b2elm+qkfgm/RfFt3vkuOxGSyZyIFFh+/twv6iABPvr6w7MZwrFaS0UP3g1VGa5TFqg6KNxod9H/gPLxv45lutXf3VvBZTJpr1pxn7aLHlFzEyIgNZbP/N1QF44GSrN/k0DfL631sZjauUXaZXbi5xGsKKCYwJ1g3q587pi6mTdTV3n0hKgVuipO8hGy5++YeOv+hXsCxDwyZ+Shv+qavd/SapxYgCdEueuwONIFfsIsWCd3SCcjKXicTTEFMu8nvBmf7xuo2hv6vEOxoijlXV+4LkGrskdB8ZMg8PywEx6DLmDokgnAhTLrTc1ShbkOtQ3yNjjyFK7BDpqobsJal6d8SpbhccUJLepaSmsk0CgJsTjhAl6EwX0EYgTo3kP5fScqrbD8VwQaT8CcE4rCV4uQGNBF+hSOgBDADHtpTT1k4x+6FN5OeURpKAaIsoPHghkJ2lb6yWmESCa+DaR6GXAKlbd0L9UMcXLqnaCn4SpZvbf8hP4fJRgWdRl5uVN/rmyVbZLUVjM8NcVdFRIrTsNyu4mLBmydc3iA/90sCTEOj9e7DSvxLmmLFjpwM5xXLd6z0l6+9G+woNmARXVS3V/RryFntyKC3ATCqVlJoQBG45Tj2gMIunpadTJXWmdioooeGW3sLeUv5MM98mSB4SjKRlJqGPNjx5lO6MmJbZeXZ/L/aO6EsXUQD2h82Wphll4rpGYWPiHTCYqZYiqNYr6E3xUpzcvWVp3uCYVJWP6Ds117p7BoyKVz00yxC9ledF3eppktZWqFVowCMihQE3676L3DDTZsnJf1/8xKUh5U2Mj3lBvjlvCECKi00qo8b1mn/OklQjJ5T4WzTrH6X+/zpez8ZkmtcOayHdUKD/64roZ9dXbXG/hp5A+UWj8oSVYKg2QNAwAnZ+aiZ2KVRE/Y61DCgFg6Ccx/cAEQEAAYkBvAQYAQgAJhYhBPP69mjoLvUSTVGHuu8m9GgjQ/aSBQJfoUjoAhsMBQkDwmcAAAoJEO8m9GgjQ/aSIPcL/3jqL2A2SmC+s0BO4vMPEfCpa2gZ/vo1azzjUieZu5WhIxb5ik0V6T75EW5F0OeZj9qXI06gW+IM8+C6ImUgaR3l47UjBiBPq+uKO9QuT/nOtbSs2dXoTNCLMQN7MlrdUBix+lnqZZGSDgh6n/uVyAYw8Sh4c3/3thHUiR7xzVKGxAKDT8LoVjhHshTzYuQq8MqlfvwVI4eESLaryQ+Y+j5+VLDzSLgPAnnIqF/ui2JQjefJxm/VLoYNaPAGdqoz/u/R0Tmz94bZUfLjgQaDoUpnxYywK2JGlf3mPZ3PNWjxJzuQTF5Ge5bz/TylnRYIyBT7KD7oaKHO62fhDbYPJ4f94iZN4B6nnTAeP34zFDlkUbX4AHudXU7bvxT5OUk9x9c2tj7xwxQHaEhq2+JsYW0EVw27RLhbymnBfLjVVUktNF0nQGvU2TEocw4pr2ZkDHQkSnlbNa4kujlL7VzbpnEgyOmi5er9GaIuVSVADovBu+pz/Ov1y/3jUe8hZ/KleQ==:iQGzBAABCAAdFiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAmM9guIACgkQ7yb0aCND9pIEdwwAiLLqFlrKu0UsQebfuUP07cvGbYy9LfbCMsQj/3/pG/zl7q2mSl2YdXOalbaYD2uyGU/sm7J/+qQZXGyIjmDA7F53sNVAXTuYnrcKmYIzAmzW4lUAzfWA7yL55MtbR/eNUE1rqp/Gu/ejj1OedLyxi+tGFcXUHU0q8EnjQnzfHCJVzOa3PGMIX10NiXPjrF2pafAyE7q2ogwkKZdjJi+8tyAw0tviu4CRGOVlsNZlF+yxePZh55XdRZLCEt4n6mnJrccu0C22rM9R2dEReqGLAj8t/WhACI+UyNXtL+hICnu9y6wjk4spoMr0s0pqTQ76SMwfmRFzk11uZ+ge846hArcUxE27+AeBf9Q1IwT5Ypsc0Efm9ZPoJvA2ggcJv1Yyb58Ggfmd02xPW4EQ8MOEMLA/ZoAhOm3t3wATPNFG1ucm/o+NFNDpF7HNby+Savqv2NrbNwDMlWvFzRmER2+AIO0CIG2HVJScMEn7UkjF8jIm+ba3BIAXbz2FUZ3dytFF +> Content-Length: 3 +> Content-Type: application/x-www-form-urlencoded +> +} [3 bytes data] +* Mark bundle as not supporting multiuse +< HTTP/1.1 200 OK +< Server: tiny-http (Rust) +< Date: Wed, 5 Oct 2022 13:15:37 GMT +< Content-Type: text/plain; charset=UTF-8 +< Access-Control-Allow-Origin: * +< Access-Control-Allow-Methods: OPTIONS, PUT, GET +< Access-Control-Allow-Headers: Content-Type,Authorization,X-Filename +< Content-Length: 64 +< +{ [64 bytes data] +100 67 100 64 100 3 3548 166 --:--:-- --:--:-- --:--:-- 3722100 67 100 64 100 3 3534 165 --:--:-- --:--:-- --:--:-- 3722 +* Connection #0 to host localhost left intact +32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 +</span><span class="gp">$ </span>curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 +<span class="go">bar +</span><span class="gp">$ </span><span class="nb">echo</span> -n bar <span class="p">|</span> sha256sum +<span class="go">fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9 +</span><span class="gp">$ </span>curl -X GET http://localhost:8000/fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9 +<span class="go">bar</span> +</pre> +<p>Now, by changing the data and signature, the data under the mutable resource changes:</p> +<pre class="code console literal-block"> +<span class="gp">$ </span><span class="nv">sig</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -n <span class="s2">"xyzzy"</span> <span class="p">|</span> gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 <span class="p">|</span> base64 -w <span class="m">0</span><span class="sb">`</span> +<span class="gp">$ </span>curl -X PUT http://localhost:8000/foo -H <span class="s2">"Authorization: PUBSIG:</span><span class="nv">$pub</span><span class="s2">:</span><span class="nv">$sig</span><span class="s2">"</span> --data xyzzy +<span class="go">32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 +</span><span class="gp">$ </span>curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 +<span class="go">xyzzy</span> +</pre> +</div> +</div> +<div class="section" id="building-your-identity"> +<h2>Building your identity</h2> +<p>We can also recreate the symbolic link hash using local tools:</p> +<pre class="code console literal-block"> +<span class="gp">$ </span><span class="nv">t</span><span class="o">=</span><span class="sb">`</span>mktemp<span class="sb">`</span> +<span class="gp">$ </span><span class="nv">d</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -n foo <span class="p">|</span> sha256sum <span class="p">|</span> awk <span class="s1">'{ printf "%s",$1; }'</span> <span class="p">|</span> sed -e <span class="s1">'s/../\\\\x&/g'</span><span class="sb">`</span> +<span class="gp">$ </span><span class="nb">echo</span> -ne <span class="nv">$d</span> > <span class="nv">$t</span> +<span class="gp">$ </span><span class="nv">k</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -n F3FAF668E82EF5124D5187BAEF26F4682343F692 <span class="p">|</span> sed -e <span class="s1">'s/../\\\\x&/g'</span><span class="sb">`</span> +<span class="gp">$ </span><span class="nb">echo</span> -ne <span class="nv">$k</span> >> <span class="nv">$t</span> +<span class="gp">$ </span>cat <span class="nv">$t</span> <span class="p">|</span> sha256sum +<span class="go">32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2 -</span> +</pre> +<p>In other words, the mutable reference is constructed using the following recipe.</p> +<ol class="arabic simple"> +<li>Calculate binary value <code>R</code> of the SHA256 digest of the keyword (<code>sha256(foo) -> 0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AE</code>)</li> +<li>Calculate binary value <code>K</code> of the key fingerprint (<code>0xF3FAF668E82EF5124D5187BAEF26F4682343F692</code>).</li> +<li>Calculate binary value of SHA256 digest of <code>R | K</code> (<code>sha256(0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AEF3FAF668E82EF5124D5187BAEF26F4682343F692) -> 0x32EA47ECC5A3EE2576AAB00C2F30EAABC2592D56E19F5C82FE4F7CF5874632B2</code>)</li> +</ol> +<p>That means that anyone who knows the keyword and the public key of the uploader can calculate the mutable reference themselves, and retrieve the data behind it.</p> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Refer to <a class="reference external" href="https://www.ethswarm.org/The-Book-of-Swarm.pdf">section 4.3 of The Book of Swarm</a> for a description of "feeds" and "single owner chunks." Swarm uses signatures to allow propagation of the data in the network. <code>wala</code> similarly uses signatures to accept update of resources "owned" by the holder of the private keys of an identity.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>At the time of writing <code>wala</code> is at version <code>0.1.1</code></td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>A custom authorization scheme <code>PUBSIG</code> has been invented for this purpose. The format is <code>Authorization: PUBSIG pgp:key-fingerprint-in-base64>:<signature-in-base64></code>. Currently this is strictly just authentication, as there is no feature in wala (yet) to use access control lists to determine which public keys to allow PUTs from.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-4" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-4">[4]</a></td><td>If you do not specify a keyword, the keyword value in the mutable resource reference will be the sha256 hash of an empty value (<code>0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</code>)</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-5" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-5">[5]</a></td><td><code>wala</code> comes with a binary tool <code>wala_send</code> that lets you specify a key fingerprint in your default pgp keyring and a keyword as arguments when you upload data. A lot less messy, but a lot less educational, too.</td></tr> +</tbody> +</table> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/wala.sha256 b/wala.sha256 @@ -0,0 +1 @@ +406a858d431883e0f1e8e05ea87fae94b152a03a408a0d239e6868cef369f198 wala.html +\ No newline at end of file