commit f143eaae307596e3450784a9ec4cb3eb19baaa51 parent b69f947acca539f702a22851bed29b3108162867 Author: lash <dev@holbrook.no> Date: Sat, 1 Oct 2022 12:41:06 +0000 New post for portable metadata entries Diffstat:
137 files changed, 7833 insertions(+), 3136 deletions(-)
diff --git a/09c60de8503f5115c11d7059476de0a930bdeaf0511609d24ed93d15aadef47b b/09c60de8503f5115c11d7059476de0a930bdeaf0511609d24ed93d15aadef47b @@ -0,0 +1,328 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - A village network protocol</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="./category/archiving.html">Archiving</a></li> + <li><a href="./category/code.html">Code</a></li> + <li class="active"><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./village-network-protocol.html" rel="bookmark" + title="Permalink to A village network protocol">A village network protocol</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2021-05-19T10:40:00+02:00"> + Wed 19 May 2021 + </time> +in <a class="category" href="./category/hygiene.html">hygiene</a> + <a href="./tag/p2p.html">p2p</a> + <a href="./tag/protocol.html">protocol</a> + <a href="./tag/network.html">network</a> + <a href="./tag/crm.html">crm</a> + <a href="./tag/privacy.html">privacy</a> + <a href="./tag/f2f.html">f2f</a> + <a href="./tag/community.html">community</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./nft-tokenid-content.html"> + The NFT token id as URI + </a> + </li> + | + + <li> + Next: <a href="./docker-offline-3-npm.html"> + Local npm repository + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>What would truly peer-to-peer CRM backend look like? And at that, once that respected the right of the "customer" to choose how it wants to represent itself in the context?</p> +<p>Perhaps by Modeling the network on how villages treat strangers and their own, and how you would go about interacting with them.</p> +<p>Here's an initial stab at describing what such a meta-protocol could look like.</p> +<!-- There is no truth. There is only perspective and predictability. --> +<div class="section" id="terms"> +<h2>Terms</h2> +<dl class="docutils"> +<dt>Individual</dt> +<dd>A subjective structure where the thing being described is the thing that described it.</dd> +<dt>User</dt> +<dd>A particular perception of an individual by an entity in the network.</dd> +<dt>Userspec</dt> +<dd>Data structure containing details about a user. May or may not be public.</dd> +<dt>Locals</dt> +<dd>A group of individuals acknowledging each other as part of a group, and who share views on users.</dd> +<dt>Local</dt> +<dd>A member of a group of locals, from the shared perspective of an individual and another local.</dd> +<dt>Localslist</dt> +<dd>The individuals making up a group of locals, from the perspective of a local.</dd> +<dt>Stranger</dt> +<dd>A user as percieved by a group of locals.</dd> +<dt>Strangerlist</dt> +<dd>All users that may speak to the locals, from the perspective of a local. To be shared among locals.</dd> +<dt>Strangerspec</dt> +<dd>Userspec created by a local describing a user. To be shared among locals.</dd> +<dt>Entity</dt> +<dd>An individual or any derived representation of it (user, stranger, local).</dd> +<dt>Peer</dt> +<dd>An entity coupled with connection information</dd> +<dt>Content address</dt> +<dd>SHA256 sum of arbitrary content</dd> +<dt>Key setup</dt> +<dd>May be an individual key, or a specification describing how to generate keys for a particular relation.</dd> +</dl> +</div> +<div class="section" id="roles"> +<h2>Roles</h2> +<div class="section" id="the-individuals"> +<h3>The individuals</h3> +<p>Are free to have their own opinion of what they are.</p> +<p>When interacting with others, they are free to choose what they say about what they are.</p> +<p>An individual initially <em>defines itself</em> in the network.</p> +</div> +<div class="section" id="the-users"> +<h3>The users</h3> +<p>A user is a view of an individual as defined by one or more other individuals.</p> +<p>It may be how an individual presents itself to an particular individual or a group.</p> +<p>It may also be how an individual or a group presents another individual to itself.</p> +</div> +<div class="section" id="the-locals"> +<h3>The locals</h3> +<p>Locals are a group of individuals who wish to share a view of themselves and prejudice about others.</p> +<p>An individual will not be treated as a local unless it and its local counterpart considers it a local.</p> +<p>Locals may acknowledge strangers with or without proof.</p> +<p>If one local changes opinion about a stranger, all locals should change their opinon to the same.</p> +<p>If one local excludes a stranger, all locals should exclude the stranger.</p> +<p>You can't keep secrets in a village. If someone says something to another local, it must assume all locals will get to know what was said.</p> +</div> +<div class="section" id="the-strangers"> +<h3>The strangers</h3> +<p>A stranger is a user that's not considered local by a group of locals.</p> +<p>A stranger may not speak to a local until the local has decided it can.</p> +<p>If a stranger may speak to one local, it may speak all locals.</p> +</div> +</div> +<div class="section" id="identifiers"> +<h2>Identifiers</h2> +<p>The <code>individual_id</code> is the identifier of an individual consists of the content address of some basic public data about the entity registering itself. The <code>individual_id</code> is immutable.</p> +<p>Any local will refer to another local by its <code>individual_id</code>.</p> +<p>A <code>user_id</code> is the content address of an <code>individual_id</code> together with the content address of some basic data describing the perception of the individual.</p> +<p>Locals describe stranger by a pair of <code>individual_id</code> and <code>user_id</code>. Let us call this <code>stranger_id</code>. The <code>stranger_id</code> is immutable.</p> +<p>The data represented by the <code>user_id</code> in a <code>stranger_id</code> is visible to locals, and may or may not be public.</p> +</div> +<div class="section" id="state-description"> +<h2>State description</h2> +<p>A user keeps a state towards other entities consisting of the following content-addresses, in the given order:</p> +<ol class="arabic simple"> +<li>Previous state</li> +<li>Userspec</li> +</ol> +<p>A local keeps a state towards its locals consisting of the following content-addresses, in the given order.</p> +<ol class="arabic simple"> +<li>Previous state</li> +<li>Userspec</li> +<li>Localslist</li> +<li>Locals state</li> +<li>Strangerslist</li> +<li>Strangers state</li> +</ol> +<p>The state is represented by the content address of the concatenation of these content-addresses, in the given order.</p> +<p>Any state change will be signed by the key resolvable through the <code>individual_id</code>.</p> +<div class="section" id="previous-state"> +<h3>Previous state</h3> +<p>The content address of the previous state. Will be <code>0</code> in the first state.</p> +</div> +<div class="section" id="userspec"> +<h3>Userspec</h3> +<p>The individual as it wants to be perceived in the current context. There is no guarantee that the individual will be viewed in this way by others.</p> +</div> +<div class="section" id="localslist"> +<h3>Localslist</h3> +<p>A lexiographically ordered array of <code>individual_id</code> that define the individuals in a group of locals.</p> +<p>All locals must keep this list in the same state.</p> +</div> +<div class="section" id="locals-state"> +<h3>Locals state</h3> +<p>A local's last recorded state of each of the locals, in the same order as <code>Localslist</code>.</p> +<p>The recorded state is the same content address as in the state description section above.</p> +<p>All locals must strive to keep this in the same state.</p> +</div> +<div class="section" id="strangerslist"> +<h3>Strangerslist</h3> +<p>A lexiographically ordered pointer array representing a local's opinion of which users that can communicate with locals. <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a></p> +<p>Combination of <code>stranger_id</code> and a <em>vote</em> on the specific user.</p> +<p>The vote can be in one of three states:</p> +<ul class="simple"> +<li>Value <code>0</code>, meaning local has acknowledged the user, explicitly or by following other locals. We call this state "deferred."</li> +<li>Value <code>-1</code>, meaning a local has vetoed the user. We call this state "vetoed."</li> +<li>Any other value, which is the content-address of the proof accepted by the local. <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a> We call this state "accepted."</li> +</ul> +<p>Typically, the initial vote value is <code>0</code>. The value may change at any time, but may never change back to <code>0</code> once set to another value.</p> +<p>A user that is part of the list, but not vetoed, is referred to as "acknowledged."</p> +<p>The Strangerslist state may be differ between locals, but they should strive to keep it the same.</p> +</div> +<div class="section" id="strangers-state"> +<h3>Strangers state</h3> +<p>The local's version of the current user state of the strangers, in the same order as the <code>Strangerlist</code>.</p> +<p>The Strangers state may differ between locals, but they should strive to keep it the same.</p> +</div> +</div> +<div class="section" id="behavior-of-locals"> +<h2>Behavior of locals</h2> +<p>A local may announce itself as <em>active</em> or <em>passive</em>. This should be part of the <code>Userspec</code> in the locals context.</p> +<p>Specifically, this affects the way it "votes" on strangers. A passive local:</p> +<ul class="simple"> +<li>Will always <em>veto</em> a user if <em>at least one active local</em> has vetoed it.</li> +<li>Will always <em>accept</em> a user if <em>all active locals</em> have accepted it. <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a></li> +</ul> +<p>Typically, a passive local will be a bot, for example an always-online node that guarantees availability of the locals so users can request state changes.</p> +<p>There must be at least two active locals in any group of locals.</p> +<div class="section" id="veto-dissent"> +<h3>Veto dissent</h3> +<p>If a local vetoes a stranger, then all locals should cut communication with that stranger.</p> +<p>Vetoes should be adopted by all locals.</p> +<p>If, however, other locals consider the veto to be issued in error, there are two possible outcomes:</p> +<ol class="arabic simple"> +<li>The vetoing local(s) abandon(s) the veto. <a class="footnote-reference" href="#footnote-4" id="footnote-reference-4">[4]</a></li> +<li>The vetoing individual(s) is/are excluded by the locals.</li> +</ol> +</div> +</div> +<div class="section" id="network"> +<h2>Network</h2> +<p>Any entity needs to define and announce itself as a <code>peer</code> in order to be available on the network.</p> +<p>A "peer" is the coupling of network connection information to an <code>individual_id</code>.</p> +<div class="section" id="discoverability"> +<h3>Discoverability</h3> +<p>Peers are kept in a <a class="reference external" href="https://en.wikipedia.org/wiki/Distributed_hash_table">DHT</a>.</p> +<p>Peers must be discoverable by their locals affiliation. <a class="footnote-reference" href="#footnote-5" id="footnote-reference-5">[5]</a></p> +</div> +<div class="section" id="users"> +<h3>Users</h3> +<p>Users open single-request connections to other users to annonunce, look up and prune peers.</p> +<p>Users may request state changes for themselves, or relay such requests for other users.</p> +</div> +<div class="section" id="locals"> +<h3>Locals</h3> +<p>Locals keep open connections (streams) with each other.</p> +<p>Locals continually exchange state changes.</p> +<p>Locals should be able to audit the chain of state changes of other locals.</p> +</div> +<div class="section" id="between-locals-and-users"> +<h3>Between locals and users</h3> +<p>Strangers <em>acknowledged</em> by locals open single-request connections to a local to announce state changes.</p> +<p>Locals may open single-request connections to a user to announce a change to the state of the <code>Userspec</code> of a stranger. <a class="footnote-reference" href="#footnote-6" id="footnote-reference-6">[6]</a></p> +<p>Individuals are free to deny connections from non-locals, for example if the entity has been sending garbage requests. The implementation of such sanctions is deferred to the application layer.</p> +</div> +</div> +<div class="section" id="encryption"> +<h2>Encryption</h2> +<p>An individual may keep content totally private on the network.</p> +<p>A user may keep different encryption keys for different contexts.</p> +<p>If a stranger shares a key setup with a local, that setup will be shared among all locals. A local will expect to be able to communicate with a strangerr using this key setup.</p> +<p>The stranger must be able to change the key setup towards locals.</p> +<p>The locals may be able to change the key setup towards a stranger.</p> +<p>The protocol should allow anything from simple symmetric key exchanges to complex forward/backward secrecy schemes.</p> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Insert in this sorted list will face scaling issues when locals need to acknowledge a lot of users. The lists can be split into <a class="reference external" href="https://en.wikipedia.org/wiki/Bucket_%28computing%29">buckets</a>, either statically (for example two-byte prefixes) or dynamically after crossing a certain threshold. It should not be necessary to store empty buckets.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>This proof may map to what's commonly known as <em>kyc information</em>.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>Perhaps with the further limitation that all active users must have accepted with the <em>same</em> proof. Otherwise, how will the passive local choose which proof to accept.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-4" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-4">[4]</a></td><td>Observe that since the vote state cannot change back to <code>0</code>, "cancelling" the veto can only be done by adding proof.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-5" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-5">[5]</a></td><td>What exactly uniquely identifies a group of locals on the network in such a way that it cannot be hi-jacked or spoofed still needs to be thought through. Perhaps it can be a combination of a specific name with a signature from the "founders;" the first (two or more) locals in it.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-6" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-6">[6]</a></td><td>A use-case here would be where the locals make a change for the user due to an out-of-band request from the user. This is <em>not</em> the <code>Strangerspec</code>; it is the user's current <code>Userspec</code> for the given context. Of course, it's up to the user to accept the change. It may in turn request a new state change to the locals to revert it.</td></tr> +</tbody> +</table> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/09c60de8503f5115c11d7059476de0a930bdeaf0511609d24ed93d15aadef47b.asc b/09c60de8503f5115c11d7059476de0a930bdeaf0511609d24ed93d15aadef47b.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NzsACgkQzcvSTdHQ +4AFVDQgAqdsTEMu0m6C1lMiCKKujR7+xyslkJjVrp14OpV4m55ZjQDMOqkK97PB9 +6PHlJDuluYsHPRTXVHQvWreA9JTD00D1M1wZjSkHarM3taYouhfIY+tlReIw5JSj ++mKHa0u4O637dlylbTRo2BFv+MQX+g2U+QyB+xiPVwD+pMLT6fEtD/kntY5gH0bL +KhCV+GTMJmc21iM9C+A+gqGEL33OnHdShwKs5bUGxgtjWdozSK4QM4md0DoAIhIW +gi/82rtiCpMlcYKimK++lT0D67ghTV8moZ4qa9HmbtMGY0pQNseUolLjyrSl65YT +Wc3jIXR3YKFNYJuOBlcxeVwB2NNLEw== +=3Wp9 +-----END PGP SIGNATURE----- diff --git a/1147e630f5d17d981407c49cfc72bb0b340e2b672044f7df387490f23c8700c0 b/1147e630f5d17d981407c49cfc72bb0b340e2b672044f7df387490f23c8700c0 @@ -0,0 +1,271 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - Proving what you link to</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li class="active"><a href="./category/archiving.html">Archiving</a></li> + <li><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./web-snapshot.html" rel="bookmark" + title="Permalink to Proving what you link to">Proving what you link to</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2021-05-03T14:22:00+02:00"> + Mon 03 May 2021 + </time> +in <a class="category" href="./category/archiving.html">archiving</a> + <a href="./tag/web.html">web</a> + <a href="./tag/hash.html">hash</a> + <a href="./tag/chromium.html">chromium</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./docker-offline-2-python.html"> + Local python repository + </a> + </li> + | + + <li> + Next: <a href="./wasm-c-bare.html"> + wasm and C - The bare necessities + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>When we send a link to someone, we are trusting that whoever is behind that link will serve that someone the content we actually saw. Give or take an ad or two.</p> +<p>Also, when we bookmark the link for later retrieval. We are trusting that the entity will be serving that content at any time in the future that you may want it.</p> +<p>This may not be a huge problem if the page is merely a list of <a class="reference external" href="https://dead-baby-joke.com/">dead baby jokes</a>. They are objectively funny, of course. But you can also get along without them.</p> +<p>But what of the case of formal, scientific texts we may depend on, and that use citations from the web as part of their source material? Usually, they refer to a source by <em>link</em> and <em>date of retrieval</em>. This is not of much use unless the actual source and/or render they saw at that time also is available.</p> +<p>That may not always be the case.</p> +<div class="section" id="take-care-of-your-shelf"> +<h2>Take care of your shelf</h2> +<blockquote> +"No worries, the <a class="reference external" href="https://archive.org">Wayback Machine</a> has me covered."</blockquote> +<p>Yes. But no. The Wayback Machine is a (thus far) centralized entity that depend on a few idealists and donations to keep going. If they stop to keep going, they depend on passing the buck to someone else. If that someone is evil, they may take it and rewrite history to suit themselves. If that someone else cannot be found, it becomes a garbage collection blip on Bezos' <a class="reference external" href="https://gizmodo.com/i-tried-to-block-amazon-from-my-life-it-was-impossible-1830565336">infrastructure monopoly</a> dashboard. <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a></p> +<p>That aside, sources like Wayback Machine are like libraries. Libraries are, of course, essential. Not only because they serve as one of the pillars of democracy, providing free access to knowledge for everyone. They are also essential because it's simply not very practical for you to pre-emptively own all the books that you may at some point want to read. Let alone ask around in your neighborhood if they happen to have a copy (although a crowd-sourced library app sounds like a fun decentralization project to explore).</p> +<p>You may however want to keep a copy of the books you <em>depend</em> on, and the ones you <em>really like</em>. Just to make really sure you have it available when you want it. Then, if some New Public Management clowns get the chance to gut public infrastructure where you live, or someone starts a good old-fashioned fascist book burning, you have yourself covered.</p> +</div> +<div class="section" id="a-lack-of-friction"> +<h2>A lack of friction</h2> +<p>Yes, stuff may disappear on the web. Just as books may.</p> +<p>On the web that stuff can get <em>rewritten</em>. Books may be rewritten, too. <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a> The previous editions of the books will still exist as independent physical objects until they degrade, as long as something is not actively done to them. But so too with data on storage media. If it is not renewed or copied during that lifetime it may degrade until it becomes illegible. And of course, it may also simply be deleted. And without the smell of smoke at that.</p> +<p>That's the difference that seems to leap out when using this imagery. How <em>easy</em> it is to change and destroy stuff at scale on the web compared with the real world of books. And how inconspicuously it can happen, without <em>anyone</em> noticing. And for those who notice, it is very hard to prove what has changed, unless you have a copy. <a class="footnote-reference" href="#footnote-5" id="footnote-reference-3">[5]</a></p> +<p>So what can we do? Copies and proofs are definitely keywords here. Fortunately, copying is what computers are all about. And making a cryptographical proof of what you see is easy enough these days, too. The tricky bit is to build <em>credibility</em> around that proof. But let's stick our head in the sand and start with the easy part and see where it takes us.</p> +</div> +<div class="section" id="look-no-head"> +<h2>Look, no head</h2> +<!-- .. WARNING:: + + Nerdy zone ahead --> +<p>A good start is to dump the document source to disk, then calculating and storing the sum of it.</p> +<p>In many cases this will be insufficient, though, as many sites populates the DOM through scripts, either in part or in full. As the use-case here is humans voting on what they see is what they get, human aids will be needed. In other words, we need to render the page to store what we actually see.</p> +<p>Printing to PDF from the browser is an option, but that is really difficult to automate. Fortunately, modern browser engines provide command line access to rendering. Since I mostly use <a class="reference external" href="https://brave.com/">Brave Browser</a> these days, we'll use <em>headless Chromium</em> here.</p> +<p>In addition to source, sum and rendering, we should also include a copy of the request headers for good measure.</p> +<p>Thus, we end up with something like this.</p> +<pre class="code bash literal-block"> +<span class="ch">#!/bin/bash +</span> +<span class="nv">f</span><span class="o">=</span><span class="si">${</span><span class="nv">WEBSHOT_OUTPUT_DIR</span><span class="k">:-</span><span class="p">/tmp</span><span class="si">}</span> +<span class="nv">url</span><span class="o">=</span><span class="nv">$1</span> +<span class="nv">title</span><span class="o">=</span><span class="nv">$2</span> +><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> using outdir <span class="nv">$f</span> + +<span class="nb">set</span> +e + +<span class="c1"># prepare +</span><span class="nv">d</span><span class="o">=</span><span class="sb">`</span><span class="nv">TZ</span><span class="o">=</span>UTC date +%Y%m%d%H%M<span class="sb">`</span> +<span class="nv">t</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> +<span class="nb">pushd</span> <span class="nv">$t</span> + +<span class="c1"># store raw outputs +</span><span class="nb">echo</span> <span class="nv">$1</span> > url.txt +curl -s -I <span class="nv">$1</span> > headers.txt +curl -s -X GET <span class="nv">$1</span> > contents.txt +<span class="nv">z</span><span class="o">=</span><span class="sb">`</span>sha256sum contents.txt<span class="sb">`</span> +<span class="nb">echo</span> <span class="nv">$z</span> > contents.txt.sha256 +<span class="nv">h</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -n <span class="nv">$z</span> <span class="p">|</span> awk <span class="s1">'{ print $1; }'</span><span class="sb">`</span> + +<span class="k">if</span> <span class="o">[</span> -z <span class="s2">"</span><span class="nv">$title</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + <span class="nv">title</span><span class="o">=</span><span class="nv">$h</span> +<span class="k">fi</span> +><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> using title <span class="nv">$title</span> + +<span class="c1"># rendered snapshot +</span>chromium --headless --print-to-pdf <span class="nv">$url</span> +<span class="nv">n</span><span class="o">=</span><span class="si">${</span><span class="nv">d</span><span class="si">}</span>_<span class="si">${</span><span class="nv">h</span><span class="si">}</span> +mv output.pdf <span class="nv">$n</span>.pdf + +<span class="c1"># store result +</span>mkdir -p <span class="s2">"</span><span class="nv">$f</span><span class="s2">/</span><span class="nv">$title</span><span class="s2">"</span> +tar -zcvf <span class="s2">"</span><span class="nv">$f</span><span class="s2">/</span><span class="nv">$title</span><span class="s2">/</span><span class="nv">$n</span><span class="s2">.tar.gz"</span> * + +<span class="c1"># clean up +</span><span class="nb">popd</span> + +<span class="nb">set</span> -e +</pre> +</div> +<div class="section" id="what-does-this-mean"> +<h2>What does this mean</h2> +<p>Let's sum up what information we've managed to store with this operation.</p> +<ul class="simple"> +<li>We have a copy of the unrendered source. It may or may not include all the information we want to store.</li> +<li>We have a fingerprint of that source.</li> +<li>We have a copy of the headers we were served when retrieving the document. <a class="footnote-reference" href="#footnote-3" id="footnote-reference-4">[3]</a></li> +<li>We have a image copy of what we actually saw when visiting the page.</li> +<li>We have a date and time for retrieval (file attributes).</li> +</ul> +<p>To link the headers together with the visual copy, we could sum the header file and image file aswell, put those sums together with the content sum in a deterministic order, and calculate the sum of those sums. E.g. <a class="footnote-reference" href="#footnote-4" id="footnote-reference-5">[4]</a></p> +<div class="highlight"><pre><span></span>$ cp contents.txt.sha256 sums.txt +$ sha256sum headers.txt <span class="p">|</span> awk <span class="s1">'{ print $1; }'</span> >> sums.txt +$ sha256sum <pdf file> <span class="p">|</span> awk <span class="s1">'{ print $1; }'</span> >> sums.txt +$ sha256sum sums.txt <span class="p">|</span> awk <span class="s1">'{ print $1; }'</span> > topsum.txt +</pre></div> +<p>If we now sign <em>this</em> sum, we are confirming that for this particuar resource:</p> +<blockquote> +"This was the source. These were the headers for that source. This is how that source served in that manner looked for <strong>ME</strong> at the time"</blockquote> +</div> +<div class="section" id="proving-links-in-this-post"> +<h2>Proving links in this post</h2> +<p>This post makes use of several external links to articles. So as a final step, let's eat our own dogfood and add proofs for them.</p> +<table border="1" class="docutils"> +<caption>Article retrieval proofs</caption> +<colgroup> +<col width="34%" /> +<col width="22%" /> +<col width="22%" /> +<col width="22%" /> +</colgroup> +<thead valign="bottom"> +<tr><th class="head">Link</th> +<th class="head">Content</th> +<th class="head">Header</th> +<th class="head">Image</th> +</tr> +</thead> +<tbody valign="top"> +<tr><td><a class="reference external" href="https://gizmodo.com/i-tried-to-block-amazon-from-my-life-it-was-impossible-1830565336">https://gizmodo.com/i-tried-to-block-amazon-from-my-life-it-was-impossible-1830565336</a></td> +<td><a class="reference external" href="./misc/web_snapshot/0c657ec25e55e72702b0473f62e6d632dece992485f67c407ed1f748b3f40bc2.txt">0c657ec2</a></td> +<td><a class="reference external" href="./misc/web_snapshot/360125fa513b8db8380eb2b62c4479164baa5b48d8544b2242bcc7305bad0de4.txt">360125fa</a></td> +<td><a class="reference external" href="./misc/web_snapshot/a2584ca07ba16b15b9fad315a767cbb0c4b7124abdfd578cab2afb4dce9d1971.pdf">a2584ca0</a></td> +</tr> +<tr><td><a class="reference external" href="https://www.thelocal.se/20111109/37244/">https://www.thelocal.se/20111109/37244/</a></td> +<td><a class="reference external" href="./misc/web_snapshot/be8741ff91c6e3eac760a3a4652b57f47bce92fa9b617662296c2ab5b3c5fe31.txt">be8741ff</a></td> +<td><a class="reference external" href="./misc/web_snapshot/6c20e7678a0235467687425b9818718ab143fd477afee2087d7f79c933abdc75.txt">6c20e767</a></td> +<td><a class="reference external" href="./misc/web_snapshot/f1c557b9555149dde570976ed956ffb17d17b99cea5f2651020f66408dacf301.pdf">f1c557b9</a></td> +</tr> +<tr><td><a class="reference external" href="https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/">https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/</a></td> +<td><a class="reference external" href="./misc/web_snapshot/595a07b85e6b75a41fe0880c8538f15c4b6da5770db230d986efa2e080ca479a.txt">595a07b8</a></td> +<td><a class="reference external" href="./misc/web_snapshot/f2aa95b42c5420cb11ccbf1cb084d5e5ccc3fc6cd575c51e9ee473b9df19c890.txt">f2aa95b4</a></td> +<td><a class="reference external" href="./misc/web_snapshot/6a610fa69d4909cc9aa1dcb7105203cc50c1bcf26733411964f95cbcbdf37eb5.pdf">6a610fa6</a></td> +</tr> +<tr><td><a class="reference external" href="https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html">https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html</a></td> +<td><a class="reference external" href="./misc/web_snapshot/5a806be410da82986a85f68658279234c1a5cf3bb6dc55da137d5274dc722f26.txt">5a806be4</a></td> +<td><a class="reference external" href="./misc/web_snapshot/078d9fe6be070de0d378a6e903f8558a7da4917cba5c95ca453ae1936541e4f6.txt">078d9fe6</a></td> +<td><a class="reference external" href="./misc/web_snapshot/2d0f0e69e7c8b6ffe9ff8ffc8702a78d6a2d46ab1edd4123e84eb211171d6cde.pdf">2d0f0e69</a></td> +</tr> +<tr><td><a class="reference external" href="https://www.breitbart.com/europe/2017/07/19/swedens-libraries-pulp-traditional-children-books-racist-phrases/">https://www.breitbart.com/europe/2017/07/19/swedens-libraries-pulp-traditional-children-books-racist-phrases/</a></td> +<td><a class="reference external" href="./misc/web_snapshot/a8c6b61cec2cce1a58bcf7a65091a6c2e8510ca5fa17f2d242d286f087d95cd5.txt">a8c6b61c</a></td> +<td><a class="reference external" href="./misc/web_snapshot/3ba9094b295a898cfe8cba256f4ebf65ef98ff05bb3034e048e517d52fc13d33.txt">3ba9094b</a></td> +<td><a class="reference external" href="./misc/web_snapshot/eb76a87f224b0e4f4e5d1673b1505aaeee28d8ad03ce544656b6f5ac7c4d9983.pdf">eb76a87f</a></td> +</tr> +</tbody> +</table> +<p>Clicking on the "image" links, we see that thanks to the recent ubiquity of cookie nag boxes screaming "accept all" at you, those very boxes are now blocking the content we want to get at. So we will need more work to get to where we want by automation. But it's s start.</p> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Early 2021 survey puts Amazon at one-third of the global market share. <a class="reference external" href="https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/">https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/</a></td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>2011 sparked a <a class="reference external" href="https://www.thelocal.se/20111109/37244/">controversy</a> around Astrid Lindgren's Pippi Longstockings. Echoing those viewpoints, the books were edited in 2015 during which some alleged "racist" content was altered. The rabid rightwing media later spun a <a class="reference external" href="https://www.breitbart.com/europe/2017/07/19/swedens-libraries-pulp-traditional-children-books-racist-phrases/">false tale of mass purges</a> of Pippi books around one single swedish library's decision to throw out copies of the originial "racist" versions. Case-in-point, a public statement in which the library tries to justify its actions is no longer available on their website, and has to be retrieved by the Wayback Machine. <a class="reference external" href="https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html">https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html</a> (in swedish)</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-4">[3]</a></td><td>Well actually, not quite. We did the same request twice, but they were two separate requests. Using a single request would improve the script.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-4" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-5">[4]</a></td><td>We use the hex representation for clarity here. A proper tool would convert the hex values to bytes before calculating sum on them.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-5" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[5]</a></td><td>Another important difference is that the book does not need to be <em>interpreted</em> by a <em>machine</em> in order to make sense for a human. Availability of tooling is definitely an equally important topic in this discussion. However this post limits focus to the actual data itself.</td></tr> +</tbody> +</table> +</blockquote> +<!-- .. _data availability: https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.433.3480 --> +<!-- https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html <- not available on link, no result on search https://www.mhpbooks.com/the-trouble-with-pippi/ --> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/1147e630f5d17d981407c49cfc72bb0b340e2b672044f7df387490f23c8700c0.asc b/1147e630f5d17d981407c49cfc72bb0b340e2b672044f7df387490f23c8700c0.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NzwACgkQzcvSTdHQ +4AEw+AgAnVLl2QbGjK++z5q1+NnQXhsWBfT/MFk6XrM0d2CCy6yLJaSU31nYzeWl +a7Z2GTw1m0VipLmag4gedo4L1Mjl9pNjr/vUmGHj56jPmOeij8sL/3XrefsMPoGD +N61sQKLxzrJ896K0Qcrqg92EIYkDk6tBkm4tn5iHBByjeScifKZojnGkYVR1Mu11 +jPI6YtLQVN64R8bSFmfWuuDZEJCRxwZcaYH4+GaIM+du7TvhQ2pKR5PXnZcugxet +QCXytgWV9W2uo5V1OwGy+YGBIayvqYJuZF4fTHGZD5idEkCOXzibXUTV+nKpUGPL +WYfS3WcX3E6R7VoNuOfio6Nb97YSNQ== +=ws7R +-----END PGP SIGNATURE----- diff --git a/132b445a5c24f134b82b8cb95e7112fd053ca2c15531afa694105a5d1a1d872a b/132b445a5c24f134b82b8cb95e7112fd053ca2c15531afa694105a5d1a1d872a @@ -1,271 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> - <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> - <title>man bytes gnu - Proving what you link to</title> - <meta charset="utf-8" /> - <meta name="generator" content="Pelican" /> - -</head> - -<body id="index" class="home"> - - <header id="banner" class="body"> - <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> - </header> - <nav id="menu"> - <ul> - <li class="active"><a href="./category/archiving.html">Archiving</a></li> - <li><a href="./category/code.html">Code</a></li> - <li><a href="./category/hygiene.html">Hygiene</a></li> - <li><a href="./category/offlining.html">Offlining</a></li> - <li><a href="./pages/identities.html">Identities</a></li> - <li><a href="./pages/shares.html">Shares</a></li> - - <li><a href="/tags.html">tags</a></li> - </ul> - <hr/> - </nav><!-- /#menu --> - <header> - <h1 class="entry-title top-body-title"> - <a href="./web-snapshot.html" rel="bookmark" - title="Permalink to Proving what you link to">Proving what you link to</a></h1> - - - <div class="category meta"> - Posted -<time class="published" datetime="2021-05-03T14:22:00+02:00"> - Mon 03 May 2021 - </time> -in <a class="category" href="./category/archiving.html">archiving</a> - <a href="./tag/web.html">web</a> - <a href="./tag/hash.html">hash</a> - <a href="./tag/chromium.html">chromium</a> - </div> - - <div class="neighbors meta"> - <ul> - <li> - Previous: <a href="./docker-offline-2-python.html"> - Local python repository - </a> - </li> - | - - <li> - Next: <a href="./wasm-c-bare.html"> - wasm and C - The bare necessities - </a> - </li> - </ul> - </div> - <div class="meta"> -<hr/> - </div> -</header> - -<section id="content" class="body"> - <div class="entry-content"> - <p>When we send a link to someone, we are trusting that whoever is behind that link will serve that someone the content we actually saw. Give or take an ad or two.</p> -<p>Also, when we bookmark the link for later retrieval. We are trusting that the entity will be serving that content at any time in the future that you may want it.</p> -<p>This may not be a huge problem if the page is merely a list of <a class="reference external" href="https://dead-baby-joke.com/">dead baby jokes</a>. They are objectively funny, of course. But you can also get along without them.</p> -<p>But what of the case of formal, scientific texts we may depend on, and that use citations from the web as part of their source material? Usually, they refer to a source by <em>link</em> and <em>date of retrieval</em>. This is not of much use unless the actual source and/or render they saw at that time also is available.</p> -<p>That may not always be the case.</p> -<div class="section" id="take-care-of-your-shelf"> -<h2>Take care of your shelf</h2> -<blockquote> -"No worries, the <a class="reference external" href="https://archive.org">Wayback Machine</a> has me covered."</blockquote> -<p>Yes. But no. The Wayback Machine is a (thus far) centralized entity that depend on a few idealists and donations to keep going. If they stop to keep going, they depend on passing the buck to someone else. If that someone is evil, they may take it and rewrite history to suit themselves. If that someone else cannot be found, it becomes a garbage collection blip on Bezos' <a class="reference external" href="https://gizmodo.com/i-tried-to-block-amazon-from-my-life-it-was-impossible-1830565336">infrastructure monopoly</a> dashboard. <a class="footnote-reference" href="#id6" id="id1">[1]</a></p> -<p>That aside, sources like Wayback Machine are like libraries. Libraries are, of course, essential. Not only because they serve as one of the pillars of democracy, providing free access to knowledge for everyone. They are also essential because it's simply not very practical for you to pre-emptively own all the books that you may at some point want to read. Let alone ask around in your neighborhood if they happen to have a copy (although a crowd-sourced library app sounds like a fun decentralization project to explore).</p> -<p>You may however want to keep a copy of the books you <em>depend</em> on, and the ones you <em>really like</em>. Just to make really sure you have it available when you want it. Then, if some New Public Management clowns get the chance to gut public infrastructure where you live, or someone starts a good old-fashioned fascist book burning, you have yourself covered.</p> -</div> -<div class="section" id="a-lack-of-friction"> -<h2>A lack of friction</h2> -<p>Yes, stuff may disappear on the web. Just as books may.</p> -<p>On the web that stuff can get <em>rewritten</em>. Books may be rewritten, too. <a class="footnote-reference" href="#id7" id="id2">[2]</a> The previous editions of the books will still exist as independent physical objects until they degrade, as long as something is not actively done to them. But so too with data on storage media. If it is not renewed or copied during that lifetime it may degrade until it becomes illegible. And of course, it may also simply be deleted. And without the smell of smoke at that.</p> -<p>That's the difference that seems to leap out when using this imagery. How <em>easy</em> it is to change and destroy stuff at scale on the web compared with the real world of books. And how inconspicuously it can happen, without <em>anyone</em> noticing. And for those who notice, it is very hard to prove what has changed, unless you have a copy. <a class="footnote-reference" href="#id10" id="id3">[5]</a></p> -<p>So what can we do? Copies and proofs are definitely keywords here. Fortunately, copying is what computers are all about. And making a cryptographical proof of what you see is easy enough these days, too. The tricky bit is to build <em>credibility</em> around that proof. But let's stick our head in the sand and start with the easy part and see where it takes us.</p> -</div> -<div class="section" id="look-no-head"> -<h2>Look, no head</h2> -<!-- .. WARNING:: - - Nerdy zone ahead --> -<p>A good start is to dump the document source to disk, then calculating and storing the sum of it.</p> -<p>In many cases this will be insufficient, though, as many sites populates the DOM through scripts, either in part or in full. As the use-case here is humans voting on what they see is what they get, human aids will be needed. In other words, we need to render the page to store what we actually see.</p> -<p>Printing to PDF from the browser is an option, but that is really difficult to automate. Fortunately, modern browser engines provide command line access to rendering. Since I mostly use <a class="reference external" href="https://brave.com/">Brave Browser</a> these days, we'll use <em>headless Chromium</em> here.</p> -<p>In addition to source, sum and rendering, we should also include a copy of the request headers for good measure.</p> -<p>Thus, we end up with something like this.</p> -<pre class="code bash literal-block"> -<span class="ch">#!/bin/bash -</span> -<span class="nv">f</span><span class="o">=</span><span class="si">${</span><span class="nv">WEBSHOT_OUTPUT_DIR</span><span class="k">:-</span><span class="p">/tmp</span><span class="si">}</span> -<span class="nv">url</span><span class="o">=</span><span class="nv">$1</span> -<span class="nv">title</span><span class="o">=</span><span class="nv">$2</span> -><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> using outdir <span class="nv">$f</span> - -<span class="nb">set</span> +e - -<span class="c1"># prepare -</span><span class="nv">d</span><span class="o">=</span><span class="sb">`</span><span class="nv">TZ</span><span class="o">=</span>UTC date +%Y%m%d%H%M<span class="sb">`</span> -<span class="nv">t</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> -<span class="nb">pushd</span> <span class="nv">$t</span> - -<span class="c1"># store raw outputs -</span><span class="nb">echo</span> <span class="nv">$1</span> > url.txt -curl -s -I <span class="nv">$1</span> > headers.txt -curl -s -X GET <span class="nv">$1</span> > contents.txt -<span class="nv">z</span><span class="o">=</span><span class="sb">`</span>sha256sum contents.txt<span class="sb">`</span> -<span class="nb">echo</span> <span class="nv">$z</span> > contents.txt.sha256 -<span class="nv">h</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -n <span class="nv">$z</span> <span class="p">|</span> awk <span class="s1">'{ print $1; }'</span><span class="sb">`</span> - -<span class="k">if</span> <span class="o">[</span> -z <span class="s2">"</span><span class="nv">$title</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - <span class="nv">title</span><span class="o">=</span><span class="nv">$h</span> -<span class="k">fi</span> -><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> using title <span class="nv">$title</span> - -<span class="c1"># rendered snapshot -</span>chromium --headless --print-to-pdf <span class="nv">$url</span> -<span class="nv">n</span><span class="o">=</span><span class="si">${</span><span class="nv">d</span><span class="si">}</span>_<span class="si">${</span><span class="nv">h</span><span class="si">}</span> -mv output.pdf <span class="nv">$n</span>.pdf - -<span class="c1"># store result -</span>mkdir -p <span class="s2">"</span><span class="nv">$f</span><span class="s2">/</span><span class="nv">$title</span><span class="s2">"</span> -tar -zcvf <span class="s2">"</span><span class="nv">$f</span><span class="s2">/</span><span class="nv">$title</span><span class="s2">/</span><span class="nv">$n</span><span class="s2">.tar.gz"</span> * - -<span class="c1"># clean up -</span><span class="nb">popd</span> - -<span class="nb">set</span> -e -</pre> -</div> -<div class="section" id="what-does-this-mean"> -<h2>What does this mean</h2> -<p>Let's sum up what information we've managed to store with this operation.</p> -<ul class="simple"> -<li>We have a copy of the unrendered source. It may or may not include all the information we want to store.</li> -<li>We have a fingerprint of that source.</li> -<li>We have a copy of the headers we were served when retrieving the document. <a class="footnote-reference" href="#id8" id="id4">[3]</a></li> -<li>We have a image copy of what we actually saw when visiting the page.</li> -<li>We have a date and time for retrieval (file attributes).</li> -</ul> -<p>To link the headers together with the visual copy, we could sum the header file and image file aswell, put those sums together with the content sum in a deterministic order, and calculate the sum of those sums. E.g. <a class="footnote-reference" href="#id9" id="id5">[4]</a></p> -<div class="highlight"><pre><span></span>$ cp contents.txt.sha256 sums.txt -$ sha256sum headers.txt <span class="p">|</span> awk <span class="s1">'{ print $1; }'</span> >> sums.txt -$ sha256sum <pdf file> <span class="p">|</span> awk <span class="s1">'{ print $1; }'</span> >> sums.txt -$ sha256sum sums.txt <span class="p">|</span> awk <span class="s1">'{ print $1; }'</span> > topsum.txt -</pre></div> -<p>If we now sign <em>this</em> sum, we are confirming that for this particuar resource:</p> -<blockquote> -"This was the source. These were the headers for that source. This is how that source served in that manner looked for <strong>ME</strong> at the time"</blockquote> -</div> -<div class="section" id="proving-links-in-this-post"> -<h2>Proving links in this post</h2> -<p>This post makes use of several external links to articles. So as a final step, let's eat our own dogfood and add proofs for them.</p> -<table border="1" class="colwidths-given docutils"> -<caption>Article retrieval proofs</caption> -<colgroup> -<col width="34%" /> -<col width="22%" /> -<col width="22%" /> -<col width="22%" /> -</colgroup> -<thead valign="bottom"> -<tr><th class="head">Link</th> -<th class="head">Content</th> -<th class="head">Header</th> -<th class="head">Image</th> -</tr> -</thead> -<tbody valign="top"> -<tr><td><a class="reference external" href="https://gizmodo.com/i-tried-to-block-amazon-from-my-life-it-was-impossible-1830565336">https://gizmodo.com/i-tried-to-block-amazon-from-my-life-it-was-impossible-1830565336</a></td> -<td><a class="reference external" href="./misc/web_snapshot/0c657ec25e55e72702b0473f62e6d632dece992485f67c407ed1f748b3f40bc2.txt">0c657ec2</a></td> -<td><a class="reference external" href="./misc/web_snapshot/360125fa513b8db8380eb2b62c4479164baa5b48d8544b2242bcc7305bad0de4.txt">360125fa</a></td> -<td><a class="reference external" href="./misc/web_snapshot/a2584ca07ba16b15b9fad315a767cbb0c4b7124abdfd578cab2afb4dce9d1971.pdf">a2584ca0</a></td> -</tr> -<tr><td><a class="reference external" href="https://www.thelocal.se/20111109/37244/">https://www.thelocal.se/20111109/37244/</a></td> -<td><a class="reference external" href="./misc/web_snapshot/be8741ff91c6e3eac760a3a4652b57f47bce92fa9b617662296c2ab5b3c5fe31.txt">be8741ff</a></td> -<td><a class="reference external" href="./misc/web_snapshot/6c20e7678a0235467687425b9818718ab143fd477afee2087d7f79c933abdc75.txt">6c20e767</a></td> -<td><a class="reference external" href="./misc/web_snapshot/f1c557b9555149dde570976ed956ffb17d17b99cea5f2651020f66408dacf301.pdf">f1c557b9</a></td> -</tr> -<tr><td><a class="reference external" href="https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/">https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/</a></td> -<td><a class="reference external" href="./misc/web_snapshot/595a07b85e6b75a41fe0880c8538f15c4b6da5770db230d986efa2e080ca479a.txt">595a07b8</a></td> -<td><a class="reference external" href="./misc/web_snapshot/f2aa95b42c5420cb11ccbf1cb084d5e5ccc3fc6cd575c51e9ee473b9df19c890.txt">f2aa95b4</a></td> -<td><a class="reference external" href="./misc/web_snapshot/6a610fa69d4909cc9aa1dcb7105203cc50c1bcf26733411964f95cbcbdf37eb5.pdf">6a610fa6</a></td> -</tr> -<tr><td><a class="reference external" href="https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html">https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html</a></td> -<td><a class="reference external" href="./misc/web_snapshot/5a806be410da82986a85f68658279234c1a5cf3bb6dc55da137d5274dc722f26.txt">5a806be4</a></td> -<td><a class="reference external" href="./misc/web_snapshot/078d9fe6be070de0d378a6e903f8558a7da4917cba5c95ca453ae1936541e4f6.txt">078d9fe6</a></td> -<td><a class="reference external" href="./misc/web_snapshot/2d0f0e69e7c8b6ffe9ff8ffc8702a78d6a2d46ab1edd4123e84eb211171d6cde.pdf">2d0f0e69</a></td> -</tr> -<tr><td><a class="reference external" href="https://www.breitbart.com/europe/2017/07/19/swedens-libraries-pulp-traditional-children-books-racist-phrases/">https://www.breitbart.com/europe/2017/07/19/swedens-libraries-pulp-traditional-children-books-racist-phrases/</a></td> -<td><a class="reference external" href="./misc/web_snapshot/a8c6b61cec2cce1a58bcf7a65091a6c2e8510ca5fa17f2d242d286f087d95cd5.txt">a8c6b61c</a></td> -<td><a class="reference external" href="./misc/web_snapshot/3ba9094b295a898cfe8cba256f4ebf65ef98ff05bb3034e048e517d52fc13d33.txt">3ba9094b</a></td> -<td><a class="reference external" href="./misc/web_snapshot/eb76a87f224b0e4f4e5d1673b1505aaeee28d8ad03ce544656b6f5ac7c4d9983.pdf">eb76a87f</a></td> -</tr> -</tbody> -</table> -<p>Clicking on the "image" links, we see that thanks to the recent ubiquity of cookie nag boxes screaming "accept all" at you, those very boxes are now blocking the content we want to get at. So we will need more work to get to where we want by automation. But it's s start.</p> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id6" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Early 2021 survey puts Amazon at one-third of the global market share. <a class="reference external" href="https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/">https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/</a></td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id7" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>2011 sparked a <a class="reference external" href="https://www.thelocal.se/20111109/37244/">controversy</a> around Astrid Lindgren's Pippi Longstockings. Echoing those viewpoints, the books were edited in 2015 during which some alleged "racist" content was altered. The rabid rightwing media later spun a <a class="reference external" href="https://www.breitbart.com/europe/2017/07/19/swedens-libraries-pulp-traditional-children-books-racist-phrases/">false tale of mass purges</a> of Pippi books around one single swedish library's decision to throw out copies of the originial "racist" versions. Case-in-point, a public statement in which the library tries to justify its actions is no longer available on their website, and has to be retrieved by the Wayback Machine. <a class="reference external" href="https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html">https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html</a> (in swedish)</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id8" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id4">[3]</a></td><td>Well actually, not quite. We did the same request twice, but they were two separate requests. Using a single request would improve the script.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id9" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id5">[4]</a></td><td>We use the hex representation for clarity here. A proper tool would convert the hex values to bytes before calculating sum on them.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id10" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id3">[5]</a></td><td>Another important difference is that the book does not need to be <em>interpreted</em> by a <em>machine</em> in order to make sense for a human. Availability of tooling is definitely an equally important topic in this discussion. However this post limits focus to the actual data itself.</td></tr> -</tbody> -</table> -</blockquote> -<!-- .. _data availability: https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.433.3480 --> -<!-- https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html <- not available on link, no result on search https://www.mhpbooks.com/the-trouble-with-pippi/ --> -</div> - - </div><!-- /.entry-content --> - <footer class="meta"> - </footer> - -</section> - <footer id="contentinfo" class="body"> - <hr/> - <address id="about" class="vcard body"> - Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> - </address><!-- /#about --> - </footer><!-- /#contentinfo --> -</body> -</html> -\ No newline at end of file diff --git a/132b445a5c24f134b82b8cb95e7112fd053ca2c15531afa694105a5d1a1d872a.asc b/132b445a5c24f134b82b8cb95e7112fd053ca2c15531afa694105a5d1a1d872a.asc @@ -1,11 +0,0 @@ ------BEGIN PGP SIGNATURE----- - -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmHkZ0UACgkQzcvSTdHQ -4AH7oggAo8ILCivCHh3RxAOQxTuwRUZbJaPgqTm5LlvehBp2H5l/Fi5ZOcrhswVw -ROt/Q9AwJOpTHQa6EKnZF1it0Ho+N6A3KbTy665iLSb0GZSYyQ6HmAFv+hihuty9 -L7B2DSbUTp3wkwvB8sy88VTFr+MgSuJKvP8MzdYEa/+sVngsldYflhxqJU0AdlYq -ib9aemr1vJNZGb9VGWJwSnyzHUM8nLlF3bM9ts11/dQ9pDr5t7TWQBSxupKsgSCL -Vv8/ci4SmPPvSpVauOZ59D/xbZ1HyqF8o9+QSv1V40o8njEXgFD7cJxJ7nC1fH2g -ieashaiyddU7pjoGpKo2WcUCoTKZzg== -=i0t/ ------END PGP SIGNATURE----- diff --git a/1c19dbdeef6f5dea9dfc688b2aff4784fe2c5dbb2a8af928974d41c536f6afdd b/1c19dbdeef6f5dea9dfc688b2aff4784fe2c5dbb2a8af928974d41c536f6afdd @@ -1,315 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> - <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> - <title>man bytes gnu - Combining duplicity and rsync</title> - <meta charset="utf-8" /> - <meta name="generator" content="Pelican" /> - -</head> - -<body id="index" class="home"> - - <header id="banner" class="body"> - <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> - </header> - <nav id="menu"> - <ul> - <li class="active"><a href="./category/archiving.html">Archiving</a></li> - <li><a href="./category/code.html">Code</a></li> - <li><a href="./category/hygiene.html">Hygiene</a></li> - <li><a href="./category/offlining.html">Offlining</a></li> - <li><a href="./pages/identities.html">Identities</a></li> - <li><a href="./pages/shares.html">Shares</a></li> - - <li><a href="/tags.html">tags</a></li> - </ul> - <hr/> - </nav><!-- /#menu --> - <header> - <h1 class="entry-title top-body-title"> - <a href="./backup-rsync-duplicity.html" rel="bookmark" - title="Permalink to Combining duplicity and rsync">Organizing backups: Combining duplicity and rsync</a></h1> - - - <div class="category meta"> - Posted -<time class="published" datetime="2022-01-15T16:57:00+01:00"> - Sat 15 January 2022 - </time> -in <a class="category" href="./category/archiving.html">archiving</a> - <a href="./tag/backup.html">backup</a> - <a href="./tag/rsync.html">rsync</a> - <a href="./tag/duplicity.html">duplicity</a> - <a href="./tag/bash.html">bash</a> - </div> - - <div class="neighbors meta"> - <ul> - <li> - Previous: <a href="./internet-up-monitor.html"> - Homemade internet state monitor - </a> - </li> - </ul> - </div> - <div class="entry-series meta"> -Part 1 from the series "Organizing backups" - <ol class="entry-series-parts"> - </ol> - </div> - <div class="meta"> -<hr/> - </div> -</header> - -<section id="content" class="body"> - <div class="entry-content"> - <p>There are two awesome, weathered tools out there that are all you really need for your personal backups. <a class="footnote-reference" href="#id4" id="id1">[1]</a> One is the <a class="reference external" href="https://rsync.samba.org/">rsync cli</a>, the other is <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a>.</p> -<p>The former should need no introduction.</p> -<p>The latter operates more like tar. But it still works over ssh like rsync. In fact, it's based on <a class="reference external" href="http://librsync.sourcefrog.net/">librsync</a> which implements the <a class="reference external" href="https://rsync.samba.org/tech_report/">rsync protocol</a>. The special sauce however is, of course, <em>encryption</em>.</p> -<div class="section" id="backup-categories"> -<h2>Backup categories</h2> -<p>Let's for the sake of argument say that our personal backups can be divided in three categories:</p> -<div class="section" id="stuff-that-can-be-public"> -<h3>Stuff that can be public</h3> -<p>Code snippets, git repositories, public data store states (e.g. blockchain ledgers), copies of OS packages and any other assets assets without redistribution issues.</p> -<p>For this we will use <a class="reference external" href="https://rsync.samba.org/">rsync</a>.</p> -</div> -<div class="section" id="sensitive-stuff"> -<h3>Sensitive stuff</h3> -<p>Passwords, keys, contacts, calendars, contracts, invoices, task lists, databases, system configurations, application data.</p> -<p>For this we will use <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a>.</p> -</div> -<div class="section" id="secret-stuff"> -<h3>Secret stuff</h3> -<p>Long-lived keys, password- and volume decryption keys, cryptocurrency keys and meta-information about the backups themselves.</p> -<p>This will not be addressed now.</p> -</div> -</div> -<div class="section" id="why-not-just-one-or-the-other"> -<h2>Why not just one or the other?</h2> -<p><a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">Duplicity</a> stores everything in an archive file format. That means that you must first authenticate, decrypt and unpack the archive in order to even browse the files inside.</p> -<p>If there is no reason to keep the files from prying eyes, then it's much more practical to be able to browse the files where they lie, with the regular filesystem tools. In such a case, <a class="reference external" href="https://rsync.samba.org/">rsync</a> will scratch your itch.</p> -<p>For the <strong>sensitive</strong> and <strong>secret stuff</strong>, there would be no real need to use <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a> if you were only operating on your local host. You'd just use an encrypted volume <a class="footnote-reference" href="#id5" id="id2">[2]</a> and <a class="reference external" href="https://rsync.samba.org/">rsync</a> everything in there.</p> -<p>But half the point here is to keep remote copies aswell as your local ones. You know, in case of fire, hardware-eating locust swarms or some totalitarian minions nabbing all your electronics. Unless "remote" here means some box hidden in some moated leisure castle of yours, you'll want to encrypt everything <em>before</em> you ship it off. And that's where <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a> comes in.</p> -</div> -<div class="section" id="vive-la-difference"> -<h2>Vive la difference</h2> -<p>Of course, it would be too much to hope for that <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a> and <a class="reference external" href="https://rsync.samba.org/">rsync cli</a> have aligned the ways they parse their invocation parameters.</p> -<p>Here are some examples <a class="footnote-reference" href="#id6" id="id3">[3]</a> of how they do <em>not</em> match:</p> -<div class="section" id="local-to-local"> -<h3>local to local</h3> -<div class="highlight"><pre><span></span>$ rsync -a src/ /path/to/dst/ - -$ duplicity src/ file:///path/to/dst/ -</pre></div> -</div> -<div class="section" id="local-to-remote-relative-path"> -<h3>local to remote, relative path</h3> -<div class="highlight"><pre><span></span>$ rsync -a src/ user@remotehost:path/to/dst/ - -$ duplicity src/ scp://user@remotehost/path/to/dst -</pre></div> -</div> -<div class="section" id="toggle-dotfiles-from-current-path"> -<h3>toggle dotfiles from current path</h3> -<div class="highlight"><pre><span></span><span class="c1"># include only .foo/foo.txt given the current structure:</span> -$ tree src/ -a -src/ -├── .bar -├── baz -└── .foo - └── foo.txt - -$ rsync --exclude<span class="o">=</span><span class="s2">".b*"</span> --include<span class="o">=</span><span class="s2">".*/***"</span> --exclude<span class="o">=</span><span class="s2">"*"</span> ./ ../dst/ - -$ duplicity --exclude<span class="o">=</span><span class="s2">"./.b*"</span> --include<span class="o">=</span><span class="s2">"./.*/***"</span> --exclude<span class="o">=</span><span class="s2">"*"</span> ./ file:///home/lash/tmp/dst/ -</pre></div> -</div> -<div class="section" id="logging"> -<h3>logging</h3> -<div class="highlight"><pre><span></span><span class="c1"># spill the beans</span> -$ rsync -vv ... - -$ duplicity -v debug -</pre></div> -</div> -</div> -<div class="section" id="batchin"> -<h2>Batchin'</h2> -<p>Since you will want to select up front which tool to use for which sensititivy category, you'll be writing the includes and excludes specifically for the tool anyway.</p> -<p>So the only real issue with the above is the way remote host is specified.</p> -<p>Let's say we choose to stick to the <a class="reference external" href="https://rsync.samba.org/">rsync cli</a> host format. That means we need to make the following translations:</p> -<table border="1" class="colwidths-given docutils"> -<colgroup> -<col width="50%" /> -<col width="50%" /> -</colgroup> -<thead valign="bottom"> -<tr><th class="head">rsync</th> -<th class="head">duplicity</th> -</tr> -</thead> -<tbody valign="top"> -<tr><td><tt class="docutils literal">foo/bar</tt></td> -<td><tt class="docutils literal"><span class="pre">file://foo/bar</span></tt></td> -</tr> -<tr><td><tt class="docutils literal">/foo/bar</tt></td> -<td><tt class="docutils literal"><span class="pre">file:///foo/bar</span></tt></td> -</tr> -<tr><td><tt class="docutils literal">user@host:foo/bar</tt></td> -<td><tt class="docutils literal"><span class="pre">scp://user@host/foo/bar</span></tt></td> -</tr> -<tr><td><tt class="docutils literal"><span class="pre">user@host:/foo/bar</span></tt></td> -<td><tt class="docutils literal"><span class="pre">scp://user@host//foo/bar</span></tt></td> -</tr> -</tbody> -</table> -<p>Expressed in <tt class="docutils literal">bash</tt> that could look like this:</p> -<pre class="code bash literal-block"> -to_duplicity_remote<span class="o">()</span> <span class="o">{</span> - - <span class="c1"># remote host is defined in rsync format -</span> <span class="c1"># ... and we will only support scp -</span> <span class="c1"># $remote_base is the path we want to parse -</span> <span class="nv">remote_duplicity_base</span><span class="o">=</span> - <span class="nv">remote_base</span><span class="o">=</span><span class="nv">$1</span> - - <span class="c1"># substring up until the first slash -</span> <span class="nv">s_firstslash</span><span class="o">=</span><span class="si">${</span><span class="nv">remote_base</span><span class="p">%%/*</span><span class="si">}</span> - - <span class="c1"># substring up until the first colon -</span> <span class="nv">s_firstcolon</span><span class="o">=</span><span class="si">${</span><span class="nv">remote_base</span><span class="p">%%:*</span><span class="si">}</span> - - <span class="c1"># string index of the first slash -</span> <span class="nv">i_firstslash</span><span class="o">=</span><span class="k">$((</span><span class="si">${#</span><span class="nv">s_firstslash</span><span class="si">}</span><span class="k">))</span> - - <span class="c1"># string index of the first colon -</span> <span class="nv">i_firstcolon</span><span class="o">=</span><span class="k">$((</span><span class="si">${#</span><span class="nv">s_firstcolon</span><span class="si">}</span><span class="k">))</span> - - <span class="c1"># if colon is before first slash that most likely means we have a remote host -</span> <span class="c1"># (Exception is if first directory of path happens to have ":" in it. Seriously, don't use ":" in filenames) -</span> <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$i_firstcolon</span><span class="s2">"</span> -gt <span class="s2">"0"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - - <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$i_firstcolon</span><span class="s2">"</span> -lt <span class="s2">"</span><span class="nv">$i_firstslash</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - - <span class="c1"># pexpect addition due to lack of implicit private key fetch, without pexpect works only with key wo pwd -</span> <span class="c1"># https://serverfault.com/questions/982591/duplicity-backup-fails-private-key-file-is-encrypted -</span> <span class="nv">remote_duplicity_base</span><span class="o">=</span><span class="s2">"pexpect+scp://"</span> - - <span class="c1"># trim url so that colon after hostname is removed -</span> <span class="c1"># (no support here for setting an alternate port number) -</span> <span class="nv">remote_duplicity_base</span><span class="o">=</span><span class="si">${</span><span class="nv">remote_duplicity_base</span><span class="si">}${</span><span class="nv">remote_base</span><span class="p">:</span><span class="nv">0</span><span class="p">:</span><span class="nv">$i_firstcolon</span><span class="si">}</span> - <span class="nv">remote_duplicity_base</span><span class="o">=</span><span class="si">${</span><span class="nv">remote_duplicity_base</span><span class="si">}</span>/<span class="si">${</span><span class="nv">remote_base</span><span class="p">:((</span><span class="nv">$i_firstcolon</span><span class="p">+1))</span><span class="si">}</span> - - <span class="c1"># indicate that we have a remote host -</span> <span class="nv">remote</span><span class="o">=</span><span class="m">1</span> - <span class="k">fi</span> - <span class="k">fi</span> - - <span class="c1"># If it's not a remote host, treat it as a file -</span> <span class="k">if</span> <span class="o">[</span> -z <span class="nv">$remote_duplicity_base</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - <span class="nv">remote_duplicity_base</span><span class="o">=</span><span class="s2">"file://</span><span class="si">${</span><span class="nv">remote_base</span><span class="si">}</span><span class="s2">"</span> - <span class="k">fi</span> -<span class="o">}</span> - -<span class="k">if</span> <span class="o">[</span> ! -z <span class="s2">"</span><span class="nv">$BAK_TEST</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - <span class="nv">src</span><span class="o">=(</span>/foo/bar foo/bar localhost:foo/bar localhost:/foo/bar<span class="o">)</span> - <span class="nv">res</span><span class="o">=(</span>file:///foo/bar file://foo/bar pexpect+scp://localhost/foo/bar pexpect+scp://localhost//foo/bar<span class="o">)</span> - - <span class="nv">i</span><span class="o">=</span><span class="m">0</span> - <span class="k">for</span> case_src <span class="k">in</span> <span class="si">${</span><span class="nv">src</span><span class="p">[@]</span><span class="si">}</span><span class="p">;</span> <span class="k">do</span> - <span class="nv">case_res</span><span class="o">=</span><span class="si">${</span><span class="nv">res</span><span class="p">[</span><span class="nv">$i</span><span class="p">]</span><span class="si">}</span> - - to_duplicity_remote <span class="nv">$case_src</span> - <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$remote_duplicity_base</span><span class="s2">"</span> !<span class="o">=</span> <span class="s2">"</span><span class="nv">$case_res</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> <span class="s2">"expected </span><span class="nv">$case_res</span><span class="s2"> got </span><span class="nv">$remote_duplicity_base</span><span class="s2"> from </span><span class="nv">$case_src</span><span class="s2">"</span> - <span class="nb">exit</span> <span class="m">1</span> - <span class="k">elif</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$remote_base</span><span class="s2">"</span> !<span class="o">=</span> <span class="s2">"</span><span class="nv">$case_src</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$case_src</span><span class="s2"> got mangled into </span><span class="nv">$remote_base</span><span class="s2">"</span> - <span class="nb">exit</span> <span class="m">1</span> - <span class="k">fi</span> - <span class="nv">i</span><span class="o">=</span><span class="k">$((</span>i+1<span class="k">))</span> - <span class="k">done</span> -<span class="k">fi</span> -</pre> -<p>Let's behave and test our code:</p> -<pre class="code bash literal-block"> -<span class="k">if</span> <span class="o">[</span> ! -z <span class="s2">"</span><span class="nv">$BAK_TEST</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - <span class="nv">src</span><span class="o">=(</span>/foo/bar foo/bar localhost:foo/bar localhost:/foo/bar<span class="o">)</span> - <span class="nv">res</span><span class="o">=(</span>file:///foo/bar file://foo/bar pexpect+scp://localhost/foo/bar pexpect+scp://localhost//foo/bar<span class="o">)</span> - - <span class="nv">i</span><span class="o">=</span><span class="m">0</span> - <span class="k">for</span> case_src <span class="k">in</span> <span class="si">${</span><span class="nv">src</span><span class="p">[@]</span><span class="si">}</span><span class="p">;</span> <span class="k">do</span> - <span class="nv">case_res</span><span class="o">=</span><span class="si">${</span><span class="nv">res</span><span class="p">[</span><span class="nv">$i</span><span class="p">]</span><span class="si">}</span> - - to_duplicity_remote <span class="nv">$case_src</span> - <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$remote_duplicity_base</span><span class="s2">"</span> !<span class="o">=</span> <span class="s2">"</span><span class="nv">$case_res</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> <span class="s2">"expected </span><span class="nv">$case_res</span><span class="s2"> got </span><span class="nv">$remote_duplicity_base</span><span class="s2"> from </span><span class="nv">$case_src</span><span class="s2">"</span> - <span class="nb">exit</span> <span class="m">1</span> - <span class="k">elif</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$remote_base</span><span class="s2">"</span> !<span class="o">=</span> <span class="s2">"</span><span class="nv">$case_src</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$case_src</span><span class="s2"> got mangled into </span><span class="nv">$remote_base</span><span class="s2">"</span> - <span class="nb">exit</span> <span class="m">1</span> - <span class="k">fi</span> - <span class="nv">i</span><span class="o">=</span><span class="k">$((</span>i+1<span class="k">))</span> - <span class="k">done</span> -<span class="k">fi</span> -</pre> -<div class="highlight"><pre><span></span><span class="c1"># 0 == good!</span> -$ <span class="nv">BAK_TEST</span><span class="o">=</span><span class="m">1</span> bash remote.sh <span class="o">&&</span> <span class="nb">echo</span> <span class="nv">$?</span> -<span class="m">0</span> -</pre></div> -<p>Now we can use the <a class="reference external" href="https://rsync.samba.org/">rsync cli</a> path input, and use that same input to a batch of single backup steps, each which may use <a class="reference external" href="https://rsync.samba.org/">rsync cli</a> or <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a></p> -<div class="highlight"><pre><span></span>to_duplicity_remote localhost:/foo/bar - -rsync -avzP pub/ <span class="nv">$remote_base</span>:src/ - -duplicity -v info secret/ <span class="nv">$remote_duplicity_base</span>:secret/ -</pre></div> -</div> -<div class="section" id="see-also"> -<h2>See also</h2> -<ul class="simple"> -<li><a class="reference external" href="https://git.defalsify.org/rsync-duplicity-backups">https://git.defalsify.org/rsync-duplicity-backups</a></li> -</ul> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id4" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Ok, I know, I assuming that you are using <tt class="docutils literal">git</tt> in daily life, too.</td></tr> -</tbody> -</table> -<table class="docutils footnote" frame="void" id="id5" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Provided, of course, that it's an encrypted volume that you don't keep unlocked all the time.</td></tr> -</tbody> -</table> -<table class="docutils footnote" frame="void" id="id6" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>Duplicity needs at a minimum a password for symmetric encryption, and will prompt for it unless it's set in the environment. Simply <tt class="docutils literal">export PASSPHRASE=test</tt> for these examples to relieve you of the annoyance.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -</blockquote> -</div> - - </div><!-- /.entry-content --> - <footer class="meta"> - </footer> - -</section> - <footer id="contentinfo" class="body"> - <hr/> - <address id="about" class="vcard body"> - Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> - </address><!-- /#about --> - </footer><!-- /#contentinfo --> -</body> -</html> -\ No newline at end of file diff --git a/1c19dbdeef6f5dea9dfc688b2aff4784fe2c5dbb2a8af928974d41c536f6afdd.asc b/1c19dbdeef6f5dea9dfc688b2aff4784fe2c5dbb2a8af928974d41c536f6afdd.asc @@ -1,11 +0,0 @@ ------BEGIN PGP SIGNATURE----- - -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmHkZ0UACgkQzcvSTdHQ -4AG4Mgf7BS4wTvEqRqo5QpUekr7cQ7DwFrcrwIt6pYnUjqbtXnaIqXrIOBBOWrcg -Cck0FXDHvLSvPBI/RKCSCJz7RaTgkxiAZBgB2e0eXJjPuILSk0p7X2vivm9TEcHB -6ckcxhG7Le6gjK/5/8ImJ+rTB9lVMmFlUSVliw359tbx/yUVidmxUK14kYxAV0k7 -cAaebixofVkjutR8YvFiskv9t5/0sPKzFK1yRt//KP9IFzKhjyF9LTf3lOm/EUT+ -mp4Is4QtSdobCuVdS+CHAHgGBZGupRgU4RLIdX61SiIrgNoGNT3rbp1sILHfwNnG -ZVyFLO+mz0xJoaKJHKo/PAa7AF2Ffw== -=6Y6p ------END PGP SIGNATURE----- diff --git a/24345ef2cf2edbf32d9ec3e55a6e0801a5a73f8337bb81bc1a7395ebbe3bf7e0 b/24345ef2cf2edbf32d9ec3e55a6e0801a5a73f8337bb81bc1a7395ebbe3bf7e0 @@ -0,0 +1,291 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - The NFT token id as URI</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="./category/archiving.html">Archiving</a></li> + <li class="active"><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./nft-tokenid-content.html" rel="bookmark" + title="Permalink to The NFT token id as URI">The NFT token id as URI</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2021-05-08T19:14:00+02:00"> + Sat 08 May 2021 + </time> +in <a class="category" href="./category/code.html">code</a> + <a href="./tag/nft.html">nft</a> + <a href="./tag/evm.html">evm</a> + <a href="./tag/hash.html">hash</a> + <a href="./tag/key-value-store.html">key-value store</a> + <a href="./tag/decentralized-storage.html">decentralized storage</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./wasm-c-bare.html"> + wasm and C - The bare necessities + </a> + </li> + | + + <li> + Next: <a href="./village-network-protocol.html"> + A village network protocol + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>Let's consider an NFT that works like a badge for participating in development of a software project.</p> +<p>This token is awarded as a proof that the task was completed.</p> +<p>To make things more fun, each NFT should have some unique, immutable content attached to it.</p> +<p>In other words, the properties of this token, once set, should never change.</p> +<p>Nor should they disappear.</p> +<p>So how do we refer to the artwork asset within the token standard?</p> +<div class="section" id="it-was-acceptable-at-the-time"> +<h2>It was acceptable at the time</h2> +<p>The ERC721 standard is not explicit about where the assets that belong with the NFT can be discovered and resolved.</p> +<p>At the time when the standard was adopted by the Ethereum community, there were multiple <em>"[...] Alternatives considered: put all metadata for each asset on the blockchain (too expensive), use URL templates to query metadata parts (URL templates do not work with all URL schemes, especially P2P URLs), multiaddr network address (not mature enough)."</em> Furthermore, they <em>"[...] considered an NFT representing ownership of a house, in this case metadata about the house (image, occupants, etc.) can naturally change."</em> <a class="citation-reference" href="#eip721" id="citation-reference-1">[EIP721]</a></p> +<p>A "changing house" doesn't sound quite like what we need. And anyway; if we stick a good old web2 URI in there, then that will end up on the great bonfire of dead links before long.</p> +</div> +<div class="section" id="image-schmimage"> +<h2>Image, schmimage</h2> +<p>To be honest, I find the presumption in the optional EIP721 metadata structure to be surprisingly short-sighted. It <em>specifically</em> defines the asset as an image, and at the same time is presupposes that only a <em>single</em> asset file will be used.</p> +<p>We may want to add <em>multiple</em> sources, so this is another obstacle for us.</p> +<p>So how to get around this, while still playing nice with existing implementations out there? Two ideas come to mind:</p> +<ul class="simple"> +<li>Embed a <em>thumbnail</em> as a preview of the artwork using a <code>base64</code> <em>data URI</em> <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a> in the metadata. Stick <code>name</code> and <code>description</code> on it, and the schema is still fulfilled.</li> +<li><em>Extend</em> the structure with a list of <em>attachments</em> that <em>our</em> application layer knows about. Of course, each of these can have the same format as above.</li> +</ul> +<p>In other words:</p> +<pre class="code json literal-block"> +<span class="p">{</span><span class="w"> + </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo image"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"image"</span><span class="p">:</span><span class="w"> </span><span class="nt">"data:image/gif;base64,R0lGODlhDgARAKEDAAAAAOjqAPP1APDw8CH+EUNyZWF0ZWQgd2l0aCBHSU1QACwAAAAADgARAAACO5wHqXdrClocodbUaGg34qoBHaJtl7VYIfqBZxepb5C6NBvbIlyucO6jPUIK1CMiSCYvDIVS4GhCJoYCADs="</span><span class="w"> + </span><span class="nt">"attachments"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> + </span><span class="p">{</span><span class="w"> + </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Bar"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Bar image"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"image"</span><span class="p">:</span><span class="w"> </span><span class="s2">"data:image/gif;base64,"</span><span class="w"> + </span><span class="p">},</span><span class="w"> + </span><span class="p">{</span><span class="w"> + </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Baz"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Baz image"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"image"</span><span class="p">:</span><span class="w"> </span><span class="s2">"data:image/gif;base64,"</span><span class="w"> + </span><span class="p">}</span><span class="w"> + </span><span class="p">]</span><span class="w"> +</span><span class="p">}</span> +</pre> +</div> +<div class="section" id="mirror-mirror"> +<h2>Mirror, mirror</h2> +<p>Since the asset reference shouldn't change, we can refer to it by its fingerprint or <a class="reference external" href="https://en.wikipedia.org/wiki/Content-addressable_storage">content address</a>. If we define that the resource can be looked up over HTTP by that fingerprint as its basename, then we are free to define and modify whatever list of mirrors for that resource that's valid for any point in time. The application layer would simply try the endpoints one after another.</p> +<p>We take the <code>sha2-256</code> <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a> of the asset reference (the json file above, free of evil whitespace and newlines):</p> +<div class="highlight"><pre><span></span>$ cat reference.json <span class="p">|</span> jq -c -j <span class="p">|</span> sha256sum <span class="p">|</span> awk <span class="s1">'{ print $1; }'</span> +3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551 +</pre></div> +<p>Imagine we had a mirror list of <a class="reference external" href="https://foo.com">https://foo.com</a> and <a class="reference external" href="https://bar.com/baz/">https://bar.com/baz/</a>. Then our application would try these urls in sequence, stopping at the first that returns a valid result:</p> +<div class="highlight"><pre><span></span>https://foo.com/3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551 +https://bar.com/baz/3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551 +</pre></div> +<p>Once we receive the content, all we have to do is hash it ourselves and verify that the sum matches the basename of the URI. If it doesn't the result is of course not valid and we continue down the list, appropriately banning the mischievous server then throrougly harassing its admin.</p> +</div> +<div class="section" id="cast-away"> +<h2>Cast away</h2> +<p>Since our fingerprint is 32 bytes, it fits exactly inside the <code>tokenId</code> (<code>uint256</code>). Let's decide to big-endian numbers when converting (I find them easier to make sense of). In that case our hash from the reference turns into this modest number:</p> +<div class="highlight"><pre><span></span><span class="c1"># python3</span> +<span class="o">>>></span> <span class="n">hx</span> <span class="o">=</span> <span class="nb">bytes</span><span class="o">.</span><span class="n">fromhex</span><span class="p">(</span><span class="s1">'3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551'</span><span class="p">)</span> +<span class="o">>>></span> <span class="nb">int</span><span class="o">.</span><span class="n">from_bytes</span><span class="p">(</span><span class="n">hx</span><span class="p">,</span> <span class="n">byteorder</span><span class="o">=</span><span class="s1">'big'</span><span class="p">)</span> +<span class="mi">28891040728719892888467057134569335350980764617882743994259054993630416573777</span> +</pre></div> +<p>As long as we're composing the <code>evm</code> inputs ourselves, we don't really have to worry about the integer representation in this particular case. But the interface is defined as an integer type, and other mortals may be using higher level interfaces, we have to be explicit about our choice.</p> +</div> +<div class="section" id="welcoming-mint"> +<h2>Welcoming mint</h2> +<p>Assume we have a method <code>mintTo(address _recipient, uint256 _tokenId)</code> on our NFT contract. The Solidity signature of that contract is <code>edb20b7e</code> <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a>. If I were to mint to myself then the input to the contract would be:</p> +<div class="highlight"><pre><span></span>edb20b7e000000000000000000000000185cbce7650ff7ad3b587e26b2877d95568805e33fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551 +</pre></div> +<p>Broken down:</p> +<div class="highlight"><pre><span></span>signature: edb20b7e +address, zero-padded: 000000000000000000000000185cbce7650ff7ad3b587e26b2877d95568805e3 +token id: 3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551 +</pre></div> +<p>The corresponding web3.js code would look like:</p> +<div class="highlight"><pre><span></span><span class="kd">const</span><span class="w"> </span><span class="nx">c</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">web3</span><span class="p">.</span><span class="nx">eth</span><span class="p">.</span><span class="nx">Contract</span><span class="p">([...],</span><span class="w"> </span><span class="s1">'0x...'</span><span class="p">);</span><span class="w"></span> +<span class="nx">c</span><span class="p">.</span><span class="nx">methods</span><span class="p">.</span><span class="nx">mintTo</span><span class="p">(</span><span class="s1">'0x185cbce7650ff7ad3b587e26b2877d95568805e3'</span><span class="p">,</span><span class="w"> </span><span class="mf">28891040728719892888467057134569335350980764617882743994259054993630416573777</span><span class="p">).</span><span class="nx">call</span><span class="p">();</span><span class="w"></span> +</pre></div> +<p>To satisfy the <cite>tokenURI</cite> method, we can generate a string that's prefix with sha256 as a "scheme" <a class="footnote-reference" href="#footnote-4" id="footnote-reference-4">[4]</a>. A bit of (unoptimized) solidity helps us out here:</p> +<pre class="code solidity literal-block"> +<span class="k">contract</span><span class="w"> </span><span class="ni">NFT</span><span class="w"> </span><span class="p">{</span><span class="w"> + + </span><span class="kt">uint256</span><span class="p">[]</span><span class="w"> </span>token<span class="p">;</span><span class="w"> + </span><span class="kt">mapping</span><span class="p">(</span><span class="kt">uint256</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="kt">uint256</span><span class="p">)</span><span class="w"> </span>tokenIndex<span class="p">;</span><span class="w"> </span><span class="c1">// map token id to master token array index position +</span><span class="w"> + </span><span class="p">[...]</span><span class="w"> + + </span><span class="kt">function</span><span class="w"> </span><span class="nv">tokenURI</span><span class="p">(</span><span class="kt">uint256</span><span class="w"> </span><span class="nv">_tokenId</span><span class="p">)</span><span class="w"> </span><span class="kt">public</span><span class="w"> </span>pure<span class="w"> </span><span class="kt">returns</span><span class="p">(</span><span class="kt">string</span><span class="w"> </span><span class="nv">memory</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span><span class="kt">bytes32</span><span class="w"> </span><span class="nv">token_bytes</span><span class="p">;</span><span class="w"> + </span><span class="kt">bytes</span><span class="w"> </span><span class="nv">memory</span><span class="w"> </span>out<span class="p">;</span><span class="w"> + </span><span class="kt">uint8</span><span class="w"> </span><span class="nv">t</span><span class="p">;</span><span class="w"> + </span><span class="kt">uint256</span><span class="w"> </span><span class="nv">c</span><span class="p">;</span><span class="w"> + + </span>token_bytes<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">bytes32</span><span class="p">(</span>token<span class="p">[</span>tokenIndex<span class="p">[</span>_tokenId<span class="p">]]);</span><span class="w"> + + </span>out<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">new</span><span class="w"> </span><span class="kt">bytes</span><span class="p">(</span><span class="m-Decimal">64</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="m-Decimal">7</span><span class="p">);</span><span class="w"> + </span>out<span class="p">[</span><span class="m-Decimal">0</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"s"</span><span class="p">;</span><span class="w"> + </span>out<span class="p">[</span><span class="m-Decimal">1</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"h"</span><span class="p">;</span><span class="w"> + </span>out<span class="p">[</span><span class="m-Decimal">2</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"a"</span><span class="p">;</span><span class="w"> + </span>out<span class="p">[</span><span class="m-Decimal">3</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"2"</span><span class="p">;</span><span class="w"> + </span>out<span class="p">[</span><span class="m-Decimal">4</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"5"</span><span class="p">;</span><span class="w"> + </span>out<span class="p">[</span><span class="m-Decimal">5</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"6"</span><span class="p">;</span><span class="w"> + </span>out<span class="p">[</span><span class="m-Decimal">6</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">":"</span><span class="p">;</span><span class="w"> + + </span>c<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m-Decimal">7</span><span class="p">;</span><span class="w"> + </span><span class="kt">for</span><span class="w"> </span><span class="p">(</span><span class="kt">uint256</span><span class="w"> </span><span class="nv">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m-Decimal">0</span><span class="p">;</span><span class="w"> </span>i<span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="m-Decimal">32</span><span class="p">;</span><span class="w"> </span>i<span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span>t<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">uint8</span><span class="p">(</span>_data<span class="p">[</span>i<span class="p">])</span><span class="w"> </span><span class="err">&</span><span class="w"> </span><span class="mh">0xf0</span><span class="p">)</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="m-Decimal">4</span><span class="p">;</span><span class="w"> + </span><span class="kt">if</span><span class="w"> </span><span class="p">(</span>t<span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="m-Decimal">10</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span>out<span class="p">[</span>c<span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>bytes1<span class="p">(</span>t<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mh">0x30</span><span class="p">);</span><span class="w"> + </span><span class="p">}</span><span class="w"> </span><span class="kt">else</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span>out<span class="p">[</span>c<span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>bytes1<span class="p">(</span>t<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mh">0x57</span><span class="p">);</span><span class="w"> + </span><span class="p">}</span><span class="w"> + </span>t<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">uint8</span><span class="p">(</span>_data<span class="p">[</span>i<span class="p">])</span><span class="w"> </span><span class="err">&</span><span class="w"> </span><span class="mh">0x0f</span><span class="p">;</span><span class="w"> + </span><span class="kt">if</span><span class="w"> </span><span class="p">(</span>t<span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="m-Decimal">10</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span>out<span class="p">[</span>c<span class="o">+</span><span class="m-Decimal">1</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>bytes1<span class="p">(</span>t<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mh">0x30</span><span class="p">);</span><span class="w"> + </span><span class="p">}</span><span class="w"> </span><span class="kt">else</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span>out<span class="p">[</span>c<span class="o">+</span><span class="m-Decimal">1</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>bytes1<span class="p">(</span>t<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mh">0x57</span><span class="p">);</span><span class="w"> + </span><span class="p">}</span><span class="w"> + </span>c<span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="m-Decimal">2</span><span class="p">;</span><span class="w"> + </span><span class="p">}</span><span class="w"> + </span><span class="kt">return</span><span class="w"> </span><span class="kt">string</span><span class="p">(</span>out<span class="p">);</span><span class="w"> + </span><span class="p">}</span><span class="w"> +</span><span class="p">}</span> +</pre> +<p>This will return <code>sha256:3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551</code> for <code>tokenId</code> <code>3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551</code> as input, provided that the <code>tokenId</code> actually exists. That may seem a bit useless at first, but consider the scenario where we want to interface with other NFTs aswell. Or perhaps we are implementing a contract that optionally can support a static web2 URI in storage. By doing it this way, all bases are covered.</p> +</div> +<div class="section" id="decentralized-identifiers"> +<h2>Decentralized identifiers</h2> +<p>Even better would be to add redundancy with autonomous decentralized storage. However, networks like <a class="reference external" href="https://ethswarm.org">Swarm</a> and <a class="reference external" href="https://ipfs.io">IPFS</a> use their own hashing recipes. That means that for every network referenced, we'd have to define an <em>alternative</em> in our reference structure.</p> +<p>Referencing the canonical <code>sha256</code> aswell as the <code>Swarmhash</code> for the same item could then look like this <a class="footnote-reference" href="#footnote-5" id="footnote-reference-5">[5]</a>:</p> +<pre class="code json literal-block"> +<span class="p">{</span><span class="w"> + </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo image"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"image"</span><span class="p">:</span><span class="w"> </span><span class="nt">"data:image/gif;base64,R0lGODlhDgARAKEDAAAAAOjqAPP1APDw8CH+EUNyZWF0ZWQgd2l0aCBHSU1QACwAAAAADgARAAACO5wHqXdrClocodbUaGg34qoBHaJtl7VYIfqBZxepb5C6NBvbIlyucO6jPUIK1CMiSCYvDIVS4GhCJoYCADs="</span><span class="w"> + </span><span class="nt">"alternatives"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> + </span><span class="p">{</span><span class="w"> + </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo image"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"image"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bzz:4b9149ee4550f2d786f9ba6584b79a30ee14ae05ff6e84a0f7c7561a14e3b779"</span><span class="w"> + </span><span class="p">},</span><span class="w"> + </span><span class="p">{</span><span class="w"> + </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo image"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"image"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sha256:d036a4ce7f929b632256225b2bebd81bdd558d3bfe3d96faae61db664708c16f"</span><span class="w"> + </span><span class="p">}</span><span class="w"> + </span><span class="p">]</span><span class="w"> +</span><span class="p">}</span> +</pre> +<hr class="docutils" /> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Yes, they are valid URIs actually: <a class="reference external" href="https://www.rfc-archive.org/getrfc.php?rfc=2397">https://www.rfc-archive.org/getrfc.php?rfc=2397</a></td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>Likely it would be prudent to start using the official <code>sha3</code> instead of <code>sha2</code> these days, also because the <code>sha2</code> hash is not a builtin for <code>evm</code>. But neither is <code>sha3</code>. The <code>keccak256</code> Bitcoin uses, which EVM has inherited, is a pre-cursor to the <code>keccak</code> published as the <em>official</em> <code>sha3</code>. Still, <code>keccak256</code> and <code>sha3</code> is used interchangeably in opcode lists (and previously in <a class="reference external" href="https://docs.soliditylang.org/en/v0.8.0/050-breaking-changes.html#functions">Solidity</a> too). This has caused me quite a fair bit of confusion, I might add. Apart from it being ambiguous, the <code>keccak256</code> tooling is also less common in the wild. Therefore <code>sha2</code> seems like a safer bet for our experiments. It's not broken yet, after all.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>The hex result of <code>keccak256("mintTo(address,uint256)")</code></td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-4" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-4">[4]</a></td><td><em>Data URI</em> is of no use here, because the hash itself is just nondescript binary data. Luckily <code><scheme>:<path></code> is still a valid URI.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-5" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-5">[5]</a></td><td>Here the hashes represent the media content itself, not the reference. That's why the <code>sha256</code> one is different than before.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils citation" frame="void" id="eip721" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#citation-reference-1">[EIP721]</a></td><td><a class="reference external" href="https://eips.ethereum.org/EIPS/eip-721">https://eips.ethereum.org/EIPS/eip-721</a></td></tr> +</tbody> +</table> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/24345ef2cf2edbf32d9ec3e55a6e0801a5a73f8337bb81bc1a7395ebbe3bf7e0.asc b/24345ef2cf2edbf32d9ec3e55a6e0801a5a73f8337bb81bc1a7395ebbe3bf7e0.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NzsACgkQzcvSTdHQ +4AFDcAf+PhGwhBAACBONtnExlRdrDjruax1tLkq5DBoTJY65m5uNEJVJ7L5vunON +UTvbF7gk4HWER7gWOs0jdfeybQJaHC5I9IKkjo8Hynfc+wHqC2HNRmYoV2UZw0nT +2iTeJULmYuj1Equoav8mtyQHTTPZOvIEAH5GPSUFnCVM83ObzZgAX1h17pfN2vVe +CpmbMUGIvHEXYL0YfhmcSRjEQNL0WAwOPRoS90ba6jdeMlVv3axy51bkxw7IoJbu +taIrevqD1437Gzc6ubNgNeftvGOC6F7sY/D1ek9CYxhaMpuhs1olWR2e93RIzV0U +jL/FJyqBID+2Rx1QMF8yjaKESY2cxw== +=sHf6 +-----END PGP SIGNATURE----- diff --git a/2868ebaf23ffa05cd78bead77dbae8a6c0e596d67979ef45c9fd8e54db2cfbc9 b/2868ebaf23ffa05cd78bead77dbae8a6c0e596d67979ef45c9fd8e54db2cfbc9 @@ -1,331 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> - <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> - <title>man bytes gnu - Local npm repository</title> - <meta charset="utf-8" /> - <meta name="generator" content="Pelican" /> - -</head> - -<body id="index" class="home"> - - <header id="banner" class="body"> - <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> - </header> - <nav id="menu"> - <ul> - <li><a href="./category/archiving.html">Archiving</a></li> - <li><a href="./category/code.html">Code</a></li> - <li><a href="./category/hygiene.html">Hygiene</a></li> - <li class="active"><a href="./category/offlining.html">Offlining</a></li> - <li><a href="./pages/identities.html">Identities</a></li> - <li><a href="./pages/shares.html">Shares</a></li> - - <li><a href="/tags.html">tags</a></li> - </ul> - <hr/> - </nav><!-- /#menu --> - <header> - <h1 class="entry-title top-body-title"> - <a href="./docker-offline-3-npm.html" rel="bookmark" - title="Permalink to Local npm repository">Offline Docker: Local npm repository</a></h1> - - - <div class="category meta"> - Posted -<time class="published" datetime="2021-05-25T11:47:00+02:00"> - Tue 25 May 2021 - </time> -in <a class="category" href="./category/offlining.html">offlining</a> - <a href="./tag/docker.html">docker</a> - <a href="./tag/npm.html">npm</a> - <a href="./tag/nodejs.html">nodejs</a> - <a href="./tag/javascript.html">javascript</a> - <a href="./tag/devops.html">devops</a> - </div> - - <div class="neighbors meta"> - <ul> - <li> - Previous: <a href="./village-network-protocol.html"> - A village network protocol - </a> - </li> - | - - <li> - Next: <a href="./eth-log-bloom.html"> - Making sense of Ethereum log bloom filters - </a> - </li> - </ul> - </div> - <div class="entry-series meta"> -Part 3 from the series "Offline Docker" - <ol class="entry-series-parts"> - <li>| <a href="docker-offline-1-routing.html" title="The routing to freedom ">Part 1</a></li> - <li>| <a href="docker-offline-2-python.html" title="Local python repository ">Part 2</a></li> - </ol> - </div> - <div class="meta"> -<hr/> - </div> -</header> - -<section id="content" class="body"> - <div class="entry-content"> - <p>As expected, serving a local <tt class="docutils literal">npm</tt> repository is a <em>lot</em> less straightforward than that for <tt class="docutils literal">pip</tt> and <tt class="docutils literal">python</tt>.</p> -<p>The npm registry uses the <code>CouchDB</code> nosql server as backend to resolve dependencies and serve resource metadata. The <code>npm</code> CLI tool expects a corresponding json response to requests that cite the <em>name</em> (and not the path) of the package.</p> -<p>Let's ask the <a class="reference external" href="https://registry.npmjs.org">registry</a> <a class="footnote-reference" href="#id6" id="id1">[1]</a> for the package <tt class="docutils literal">ftp</tt>, and have a look at the response (excerpt):</p> -<div class="highlight"><pre><span></span><span class="gp">$ </span>curl -X GET https://registry.npmjs.org/ftp <span class="p">|</span> jq -<span class="go">{</span> -<span class="go"> "_id": "ftp",</span> -<span class="go"> "_rev": "113-89fe76508a7ece41b4c9a157114f966f",</span> -<span class="go"> "name": "ftp",</span> -<span class="go"> "description": "An FTP client module for node.js",</span> -<span class="go"> "dist-tags": {</span> -<span class="go"> "latest": "0.3.10"</span> -<span class="go"> },</span> -<span class="go"> "versions": {</span> - - - -<span class="go"> "0.3.10": {</span> -<span class="go"> "name": "ftp",</span> -<span class="go"> "version": "0.3.10",</span> -<span class="go"> "author": {</span> -<span class="go"> "name": "Brian White",</span> -<span class="go"> "email": "mscdex@mscdex.net"</span> -<span class="go"> },</span> -<span class="go"> "description": "An FTP client module for node.js",</span> -<span class="go"> "main": "./lib/connection",</span> -<span class="go"> "engines": {</span> -<span class="go"> "node": ">=0.8.0"</span> -<span class="go"> },</span> -<span class="go"> "dependencies": {</span> -<span class="go"> "xregexp": "2.0.0",</span> -<span class="go"> "readable-stream": "1.1.x"</span> -<span class="go"> },</span> -<span class="go"> "scripts": {</span> -<span class="go"> "test": "node test/test.js"</span> -<span class="go"> },</span> -<span class="go"> "keywords": [</span> -<span class="go"> "ftp",</span> -<span class="go"> "client",</span> -<span class="go"> "transfer"</span> -<span class="go"> ],</span> -<span class="go"> "licenses": [</span> -<span class="go"> {</span> -<span class="go"> "type": "MIT",</span> -<span class="go"> "url": "http://github.com/mscdex/node-ftp/raw/master/LICENSE"</span> -<span class="go"> }</span> -<span class="go"> ],</span> -<span class="go"> "repository": {</span> -<span class="go"> "type": "git",</span> -<span class="go"> "url": "http://github.com/mscdex/node-ftp.git"</span> -<span class="go"> },</span> -<span class="go"> "bugs": {</span> -<span class="go"> "url": "https://github.com/mscdex/node-ftp/issues"</span> -<span class="go"> },</span> -<span class="go"> "homepage": "https://github.com/mscdex/node-ftp",</span> -<span class="go"> "_id": "ftp@0.3.10",</span> -<span class="go"> "_shasum": "9197d861ad8142f3e63d5a83bfe4c59f7330885d",</span> -<span class="go"> "_from": "https://github.com/mscdex/node-ftp/tarball/v0.3.10",</span> -<span class="go"> "_resolved": "https://github.com/mscdex/node-ftp/tarball/v0.3.10",</span> -<span class="go"> "_npmVersion": "1.4.28",</span> -<span class="go"> "_npmUser": {</span> -<span class="go"> "name": "mscdex",</span> -<span class="go"> "email": "mscdex@mscdex.net"</span> -<span class="go"> },</span> -<span class="go"> "maintainers": [</span> -<span class="go"> {</span> -<span class="go"> "name": "mscdex",</span> -<span class="go"> "email": "mscdex@mscdex.net"</span> -<span class="go"> }</span> -<span class="go"> ],</span> -<span class="go"> "dist": {</span> -<span class="go"> "shasum": "9197d861ad8142f3e63d5a83bfe4c59f7330885d",</span> -<span class="go"> "tarball": "https://registry.npmjs.org/ftp/-/ftp-0.3.10.tgz"</span> -<span class="go"> },</span> -<span class="go"> "directories": {}</span> -<span class="go"> }</span> -<span class="go"> },</span> -</pre></div> -<p>That looks a lot like embellished contents of the basic package.json set up by <code>npm init</code>, except it's wrapped in a versions array. It also explicitly defines an <em>absolute</em> url to a <cite>tarball</cite> under the <cite>dist</cite> key. <a class="footnote-reference" href="#id7" id="id2">[2]</a></p> -<div class="section" id="how-low-can-you-go"> -<h2>How low can you go</h2> -<p>It seems that all that's needed is a json index page served for the package name subpath of repository path, together with the tarball itself. We check this by putting together an utterly pointless package <tt class="docutils literal">foobarbarbar</tt>:</p> -<p><strong>foobarbarbar/index.js</strong></p> -<pre class="code javascript literal-block" id="foobarbarbar-index-js"> -<span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span> - <span class="s1">'smth'</span><span class="o">:</span> <span class="nx">smth</span><span class="p">,</span> -<span class="p">}</span> - -<span class="kd">function</span> <span class="nx">smth</span><span class="p">()</span> <span class="p">{</span> - <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'foo'</span><span class="p">);</span> -<span class="p">}</span> -</pre> -<p><strong>foobarbarbar/package.json</strong></p> -<pre class="code json literal-block" id="foobarbarbar-package-json"> -<span class="p">{</span> - <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"foobarbarbar"</span><span class="p">,</span> - <span class="nt">"version"</span><span class="p">:</span> <span class="s2">"1.0.0"</span><span class="p">,</span> - <span class="nt">"description"</span><span class="p">:</span> <span class="s2">"Foo repo"</span><span class="p">,</span> - <span class="nt">"main"</span><span class="p">:</span> <span class="s2">"index.js"</span><span class="p">,</span> - <span class="nt">"author"</span><span class="p">:</span> <span class="s2">"Foo Bar"</span><span class="p">,</span> - <span class="nt">"license"</span><span class="p">:</span> <span class="s2">"GPL3"</span><span class="p">,</span> -<span class="p">}</span> -</pre> -<p>Then make a tarball from those two files named <tt class="docutils literal"><span class="pre">foobarbarbar-1.0.0.tgz</span></tt>, take the sha1 sum of it,</p> -<p>Remembering our <a class="reference external" href="./docker-offline-2-python.html">virtual interface setup from the pip example</a> we stick it behind our webserver (here with npm sub-path) and add our minimal version json wrapper:</p> -<p><strong>package.json</strong></p> -<pre class="code json literal-block" id="package-json"> -<span class="p">{</span> - <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"foobarbarbar"</span><span class="p">,</span> - <span class="nt">"versions"</span><span class="p">:</span> <span class="p">{</span> - <span class="nt">"1.0.0"</span><span class="p">:</span> <span class="p">{</span> - <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"foobarbarbar"</span><span class="p">,</span> - <span class="nt">"version"</span><span class="p">:</span> <span class="s2">"1.0.0"</span><span class="p">,</span> - <span class="nt">"description"</span><span class="p">:</span> <span class="s2">"Foo repo"</span><span class="p">,</span> - <span class="nt">"main"</span><span class="p">:</span> <span class="s2">"index.js"</span><span class="p">,</span> - <span class="nt">"author"</span><span class="p">:</span> <span class="s2">"Foo Bar"</span><span class="p">,</span> - <span class="nt">"license"</span><span class="p">:</span> <span class="s2">"GPL3"</span><span class="p">,</span> - <span class="nt">"dist"</span><span class="p">:</span> <span class="p">{</span> - <span class="nt">"shasum"</span><span class="p">:</span> <span class="s2">"2ccd68498ef5f2bfa00f0e1e59f44686fdb296ee"</span><span class="p">,</span> - <span class="nt">"tarball"</span><span class="p">:</span> <span class="s2">"http://10.1.2.1/npm/foobarbarbar-1.0.0.tgz"</span> - <span class="p">}</span> - - <span class="p">}</span> - <span class="p">}</span> -<span class="p">}</span> -</pre> -</div> -<div class="section" id="making-introductions"> -<h2>Making introductions</h2> -<p>The central trick here is to serve a json document as the "directory index" of the HTTP server. It's useful to remind ourselves at this point that we are <em>not</em> setting up a <em>registry</em> but a <em>repository</em> that contains provisions for a <em>locked</em> or <em>frozen</em> dependency graph. The former would surely need some of the <tt class="docutils literal">CouchDB</tt> magic to resolve dependencies. Our assumption is that the latter can theoretically be realized using static files.</p> -<p>In other words; just as with the previous <a class="reference external" href="./docker-offline-2-python.html">python repository example</a>, we don't try to handle dependency resolutions for pip, but merely serve the actual package files after dependences have been resolved.</p> -<p>With Apache Web Server, using the <tt class="docutils literal">package.json</tt> as the directory index is as easy as:</p> -<div class="highlight"><pre><span></span><Directory "/srv/http/npm"> - DirectoryIndex package.json -</Directory> -</pre></div> -<p>Of course adjusting the <cite>Directory</cite> path as needed to match the local setup.</p> -<p>Make sure that the tarball and the latter <cite>package.json</cite> can be found in the <cite>foobarbarbar</cite> virtual subfolder of the above <cite>Directory</cite> directive path:</p> -<div class="highlight"><pre><span></span><span class="gp">$ </span>ls /srv/http/npm/foobarbarbar/ -<span class="go">-rw-r--r-- 1 root root 351 May 24 18:38 foobarbarbar-1.0.0.tgz</span> -<span class="go">-rw-r--r-- 1 root root 380 May 25 09:36 package.json</span> -</pre></div> -<p>Set the registry entry in your <tt class="docutils literal"><span class="pre">~./npmrc</span></tt> as follows:</p> -<div class="highlight"><pre><span></span><span class="na">registry</span><span class="o">=</span><span class="s">http://10.1.2.1/npm</span> -</pre></div> -<p>Then restart the Apache server and give it a go:</p> -<div class="highlight"><pre><span></span><span class="gp">$ </span>npm install --verbose foobarbarbar -<span class="go">npm verb cli [ '/usr/bin/node', '/usr/bin/npm', 'install', '--verbose', 'foobarbarbar' ]</span> -<span class="go">npm info using npm@7.13.0</span> -<span class="go">npm info using node@v16.1.0</span> -<span class="go">[...]</span> -<span class="go">npm http fetch GET 200 http://10.1.2.1/npm/foobarbarbar/ 6ms</span> -<span class="go">[...]</span> -<span class="go">npm http fetch GET 200 http://10.1.2.1/npm/foobarbarbar/foobarbarbar-1.0.0.tgz 25ms</span> -<span class="go">[...]</span> -<span class="go">added 2 packages in 545ms</span> -<span class="go">npm timing command:install Completed in 70ms</span> -<span class="go">[...]</span> -</pre></div> -</div> -<div class="section" id="cache-dodging"> -<h2>Cache dodging</h2> -<p>Seriously, <tt class="docutils literal">npm <span class="pre">--help</span></tt> leaves a lot to be desired. However, the online <tt class="docutils literal">npm install</tt> documentation does not yield any more clues as to whether it gives you an option of ignoring local cache like you can with <tt class="docutils literal">pip <span class="pre">--no-cache</span></tt>. This is a major pain in the ass, as you have to remember to keep your <tt class="docutils literal"><span class="pre">~/.npm</span></tt> folder clean between each install attempt. Otherwise, changes you make to the repository won't be used by <tt class="docutils literal">npm install</tt>.</p> -<p>While testing, it pays to use a folder fairly close to the fs root, like a subfolder in <cite>tmp</cite>. That way, you won't be thrown off by some stray <cite>node_modules</cite> folder somewhere down the tree.</p> -</div> -<div class="section" id="think-locally-act-locally"> -<h2>Think locally, act locally</h2> -<p>Serving packages globally apparently comes down to this:</p> -<ol class="arabic simple"> -<li>Pull the <tt class="docutils literal">package.json</tt> versions list from a proper registry.</li> -<li>Get the tarball</li> -<li>Transform the absolute package url to our host instead. (sigh)</li> -<li>(optional) Prune all the versions and metadata which is not needed (anything that's not in our minimal <tt class="docutils literal">foobarbarbar</tt> example above).</li> -</ol> -<p>Obviously, this is an annoyingly large amount of work to code up parsing and retrieval for.</p> -<p>Incidentally, there is a very nice tool called <a class="reference external" href="https://verdaccio.org">verdaccio</a> which gets us most of the way there. <a class="footnote-reference" href="#id8" id="id3">[3]</a> Its <em>storage</em> directory <a class="footnote-reference" href="#id9" id="id4">[4]</a> uses exactly the directory structure we need. After retrieving a package collection using it as a proxy, getting the files in place is merely a case of copying <a class="footnote-reference" href="#id10" id="id5">[5]</a> the files from the <em>storage</em> directory to the corresponding webserver directory.</p> -<p>Looking at the <tt class="docutils literal">package.json</tt> versions wrapper saved by <tt class="docutils literal">verdaccio</tt>, however, we see that it still preserves the absolute path of the upstream registry. Sadly we are still left with doing the third task ourselves.</p> -<p>As parsing json is one of my least favorite things in the world, I won't include the code for this last step here. For now it's sufficient that we understand the minimums required for manually setting up and serving a static, local, offline npm repository. Regardless of the pain involved.</p> -</div> -<div class="section" id="making-it-personal"> -<h2>Making it personal</h2> -<p>Whether we massage jsons ourselves or lazily resort to <tt class="docutils literal">verdaccio</tt>, the final step we need to take is the same: Setting the registry url in the npm configuration of the docker image.</p> -<p>This is merely a case of setting the registry url in the <a class="reference external" href="https://docs.npmjs.com/cli/v6/configuring-npm/npmrc#files">npmrc</a> inside the container.</p> -<div class="highlight"><pre><span></span><span class="gp">$ </span>docker run -it <span class="o">[</span>...<span class="o">]</span> npm config ls -l <span class="p">|</span> grep etc/npmrc -<span class="go">globalconfig = "/usr/etc/npmrc"</span> -</pre></div> -<p>To get at our manually provided <tt class="docutils literal">foobarbarbar</tt> package from before, the dockerfile will be (using archlinux):</p> -<div class="highlight"><pre><span></span><span class="o">[</span>...<span class="o">]</span> -<span class="k">RUN</span> pacman -S nodejs npm -<span class="k">RUN</span> mkdir -vp /usr/etc -<span class="k">RUN</span> <span class="nb">echo</span> <span class="s2">"registry=http:/10.1.2.1/npm"</span> > /usr/etc/npmrc -<span class="k">WORKDIR</span><span class="s"> /root</span> -<span class="k">RUN</span> npm install foobarbarbar -</pre></div> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id6" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>This was the registry address I had in my <cite>~/.npmrc</cite> or whether I put it in myself.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id7" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>I have tried using a relative path instead, both with and without a leading <tt class="docutils literal">/</tt>, but in either case the install then errors our complaining that the lock file is "corrupt." Whether the schema allows a base-url setting I do not know, as the schema documentation doesn't seem to be readily available.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id8" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>In fact, <tt class="docutils literal">verdaccio</tt> by itself solves the particular problem that we're trying to solve; providing an offline repository alternative. The task at hand, however, is to understand how to cope <em>without</em> using other tools than what can be considered base infrastructure (i.e. a web server).</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id9" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>Should be <tt class="docutils literal">/etc/verdaccio/storage</tt>; see <tt class="docutils literal">/etc/verdaccio/config.yaml</tt></td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id10" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td>An alternative approach would be to set the storage path to point to the same folder as the web server is serving. However, since we need to mess with the tarball paths</td></tr> -</tbody> -</table> -</blockquote> -</div> - - </div><!-- /.entry-content --> - <footer class="meta"> - </footer> - -</section> - <footer id="contentinfo" class="body"> - <hr/> - <address id="about" class="vcard body"> - Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> - </address><!-- /#about --> - </footer><!-- /#contentinfo --> -</body> -</html> -\ No newline at end of file diff --git a/2868ebaf23ffa05cd78bead77dbae8a6c0e596d67979ef45c9fd8e54db2cfbc9.asc b/2868ebaf23ffa05cd78bead77dbae8a6c0e596d67979ef45c9fd8e54db2cfbc9.asc @@ -1,11 +0,0 @@ ------BEGIN PGP SIGNATURE----- - -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmHkZ0UACgkQzcvSTdHQ -4AH+Igf/T59Ll3NXuF2UsG48pMveE2ywxCzER2Twm2PYM2omdUWgZ2bT5bs1ieTh -lrMCrU7yb+RxcMlP2mhca86ju+YHdAkPZA+21ZDwpSCfSnKp2E/8xN8qNIXo6TeB -S5QBDKmLNK1fLqPXnjpbwptD569XAifYzFvdvP/XWXXPKn7FzG339KFLc4xJYxpB -F5gRngN25pArUqR5Fr84PP8VI47DSV90+RKcot6klKbog7nWMVbCf2oMb+RrEqn7 -YJtuYhYhBLfLuJuY5h1SyC8bBQuOkXbLdf5+Nl6gUN+Q4FlEuZJttSBYnmaz9tj+ -x8oBBSwylLJsGfiPGNRxHvbjB+MI/g== -=/Jj5 ------END PGP SIGNATURE----- diff --git a/2918deb778c2f4961cc75fcf66f21c6bbd1c6a848e0a5aebb2e3a9afdac25ce6 b/2918deb778c2f4961cc75fcf66f21c6bbd1c6a848e0a5aebb2e3a9afdac25ce6 @@ -1,363 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> - <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> - <title>man bytes gnu - wasm and C - The bare necessities</title> - <meta charset="utf-8" /> - <meta name="generator" content="Pelican" /> - -</head> - -<body id="index" class="home"> - - <header id="banner" class="body"> - <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> - </header> - <nav id="menu"> - <ul> - <li><a href="./category/archiving.html">Archiving</a></li> - <li class="active"><a href="./category/code.html">Code</a></li> - <li><a href="./category/hygiene.html">Hygiene</a></li> - <li><a href="./category/offlining.html">Offlining</a></li> - <li><a href="./pages/identities.html">Identities</a></li> - <li><a href="./pages/shares.html">Shares</a></li> - - <li><a href="/tags.html">tags</a></li> - </ul> - <hr/> - </nav><!-- /#menu --> - <header> - <h1 class="entry-title top-body-title"> - <a href="./wasm-c-bare.html" rel="bookmark" - title="Permalink to wasm and C - The bare necessities">wasm and C - The bare necessities</a></h1> - - - <div class="category meta"> - Posted -<time class="published" datetime="2021-05-05T06:15:00+02:00"> - Wed 05 May 2021 - </time> -in <a class="category" href="./category/code.html">code</a> - <a href="./tag/wasm.html">wasm</a> - <a href="./tag/c.html">c</a> - <a href="./tag/clang.html">clang</a> - <a href="./tag/llvm.html">llvm</a> - </div> - - <div class="neighbors meta"> - <ul> - <li> - Previous: <a href="./web-snapshot.html"> - Proving what you link to - </a> - </li> - | - - <li> - Next: <a href="./nft-tokenid-content.html"> - The NFT token id as URI - </a> - </li> - </ul> - </div> - <div class="meta"> -<hr/> - </div> -</header> - -<section id="content" class="body"> - <div class="entry-content"> - <p>I am currently resuming my self-improvement task of learning Webassembly. Or, I should rather say, <em>using</em> Webassembly.</p> -<p>Since I have some optimized small code chunks that I want to use in an embedded environment, it seemed sensible that the first step would be to establish two-way communication with C.</p> -<p>In my last outing around three years ago, I was using <a class="reference external" href="https://emscripten.org/">Emscripten</a> to bridge the gap. That tool adds quite a few bells and whistles, and doesn't quite yield that warm, fuzzy bare-metal rush. <a class="reference external" href="https://emscripten.org/">Emscripten</a> relies on <a class="reference external" href="http://clang.org/">Clang</a> and <a class="reference external" href="https://llvm.org/">LLVM</a>, all of which seem to have gotten their <code>wasm</code> support built-in in the meantime (at least on my archlinux system). This it integrates nicely with <a class="reference external" href="https://github.com/WebAssembly/wabt">wabt</a> - the swiss-army knife of Webassembly.</p> -<p>So how far do we get with just <a class="reference external" href="http://clang.org/">clang</a>, <a class="reference external" href="https://llvm.org/">LLVM</a> and <a class="reference external" href="https://github.com/WebAssembly/wabt">wabt</a> ? Let's see if we at least can set up a code snippet which simply writes <em>"foobar"</em> to memory. The host will write <em>"foo"</em>, and <code>wasm</code> will write <em>"bar"</em>.</p> -<div class="section" id="without-libc"> -<h2>Without libc</h2> -<p>This excellent <a class="reference external" href="https://surma.dev/things/c-to-webassembly/">tutorial by Surma</a> provides a good starting point. Go ahead and <strong>read that first</strong>. This text is not a Webassembly primer, so the following will make a lot more sense if you do.</p> -<p>That setup still adds some magic. Namely, the <em>memory</em> and <em>symbol table</em> are here added by the <em>wasm linker</em>. It would be even more fun to pass this from the host system instead.</p> -<p>And so we start: <a class="footnote-reference" href="#id4" id="id1">[1]</a></p> -<pre class="code c literal-block"> -<span class="ln"> 0 </span><span class="cp">#include</span><span class="w"> </span><span class="cpf">"string.h"</span><span class="cp"> -</span><span class="ln"> 1 </span><span class="cp"></span><span class="w"> -</span><span class="ln"> 2 </span><span class="w"></span><span class="k">extern</span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">__heap_base</span><span class="p">;</span><span class="w"> -</span><span class="ln"> 3 </span><span class="w"></span><span class="k">extern</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">call_me_sometime</span><span class="p">(</span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">b</span><span class="p">);</span><span class="w"> -</span><span class="ln"> 4 </span><span class="w"> -</span><span class="ln"> 5 </span><span class="w"></span><span class="kt">void</span><span class="w"> </span><span class="nf">foo</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"> -</span><span class="ln"> 6 </span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="o">*</span><span class="p">)</span><span class="o">&</span><span class="n">__heap_base</span><span class="p">;</span><span class="w"> -</span><span class="ln"> 7 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">3</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'b'</span><span class="p">;</span><span class="w"> -</span><span class="ln"> 8 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">4</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'a'</span><span class="p">;</span><span class="w"> -</span><span class="ln"> 9 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">5</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'r'</span><span class="p">;</span><span class="w"> -</span><span class="ln">10 </span><span class="w"> </span><span class="n">call_me_sometime</span><span class="p">(</span><span class="n">buf</span><span class="p">);</span><span class="w"> -</span><span class="ln">11 </span><span class="w"></span><span class="p">}</span> -</pre> -<p>Compiling this without linking gives us a hint on what needs to be defined.</p> -<div class="highlight"><pre><span></span>$ clang --target<span class="o">=</span>wasm32 -nostdlib -nostartfiles -o bare.wasm -c bare.c -$ wasm-objdump -x bare.wasm -bare.wasm: file format wasm 0x1 - -Section Details: - -Type<span class="o">[</span><span class="m">2</span><span class="o">]</span>: - - type<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="o">()</span> -> nil - - type<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="o">(</span>i32<span class="o">)</span> -> nil -Import<span class="o">[</span><span class="m">4</span><span class="o">]</span>: - - memory<span class="o">[</span><span class="m">0</span><span class="o">]</span> pages: <span class="nv">initial</span><span class="o">=</span><span class="m">0</span> <- env.__linear_memory - - table<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="nv">type</span><span class="o">=</span>funcref <span class="nv">initial</span><span class="o">=</span><span class="m">0</span> <- env.__indirect_function_table - - global<span class="o">[</span><span class="m">0</span><span class="o">]</span> i32 <span class="nv">mutable</span><span class="o">=</span><span class="m">1</span> <- env.__stack_pointer - - func<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">1</span> <env.call_me_sometime> <- env.call_me_sometime -Function<span class="o">[</span><span class="m">1</span><span class="o">]</span>: - - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">0</span> <foo> -Code<span class="o">[</span><span class="m">1</span><span class="o">]</span>: - - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">138</span> <foo> -Custom: - - name: <span class="s2">"linking"</span> - - symbol table <span class="o">[</span><span class="nv">count</span><span class="o">=</span><span class="m">4</span><span class="o">]</span> - - <span class="m">0</span>: F <foo> <span class="nv">func</span><span class="o">=</span><span class="m">1</span> <span class="nv">binding</span><span class="o">=</span>global <span class="nv">vis</span><span class="o">=</span>hidden - - <span class="m">1</span>: G <env.__stack_pointer> <span class="nv">global</span><span class="o">=</span><span class="m">0</span> undefined <span class="nv">binding</span><span class="o">=</span>global <span class="nv">vis</span><span class="o">=</span>default - - <span class="m">2</span>: D <__heap_base> undefined <span class="nv">binding</span><span class="o">=</span>global <span class="nv">vis</span><span class="o">=</span>default - - <span class="m">3</span>: F <env.call_me_sometime> <span class="nv">func</span><span class="o">=</span><span class="m">0</span> undefined <span class="nv">binding</span><span class="o">=</span>global <span class="nv">vis</span><span class="o">=</span>default -Custom: - - name: <span class="s2">"reloc.CODE"</span> - - relocations <span class="k">for</span> section: <span class="m">3</span> <span class="o">(</span>Code<span class="o">)</span> <span class="o">[</span><span class="m">5</span><span class="o">]</span> - - R_wasm_GLOBAL_INDEX_LEB <span class="nv">offset</span><span class="o">=</span>0x000007<span class="o">(</span><span class="nv">file</span><span class="o">=</span>0x000099<span class="o">)</span> <span class="nv">symbol</span><span class="o">=</span><span class="m">1</span> <env.__stack_pointer> - - R_wasm_GLOBAL_INDEX_LEB <span class="nv">offset</span><span class="o">=</span>0x00001c<span class="o">(</span><span class="nv">file</span><span class="o">=</span>0x0000ae<span class="o">)</span> <span class="nv">symbol</span><span class="o">=</span><span class="m">1</span> <env.__stack_pointer> - - R_wasm_MEMORY_ADDR_SLEB <span class="nv">offset</span><span class="o">=</span>0x000031<span class="o">(</span><span class="nv">file</span><span class="o">=</span>0x0000c3<span class="o">)</span> <span class="nv">symbol</span><span class="o">=</span><span class="m">2</span> <__heap_base> - - R_wasm_FUNCTION_INDEX_LEB <span class="nv">offset</span><span class="o">=</span>0x000073<span class="o">(</span><span class="nv">file</span><span class="o">=</span>0x000105<span class="o">)</span> <span class="nv">symbol</span><span class="o">=</span><span class="m">3</span> <env.call_me_sometime> - - R_wasm_GLOBAL_INDEX_LEB <span class="nv">offset</span><span class="o">=</span>0x000086<span class="o">(</span><span class="nv">file</span><span class="o">=</span>0x000118<span class="o">)</span> <span class="nv">symbol</span><span class="o">=</span><span class="m">1</span> <env.__stack_pointer> -Custom: - - name: <span class="s2">"producers"</span> -</pre></div> -<p>Using nodejs as the host, we check if we can instantiate a <code>WebAssembly</code> object</p> -<pre class="code javascript literal-block"> -<span class="ln"> 0 </span><span class="kd">const</span> <span class="nx">fs</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'fs'</span><span class="p">);</span> -<span class="ln"> 1 </span> -<span class="ln"> 2 </span><span class="kd">const</span> <span class="nx">imports</span> <span class="o">=</span> <span class="p">{}</span> -<span class="ln"> 3 </span> -<span class="ln"> 4 </span><span class="k">async</span> <span class="kd">function</span> <span class="nx">init</span><span class="p">()</span> <span class="p">{</span> -<span class="ln"> 5 </span> <span class="kd">const</span> <span class="nx">code</span> <span class="o">=</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="s1">'./bare.wasm'</span><span class="p">);</span> -<span class="ln"> 6 </span> <span class="kd">const</span> <span class="nx">m</span> <span class="o">=</span> <span class="ow">new</span> <span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Module</span><span class="p">(</span><span class="nx">code</span><span class="p">);</span> -<span class="ln"> 7 </span> <span class="kd">const</span> <span class="nx">i</span> <span class="o">=</span> <span class="ow">new</span> <span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Instance</span><span class="p">(</span><span class="nx">m</span><span class="p">,</span> <span class="nx">imports</span><span class="p">);</span> -<span class="ln"> 8 </span><span class="p">}</span> -<span class="ln"> 9 </span><span class="nx">init</span><span class="p">();</span> -</pre> -<p>Running this tells us we are apparently missing a property <code>env</code> in the imports object.</p> -<div class="highlight"><pre><span></span>$ node bare_naive.js -/home/lash/src/tests/wasm/bare/bare_naive.js:8 -const <span class="nv">i</span> <span class="o">=</span> new WebAssembly.Instance<span class="o">(</span>m, imports<span class="o">)</span><span class="p">;</span> - ^ - -TypeError: WebAssembly.Instance<span class="o">()</span>: Import <span class="c1">#0 module="env" error: module is not an object or function</span> -</pre></div> -<p>That seems to match with the <code>Import</code> section in the <code>objdump</code> output above. Let's stick the <em>memory</em> and <em>table</em> in there. <a class="footnote-reference" href="#id5" id="id2">[2]</a></p> -<p>And let's make a bold guess that the callback function <code>call_me_sometime</code> needs to go in there aswell.</p> -<pre class="code javascript literal-block"> -<span class="ln"> 0 </span><span class="kd">const</span> <span class="nx">fs</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'fs'</span><span class="p">);</span> -<span class="ln"> 1 </span> -<span class="ln"> 2 </span><span class="kd">const</span> <span class="nx">memory</span> <span class="o">=</span> <span class="ow">new</span> <span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Memory</span><span class="p">({</span><span class="nx">initial</span><span class="o">:</span> <span class="mf">2</span><span class="p">});</span> -<span class="ln"> 3 </span><span class="kd">const</span> <span class="nx">table</span> <span class="o">=</span> <span class="ow">new</span> <span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Table</span><span class="p">({</span><span class="nx">initial</span><span class="o">:</span> <span class="mf">3</span><span class="p">,</span> <span class="nx">element</span><span class="o">:</span> <span class="s1">'anyfunc'</span><span class="p">});</span> -<span class="ln"> 4 </span><span class="kd">const</span> <span class="nx">importsObj</span> <span class="o">=</span> <span class="p">{</span> -<span class="ln"> 5 </span> <span class="nx">env</span><span class="o">:</span> <span class="p">{</span> -<span class="ln"> 6 </span> <span class="nx">memory</span><span class="o">:</span> <span class="nx">memory</span><span class="p">,</span> -<span class="ln"> 7 </span> <span class="nx">__linear_memory</span><span class="o">:</span> <span class="nx">memory</span><span class="p">,</span> -<span class="ln"> 8 </span> <span class="nx">__indirect_function_table</span><span class="o">:</span> <span class="nx">table</span><span class="p">,</span> -<span class="ln"> 9 </span> <span class="nx">call_me_sometime</span><span class="o">:</span> <span class="p">(</span><span class="nx">n</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span> -<span class="ln">10 </span> <span class="kd">let</span> <span class="nx">a</span> <span class="o">=</span> <span class="ow">new</span> <span class="nb">Uint8Array</span><span class="p">(</span><span class="nx">memory</span><span class="p">.</span><span class="nx">buffer</span><span class="p">,</span> <span class="nx">n</span><span class="p">,</span> <span class="mf">9</span><span class="p">)</span> -<span class="ln">11 </span> <span class="nx">a</span><span class="p">.</span><span class="nx">set</span><span class="p">([</span><span class="mh">0x66</span><span class="p">,</span> <span class="mh">0x6f</span><span class="p">,</span> <span class="mh">0x6f</span><span class="p">],</span> <span class="mf">0</span><span class="p">);</span> -<span class="ln">12 </span> <span class="nx">console</span><span class="p">.</span><span class="nx">debug</span><span class="p">(</span><span class="s1">'heap is at: '</span> <span class="o">+</span> <span class="nx">n</span><span class="p">);</span> -<span class="ln">13 </span> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'heap contains: '</span> <span class="o">+</span> <span class="ow">new</span> <span class="nx">TextDecoder</span><span class="p">().</span><span class="nx">decode</span><span class="p">(</span><span class="nx">a</span><span class="p">));</span> -<span class="ln">14 </span> <span class="p">},</span> -<span class="ln">15 </span> <span class="p">},</span> -<span class="ln">16 </span><span class="p">}</span> -<span class="ln">17 </span> -<span class="ln">18 </span><span class="k">async</span> <span class="kd">function</span> <span class="nx">init</span><span class="p">()</span> <span class="p">{</span> -<span class="ln">19 </span> <span class="kd">const</span> <span class="nx">code</span> <span class="o">=</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="s1">'./bare.wasm'</span><span class="p">);</span> -<span class="ln">20 </span> <span class="kd">const</span> <span class="nx">m</span> <span class="o">=</span> <span class="ow">new</span> <span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Module</span><span class="p">(</span><span class="nx">code</span><span class="p">);</span> -<span class="ln">21 </span> <span class="kd">const</span> <span class="nx">i</span> <span class="o">=</span> <span class="ow">new</span> <span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Instance</span><span class="p">(</span><span class="nx">m</span><span class="p">,</span> <span class="nx">importsObj</span><span class="p">);</span> -<span class="ln">22 </span> <span class="nx">i</span><span class="p">.</span><span class="nx">exports</span><span class="p">.</span><span class="nx">foo</span><span class="p">();</span> -<span class="ln">23 </span><span class="p">}</span> -<span class="ln">24 </span><span class="nx">init</span><span class="p">();</span> -</pre> -<p>The linker needs a little help from us for this:</p> -<ul class="simple"> -<li>Our callback function will not be available at link time, so we have to <code>--allow-undefined</code> to promise that the host has got this covered.</li> -<li><code>--import-memory</code> and <code>--import-table</code> to enable us to get memory and symbol table from the host.</li> -<li><code>--export="foo"</code> to make sure we only export exactly what we intend to from our <code>wasm</code>.</li> -</ul> -<div class="highlight"><pre><span></span>$ clang --target<span class="o">=</span>wasm32 -nostdlib -nostartfiles -Wl,--no-entry -Wl,--export<span class="o">=</span><span class="s2">"foo"</span> -Wl,--import-memory -Wl,--import-table -Wl,--allow-undefined -o bare.wasm bare.c -</pre></div> -<p>And that should give us:</p> -<div class="highlight"><pre><span></span>$ node bare.js -heap is at: <span class="m">66560</span> -heap contains: foobar -</pre></div> -<p>This way of pointing to memory is of course grossly inadequate <em>and</em> unsafe <em>and</em> ridiculous for any purpose more advanced that this one. So some proper memory management would not be a bad thing.</p> -</div> -<div class="section" id="adding-libc"> -<h2>Adding libc</h2> -<p>And what do you know. In other news since last time I looked at this is the addition of "a libc for WebAssembly programs built on top of WASI system calls." <a class="citation-reference" href="#wasi-libc" id="id3">[wasi-libc]</a>. Let's see if we can add a slightly less manual way of handling memory with <code>malloc</code> and <code>memcpy</code></p> -<pre class="code c literal-block"> -<span class="ln"> 0 </span><span class="cp">#ifdef HAVE_LIBC -</span><span class="ln"> 1 </span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><string.h></span><span class="cp"> -</span><span class="ln"> 2 </span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdlib.h></span><span class="cp"> -</span><span class="ln"> 3 </span><span class="cp">#endif -</span><span class="ln"> 4 </span><span class="cp"></span><span class="w"> -</span><span class="ln"> 5 </span><span class="w"></span><span class="k">extern</span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">__heap_base</span><span class="p">;</span><span class="w"> -</span><span class="ln"> 6 </span><span class="w"></span><span class="k">extern</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">call_me_sometime</span><span class="p">(</span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">b</span><span class="p">);</span><span class="w"> -</span><span class="ln"> 7 </span><span class="w"> -</span><span class="ln"> 8 </span><span class="w"></span><span class="kt">void</span><span class="w"> </span><span class="nf">foo</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"> -</span><span class="ln"> 9 </span><span class="w"> -</span><span class="ln">10 </span><span class="w"></span><span class="cp">#ifdef HAVE_LIBC -</span><span class="ln">11 </span><span class="cp"></span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">buf</span><span class="p">;</span><span class="w"> -</span><span class="ln">12 </span><span class="w"> </span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">malloc</span><span class="p">(</span><span class="mi">9</span><span class="p">);</span><span class="w"> -</span><span class="ln">13 </span><span class="w"> </span><span class="n">memcpy</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="s">"bazbar"</span><span class="p">,</span><span class="w"> </span><span class="mi">6</span><span class="p">);</span><span class="w"> -</span><span class="ln">14 </span><span class="w"></span><span class="cp">#else -</span><span class="ln">15 </span><span class="cp"></span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="o">*</span><span class="p">)</span><span class="o">&</span><span class="n">__heap_base</span><span class="p">;</span><span class="w"> -</span><span class="ln">16 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">3</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'b'</span><span class="p">;</span><span class="w"> -</span><span class="ln">17 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">4</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'a'</span><span class="p">;</span><span class="w"> -</span><span class="ln">18 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">5</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'r'</span><span class="p">;</span><span class="w"> -</span><span class="ln">19 </span><span class="w"></span><span class="cp">#endif -</span><span class="ln">20 </span><span class="cp"></span><span class="w"> </span><span class="n">call_me_sometime</span><span class="p">(</span><span class="n">buf</span><span class="p">);</span><span class="w"> -</span><span class="ln">21 </span><span class="w"> -</span><span class="ln">22 </span><span class="w"></span><span class="cp">#ifdef HAVE_LIBC -</span><span class="ln">23 </span><span class="cp"></span><span class="w"> </span><span class="n">free</span><span class="p">(</span><span class="n">buf</span><span class="p">);</span><span class="w"> -</span><span class="ln">24 </span><span class="w"></span><span class="cp">#endif -</span><span class="ln">25 </span><span class="cp"></span><span class="w"> -</span><span class="ln">26 </span><span class="w"></span><span class="p">}</span> -</pre> -<p>As you see, we need a few more parameters for the compiler and linker at this point. The <code>--target=wasm32-unknown-wasi --sysroot /opt/wasi-libc .. /opt/wasi-libc/lib/wasm32-wasi/libc.a</code> is needed to hook us up with headers and symbols for the libc.</p> -<p>My archlinux puts that sysroot in <code>/opt/wasi-libc</code>, that may of course not be the case elsewhere.</p> -<div class="highlight"><pre><span></span>$ clang -DHAVE_LIBC<span class="o">=</span><span class="m">1</span> --target<span class="o">=</span>wasm32-unknown-wasi --sysroot /opt/wasi-libc -nostdlib -nostartfiles -Wl,--no-entry -Wl,--export<span class="o">=</span><span class="s2">"foo"</span> -Wl,--import-memory -Wl,--import-table -Wl,--allow-undefined -o bare.wasm bare.c /opt/wasi-libc/lib/wasm32-wasi/libc.a -$ wasm-objdump -x bare.wasm - -bare.wasm: file format wasm 0x1 - -Section Details: - -Type<span class="o">[</span><span class="m">3</span><span class="o">]</span>: - - type<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="o">(</span>i32<span class="o">)</span> -> nil - - type<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="o">()</span> -> nil - - type<span class="o">[</span><span class="m">2</span><span class="o">]</span> <span class="o">(</span>i32<span class="o">)</span> -> i32 -Import<span class="o">[</span><span class="m">3</span><span class="o">]</span>: - - memory<span class="o">[</span><span class="m">0</span><span class="o">]</span> pages: <span class="nv">initial</span><span class="o">=</span><span class="m">2</span> <- env.memory - - table<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="nv">type</span><span class="o">=</span>funcref <span class="nv">initial</span><span class="o">=</span><span class="m">1</span> <- env.__indirect_function_table - - func<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">0</span> <call_me_sometime> <- env.call_me_sometime -Function<span class="o">[</span><span class="m">7</span><span class="o">]</span>: - - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">1</span> <foo> - - func<span class="o">[</span><span class="m">2</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">2</span> <malloc> - - func<span class="o">[</span><span class="m">3</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">2</span> <dlmalloc> - - func<span class="o">[</span><span class="m">4</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">0</span> <free> - - func<span class="o">[</span><span class="m">5</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">0</span> <dlfree> - - func<span class="o">[</span><span class="m">6</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">1</span> <abort> - - func<span class="o">[</span><span class="m">7</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">2</span> <sbrk> -Global<span class="o">[</span><span class="m">2</span><span class="o">]</span>: - - global<span class="o">[</span><span class="m">0</span><span class="o">]</span> i32 <span class="nv">mutable</span><span class="o">=</span><span class="m">1</span> - init <span class="nv">i32</span><span class="o">=</span><span class="m">67072</span> - - global<span class="o">[</span><span class="m">1</span><span class="o">]</span> i32 <span class="nv">mutable</span><span class="o">=</span><span class="m">0</span> <__heap_base> - init <span class="nv">i32</span><span class="o">=</span><span class="m">67072</span> -Export<span class="o">[</span><span class="m">2</span><span class="o">]</span>: - - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <foo> -> <span class="s2">"foo"</span> - - global<span class="o">[</span><span class="m">1</span><span class="o">]</span> -> <span class="s2">"__heap_base"</span> -Code<span class="o">[</span><span class="m">7</span><span class="o">]</span>: - - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">171</span> <foo> - - func<span class="o">[</span><span class="m">2</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">10</span> <malloc> - - func<span class="o">[</span><span class="m">3</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">6984</span> <dlmalloc> - - func<span class="o">[</span><span class="m">4</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">10</span> <free> - - func<span class="o">[</span><span class="m">5</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">1908</span> <dlfree> - - func<span class="o">[</span><span class="m">6</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">4</span> <abort> - - func<span class="o">[</span><span class="m">7</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">78</span> <sbrk> -Data<span class="o">[</span><span class="m">2</span><span class="o">]</span>: - - segment<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="nv">memory</span><span class="o">=</span><span class="m">0</span> <span class="nv">size</span><span class="o">=</span><span class="m">7</span> - init <span class="nv">i32</span><span class="o">=</span><span class="m">1024</span> - - <span class="m">0000400</span>: <span class="m">6261</span> 7a62 <span class="m">6172</span> <span class="m">00</span> bazbar. - - segment<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="nv">memory</span><span class="o">=</span><span class="m">0</span> <span class="nv">size</span><span class="o">=</span><span class="m">500</span> - init <span class="nv">i32</span><span class="o">=</span><span class="m">1032</span> - - <span class="m">0000408</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000418</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000428</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000438</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000448</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000458</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000468</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000478</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000488</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000498</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00004a8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00004b8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00004c8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00004d8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00004e8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00004f8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000508</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000518</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000528</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000538</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000548</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000558</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000568</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000578</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000588</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - <span class="m">0000598</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00005a8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00005b8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00005c8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00005d8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00005e8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ - - 00005f8: <span class="m">0000</span> <span class="m">0000</span> .... -Custom: - - name: <span class="s2">"name"</span> - - func<span class="o">[</span><span class="m">0</span><span class="o">]</span> <call_me_sometime> - - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <foo> - - func<span class="o">[</span><span class="m">2</span><span class="o">]</span> <malloc> - - func<span class="o">[</span><span class="m">3</span><span class="o">]</span> <dlmalloc> - - func<span class="o">[</span><span class="m">4</span><span class="o">]</span> <free> - - func<span class="o">[</span><span class="m">5</span><span class="o">]</span> <dlfree> - - func<span class="o">[</span><span class="m">6</span><span class="o">]</span> <abort> - - func<span class="o">[</span><span class="m">7</span><span class="o">]</span> <sbrk> -Custom: - - name: <span class="s2">"producers"</span> -</pre></div> -<p>What luxury. And of course, our <code>bare.wasm</code> file just grew from 350 bytes to 10k...</p> -<p>We don't have to change our <code>javascript</code> code at this point. Simply run again, and get:</p> -<div class="highlight"><pre><span></span>$ node bare.js -heap is at: <span class="m">67088</span> -heap contains: foobazbar -</pre></div> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id4" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td><code>__heap_base</code> will be set by default by the wasm environment, and is thus available as an external symbol.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id5" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>After linking the memory symbol meeds to be called <code>memory</code> instead of <code>__linear_memory</code> for some reason. Thus we add both here for clarity.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils citation" frame="void" id="wasi-libc" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id3">[wasi-libc]</a></td><td><a class="reference external" href="https://github.com/WebAssembly/wasi-libc">https://github.com/WebAssembly/wasi-libc</a></td></tr> -</tbody> -</table> -</blockquote> -</div> - - </div><!-- /.entry-content --> - <footer class="meta"> - </footer> - -</section> - <footer id="contentinfo" class="body"> - <hr/> - <address id="about" class="vcard body"> - Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> - </address><!-- /#about --> - </footer><!-- /#contentinfo --> -</body> -</html> -\ No newline at end of file diff --git a/2918deb778c2f4961cc75fcf66f21c6bbd1c6a848e0a5aebb2e3a9afdac25ce6.asc b/2918deb778c2f4961cc75fcf66f21c6bbd1c6a848e0a5aebb2e3a9afdac25ce6.asc @@ -1,11 +0,0 @@ ------BEGIN PGP SIGNATURE----- - -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmHkZ0UACgkQzcvSTdHQ -4AFIYQgAiAtYWkg5t6Uq6X09SfkBsQE58UdEnbIoppr2KN5cmOpDtqf8ZPvspRXh -xaiGPnkfIm6t0XjPWMm5hx5r+rCdnsW5x9W1ZeDWHdq39h7xCFqTif2vyD8scX4k -ib6czeGaCt/PAeDRMDYqwMLYUu0aebPQjeJES7Y2EDrIeNZVbflTYWBahG8uuhND -msy6ltPXR40L/cSJoEeBVY/X9oJbjZbESpvCYNr65EXzyEE3DTNSMyTdQfNaeiei -AdjbS4s3h127sLza18phyHlCeZA1P/+toKTjsNoFRVwxeiloVRZVz2Kc9verco6h -r6NRTAQ3pnTYqaHdqLfv749As/Ybmg== -=UmQu ------END PGP SIGNATURE----- diff --git a/5f38d7d3ab58689e73a237c1e10bf588e66a50457b4c4fea08fe423bc8dbd460 b/5f38d7d3ab58689e73a237c1e10bf588e66a50457b4c4fea08fe423bc8dbd460 @@ -1,291 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> - <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> - <title>man bytes gnu - The NFT token id as URI</title> - <meta charset="utf-8" /> - <meta name="generator" content="Pelican" /> - -</head> - -<body id="index" class="home"> - - <header id="banner" class="body"> - <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> - </header> - <nav id="menu"> - <ul> - <li><a href="./category/archiving.html">Archiving</a></li> - <li class="active"><a href="./category/code.html">Code</a></li> - <li><a href="./category/hygiene.html">Hygiene</a></li> - <li><a href="./category/offlining.html">Offlining</a></li> - <li><a href="./pages/identities.html">Identities</a></li> - <li><a href="./pages/shares.html">Shares</a></li> - - <li><a href="/tags.html">tags</a></li> - </ul> - <hr/> - </nav><!-- /#menu --> - <header> - <h1 class="entry-title top-body-title"> - <a href="./nft-tokenid-content.html" rel="bookmark" - title="Permalink to The NFT token id as URI">The NFT token id as URI</a></h1> - - - <div class="category meta"> - Posted -<time class="published" datetime="2021-05-08T19:14:00+02:00"> - Sat 08 May 2021 - </time> -in <a class="category" href="./category/code.html">code</a> - <a href="./tag/nft.html">nft</a> - <a href="./tag/evm.html">evm</a> - <a href="./tag/hash.html">hash</a> - <a href="./tag/key-value-store.html">key-value store</a> - <a href="./tag/decentralized-storage.html">decentralized storage</a> - </div> - - <div class="neighbors meta"> - <ul> - <li> - Previous: <a href="./wasm-c-bare.html"> - wasm and C - The bare necessities - </a> - </li> - | - - <li> - Next: <a href="./village-network-protocol.html"> - A village network protocol - </a> - </li> - </ul> - </div> - <div class="meta"> -<hr/> - </div> -</header> - -<section id="content" class="body"> - <div class="entry-content"> - <p>Let's consider an NFT that works like a badge for participating in development of a software project.</p> -<p>This token is awarded as a proof that the task was completed.</p> -<p>To make things more fun, each NFT should have some unique, immutable content attached to it.</p> -<p>In other words, the properties of this token, once set, should never change.</p> -<p>Nor should they disappear.</p> -<p>So how do we refer to the artwork asset within the token standard?</p> -<div class="section" id="it-was-acceptable-at-the-time"> -<h2>It was acceptable at the time</h2> -<p>The ERC721 standard is not explicit about where the assets that belong with the NFT can be discovered and resolved.</p> -<p>At the time when the standard was adopted by the Ethereum community, there were multiple <em>"[...] Alternatives considered: put all metadata for each asset on the blockchain (too expensive), use URL templates to query metadata parts (URL templates do not work with all URL schemes, especially P2P URLs), multiaddr network address (not mature enough)."</em> Furthermore, they <em>"[...] considered an NFT representing ownership of a house, in this case metadata about the house (image, occupants, etc.) can naturally change."</em> <a class="citation-reference" href="#eip721" id="id1">[EIP721]</a></p> -<p>A "changing house" doesn't sound quite like what we need. And anyway; if we stick a good old web2 URI in there, then that will end up on the great bonfire of dead links before long.</p> -</div> -<div class="section" id="image-schmimage"> -<h2>Image, schmimage</h2> -<p>To be honest, I find the presumption in the optional EIP721 metadata structure to be surprisingly short-sighted. It <em>specifically</em> defines the asset as an image, and at the same time is presupposes that only a <em>single</em> asset file will be used.</p> -<p>We may want to add <em>multiple</em> sources, so this is another obstacle for us.</p> -<p>So how to get around this, while still playing nice with existing implementations out there? Two ideas come to mind:</p> -<ul class="simple"> -<li>Embed a <em>thumbnail</em> as a preview of the artwork using a <code>base64</code> <em>data URI</em> <a class="footnote-reference" href="#id7" id="id2">[1]</a> in the metadata. Stick <code>name</code> and <code>description</code> on it, and the schema is still fulfilled.</li> -<li><em>Extend</em> the structure with a list of <em>attachments</em> that <em>our</em> application layer knows about. Of course, each of these can have the same format as above.</li> -</ul> -<p>In other words:</p> -<pre class="code json literal-block"> -<span class="p">{</span> - <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Foo"</span><span class="p">,</span> - <span class="nt">"description"</span><span class="p">:</span> <span class="s2">"Foo image"</span><span class="p">,</span> - <span class="nt">"image"</span><span class="p">:</span> <span class="nt">"data:image/gif;base64,R0lGODlhDgARAKEDAAAAAOjqAPP1APDw8CH+EUNyZWF0ZWQgd2l0aCBHSU1QACwAAAAADgARAAACO5wHqXdrClocodbUaGg34qoBHaJtl7VYIfqBZxepb5C6NBvbIlyucO6jPUIK1CMiSCYvDIVS4GhCJoYCADs="</span> - <span class="nt">"attachments"</span><span class="p">:</span> <span class="p">[</span> - <span class="p">{</span> - <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Bar"</span><span class="p">,</span> - <span class="nt">"description"</span><span class="p">:</span> <span class="s2">"Bar image"</span><span class="p">,</span> - <span class="nt">"image"</span><span class="p">:</span> <span class="s2">"data:image/gif;base64,"</span> - <span class="p">},</span> - <span class="p">{</span> - <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Baz"</span><span class="p">,</span> - <span class="nt">"description"</span><span class="p">:</span> <span class="s2">"Baz image"</span><span class="p">,</span> - <span class="nt">"image"</span><span class="p">:</span> <span class="s2">"data:image/gif;base64,"</span> - <span class="p">}</span> - <span class="p">]</span> -<span class="p">}</span> -</pre> -</div> -<div class="section" id="mirror-mirror"> -<h2>Mirror, mirror</h2> -<p>Since the asset reference shouldn't change, we can refer to it by its fingerprint or <a class="reference external" href="https://en.wikipedia.org/wiki/Content-addressable_storage">content address</a>. If we define that the resource can be looked up over HTTP by that fingerprint as its basename, then we are free to define and modify whatever list of mirrors for that resource that's valid for any point in time. The application layer would simply try the endpoints one after another.</p> -<p>We take the <code>sha2-256</code> <a class="footnote-reference" href="#id8" id="id3">[2]</a> of the asset reference (the json file above, free of evil whitespace and newlines):</p> -<div class="highlight"><pre><span></span>$ cat reference.json <span class="p">|</span> jq -c -j <span class="p">|</span> sha256sum <span class="p">|</span> awk <span class="s1">'{ print $1; }'</span> -3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551 -</pre></div> -<p>Imagine we had a mirror list of <a class="reference external" href="https://foo.com">https://foo.com</a> and <a class="reference external" href="https://bar.com/baz/">https://bar.com/baz/</a>. Then our application would try these urls in sequence, stopping at the first that returns a valid result:</p> -<div class="highlight"><pre><span></span>https://foo.com/3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551 -https://bar.com/baz/3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551 -</pre></div> -<p>Once we receive the content, all we have to do is hash it ourselves and verify that the sum matches the basename of the URI. If it doesn't the result is of course not valid and we continue down the list, appropriately banning the mischievous server then throrougly harassing its admin.</p> -</div> -<div class="section" id="cast-away"> -<h2>Cast away</h2> -<p>Since our fingerprint is 32 bytes, it fits exactly inside the <code>tokenId</code> (<code>uint256</code>). Let's decide to big-endian numbers when converting (I find them easier to make sense of). In that case our hash from the reference turns into this modest number:</p> -<div class="highlight"><pre><span></span><span class="c1"># python3</span> -<span class="o">>>></span> <span class="n">hx</span> <span class="o">=</span> <span class="nb">bytes</span><span class="o">.</span><span class="n">fromhex</span><span class="p">(</span><span class="s1">'3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551'</span><span class="p">)</span> -<span class="o">>>></span> <span class="nb">int</span><span class="o">.</span><span class="n">from_bytes</span><span class="p">(</span><span class="n">hx</span><span class="p">,</span> <span class="n">byteorder</span><span class="o">=</span><span class="s1">'big'</span><span class="p">)</span> -<span class="mi">28891040728719892888467057134569335350980764617882743994259054993630416573777</span> -</pre></div> -<p>As long as we're composing the <code>evm</code> inputs ourselves, we don't really have to worry about the integer representation in this particular case. But the interface is defined as an integer type, and other mortals may be using higher level interfaces, we have to be explicit about our choice.</p> -</div> -<div class="section" id="welcoming-mint"> -<h2>Welcoming mint</h2> -<p>Assume we have a method <code>mintTo(address _recipient, uint256 _tokenId)</code> on our NFT contract. The Solidity signature of that contract is <code>edb20b7e</code> <a class="footnote-reference" href="#id9" id="id4">[3]</a>. If I were to mint to myself then the input to the contract would be:</p> -<div class="highlight"><pre><span></span>edb20b7e000000000000000000000000185cbce7650ff7ad3b587e26b2877d95568805e33fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551 -</pre></div> -<p>Broken down:</p> -<div class="highlight"><pre><span></span>signature: edb20b7e -address, zero-padded: 000000000000000000000000185cbce7650ff7ad3b587e26b2877d95568805e3 -token id: 3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551 -</pre></div> -<p>The corresponding web3.js code would look like:</p> -<div class="highlight"><pre><span></span><span class="kd">const</span> <span class="nx">c</span> <span class="o">=</span> <span class="ow">new</span> <span class="nx">web3</span><span class="p">.</span><span class="nx">eth</span><span class="p">.</span><span class="nx">Contract</span><span class="p">([...],</span> <span class="s1">'0x...'</span><span class="p">);</span> -<span class="nx">c</span><span class="p">.</span><span class="nx">methods</span><span class="p">.</span><span class="nx">mintTo</span><span class="p">(</span><span class="s1">'0x185cbce7650ff7ad3b587e26b2877d95568805e3'</span><span class="p">,</span> <span class="mf">28891040728719892888467057134569335350980764617882743994259054993630416573777</span><span class="p">).</span><span class="nx">call</span><span class="p">();</span> -</pre></div> -<p>To satisfy the <cite>tokenURI</cite> method, we can generate a string that's prefix with sha256 as a "scheme" <a class="footnote-reference" href="#id10" id="id5">[4]</a>. A bit of (unoptimized) solidity helps us out here:</p> -<pre class="code solidity literal-block"> -<span class="k">contract</span><span class="w"> </span><span class="ni">NFT</span><span class="w"> </span><span class="p">{</span><span class="w"> - - </span><span class="kt">uint256</span><span class="p">[]</span><span class="w"> </span>token<span class="p">;</span><span class="w"> - </span><span class="kt">mapping</span><span class="p">(</span><span class="kt">uint256</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="kt">uint256</span><span class="p">)</span><span class="w"> </span>tokenIndex<span class="p">;</span><span class="w"> </span><span class="c1">// map token id to master token array index position -</span><span class="w"> - </span><span class="p">[...]</span><span class="w"> - - </span><span class="kt">function</span><span class="w"> </span><span class="nv">tokenURI</span><span class="p">(</span><span class="kt">uint256</span><span class="w"> </span><span class="nv">_tokenId</span><span class="p">)</span><span class="w"> </span><span class="kt">public</span><span class="w"> </span>pure<span class="w"> </span><span class="kt">returns</span><span class="p">(</span><span class="kt">string</span><span class="w"> </span><span class="nv">memory</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span><span class="kt">bytes32</span><span class="w"> </span><span class="nv">token_bytes</span><span class="p">;</span><span class="w"> - </span><span class="kt">bytes</span><span class="w"> </span><span class="nv">memory</span><span class="w"> </span>out<span class="p">;</span><span class="w"> - </span><span class="kt">uint8</span><span class="w"> </span><span class="nv">t</span><span class="p">;</span><span class="w"> - </span><span class="kt">uint256</span><span class="w"> </span><span class="nv">c</span><span class="p">;</span><span class="w"> - - </span>token_bytes<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">bytes32</span><span class="p">(</span>token<span class="p">[</span>tokenIndex<span class="p">[</span>_tokenId<span class="p">]]);</span><span class="w"> - - </span>out<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">new</span><span class="w"> </span><span class="kt">bytes</span><span class="p">(</span><span class="m-Decimal">64</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="m-Decimal">7</span><span class="p">);</span><span class="w"> - </span>out<span class="p">[</span><span class="m-Decimal">0</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"s"</span><span class="p">;</span><span class="w"> - </span>out<span class="p">[</span><span class="m-Decimal">1</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"h"</span><span class="p">;</span><span class="w"> - </span>out<span class="p">[</span><span class="m-Decimal">2</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"a"</span><span class="p">;</span><span class="w"> - </span>out<span class="p">[</span><span class="m-Decimal">3</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"2"</span><span class="p">;</span><span class="w"> - </span>out<span class="p">[</span><span class="m-Decimal">4</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"5"</span><span class="p">;</span><span class="w"> - </span>out<span class="p">[</span><span class="m-Decimal">5</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"6"</span><span class="p">;</span><span class="w"> - </span>out<span class="p">[</span><span class="m-Decimal">6</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">":"</span><span class="p">;</span><span class="w"> - - </span>c<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m-Decimal">7</span><span class="p">;</span><span class="w"> - </span><span class="kt">for</span><span class="w"> </span><span class="p">(</span><span class="kt">uint256</span><span class="w"> </span><span class="nv">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m-Decimal">0</span><span class="p">;</span><span class="w"> </span>i<span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="m-Decimal">32</span><span class="p">;</span><span class="w"> </span>i<span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span>t<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">uint8</span><span class="p">(</span>_data<span class="p">[</span>i<span class="p">])</span><span class="w"> </span><span class="err">&</span><span class="w"> </span><span class="mh">0xf0</span><span class="p">)</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="m-Decimal">4</span><span class="p">;</span><span class="w"> - </span><span class="kt">if</span><span class="w"> </span><span class="p">(</span>t<span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="m-Decimal">10</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span>out<span class="p">[</span>c<span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>bytes1<span class="p">(</span>t<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mh">0x30</span><span class="p">);</span><span class="w"> - </span><span class="p">}</span><span class="w"> </span><span class="kt">else</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span>out<span class="p">[</span>c<span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>bytes1<span class="p">(</span>t<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mh">0x57</span><span class="p">);</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span>t<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">uint8</span><span class="p">(</span>_data<span class="p">[</span>i<span class="p">])</span><span class="w"> </span><span class="err">&</span><span class="w"> </span><span class="mh">0x0f</span><span class="p">;</span><span class="w"> - </span><span class="kt">if</span><span class="w"> </span><span class="p">(</span>t<span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="m-Decimal">10</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span>out<span class="p">[</span>c<span class="o">+</span><span class="m-Decimal">1</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>bytes1<span class="p">(</span>t<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mh">0x30</span><span class="p">);</span><span class="w"> - </span><span class="p">}</span><span class="w"> </span><span class="kt">else</span><span class="w"> </span><span class="p">{</span><span class="w"> - </span>out<span class="p">[</span>c<span class="o">+</span><span class="m-Decimal">1</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>bytes1<span class="p">(</span>t<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mh">0x57</span><span class="p">);</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span>c<span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="m-Decimal">2</span><span class="p">;</span><span class="w"> - </span><span class="p">}</span><span class="w"> - </span><span class="kt">return</span><span class="w"> </span><span class="kt">string</span><span class="p">(</span>out<span class="p">);</span><span class="w"> - </span><span class="p">}</span><span class="w"> -</span><span class="p">}</span> -</pre> -<p>This will return <code>sha256:3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551</code> for <code>tokenId</code> <code>3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551</code> as input, provided that the <code>tokenId</code> actually exists. That may seem a bit useless at first, but consider the scenario where we want to interface with other NFTs aswell. Or perhaps we are implementing a contract that optionally can support a static web2 URI in storage. By doing it this way, all bases are covered.</p> -</div> -<div class="section" id="decentralized-identifiers"> -<h2>Decentralized identifiers</h2> -<p>Even better would be to add redundancy with autonomous decentralized storage. However, networks like <a class="reference external" href="https://ethswarm.org">Swarm</a> and <a class="reference external" href="https://ipfs.io">IPFS</a> use their own hashing recipes. That means that for every network referenced, we'd have to define an <em>alternative</em> in our reference structure.</p> -<p>Referencing the canonical <code>sha256</code> aswell as the <code>Swarmhash</code> for the same item could then look like this <a class="footnote-reference" href="#id11" id="id6">[5]</a>:</p> -<pre class="code json literal-block"> -<span class="p">{</span> - <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Foo"</span><span class="p">,</span> - <span class="nt">"description"</span><span class="p">:</span> <span class="s2">"Foo image"</span><span class="p">,</span> - <span class="nt">"image"</span><span class="p">:</span> <span class="nt">"data:image/gif;base64,R0lGODlhDgARAKEDAAAAAOjqAPP1APDw8CH+EUNyZWF0ZWQgd2l0aCBHSU1QACwAAAAADgARAAACO5wHqXdrClocodbUaGg34qoBHaJtl7VYIfqBZxepb5C6NBvbIlyucO6jPUIK1CMiSCYvDIVS4GhCJoYCADs="</span> - <span class="nt">"alternatives"</span><span class="p">:</span> <span class="p">[</span> - <span class="p">{</span> - <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Foo"</span><span class="p">,</span> - <span class="nt">"description"</span><span class="p">:</span> <span class="s2">"Foo image"</span><span class="p">,</span> - <span class="nt">"image"</span><span class="p">:</span> <span class="s2">"bzz:4b9149ee4550f2d786f9ba6584b79a30ee14ae05ff6e84a0f7c7561a14e3b779"</span> - <span class="p">},</span> - <span class="p">{</span> - <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Foo"</span><span class="p">,</span> - <span class="nt">"description"</span><span class="p">:</span> <span class="s2">"Foo image"</span><span class="p">,</span> - <span class="nt">"image"</span><span class="p">:</span> <span class="s2">"sha256:d036a4ce7f929b632256225b2bebd81bdd558d3bfe3d96faae61db664708c16f"</span> - <span class="p">}</span> - <span class="p">]</span> -<span class="p">}</span> -</pre> -<hr class="docutils" /> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id7" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id2">[1]</a></td><td>Yes, they are valid URIs actually: <a class="reference external" href="https://www.rfc-archive.org/getrfc.php?rfc=2397">https://www.rfc-archive.org/getrfc.php?rfc=2397</a></td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id8" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id3">[2]</a></td><td>Likely it would be prudent to start using the official <code>sha3</code> instead of <code>sha2</code> these days, also because the <code>sha2</code> hash is not a builtin for <code>evm</code>. But neither is <code>sha3</code>. The <code>keccak256</code> Bitcoin uses, which EVM has inherited, is a pre-cursor to the <code>keccak</code> published as the <em>official</em> <code>sha3</code>. Still, <code>keccak256</code> and <code>sha3</code> is used interchangeably in opcode lists (and previously in <a class="reference external" href="https://docs.soliditylang.org/en/v0.8.0/050-breaking-changes.html#functions">Solidity</a> too). This has caused me quite a fair bit of confusion, I might add. Apart from it being ambiguous, the <code>keccak256</code> tooling is also less common in the wild. Therefore <code>sha2</code> seems like a safer bet for our experiments. It's not broken yet, after all.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id9" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id4">[3]</a></td><td>The hex result of <code>keccak256("mintTo(address,uint256)")</code></td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id10" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id5">[4]</a></td><td><em>Data URI</em> is of no use here, because the hash itself is just nondescript binary data. Luckily <code><scheme>:<path></code> is still a valid URI.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id11" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id6">[5]</a></td><td>Here the hashes represent the media content itself, not the reference. That's why the <code>sha256</code> one is different than before.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils citation" frame="void" id="eip721" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id1">[EIP721]</a></td><td><a class="reference external" href="https://eips.ethereum.org/EIPS/eip-721">https://eips.ethereum.org/EIPS/eip-721</a></td></tr> -</tbody> -</table> -</blockquote> -</div> - - </div><!-- /.entry-content --> - <footer class="meta"> - </footer> - -</section> - <footer id="contentinfo" class="body"> - <hr/> - <address id="about" class="vcard body"> - Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> - </address><!-- /#about --> - </footer><!-- /#contentinfo --> -</body> -</html> -\ No newline at end of file diff --git a/5f38d7d3ab58689e73a237c1e10bf588e66a50457b4c4fea08fe423bc8dbd460.asc b/5f38d7d3ab58689e73a237c1e10bf588e66a50457b4c4fea08fe423bc8dbd460.asc @@ -1,11 +0,0 @@ ------BEGIN PGP SIGNATURE----- - -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmHkZ0UACgkQzcvSTdHQ -4AGo8wf+Oe9vfl8BEv36O4lGLFcoX5s2OyBjN/PwTbS/YJga232YqX7x/mAEm5Ul -C7rUOyrtx6RaF4lqDDs2HiWMQX7QVspH8pLYrRfWLKq9/fYeKihM44V91/vj/X/t -1yClp0tEmuBx1g1lpjih1SxOgOjslwf1nOXgeJjlmP7nNxNQQZTBHJFhtEAal2MW -2Ryj1MvMOLbxKcNi4uzUvpbrOmPuZwo9X+x3gOryp4Vrz16yW2iMVJICE44TDetw -CJ6j+udujyTk3faGD/y8u7fMOtCl8olTUHEgGhNPMbJGdHlx/dWJzYoDhGoL2ToL -5wv5WgQVUNNRO/ZerxlmuvheLvhGMA== -=ZBzC ------END PGP SIGNATURE----- diff --git a/617878ecd470e80c0f5fadc0cf7e669ceea086a54343e56a35aeaf8131038d6c b/617878ecd470e80c0f5fadc0cf7e669ceea086a54343e56a35aeaf8131038d6c @@ -0,0 +1,224 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - Making sense of Ethereum log bloom filters</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="./category/archiving.html">Archiving</a></li> + <li class="active"><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./eth-log-bloom.html" rel="bookmark" + title="Permalink to Making sense of Ethereum log bloom filters">Making sense of Ethereum log bloom filters</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2021-06-27T11:17:00+02:00"> + Sun 27 June 2021 + </time> +in <a class="category" href="./category/code.html">code</a> + <a href="./tag/crypto.html">crypto</a> + <a href="./tag/ethereum.html">ethereum</a> + <a href="./tag/bloom-filter.html">bloom filter</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./docker-offline-3-npm.html"> + Local npm repository + </a> + </li> + | + + <li> + Next: <a href="./internet-up-monitor.html"> + Homemade internet state monitor + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>Ethereum blocks and transaction receipts both carry a bloom filter. They provide a shortcut to check whether a particular event of a particular contract was triggered. The filter in the transaction indexes all events that occurred in the transaction. The filter in the block carries all the filter bits of the individual transactions within the block.</p> +<p>Somewhat surprisingly, there are practically no friendly descriptions out there that tell you how to generate filters to match them with. In fact, the <a class="reference external" href="https://holbrook.no/doc/ethereum/yellowpaper.pdf">Ethereum Yellow Paper</a> was the only source I found.</p> +<p><strong>So let's make a friendly description</strong></p> +<p>Before we get to the friendliness, though, an ever so slight dose of brain melt is due:</p> +<div class="section" id="the-formalities"> +<h2>The formalities</h2> +<p>The formal definition of the filter calculation in the <a class="reference external" href="https://holbrook.no/doc/ethereum/yellowpaper.pdf">Ethereum Yellow Paper section 4.3.1</a> is (here cited in painfully inadequate math formatting):</p> +<blockquote> +<div class="formula"> +<i>M</i>(<i>O</i>) ≡ <span class="limits"><span class="limit"><span class="bigoperator">⋁</span></span></span><i>x</i> ∈ {<i>Oa</i>}∪<i>Ot</i>(<i>M</i><sub>3 : 2048</sub>(<i>x</i>)) +</div> +</blockquote> +<p>Where <span class="formula"><i>Oa</i></span> is the contract address and <span class="formula"><i>Ot</i></span> is all of the topics.</p> +<p>In the context of <em>Solidity</em>, "all" of the topics means the actual event signature, along with all the <em>indexed</em> parameters to the event.</p> +<p>The <span class="formula"><i>M</i></span> is the filter bit generator. The filter 2048 bits long, and for each entry that is added, 3 bits need to be set.</p> +<p>Reading further, we learn that the bits to set are the "first three" pairs of bytes of the keccak256 hash <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>, modulo the length of the filter - 2048.</p> +<p>In the context of hashes, I usually assume "first" means from the left. That happens to be the case here, too. In other words, the "first" bytes are actually the bytes of most significance in big-endian order.</p> +<p>What's less clear, however, is which bits to set in the filter. The paper says:</p> +<blockquote> +<span class="formula">ℬ</span> is the bit reference function such that <span class="formula">ℬ<sub><i>j</i></sub>(<i>x</i>)</span> equals the bit of index <span class="formula"><i>j</i></span> (indexed from 0) in the byte array <span class="formula"><i>x</i></span>.</blockquote> +<p>Turns out after some trial and error that "index 0" means the <em>right</em> end of the byte array. So when setting the bits, we have to count from the right side.</p> +</div> +<div class="section" id="the-friendly-recipe"> +<h2>The friendly recipe</h2> +<p>Now let's try to bring this down to a mortal level:</p> +<ol class="arabic simple"> +<li>Take the keccak256 hash of the contract address</li> +<li>Take the keccak256 hash of the first topic value</li> +<li>Optionally, take the keccak256 hash of the remaining topic values (the indexed parameters of the event) <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a></li> +<li>Instantiate a byte array with length 256</li> +<li>Now, <em>for each</em> of the hashes:<ol class="arabic"> +<li>Get the <em>big-endian</em> integer of the <em>first two</em> bytes of the hash</li> +<li>Calculate the 2048-modulo of that integer <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a></li> +<li><em>Bitwise or</em> the filter bit at the index corresponding to the modulated value, counting from <em>right to left</em> (i.e. the value <span class="formula">0</span> corresponds to the <em>rightmost bit</em>).</li> +<li>Repeat 2 and 3 for the two next pairs of bytes of the hash.</li> +</ol> +</li> +</ol> +</div> +<div class="section" id="the-code"> +<h2>The code</h2> +<p>All that taken into account, we can take a stab at implementing a filter generator:</p> +<pre class="code python literal-block"> +<span class="ln"> 0 </span><span class="c1"># external imports</span> +<span class="ln"> 1 </span><span class="kn">import</span> <span class="nn">sha3</span> +<span class="ln"> 2 </span> +<span class="ln"> 3 </span> +<span class="ln"> 4 </span><span class="k">class</span> <span class="nc">LogBloom</span><span class="p">:</span> +<span class="ln"> 5 </span> +<span class="ln"> 6 </span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> +<span class="ln"> 7 </span> <span class="bp">self</span><span class="o">.</span><span class="n">content</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">(</span><span class="mi">256</span><span class="p">)</span> +<span class="ln"> 8 </span> +<span class="ln"> 9 </span> +<span class="ln">10 </span> <span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">):</span> +<span class="ln">11 </span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">element</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">):</span> +<span class="ln">12 </span> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'element must be bytes'</span><span class="p">)</span> +<span class="ln">13 </span> <span class="n">h</span> <span class="o">=</span> <span class="n">sha3</span><span class="o">.</span><span class="n">keccak_256</span><span class="p">()</span> +<span class="ln">14 </span> <span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">element</span><span class="p">)</span> +<span class="ln">15 </span> <span class="n">z</span> <span class="o">=</span> <span class="n">h</span><span class="o">.</span><span class="n">digest</span><span class="p">()</span> +<span class="ln">16 </span> +<span class="ln">17 </span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span> +<span class="ln">18 </span> <span class="n">c</span> <span class="o">=</span> <span class="n">j</span> <span class="o">*</span> <span class="mi">2</span> +<span class="ln">19 </span> <span class="n">v</span> <span class="o">=</span> <span class="nb">int</span><span class="o">.</span><span class="n">from_bytes</span><span class="p">(</span><span class="n">z</span><span class="p">[</span><span class="n">c</span><span class="p">:</span><span class="n">c</span><span class="o">+</span><span class="mi">2</span><span class="p">],</span> <span class="n">byteorder</span><span class="o">=</span><span class="s1">'big'</span><span class="p">)</span> +<span class="ln">20 </span> <span class="n">v</span> <span class="o">&=</span> <span class="mh">0x07ff</span> +<span class="ln">21 </span> <span class="n">m</span> <span class="o">=</span> <span class="mi">255</span> <span class="o">-</span> <span class="nb">int</span><span class="p">(</span><span class="n">v</span> <span class="o">/</span> <span class="mi">8</span><span class="p">)</span> +<span class="ln">22 </span> <span class="n">n</span> <span class="o">=</span> <span class="n">v</span> <span class="o">%</span> <span class="mi">8</span> +<span class="ln">23 </span> <span class="bp">self</span><span class="o">.</span><span class="n">content</span><span class="p">[</span><span class="n">m</span><span class="p">]</span> <span class="o">|=</span> <span class="p">(</span><span class="mi">1</span> <span class="o"><<</span> <span class="n">n</span><span class="p">)</span> +</pre> +<p>Let's say we'd want to check if an Solidity-defined event <code>Foo(uint256,bytes32)</code> emitted by a contract at address <code>0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee</code> exists in a filter.</p> +<p>Using the class above, we would have to do something like this:</p> +<pre class="code python literal-block"> +<span class="c1"># external imports</span> +<span class="kn">import</span> <span class="nn">sha3</span> +<span class="kn">import</span> <span class="nn">some_block_getter</span> + +<span class="c1"># local imports</span> +<span class="kn">from</span> <span class="nn">the_above</span> <span class="kn">import</span> <span class="n">LogBloom</span> + +<span class="c1"># all set bits in our filter must be set in theirs</span> +<span class="k">def</span> <span class="nf">filter_match</span><span class="p">(</span><span class="n">theirs</span><span class="p">,</span> <span class="n">ours</span><span class="p">):</span> + <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">ours</span><span class="p">)):</span> + <span class="k">if</span> <span class="n">ours</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">&</span> <span class="n">theirs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">!=</span> <span class="n">ours</span><span class="p">[</span><span class="n">i</span><span class="p">]:</span> + <span class="k">return</span> <span class="kc">False</span> + <span class="k">return</span> <span class="kc">True</span> + +<span class="n">fltr</span> <span class="o">=</span> <span class="n">LogBloom</span><span class="p">()</span> + +<span class="c1"># add the contract address to the filter</span> +<span class="n">address</span> <span class="o">=</span> <span class="nb">bytes</span><span class="o">.</span><span class="n">fromhex</span><span class="p">(</span><span class="s1">'eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee'</span><span class="p">)</span> +<span class="n">h</span> <span class="o">=</span> <span class="n">sha3</span><span class="o">.</span><span class="n">keccak_256</span><span class="p">()</span> +<span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">address</span><span class="p">)</span> +<span class="n">address_element</span> <span class="o">=</span> <span class="n">h</span><span class="o">.</span><span class="n">digest</span><span class="p">()</span> +<span class="n">fltr</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">address_element</span><span class="p">)</span> + +<span class="c1"># create the topic signature for the event</span> +<span class="n">h</span> <span class="o">=</span> <span class="n">sha3</span><span class="o">.</span><span class="n">keccak_256</span><span class="p">()</span> +<span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="sa">b</span><span class="s1">'Foo(uint256,bytes32)'</span><span class="p">)</span> +<span class="n">topic</span> <span class="o">=</span> <span class="n">h</span><span class="o">.</span><span class="n">digest</span><span class="p">()</span> + +<span class="c1"># add the topic to the filter</span> +<span class="n">h</span> <span class="o">=</span> <span class="n">sha3</span><span class="o">.</span><span class="n">keccak_256</span><span class="p">()</span> +<span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">topic</span><span class="p">)</span> +<span class="n">topic_element</span> <span class="o">=</span> <span class="n">h</span><span class="o">.</span><span class="n">digest</span><span class="p">()</span> +<span class="n">fltr</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">topic_element</span><span class="p">)</span> + +<span class="c1"># get the filter from a block</span> +<span class="n">block</span> <span class="o">=</span> <span class="n">some_block_getter</span><span class="p">()</span> +<span class="n">block_bloom</span> <span class="o">=</span> <span class="nb">bytes</span><span class="o">.</span><span class="n">fromhex</span><span class="p">(</span><span class="n">block</span><span class="p">[</span><span class="s1">'logsBloom'</span><span class="p">][</span><span class="mi">2</span><span class="p">:])</span> <span class="c1"># assumes there is a 0x-prefix</span> + +<span class="c1"># check if it matches</span> +<span class="n">match</span> <span class="o">=</span> <span class="n">filter_match</span><span class="p">(</span><span class="n">block_bloom</span><span class="p">,</span> <span class="n">fltr</span><span class="o">.</span><span class="n">contents</span><span class="p">)</span> +<span class="nb">print</span><span class="p">(</span><span class="n">match</span><span class="p">)</span> +</pre> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Remember the keccak hash used in Ethereum/Bitcoin is _not_ the official <cite>sha3</cite> hash. The padding parameter is different. More on that <a class="reference external" href="./nft-tokenid-content.html#id8">here</a></td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>Solidity supports up to 3 indexed parameters, which maps to the <em>evm</em> opcodes <code>LOGn</code> where <span class="formula"><i>n</i> ∈ {1, 2, 3, 4}</span></td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>Since <span class="formula">2<sup>11</sup> = 2048</span>, this is the same as zeroing out all bits above bit number 11; <span class="formula"><i>x</i>∧2048</span>. In other words, the result is an <em>11 bit integer</em>.</td></tr> +</tbody> +</table> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/617878ecd470e80c0f5fadc0cf7e669ceea086a54343e56a35aeaf8131038d6c.asc b/617878ecd470e80c0f5fadc0cf7e669ceea086a54343e56a35aeaf8131038d6c.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NzsACgkQzcvSTdHQ +4AH2HQf5Afq7lMppFTu2kyMAHdjtFjzbSrMjrZ88lg7SeJjTLTurNc75rMsvu+lU +C19WFpg7uVecQi3aJlsMKA7zUDJ0i2bPW9waM/VO4OlKcMjm4TLE/w4Os/lm+Ljn +TY2/4X/pJuk011PWAowOkmJpIhXclTxBATiNI6SCb0QzoliY7jsazFzGqW5mYDIk +Dn4IW2CrXRBMgKn+mRZud5dk5dbS4Kw90hcjbKzo8XCaQPuxXlwWjiQr3bJTd0dC +/DwZtQxd9U2CM5/2a8WPszP1ZHqSJ7Lx4vbI0hFCZttEeznTzDlwfVPdCB8v2MoI +DzjUzNiUm4TatsXHyeY/cnEO3rhPTA== +=vBiY +-----END PGP SIGNATURE----- diff --git a/6cd7d22162d74eba04b00bdc004e18907c32ba245e0bf210ccb26e431789b18b b/6cd7d22162d74eba04b00bdc004e18907c32ba245e0bf210ccb26e431789b18b @@ -1,328 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> - <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> - <title>man bytes gnu - A village network protocol</title> - <meta charset="utf-8" /> - <meta name="generator" content="Pelican" /> - -</head> - -<body id="index" class="home"> - - <header id="banner" class="body"> - <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> - </header> - <nav id="menu"> - <ul> - <li><a href="./category/archiving.html">Archiving</a></li> - <li><a href="./category/code.html">Code</a></li> - <li class="active"><a href="./category/hygiene.html">Hygiene</a></li> - <li><a href="./category/offlining.html">Offlining</a></li> - <li><a href="./pages/identities.html">Identities</a></li> - <li><a href="./pages/shares.html">Shares</a></li> - - <li><a href="/tags.html">tags</a></li> - </ul> - <hr/> - </nav><!-- /#menu --> - <header> - <h1 class="entry-title top-body-title"> - <a href="./village-network-protocol.html" rel="bookmark" - title="Permalink to A village network protocol">A village network protocol</a></h1> - - - <div class="category meta"> - Posted -<time class="published" datetime="2021-05-19T10:40:00+02:00"> - Wed 19 May 2021 - </time> -in <a class="category" href="./category/hygiene.html">hygiene</a> - <a href="./tag/p2p.html">p2p</a> - <a href="./tag/protocol.html">protocol</a> - <a href="./tag/network.html">network</a> - <a href="./tag/crm.html">crm</a> - <a href="./tag/privacy.html">privacy</a> - <a href="./tag/f2f.html">f2f</a> - <a href="./tag/community.html">community</a> - </div> - - <div class="neighbors meta"> - <ul> - <li> - Previous: <a href="./nft-tokenid-content.html"> - The NFT token id as URI - </a> - </li> - | - - <li> - Next: <a href="./docker-offline-3-npm.html"> - Local npm repository - </a> - </li> - </ul> - </div> - <div class="meta"> -<hr/> - </div> -</header> - -<section id="content" class="body"> - <div class="entry-content"> - <p>What would truly peer-to-peer CRM backend look like? And at that, once that respected the right of the "customer" to choose how it wants to represent itself in the context?</p> -<p>Perhaps by Modeling the network on how villages treat strangers and their own, and how you would go about interacting with them.</p> -<p>Here's an initial stab at describing what such a meta-protocol could look like.</p> -<!-- There is no truth. There is only perspective and predictability. --> -<div class="section" id="terms"> -<h2>Terms</h2> -<dl class="docutils"> -<dt>Individual</dt> -<dd>A subjective structure where the thing being described is the thing that described it.</dd> -<dt>User</dt> -<dd>A particular perception of an individual by an entity in the network.</dd> -<dt>Userspec</dt> -<dd>Data structure containing details about a user. May or may not be public.</dd> -<dt>Locals</dt> -<dd>A group of individuals acknowledging each other as part of a group, and who share views on users.</dd> -<dt>Local</dt> -<dd>A member of a group of locals, from the shared perspective of an individual and another local.</dd> -<dt>Localslist</dt> -<dd>The individuals making up a group of locals, from the perspective of a local.</dd> -<dt>Stranger</dt> -<dd>A user as percieved by a group of locals.</dd> -<dt>Strangerlist</dt> -<dd>All users that may speak to the locals, from the perspective of a local. To be shared among locals.</dd> -<dt>Strangerspec</dt> -<dd>Userspec created by a local describing a user. To be shared among locals.</dd> -<dt>Entity</dt> -<dd>An individual or any derived representation of it (user, stranger, local).</dd> -<dt>Peer</dt> -<dd>An entity coupled with connection information</dd> -<dt>Content address</dt> -<dd>SHA256 sum of arbitrary content</dd> -<dt>Key setup</dt> -<dd>May be an individual key, or a specification describing how to generate keys for a particular relation.</dd> -</dl> -</div> -<div class="section" id="roles"> -<h2>Roles</h2> -<div class="section" id="the-individuals"> -<h3>The individuals</h3> -<p>Are free to have their own opinion of what they are.</p> -<p>When interacting with others, they are free to choose what they say about what they are.</p> -<p>An individual initially <em>defines itself</em> in the network.</p> -</div> -<div class="section" id="the-users"> -<h3>The users</h3> -<p>A user is a view of an individual as defined by one or more other individuals.</p> -<p>It may be how an individual presents itself to an particular individual or a group.</p> -<p>It may also be how an individual or a group presents another individual to itself.</p> -</div> -<div class="section" id="the-locals"> -<h3>The locals</h3> -<p>Locals are a group of individuals who wish to share a view of themselves and prejudice about others.</p> -<p>An individual will not be treated as a local unless it and its local counterpart considers it a local.</p> -<p>Locals may acknowledge strangers with or without proof.</p> -<p>If one local changes opinion about a stranger, all locals should change their opinon to the same.</p> -<p>If one local excludes a stranger, all locals should exclude the stranger.</p> -<p>You can't keep secrets in a village. If someone says something to another local, it must assume all locals will get to know what was said.</p> -</div> -<div class="section" id="the-strangers"> -<h3>The strangers</h3> -<p>A stranger is a user that's not considered local by a group of locals.</p> -<p>A stranger may not speak to a local until the local has decided it can.</p> -<p>If a stranger may speak to one local, it may speak all locals.</p> -</div> -</div> -<div class="section" id="identifiers"> -<h2>Identifiers</h2> -<p>The <code>individual_id</code> is the identifier of an individual consists of the content address of some basic public data about the entity registering itself. The <code>individual_id</code> is immutable.</p> -<p>Any local will refer to another local by its <code>individual_id</code>.</p> -<p>A <code>user_id</code> is the content address of an <code>individual_id</code> together with the content address of some basic data describing the perception of the individual.</p> -<p>Locals describe stranger by a pair of <code>individual_id</code> and <code>user_id</code>. Let us call this <code>stranger_id</code>. The <code>stranger_id</code> is immutable.</p> -<p>The data represented by the <code>user_id</code> in a <code>stranger_id</code> is visible to locals, and may or may not be public.</p> -</div> -<div class="section" id="state-description"> -<h2>State description</h2> -<p>A user keeps a state towards other entities consisting of the following content-addresses, in the given order:</p> -<ol class="arabic simple"> -<li>Previous state</li> -<li>Userspec</li> -</ol> -<p>A local keeps a state towards its locals consisting of the following content-addresses, in the given order.</p> -<ol class="arabic simple"> -<li>Previous state</li> -<li>Userspec</li> -<li>Localslist</li> -<li>Locals state</li> -<li>Strangerslist</li> -<li>Strangers state</li> -</ol> -<p>The state is represented by the content address of the concatenation of these content-addresses, in the given order.</p> -<p>Any state change will be signed by the key resolvable through the <code>individual_id</code>.</p> -<div class="section" id="previous-state"> -<h3>Previous state</h3> -<p>The content address of the previous state. Will be <code>0</code> in the first state.</p> -</div> -<div class="section" id="userspec"> -<h3>Userspec</h3> -<p>The individual as it wants to be perceived in the current context. There is no guarantee that the individual will be viewed in this way by others.</p> -</div> -<div class="section" id="localslist"> -<h3>Localslist</h3> -<p>A lexiographically ordered array of <code>individual_id</code> that define the individuals in a group of locals.</p> -<p>All locals must keep this list in the same state.</p> -</div> -<div class="section" id="locals-state"> -<h3>Locals state</h3> -<p>A local's last recorded state of each of the locals, in the same order as <code>Localslist</code>.</p> -<p>The recorded state is the same content address as in the state description section above.</p> -<p>All locals must strive to keep this in the same state.</p> -</div> -<div class="section" id="strangerslist"> -<h3>Strangerslist</h3> -<p>A lexiographically ordered pointer array representing a local's opinion of which users that can communicate with locals. <a class="footnote-reference" href="#id7" id="id1">[1]</a></p> -<p>Combination of <code>stranger_id</code> and a <em>vote</em> on the specific user.</p> -<p>The vote can be in one of three states:</p> -<ul class="simple"> -<li>Value <code>0</code>, meaning local has acknowledged the user, explicitly or by following other locals. We call this state "deferred."</li> -<li>Value <code>-1</code>, meaning a local has vetoed the user. We call this state "vetoed."</li> -<li>Any other value, which is the content-address of the proof accepted by the local. <a class="footnote-reference" href="#id8" id="id2">[2]</a> We call this state "accepted."</li> -</ul> -<p>Typically, the initial vote value is <code>0</code>. The value may change at any time, but may never change back to <code>0</code> once set to another value.</p> -<p>A user that is part of the list, but not vetoed, is referred to as "acknowledged."</p> -<p>The Strangerslist state may be differ between locals, but they should strive to keep it the same.</p> -</div> -<div class="section" id="strangers-state"> -<h3>Strangers state</h3> -<p>The local's version of the current user state of the strangers, in the same order as the <code>Strangerlist</code>.</p> -<p>The Strangers state may differ between locals, but they should strive to keep it the same.</p> -</div> -</div> -<div class="section" id="behavior-of-locals"> -<h2>Behavior of locals</h2> -<p>A local may announce itself as <em>active</em> or <em>passive</em>. This should be part of the <code>Userspec</code> in the locals context.</p> -<p>Specifically, this affects the way it "votes" on strangers. A passive local:</p> -<ul class="simple"> -<li>Will always <em>veto</em> a user if <em>at least one active local</em> has vetoed it.</li> -<li>Will always <em>accept</em> a user if <em>all active locals</em> have accepted it. <a class="footnote-reference" href="#id9" id="id3">[3]</a></li> -</ul> -<p>Typically, a passive local will be a bot, for example an always-online node that guarantees availability of the locals so users can request state changes.</p> -<p>There must be at least two active locals in any group of locals.</p> -<div class="section" id="veto-dissent"> -<h3>Veto dissent</h3> -<p>If a local vetoes a stranger, then all locals should cut communication with that stranger.</p> -<p>Vetoes should be adopted by all locals.</p> -<p>If, however, other locals consider the veto to be issued in error, there are two possible outcomes:</p> -<ol class="arabic simple"> -<li>The vetoing local(s) abandon(s) the veto. <a class="footnote-reference" href="#id10" id="id4">[4]</a></li> -<li>The vetoing individual(s) is/are excluded by the locals.</li> -</ol> -</div> -</div> -<div class="section" id="network"> -<h2>Network</h2> -<p>Any entity needs to define and announce itself as a <code>peer</code> in order to be available on the network.</p> -<p>A "peer" is the coupling of network connection information to an <code>individual_id</code>.</p> -<div class="section" id="discoverability"> -<h3>Discoverability</h3> -<p>Peers are kept in a <a class="reference external" href="https://en.wikipedia.org/wiki/Distributed_hash_table">DHT</a>.</p> -<p>Peers must be discoverable by their locals affiliation. <a class="footnote-reference" href="#id11" id="id5">[5]</a></p> -</div> -<div class="section" id="users"> -<h3>Users</h3> -<p>Users open single-request connections to other users to annonunce, look up and prune peers.</p> -<p>Users may request state changes for themselves, or relay such requests for other users.</p> -</div> -<div class="section" id="locals"> -<h3>Locals</h3> -<p>Locals keep open connections (streams) with each other.</p> -<p>Locals continually exchange state changes.</p> -<p>Locals should be able to audit the chain of state changes of other locals.</p> -</div> -<div class="section" id="between-locals-and-users"> -<h3>Between locals and users</h3> -<p>Strangers <em>acknowledged</em> by locals open single-request connections to a local to announce state changes.</p> -<p>Locals may open single-request connections to a user to announce a change to the state of the <code>Userspec</code> of a stranger. <a class="footnote-reference" href="#id12" id="id6">[6]</a></p> -<p>Individuals are free to deny connections from non-locals, for example if the entity has been sending garbage requests. The implementation of such sanctions is deferred to the application layer.</p> -</div> -</div> -<div class="section" id="encryption"> -<h2>Encryption</h2> -<p>An individual may keep content totally private on the network.</p> -<p>A user may keep different encryption keys for different contexts.</p> -<p>If a stranger shares a key setup with a local, that setup will be shared among all locals. A local will expect to be able to communicate with a strangerr using this key setup.</p> -<p>The stranger must be able to change the key setup towards locals.</p> -<p>The locals may be able to change the key setup towards a stranger.</p> -<p>The protocol should allow anything from simple symmetric key exchanges to complex forward/backward secrecy schemes.</p> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id7" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Insert in this sorted list will face scaling issues when locals need to acknowledge a lot of users. The lists can be split into <a class="reference external" href="https://en.wikipedia.org/wiki/Bucket_%28computing%29">buckets</a>, either statically (for example two-byte prefixes) or dynamically after crossing a certain threshold. It should not be necessary to store empty buckets.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id8" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>This proof may map to what's commonly known as <em>kyc information</em>.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id9" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>Perhaps with the further limitation that all active users must have accepted with the <em>same</em> proof. Otherwise, how will the passive local choose which proof to accept.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id10" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>Observe that since the vote state cannot change back to <code>0</code>, "cancelling" the veto can only be done by adding proof.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id11" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td>What exactly uniquely identifies a group of locals on the network in such a way that it cannot be hi-jacked or spoofed still needs to be thought through. Perhaps it can be a combination of a specific name with a signature from the "founders;" the first (two or more) locals in it.</td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id12" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id6">[6]</a></td><td>A use-case here would be where the locals make a change for the user due to an out-of-band request from the user. This is <em>not</em> the <code>Strangerspec</code>; it is the user's current <code>Userspec</code> for the given context. Of course, it's up to the user to accept the change. It may in turn request a new state change to the locals to revert it.</td></tr> -</tbody> -</table> -</blockquote> -</div> - - </div><!-- /.entry-content --> - <footer class="meta"> - </footer> - -</section> - <footer id="contentinfo" class="body"> - <hr/> - <address id="about" class="vcard body"> - Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> - </address><!-- /#about --> - </footer><!-- /#contentinfo --> -</body> -</html> -\ No newline at end of file diff --git a/6cd7d22162d74eba04b00bdc004e18907c32ba245e0bf210ccb26e431789b18b.asc b/6cd7d22162d74eba04b00bdc004e18907c32ba245e0bf210ccb26e431789b18b.asc @@ -1,11 +0,0 @@ ------BEGIN PGP SIGNATURE----- - -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmHkZ0UACgkQzcvSTdHQ -4AHHzAf/WsYO1rqEMJwV+R2WmMd+ofNiRWVcUIzwzN544JRNn4tySyh7P6gX72u3 -6FDjt3SqMVdAc8Lg1oxGIIuQMVmUhNZl1tEZjJs2cog2n2waFxL0bJsr09bXWjwS -2Gt538pfZZ8quUckO4Y5Mzk3F3H6MHyoRgxnjuMuBr2tmv2euaqORl8MawL5T3o8 -mDMP0XxNfiqqZiVkGSRigxpicYUHqX42QsnuZoTditAAE1D/GQh6BiT36Ji9miLb -D1C2gFCorThSbs+syhzOODtb8M2M7IxzTwa4RLlkfTU4wvdn5l609UrJwS3HcQWD -6/qKhtWny8Mm01w+Lxc/eHT0aic4yQ== -=Zgwg ------END PGP SIGNATURE----- diff --git a/788f1df1d56d3aeb5a7374de267f9ebd41865fefabacd46810270f1596856a71 b/788f1df1d56d3aeb5a7374de267f9ebd41865fefabacd46810270f1596856a71 @@ -0,0 +1,249 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - Homemade internet state monitor</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="./category/archiving.html">Archiving</a></li> + <li><a href="./category/code.html">Code</a></li> + <li class="active"><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./internet-up-monitor.html" rel="bookmark" + title="Permalink to Homemade internet state monitor">Homemade internet state monitor</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2021-07-01T13:59:00+02:00"> + Thu 01 July 2021 + </time> +in <a class="category" href="./category/hygiene.html">hygiene</a> + <a href="./tag/bash.html">bash</a> + <a href="./tag/systemd.html">systemd</a> + <a href="./tag/network.html">network</a> + <a href="./tag/i3wm.html">i3wm</a> + <a href="./tag/tmpfs.html">tmpfs</a> + <a href="./tag/openvpn.html">openvpn</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./eth-log-bloom.html"> + Making sense of Ethereum log bloom filters + </a> + </li> + | + + <li> + Next: <a href="./backup-rsync-duplicity.html"> + Combining duplicity and rsync + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>I use <a class="reference external" href="https://i3wm.org">i3wm</a>. I use it because it has a very low resource footprint, it has great documentation, and it's simple to configure.</p> +<p>Complementing <em>i3wm</em> is the <em>i3status</em> status provider to the <em>i3bar</em>. It provides a good suite of preset components for things like temperatur measurement, disk capacity. It also provides a couple of generic ones, namely "output the contents of a file" or "yes if file exists, no if not."</p> +<div class="section" id="it-s-not-you-it-s-me"> +<h2>It's not you, it's me?</h2> +<div class="admonition warning"> +<p class="first admonition-title">Warning</p> +<p class="last"><strong>Always use a VPN! Always!!</strong></p> +</div> +<p>Now, I <em>always</em> use VPN <em>no matter what</em>, and I configure network manually. That is, I do not use a network manager. For this reason, when network applications start misbehaving, it's sometimes hard to tell whether it is because the host is gone, the local network is flaky, the internet route is flaky, or simply that the connection is gone for good.</p> +<p>Evidently, it would be practical to have <strong>an item in the status bar telling me whether I have access to internet right now or not</strong>.</p> +<p>The "yes if file exists, no if not" option seems to be the right one to go for. Basically, we need to stick an empty file in a runtime file store whenever we detect that internet is available.</p> +</div> +<div class="section" id="ping-rinse-repeat"> +<h2>Ping, rinse, repeat</h2> +<p>There's not much trickery involved here. The check is a ping to a remote location every now and then.</p> +<p>Some caveats to cover to grant it some minimum of intelligence:</p> +<ul class="simple"> +<li>We need a few alternative destinations for the ping, that can take over if others happen to be down or take too long to repond.</li> +<li>To make the script useful for general purpose use, we'd also want to be able to supply a custom list of hosts to ping.</li> +<li>We also want to make sure that when the script is interrupted, it deletes the internet indicator file. That way, when it stops running, the internet indicator will be frozen in the "no" state.</li> +<li>Lastly, it can be useful to log to syslog every time the state of the internet connection (or the script itself) changes.</li> +</ul> +<p>That can translate to a shell script like this;</p> +<pre class="code bash literal-block"> +<span class="ch">#!/bin/bash +</span> +<span class="c1"># this is our runtime file, if - which exists - tells us internet is up +</span><span class="nv">f</span><span class="o">=</span><span class="s2">"/run/user/</span><span class="nv">$UID</span><span class="s2">/probe_up"</span> +rm -f <span class="nv">$f</span> + +up<span class="o">()</span> <span class="o">{</span> + <span class="k">if</span> <span class="o">[</span> ! -f <span class="nv">$f</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + logger -p info <span class="s2">"iup up"</span> + <span class="k">fi</span> + touch <span class="nv">$f</span> +<span class="o">}</span> +down<span class="o">()</span> <span class="o">{</span> + <span class="k">if</span> <span class="o">[</span> -f <span class="nv">$f</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + logger -p info <span class="s2">"iup down"</span> + <span class="k">fi</span> + rm -f <span class="nv">$f</span> +<span class="o">}</span> + +argh<span class="o">()</span> <span class="o">{</span> + logger -p info <span class="s2">"iup die"</span> + rm -f <span class="nv">$f</span> + <span class="nb">exit</span> <span class="m">0</span> +<span class="o">}</span> + +<span class="nv">IUP_DELAY</span><span class="o">=</span><span class="si">${</span><span class="nv">IUP_DELAY</span><span class="k">:-</span><span class="nv">$1</span><span class="si">}</span> +<span class="k">if</span> <span class="o">[</span> -z <span class="nv">$IUP_DELAY</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + <span class="nv">IUP_DELAY</span><span class="o">=</span><span class="m">5</span> +<span class="k">fi</span> +logger -p debug iup started with delay <span class="nv">$IUP_DELAY</span> + +<span class="nv">hosts</span><span class="o">=()</span> +<span class="k">while</span> <span class="nb">read</span> h<span class="p">;</span> <span class="k">do</span> + <span class="nv">hosts</span><span class="o">+=(</span><span class="nv">$h</span><span class="o">)</span> +<span class="k">done</span> < <span class="nv">$HOME</span>/.config/iup <span class="m">2</span>> /dev/null +<span class="k">if</span> <span class="o">[</span> <span class="si">${#</span><span class="nv">hosts</span><span class="p">[@]</span><span class="si">}</span> -eq <span class="s2">"0"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + <span class="nv">hosts</span><span class="o">=(</span><span class="m">8</span>.8.8.8 <span class="m">151</span>.101.193.67<span class="o">)</span> <span class="c1"># google nameserver and cnn.com +</span> logger -p warn missing hosts input file, using evil default: <span class="si">${</span><span class="nv">hosts</span><span class="p">[@]</span><span class="si">}</span> +<span class="k">fi</span> + +<span class="nb">trap</span> argh SIGINT +<span class="nb">trap</span> argh SIGTERM +<span class="nb">trap</span> argh SIGQUIT + +<span class="k">while</span> true<span class="p">;</span> <span class="k">do</span> + <span class="nv">isup</span><span class="o">=</span><span class="s2">""</span> + <span class="k">for</span> h <span class="k">in</span> <span class="si">${</span><span class="nv">hosts</span><span class="p">[@]</span><span class="si">}</span><span class="p">;</span> <span class="k">do</span> + ping -c1 <span class="nv">$h</span> -w1 -q <span class="p">&</span>> /dev/null + <span class="k">if</span> <span class="o">[</span> <span class="nv">$?</span> -eq <span class="s1">'0'</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + up + <span class="nv">isup</span><span class="o">=</span><span class="m">1</span> + <span class="c1">#logger -p debug "ping success $h" +</span> <span class="nb">break</span> + <span class="k">fi</span> + <span class="k">done</span> + <span class="k">if</span> <span class="o">[</span> -z <span class="nv">$isup</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + down + <span class="k">fi</span> + sleep <span class="nv">$IUP_DELAY</span> +<span class="k">done</span> +</pre> +<p>Since we'll want this to run when network interfaces allegedly are up, it makes sense to link it to systemd and the network target:</p> +<pre class="code ini literal-block"> +<span class="k">[Unit]</span><span class="w"> +</span><span class="na">Description</span><span class="o">=</span><span class="s">Internet connection poller</span><span class="w"> +</span><span class="na">After</span><span class="o">=</span><span class="s">multi-user.target</span><span class="w"> + +</span><span class="k">[Service]</span><span class="w"> +</span><span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/iup.sh</span><span class="w"> +</span><span class="na">Restart</span><span class="o">=</span><span class="s">always</span><span class="w"> + +</span><span class="k">[Install]</span><span class="w"> +</span><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span> +</pre> +<p>Stick this in <code>~/.config/systemd/user/iup.service</code> and enable and start the service:</p> +<div class="highlight"><pre><span></span>$ systemctl --user <span class="nb">enable</span> iup.service +$ systemctl --user start iup.service +</pre></div> +</div> +<div class="section" id="checking-what-status-our-status-is-in"> +<h2>Checking what status our status is in</h2> +<p>Now, in the i3status configuration, assuming that your system uid is 1000 <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>, add:</p> +<div class="highlight"><pre><span></span>order += "run_watch INET" + +kj + +run_watch INET { + pidfile = "/run/user/1000/probe_up" +} +</pre></div> +<p>To reload the config on the fly, press <code>mod+r</code> (that's <code>ctrl+r</code> on mine). The result should be a new item in your bar showing <code>INET: yes</code> (most likely yes, anyway, since you're currently reading this).</p> +</div> +<div class="section" id="safety-first-eh-second"> +<h2>Safety first, eh ... second</h2> +<div class="admonition warning"> +<p class="first admonition-title">Warning</p> +<p class="last"><strong>Did I mention that you should always use a VPN?</strong></p> +</div> +<p>The same <code>run_watch</code> trick can be used for VPN.</p> +<p>I use <code>openvpn</code>, and it defines a flag <code>--writepid</code>. If you pass a file path to this parameter, either through the <code>writepid</code> config directiry or on the <code>--writepid</code> flag on the command line, it will write the openvpn pid to that file, and similarly remove it when the openvpn service ends.</p> +<p>However, this raises another issue. Namely the fact that a location is needed for openvpn to write the file to, for which it also has access.</p> +<p>If we want to use the <code>/run</code> directory again, now we also has to make sure that this directory exists.</p> +<div class="section" id="got-the-runs"> +<h3>Got the runs?</h3> +<p>This is exactly what <code>systemd-tmpfiles</code> is for. You can add files to <code>/etc/tmpfiles.d</code> which describe what kind of temporary files or directories to add <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>.</p> +<p>Create a file <code>/etc/tmpfiles.d/openvpn.conf</code> and add a single line to it:</p> +<div class="highlight"><pre><span></span>d /run/openvpn 0755 openvpn openvpn +</pre></div> +<p>This will ensure that the folder is created at startup.</p> +<p>Now, to create one right away for the current session, run <code>systemd-tmpfiles --create</code>.</p> +<p>Now we have everything we need to add the VPN status to <em>i3status</em> aswell. Provided that we chose <code>/run/openvpn/openvpn.pid</code> as our argument to <code>openvpn --writepid</code>, the setup is more or less the same as before:</p> +<div class="highlight"><pre><span></span>order += "run_watch VPN" + +kj + +run_watch VPN { + pidfile = "/run/openvpn/openvpn.pid" +} +</pre></div> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>The <code>/run/user/$UID</code> folder will be created on system startup, and will be writable by that user. It provides a tmpfs storage location for processes that run in early stages of boot, before the entire filesystem tree (which may include <code>/var/run</code> because <code>/var</code> may be on a separate partition) has been mounted. More in that here: <a class="reference external" href="https://lwn.net/Articles/436012/">https://lwn.net/Articles/436012/</a></td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>This service has an impressive amount of flexibility, which you can inspect yourself by visiting the <code>tmpfiles.d</code> (and <code>systemd-tmpfiles</code>) manpage.</td></tr> +</tbody> +</table> +</blockquote> +</div> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/788f1df1d56d3aeb5a7374de267f9ebd41865fefabacd46810270f1596856a71.asc b/788f1df1d56d3aeb5a7374de267f9ebd41865fefabacd46810270f1596856a71.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NzsACgkQzcvSTdHQ +4AGi7QgAgf5uOFOstKB1yKJ1UfV88uDzuhctz62FcxQ/rt6dEcjEhU3ETebqlp+G +3QtkKhUuQOLtBpyltdyqogbcN/AgTYht1GaGaYrlI2C8pqlhmfF4PZQ/Y72okV8K +AzPe/Q5i09Qgw2ccOU7iCKsZ9NO4gZ/GVMjjOBEjPSTWAUW124MPTP382qCyQkc7 +SgrbjV0Fmyfph4KNnEcjt3Gj/5Ua0JFlrLgEhIy8DG0/RIaXAi4NnY6ZivYDC2p4 +zWftPI3NWYbcvbWoRk0TzD1uGQxzBTp7ZoVZGVskoKttjE02dddwhTVVA4qPWz8n +s1j2cBGLV0TVpcWithHMFbBmjxgl2g== +=FUj+ +-----END PGP SIGNATURE----- diff --git a/80d5137e80648103735073b716f744b6e2210092a4c2ea556d87a931a3878c53 b/80d5137e80648103735073b716f744b6e2210092a4c2ea556d87a931a3878c53 @@ -0,0 +1,176 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - The routing to freedom</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="./category/archiving.html">Archiving</a></li> + <li><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li class="active"><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./docker-offline-1-routing.html" rel="bookmark" + title="Permalink to The routing to freedom">Offline Docker: The routing to freedom</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2021-04-26T07:54:00+02:00"> + Mon 26 April 2021 + </time> +in <a class="category" href="./category/offlining.html">offlining</a> + <a href="./tag/docker.html">docker</a> + <a href="./tag/networking.html">networking</a> + <a href="./tag/iptables.html">iptables</a> + <a href="./tag/iproute.html">iproute</a> + </div> + + <div class="neighbors meta"> + <ul> + + <li> + Next: <a href="./docker-offline-2-python.html"> + Local python repository + </a> + </li> + </ul> + </div> + <div class="entry-series meta"> +Part 1 from the series "Offline Docker" + <ol class="entry-series-parts"> + <li>| <a href="docker-offline-2-python.html" title="Local python repository ">Part 2</a></li> + <li>| <a href="docker-offline-3-npm.html" title="Local npm repository ">Part 3</a></li> + </ol> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <!-- .. CAUTION:: + + This is a purely technical article. You should probably be `this geeky <https://g33k.holbrook.no/8319a926>`_ to continue reading. --> +<p>Five years ago I decided that I wanted to be able to work from anywhere, anytime. Four years ago, that promise was kept. In part.</p> +<p>I do not need to go to an office somewhere. I can work outside in a park if I want to. I can ride on to a new town every day. I only ever need to bring my trusty old <a class="reference external" href="https://www.tuxedocomputers.com/en">Tuxedo Laptop</a> whereever I go.</p> +<p>All of this as true, as long as there is internet available. And, as it turns out, <em>good</em> internet available.</p> +<p>This has become especially obvious to me once I started to work with a project that involves a collection of microservices contained in a Docker environment, which also makes extensive use of custom packages that change frequently alongside the development process. Turns out, every time I want to rebuild my cluster of containers when sitting in the sun in a park, I need my LTE modem to play along. If it doesn't, then a single package that can't be reached will thwart the build.</p> +<p>This does not feel much like freedom after all. So let's see how we can serve all of these locally from our host instead.</p> +<p>First of all, we have to be able to reach our local host from the Docker containers. This is less straightforward than it may seem at first. The most obvious solution is to use the <tt class="docutils literal">host</tt> network driver, but this exposes your <em>whole</em> localhost interface and routes to internet, too. Aside from the security issues that raises, it can also also trick you into assuming that some resources are available when they in fact will not be when you move on to a different environment. What we want is to <em>block</em> access to internet, while <em>choosing</em> which services to let the Docker container use.</p> +<p>Once we have this in place, we want to create local repositories for all the stuff we otherwise need to download. In this particular case, that means a <strong>Docker</strong> repository, a <strong>Python</strong> repository, a <strong>nodejs</strong> repository and a <strong>linux</strong> repository. We'll use <a class="reference external" href="https://archlinux.org">Archlinux</a> for this exercise, because that's been my home environment for the last four years.</p> +<p>In fact, having your own mirror of all these and anything else you base most of your work on is not only a good idea for the purpose of <em>offlining</em> in itself, but wasting on bandwidth for items you've already downloaded hundreds of times is not exactly a nod to climate awareness either. And even more importantly, ensuring <em>availability</em> of software is something we should all participate in, and not merely defer to a couple of git repository giants.</p> +<blockquote> +</blockquote> +<!-- --> +<blockquote> +</blockquote> +<div class="section" id="reaching-the-local-host"> +<h2>Reaching the local host</h2> +<p><em>Local host</em> not <em>localhost</em>, mind you. Which means we need a different interface to connect to. And since we are not wiring up in any physical sense, a virtual interface seems to be the reasonable way to go.</p> +<p>First, let's prepare a base Docker layer with some tools that you should never leave home without.</p> +<div class="section" id="prepare-the-docker-image"> +<h3>Prepare the docker image</h3> +<pre class="code docker literal-block"> +<span class="k">FROM</span><span class="w"> </span><span class="s">archlinux:latest</span> + +<span class="k">RUN</span><span class="w"> </span>pacman -Sy <span class="o">&&</span> <span class="se">\ +</span> pacman -S --noconfirm gnu-netcat socat inetutils iproute2 +</pre> +<p>Let's build this as an image called <tt class="docutils literal">archbase</tt>. Provided the content above is a file called <tt class="docutils literal">Dockerfile.archbase</tt> in your current directory:</p> +<div class="highlight"><pre><span></span>$ docker build -t archbase -f Dockerfile.archbase . +</pre></div> +</div> +<div class="section" id="set-up-network-interfaces"> +<h3>Set up network interfaces</h3> +<p>Bring up a virtual interface</p> +<div class="highlight"><pre><span></span>$ ip link add foo <span class="nb">type</span> dummy +</pre></div> +<p>Find the subnet of the <tt class="docutils literal"><span class="pre">no-internet</span></tt> Docker network. This network driver is a builtin that provides exactly what it advertises.</p> +<div class="highlight"><pre><span></span>$ docker network inspect no-internet +<span class="o">[</span>...<span class="o">]</span> +<span class="s2">"Config"</span>: <span class="o">[</span> + <span class="o">{</span> + <span class="s2">"Subnet"</span>: <span class="s2">"10.1.1.0/24"</span>, + <span class="s2">"Gateway"</span>: <span class="s2">"10.1.1.1"</span> + <span class="o">}</span> +<span class="o">]</span> +</pre></div> +<p>Assign an IP address to the dummy interface in a <em>different</em> subnet than the one the Docker network uses.</p> +<div class="highlight"><pre><span></span>$ ip addr add <span class="m">10</span>.1.2.1/24 dev foo +</pre></div> +</div> +<div class="section" id="traverse-the-firewall"> +<h3>Traverse the firewall</h3> +<p>Find the bridge used by the Docker container. Look for an ip address that matches the gateway of the docker network config.</p> +<div class="highlight"><pre><span></span>$ ip addr ls +<span class="m">17</span>: br-d4ddb68f9938: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu <span class="m">1500</span> qdisc noqueue state UP group default +link/ether <span class="m">02</span>:42:9c:1a:58:d2 brd ff:ff:ff:ff:ff:ff +inet <span class="m">10</span>.1.1.1/24 brd <span class="m">10</span>.1.1.255 scope global br-d4ddb68f9938 +valid_lft forever preferred_lft forever +inet6 fe80::42:9cff:fe1a:58d2/64 scope link +valid_lft forever preferred_lft forever +<span class="o">[</span>...<span class="o">]</span> +<span class="m">7614</span>: foo: <BROADCAST,NOARP,UP,LOWER_UP> mtu <span class="m">1500</span> qdisc noqueue master br-496e528b928c state UNKNOWN group default qlen <span class="m">1000</span> +link/ether 1a:50:53:2b:96:98 brd ff:ff:ff:ff:ff:ff +inet <span class="m">10</span>.1.3.1/24 scope global foo +valid_lft forever preferred_lft forever +inet6 fe80::1850:53ff:fe2b:9698/64 scope link +valid_lft forever preferred_lft forever +</pre></div> +<p>Add the virtual interface to the bridge.</p> +<div class="highlight"><pre><span></span>$ ip link <span class="nb">set</span> foo master br-d4ddb68f9938 +</pre></div> +<p>Long story short, the previous step will make the traffic from the container reach the <tt class="docutils literal">INPUT</tt> chain in <tt class="docutils literal">iptables</tt>. Now we can make an exception for incoming traffic from the <tt class="docutils literal"><span class="pre">no-internet</span></tt> Docker bridge.</p> +<div class="highlight"><pre><span></span>$ iptables -I INPUT <span class="m">1</span> --source br-d4ddb68f9938 --destination <span class="m">10</span>.1.2.1/24 -j ACCEPT +</pre></div> +</div> +<div class="section" id="verify"> +<h3>Verify</h3> +<p>Provided you don't have any other hurdles in your local <tt class="docutils literal">ìptables</tt> setup, a port on device <tt class="docutils literal">foo</tt> should be reachable from the docker container. We can use socat to check.</p> +<p>On local host:</p> +<div class="highlight"><pre><span></span>$ socat TCP4-LISTEN:8000,bind<span class="o">=</span><span class="m">10</span>.1.2.1,reuseaddr - +</pre></div> +<p>Start the docker container with shell prompt</p> +<div class="highlight"><pre><span></span>$ docker run --network no-internet -it archbase /bin/bash +</pre></div> +<p>The moment of truth</p> +<div class="highlight"><pre><span></span>$ <span class="nb">echo</span> bar <span class="p">|</span> socat - TCP4:10.1.2.1:8000 +</pre></div> +<p>Spoiler: <tt class="docutils literal">bar</tt> should pop up on the local host side.</p> +</div> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/80d5137e80648103735073b716f744b6e2210092a4c2ea556d87a931a3878c53.asc b/80d5137e80648103735073b716f744b6e2210092a4c2ea556d87a931a3878c53.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NzwACgkQzcvSTdHQ +4AGsTgf/eLMQ0jVh7sf+WSUFvNLE/xue01c5D9aegvk1Jb+oQ+EB4PJAEnTvDdJ4 +agTiWv4wpVP0Ktmw7DFDJo2tjVakjuRvLLs59WHsGlGsWAvCGPi1T9oVn6SBcnGy +ntM0jMPlLj/6rRFyhE055DlX5Zzy0eDceAuFWkz/yObewGJ7kxPgFkcvQPqklF6c +uvElCtp9DCe7GtEt3wQcdW/s3e1ZEiJw6p7SIV2H1wTwilR7akEA4CILrwEGVta/ +DxAX5hLsErx9U4uJu/oLZYjxUSzswAoUZZs3zh0yCmYKQZFhmMVyGLMrkaGrwh+8 +q7ceklr/bUV8Ic134VODkzn1HmNlNg== +=Cy6B +-----END PGP SIGNATURE----- diff --git a/92eb83e7074b2760e6001d40b17eecffb874b23f036abd5b6d2328f28f49bac8 b/92eb83e7074b2760e6001d40b17eecffb874b23f036abd5b6d2328f28f49bac8 @@ -0,0 +1,331 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - Local npm repository</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="./category/archiving.html">Archiving</a></li> + <li><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li class="active"><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./docker-offline-3-npm.html" rel="bookmark" + title="Permalink to Local npm repository">Offline Docker: Local npm repository</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2021-05-25T11:47:00+02:00"> + Tue 25 May 2021 + </time> +in <a class="category" href="./category/offlining.html">offlining</a> + <a href="./tag/docker.html">docker</a> + <a href="./tag/npm.html">npm</a> + <a href="./tag/nodejs.html">nodejs</a> + <a href="./tag/javascript.html">javascript</a> + <a href="./tag/devops.html">devops</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./village-network-protocol.html"> + A village network protocol + </a> + </li> + | + + <li> + Next: <a href="./eth-log-bloom.html"> + Making sense of Ethereum log bloom filters + </a> + </li> + </ul> + </div> + <div class="entry-series meta"> +Part 3 from the series "Offline Docker" + <ol class="entry-series-parts"> + <li>| <a href="docker-offline-1-routing.html" title="The routing to freedom ">Part 1</a></li> + <li>| <a href="docker-offline-2-python.html" title="Local python repository ">Part 2</a></li> + </ol> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>As expected, serving a local <tt class="docutils literal">npm</tt> repository is a <em>lot</em> less straightforward than that for <tt class="docutils literal">pip</tt> and <tt class="docutils literal">python</tt>.</p> +<p>The npm registry uses the <code>CouchDB</code> nosql server as backend to resolve dependencies and serve resource metadata. The <code>npm</code> CLI tool expects a corresponding json response to requests that cite the <em>name</em> (and not the path) of the package.</p> +<p>Let's ask the <a class="reference external" href="https://registry.npmjs.org">registry</a> <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a> for the package <tt class="docutils literal">ftp</tt>, and have a look at the response (excerpt):</p> +<div class="highlight"><pre><span></span><span class="gp">$ </span>curl -X GET https://registry.npmjs.org/ftp <span class="p">|</span> jq +<span class="go">{</span> +<span class="go"> "_id": "ftp",</span> +<span class="go"> "_rev": "113-89fe76508a7ece41b4c9a157114f966f",</span> +<span class="go"> "name": "ftp",</span> +<span class="go"> "description": "An FTP client module for node.js",</span> +<span class="go"> "dist-tags": {</span> +<span class="go"> "latest": "0.3.10"</span> +<span class="go"> },</span> +<span class="go"> "versions": {</span> + + + +<span class="go"> "0.3.10": {</span> +<span class="go"> "name": "ftp",</span> +<span class="go"> "version": "0.3.10",</span> +<span class="go"> "author": {</span> +<span class="go"> "name": "Brian White",</span> +<span class="go"> "email": "mscdex@mscdex.net"</span> +<span class="go"> },</span> +<span class="go"> "description": "An FTP client module for node.js",</span> +<span class="go"> "main": "./lib/connection",</span> +<span class="go"> "engines": {</span> +<span class="go"> "node": ">=0.8.0"</span> +<span class="go"> },</span> +<span class="go"> "dependencies": {</span> +<span class="go"> "xregexp": "2.0.0",</span> +<span class="go"> "readable-stream": "1.1.x"</span> +<span class="go"> },</span> +<span class="go"> "scripts": {</span> +<span class="go"> "test": "node test/test.js"</span> +<span class="go"> },</span> +<span class="go"> "keywords": [</span> +<span class="go"> "ftp",</span> +<span class="go"> "client",</span> +<span class="go"> "transfer"</span> +<span class="go"> ],</span> +<span class="go"> "licenses": [</span> +<span class="go"> {</span> +<span class="go"> "type": "MIT",</span> +<span class="go"> "url": "http://github.com/mscdex/node-ftp/raw/master/LICENSE"</span> +<span class="go"> }</span> +<span class="go"> ],</span> +<span class="go"> "repository": {</span> +<span class="go"> "type": "git",</span> +<span class="go"> "url": "http://github.com/mscdex/node-ftp.git"</span> +<span class="go"> },</span> +<span class="go"> "bugs": {</span> +<span class="go"> "url": "https://github.com/mscdex/node-ftp/issues"</span> +<span class="go"> },</span> +<span class="go"> "homepage": "https://github.com/mscdex/node-ftp",</span> +<span class="go"> "_id": "ftp@0.3.10",</span> +<span class="go"> "_shasum": "9197d861ad8142f3e63d5a83bfe4c59f7330885d",</span> +<span class="go"> "_from": "https://github.com/mscdex/node-ftp/tarball/v0.3.10",</span> +<span class="go"> "_resolved": "https://github.com/mscdex/node-ftp/tarball/v0.3.10",</span> +<span class="go"> "_npmVersion": "1.4.28",</span> +<span class="go"> "_npmUser": {</span> +<span class="go"> "name": "mscdex",</span> +<span class="go"> "email": "mscdex@mscdex.net"</span> +<span class="go"> },</span> +<span class="go"> "maintainers": [</span> +<span class="go"> {</span> +<span class="go"> "name": "mscdex",</span> +<span class="go"> "email": "mscdex@mscdex.net"</span> +<span class="go"> }</span> +<span class="go"> ],</span> +<span class="go"> "dist": {</span> +<span class="go"> "shasum": "9197d861ad8142f3e63d5a83bfe4c59f7330885d",</span> +<span class="go"> "tarball": "https://registry.npmjs.org/ftp/-/ftp-0.3.10.tgz"</span> +<span class="go"> },</span> +<span class="go"> "directories": {}</span> +<span class="go"> }</span> +<span class="go"> },</span> +</pre></div> +<p>That looks a lot like embellished contents of the basic package.json set up by <code>npm init</code>, except it's wrapped in a versions array. It also explicitly defines an <em>absolute</em> url to a <cite>tarball</cite> under the <cite>dist</cite> key. <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a></p> +<div class="section" id="how-low-can-you-go"> +<h2>How low can you go</h2> +<p>It seems that all that's needed is a json index page served for the package name subpath of repository path, together with the tarball itself. We check this by putting together an utterly pointless package <tt class="docutils literal">foobarbarbar</tt>:</p> +<p><strong>foobarbarbar/index.js</strong></p> +<pre class="code javascript literal-block" id="foobarbarbar-index-js"> +<span class="nx">module</span><span class="p">.</span><span class="nx">exports</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span><span class="s1">'smth'</span><span class="o">:</span><span class="w"> </span><span class="nx">smth</span><span class="p">,</span><span class="w"> +</span><span class="p">}</span><span class="w"> + +</span><span class="kd">function</span><span class="w"> </span><span class="nx">smth</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'foo'</span><span class="p">);</span><span class="w"> +</span><span class="p">}</span> +</pre> +<p><strong>foobarbarbar/package.json</strong></p> +<pre class="code json literal-block" id="foobarbarbar-package-json"> +<span class="p">{</span><span class="w"> + </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"foobarbarbar"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0.0"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo repo"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"main"</span><span class="p">:</span><span class="w"> </span><span class="s2">"index.js"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"author"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo Bar"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"license"</span><span class="p">:</span><span class="w"> </span><span class="s2">"GPL3"</span><span class="p">,</span><span class="w"> +</span><span class="p">}</span> +</pre> +<p>Then make a tarball from those two files named <tt class="docutils literal"><span class="pre">foobarbarbar-1.0.0.tgz</span></tt>, take the sha1 sum of it,</p> +<p>Remembering our <a class="reference external" href="./docker-offline-2-python.html">virtual interface setup from the pip example</a> we stick it behind our webserver (here with npm sub-path) and add our minimal version json wrapper:</p> +<p><strong>package.json</strong></p> +<pre class="code json literal-block" id="package-json"> +<span class="p">{</span><span class="w"> + </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"foobarbarbar"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"versions"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span><span class="nt">"1.0.0"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"foobarbarbar"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0.0"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo repo"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"main"</span><span class="p">:</span><span class="w"> </span><span class="s2">"index.js"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"author"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Foo Bar"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"license"</span><span class="p">:</span><span class="w"> </span><span class="s2">"GPL3"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"dist"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> + </span><span class="nt">"shasum"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2ccd68498ef5f2bfa00f0e1e59f44686fdb296ee"</span><span class="p">,</span><span class="w"> + </span><span class="nt">"tarball"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://10.1.2.1/npm/foobarbarbar-1.0.0.tgz"</span><span class="w"> + </span><span class="p">}</span><span class="w"> + + </span><span class="p">}</span><span class="w"> + </span><span class="p">}</span><span class="w"> +</span><span class="p">}</span> +</pre> +</div> +<div class="section" id="making-introductions"> +<h2>Making introductions</h2> +<p>The central trick here is to serve a json document as the "directory index" of the HTTP server. It's useful to remind ourselves at this point that we are <em>not</em> setting up a <em>registry</em> but a <em>repository</em> that contains provisions for a <em>locked</em> or <em>frozen</em> dependency graph. The former would surely need some of the <tt class="docutils literal">CouchDB</tt> magic to resolve dependencies. Our assumption is that the latter can theoretically be realized using static files.</p> +<p>In other words; just as with the previous <a class="reference external" href="./docker-offline-2-python.html">python repository example</a>, we don't try to handle dependency resolutions for pip, but merely serve the actual package files after dependences have been resolved.</p> +<p>With Apache Web Server, using the <tt class="docutils literal">package.json</tt> as the directory index is as easy as:</p> +<div class="highlight"><pre><span></span><Directory "/srv/http/npm"> + DirectoryIndex package.json +</Directory> +</pre></div> +<p>Of course adjusting the <cite>Directory</cite> path as needed to match the local setup.</p> +<p>Make sure that the tarball and the latter <cite>package.json</cite> can be found in the <cite>foobarbarbar</cite> virtual subfolder of the above <cite>Directory</cite> directive path:</p> +<div class="highlight"><pre><span></span><span class="gp">$ </span>ls /srv/http/npm/foobarbarbar/ +<span class="go">-rw-r--r-- 1 root root 351 May 24 18:38 foobarbarbar-1.0.0.tgz</span> +<span class="go">-rw-r--r-- 1 root root 380 May 25 09:36 package.json</span> +</pre></div> +<p>Set the registry entry in your <tt class="docutils literal"><span class="pre">~./npmrc</span></tt> as follows:</p> +<div class="highlight"><pre><span></span><span class="na">registry</span><span class="o">=</span><span class="s">http://10.1.2.1/npm</span><span class="w"></span> +</pre></div> +<p>Then restart the Apache server and give it a go:</p> +<div class="highlight"><pre><span></span><span class="gp">$ </span>npm install --verbose foobarbarbar +<span class="go">npm verb cli [ '/usr/bin/node', '/usr/bin/npm', 'install', '--verbose', 'foobarbarbar' ]</span> +<span class="go">npm info using npm@7.13.0</span> +<span class="go">npm info using node@v16.1.0</span> +<span class="go">[...]</span> +<span class="go">npm http fetch GET 200 http://10.1.2.1/npm/foobarbarbar/ 6ms</span> +<span class="go">[...]</span> +<span class="go">npm http fetch GET 200 http://10.1.2.1/npm/foobarbarbar/foobarbarbar-1.0.0.tgz 25ms</span> +<span class="go">[...]</span> +<span class="go">added 2 packages in 545ms</span> +<span class="go">npm timing command:install Completed in 70ms</span> +<span class="go">[...]</span> +</pre></div> +</div> +<div class="section" id="cache-dodging"> +<h2>Cache dodging</h2> +<p>Seriously, <tt class="docutils literal">npm <span class="pre">--help</span></tt> leaves a lot to be desired. However, the online <tt class="docutils literal">npm install</tt> documentation does not yield any more clues as to whether it gives you an option of ignoring local cache like you can with <tt class="docutils literal">pip <span class="pre">--no-cache</span></tt>. This is a major pain in the ass, as you have to remember to keep your <tt class="docutils literal"><span class="pre">~/.npm</span></tt> folder clean between each install attempt. Otherwise, changes you make to the repository won't be used by <tt class="docutils literal">npm install</tt>.</p> +<p>While testing, it pays to use a folder fairly close to the fs root, like a subfolder in <cite>tmp</cite>. That way, you won't be thrown off by some stray <cite>node_modules</cite> folder somewhere down the tree.</p> +</div> +<div class="section" id="think-locally-act-locally"> +<h2>Think locally, act locally</h2> +<p>Serving packages globally apparently comes down to this:</p> +<ol class="arabic simple"> +<li>Pull the <tt class="docutils literal">package.json</tt> versions list from a proper registry.</li> +<li>Get the tarball</li> +<li>Transform the absolute package url to our host instead. (sigh)</li> +<li>(optional) Prune all the versions and metadata which is not needed (anything that's not in our minimal <tt class="docutils literal">foobarbarbar</tt> example above).</li> +</ol> +<p>Obviously, this is an annoyingly large amount of work to code up parsing and retrieval for.</p> +<p>Incidentally, there is a very nice tool called <a class="reference external" href="https://verdaccio.org">verdaccio</a> which gets us most of the way there. <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a> Its <em>storage</em> directory <a class="footnote-reference" href="#footnote-4" id="footnote-reference-4">[4]</a> uses exactly the directory structure we need. After retrieving a package collection using it as a proxy, getting the files in place is merely a case of copying <a class="footnote-reference" href="#footnote-5" id="footnote-reference-5">[5]</a> the files from the <em>storage</em> directory to the corresponding webserver directory.</p> +<p>Looking at the <tt class="docutils literal">package.json</tt> versions wrapper saved by <tt class="docutils literal">verdaccio</tt>, however, we see that it still preserves the absolute path of the upstream registry. Sadly we are still left with doing the third task ourselves.</p> +<p>As parsing json is one of my least favorite things in the world, I won't include the code for this last step here. For now it's sufficient that we understand the minimums required for manually setting up and serving a static, local, offline npm repository. Regardless of the pain involved.</p> +</div> +<div class="section" id="making-it-personal"> +<h2>Making it personal</h2> +<p>Whether we massage jsons ourselves or lazily resort to <tt class="docutils literal">verdaccio</tt>, the final step we need to take is the same: Setting the registry url in the npm configuration of the docker image.</p> +<p>This is merely a case of setting the registry url in the <a class="reference external" href="https://docs.npmjs.com/cli/v6/configuring-npm/npmrc#files">npmrc</a> inside the container.</p> +<div class="highlight"><pre><span></span><span class="gp">$ </span>docker run -it <span class="o">[</span>...<span class="o">]</span> npm config ls -l <span class="p">|</span> grep etc/npmrc +<span class="go">globalconfig = "/usr/etc/npmrc"</span> +</pre></div> +<p>To get at our manually provided <tt class="docutils literal">foobarbarbar</tt> package from before, the dockerfile will be (using archlinux):</p> +<div class="highlight"><pre><span></span><span class="o">[</span>...<span class="o">]</span> +<span class="k">RUN</span><span class="w"> </span>pacman -S nodejs npm +<span class="k">RUN</span><span class="w"> </span>mkdir -vp /usr/etc +<span class="k">RUN</span><span class="w"> </span><span class="nb">echo</span> <span class="s2">"registry=http:/10.1.2.1/npm"</span> > /usr/etc/npmrc +<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/root</span> +<span class="k">RUN</span><span class="w"> </span>npm install foobarbarbar +</pre></div> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>This was the registry address I had in my <cite>~/.npmrc</cite> or whether I put it in myself.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>I have tried using a relative path instead, both with and without a leading <tt class="docutils literal">/</tt>, but in either case the install then errors our complaining that the lock file is "corrupt." Whether the schema allows a base-url setting I do not know, as the schema documentation doesn't seem to be readily available.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>In fact, <tt class="docutils literal">verdaccio</tt> by itself solves the particular problem that we're trying to solve; providing an offline repository alternative. The task at hand, however, is to understand how to cope <em>without</em> using other tools than what can be considered base infrastructure (i.e. a web server).</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-4" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-4">[4]</a></td><td>Should be <tt class="docutils literal">/etc/verdaccio/storage</tt>; see <tt class="docutils literal">/etc/verdaccio/config.yaml</tt></td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-5" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-5">[5]</a></td><td>An alternative approach would be to set the storage path to point to the same folder as the web server is serving. However, since we need to mess with the tarball paths</td></tr> +</tbody> +</table> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/92eb83e7074b2760e6001d40b17eecffb874b23f036abd5b6d2328f28f49bac8.asc b/92eb83e7074b2760e6001d40b17eecffb874b23f036abd5b6d2328f28f49bac8.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NzsACgkQzcvSTdHQ +4AFSegf6Ar0VsD+2aMnMxyLrNduJsmqtgw4DaI3F6qwd801H34FfPfOh8dMCxXp2 +Ko8/rroEvEzXxnA7F6v3XpGky4ETL7E3Y+XN0GLYAoAwe3L7aEXWT0dC+DqMABL/ +ZWBcGFCxdWmKDdYzbiPdKep/BNFYfBlC1vTTaEZfTqRSLb1KyYhh2ryTJFfrRkBz +MpwzEGEJuhs8XMcSwAnjrFeuTuQQftLOV9+IyWcYZEVqDfuHA2O25Cbp6MTTIS9a +s3bzFgdKUutlbJH1orcrh0cPqodS8S7tYcJeGj3FZ2zmjE1jFA5/zj5/kFEJl5Tx +uCTSWBRw5ZW9dH82ZZHmQ4ZVfXZQ7Q== +=nige +-----END PGP SIGNATURE----- diff --git a/970551f6b9e85da64aa25d67d89191d0a1a3223b899604a354fc694903cbf615 b/970551f6b9e85da64aa25d67d89191d0a1a3223b899604a354fc694903cbf615 @@ -0,0 +1,227 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - A portable book metadata exercise</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li class="active"><a href="./category/archiving.html">Archiving</a></li> + <li><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./portable-book-metadata.html" rel="bookmark" + title="Permalink to A portable book metadata exercise">A portable book metadata exercise</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2022-10-01T12:40:00+02:00"> + Sat 01 October 2022 + </time> +in <a class="category" href="./category/archiving.html">archiving</a> + <a href="./tag/hash.html">hash</a> + <a href="./tag/kitab.html">kitab</a> + <a href="./tag/literature.html">literature</a> + <a href="./tag/metadata.html">metadata</a> + <a href="./tag/dublincore.html">dublincore</a> + <a href="./tag/libgen.html">libgen</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./backup-rsync-duplicity.html"> + Combining duplicity and rsync + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>One of the things I have been working on the last few weeks is a rust application I have dubbed <a class="reference external" href="https://git.defalsify.net/kitab">kitab</a>.</p> +<p>In short, the application makes it easy to extract literary metadata to a separate file structure.</p> +<p>The metadata can in turn be applied as <em>extended attributes</em> recursively on a directory for files that match.</p> +<p>The way it's accomplished it simple: The file name of the metadata is the hex representation of the digest of the file. The same digest is used to match files to metadata when applying it back to the file.</p> +<p>There are two advantages to this:</p> +<ol class="arabic simple"> +<li>The digest of the media file need not be affected by the metadata, i.e. by embedding metadata in the file itself.</li> +<li>You do not need to use the file name to keep record of what a file is.</li> +</ol> +<div class="section" id="yarr-ye-matey-data"> +<h2>Yarr, ye matey-data</h2> +<p>Let's demonstrate with an example.</p> +<p>The fabulous <a class="reference external" href="https://libgen.rs">Library Genesis</a> project has made available an endpoint to retrieve <tt class="docutils literal">bibtex</tt> entries based on the <tt class="docutils literal">md5</tt> hash of the book media file.</p> +<p>A version of the <a class="reference external" href="https://libgen.rs/book/index.php?md5=BCD99F1AB4155F2A2A362E5B7938A852">Bitcoin White Paper</a>, under the <code>md5</code> hash <code>bcd99f1ab4155f2a2a362e5b7938a852</code>, can be found there.</p> +<p>If you download this file using a synchronous download link, the browser will provide you with a filename to go with the download.</p> +<p>However, if you use the torrent alternative, the filename will be the <tt class="docutils literal">md5</tt> hash itself. If you are torrenting a bunch of those files, it quickly becomes a nuisance to distinguish them.</p> +<p>And, of course: In either case there is no guarantee the any metadata comes with the file.</p> +<div class="section" id="inside-the-book"> +<h3>Inside the book</h3> +<p>Kitab (v0.0.2) is able to read metadata from both a bibtex source and xattr entries on a file, as well as its native <a class="reference external" href="https://www.w3.org/TR/turtle/">rdf-turtle</a> format.</p> +<p>In kitab's data store, every media file entity in rdf-turtle is keyed with a <a class="reference external" href="https://www.rfc-editor.org/info/rfc8141">URN</a> specifying a digest for the file.</p> +<p>To see exactly what that looks like, let's download and import the bibtex metadata for the paper <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>:</p> +<pre class="code bash literal-block"> +<span class="nv">bibtex_file</span><span class="o">=</span><span class="sb">`</span>mktemp<span class="sb">`</span> +<span class="nv">kitab_dir</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> +curl -s -X GET https://libgen.rs/book/bibtex.php?md5<span class="o">=</span>BCD99F1AB4155F2A2A362E5B7938A852 -o <span class="nv">$bibtex_file</span> +kitab --store <span class="nv">$kitab_dir</span> import --digest md5:BCD99F1AB4155F2A2A362E5B7938A852 <span class="nv">$bibtex_file</span> +cat <span class="nv">$kitab_dir</span>/* +</pre> +<p>The output of the above should be:</p> +<pre class="code turtle literal-block"> +<span class="nv"><URN:md5:bcd99f1ab4155f2a2a362e5b7938a852></span> <span class="nv"><https://purl.org/dc/terms/title></span> <span class="s">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> +<span class="nv"><https://purl.org/dc/terms/creator></span> <span class="s">"Satoshi Nakamoto"</span> <span class="p">;</span> +<span class="nv"><https://purl.org/dc/terms/type></span> <span class="s">"book"</span> <span class="p">.</span> +</pre> +<p>Now let's say the media file itself has been downloaded to <tt class="docutils literal"><span class="pre">~/.local/share/transmission</span></tt>. We can apply this metadata as extended attributes.</p> +<p>This time we turn on logging to see what's going on:</p> +<pre class="code console literal-block"> +<span class="gp">$ </span><span class="nv">RUST_LOG</span><span class="o">=</span>info kitab --store <span class="nv">$kitab_dir</span> apply --digest md5 ~/.local/share/transmission +<span class="go">[2022-10-01T11:14:59Z INFO kitab] have index directory "/tmp/tmp.r0jBm6q4hW" +[2022-10-01T11:14:59Z INFO kitab] using digest type md5 +[2022-10-01T11:14:59Z INFO kitab] apply from path "/home/lash/.local/share/transmission/" +[2022-10-01T11:14:59Z INFO kitab] apply DirEntry("/home/lash/.local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852") -> title "Bitcoin: A Peer-to-Peer Electronic Cash System" author "Satoshi Nakamoto" digest md5:bcd99f1ab4155f2a2a362e5b7938a852 + +</span><span class="gp">$ </span>find ~/.local/share/transmission -type f -regextype sed -regex <span class="s2">".*/[a-f0-9]\{32\}</span>$<span class="s2">"</span> -exec getfattr -d <span class="o">{}</span> <span class="se">\;</span> +<span class="gp"># </span>file: .local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852 +<span class="go">user.dcterms:creator="Satoshi Nakamoto" +user.dcterms:title="Bitcoin: A Peer-to-Peer Electronic Cash System" +user.dcterms:type="book"</span> +</pre> +</div> +<div class="section" id="let-the-right-one-in"> +<h3>Let the right one in</h3> +<p>Conversely, the metadata can be re-imported directly from the extended attributes. And this time, let's store it both under the <tt class="docutils literal">md5</tt> and the <tt class="docutils literal">sha512</tt> hash:</p> +<pre class="code bash literal-block"> +$ <span class="nv">kitab_dir_new</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> +$ kitab --store <span class="nv">$kitab_dir_new</span> import --digest md5 --digest sha512 .local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852 +$ find <span class="nv">$kitab_dir_new</span> -type f -exec cat <span class="o">{}</span> <span class="se">\;</span> +/tmp/tmp.B6j41YMmEM/493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792 +<URN:sha512:493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792> <https://purl.org/dc/terms/title> <span class="s2">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> + <https://purl.org/dc/terms/creator> <span class="s2">"Satoshi Nakamoto"</span> <span class="p">;</span> + <https://purl.org/dc/terms/type> <span class="s2">"book"</span> <span class="p">;</span> + <https://purl.org/dc/terms/MediaType> <span class="s2">"application/epub+zip"</span> . +/tmp/tmp.B6j41YMmEM/bcd99f1ab4155f2a2a362e5b7938a852 +<URN:md5:bcd99f1ab4155f2a2a362e5b7938a852> <https://purl.org/dc/terms/title> <span class="s2">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> + <https://purl.org/dc/terms/creator> <span class="s2">"Satoshi Nakamoto"</span> <span class="p">;</span> + <https://purl.org/dc/terms/type> <span class="s2">"book"</span> <span class="p">;</span> + <https://purl.org/dc/terms/MediaType> <span class="s2">"application/epub+zip"</span> . +</pre> +</div> +</div> +<div class="section" id="level-up"> +<h2>Level up</h2> +<p>Finally, a bash script <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a> example that lets you retrieve and apply metadata for a batch of files. This script even renames the files according to the metadata applied.</p> +<pre class="code bash literal-block"> +<span class="ln"> 0 </span><span class="c1"># NOTE! this will only work if your fs supports xattr. +</span><span class="ln"> 1 </span><span class="c1"># That's why we cannot use tmpfs (mktemp) here; tmpfs does not support xattr. +</span><span class="ln"> 2 </span><span class="c1"></span><span class="nv">outdir</span><span class="o">=</span>./<span class="k">$(</span>uuidgen<span class="k">)</span> +<span class="ln"> 3 </span>mkdir -vp <span class="nv">$outdir</span> +<span class="ln"> 4 </span> +<span class="ln"> 5 </span><span class="nv">IFS</span><span class="o">=</span><span class="s1">$'\n'</span> +<span class="ln"> 6 </span> +<span class="ln"> 7 </span><span class="k">for</span> f <span class="k">in</span> <span class="k">$(</span>find <span class="nv">$1</span> -type f<span class="k">)</span><span class="p">;</span> <span class="k">do</span> +<span class="ln"> 8 </span> <span class="nv">sum</span><span class="o">=</span><span class="k">$(</span>md5sum <span class="nv">$f</span> <span class="p">|</span> awk <span class="s1">'{print $1;}'</span><span class="k">)</span> +<span class="ln"> 9 </span> <span class="nb">echo</span> <span class="s2">"downloading metadata for </span><span class="nv">$1</span><span class="s2">/</span><span class="nv">$f</span><span class="s2">"</span> +<span class="ln">10 </span> <span class="nv">srct</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span> +<span class="ln">11 </span> curl -s -X GET https://libgen.rs/book/bibtex.php?md5<span class="o">=</span><span class="nv">$sum</span> -o <span class="nv">$srct</span> +<span class="ln">12 </span> <span class="nv">dstt</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span> +<span class="ln">13 </span> xmllint --html --xpath <span class="s1">'string(/html/body/textarea[@id="bibtext"])'</span> <span class="nv">$srct</span> > <span class="nv">$dstt</span> +<span class="ln">14 </span> kitab import --digest md5:<span class="nv">$sum</span> <span class="nv">$dstt</span> +<span class="ln">15 </span> cp <span class="nv">$f</span> <span class="nv">$outdir</span>/ +<span class="ln">16 </span><span class="k">done</span> +<span class="ln">17 </span> +<span class="ln">18 </span><span class="nv">RUST_LOG</span><span class="o">=</span>info kitab apply --digest md5 <span class="nv">$outdir</span>/ +<span class="ln">19 </span> +<span class="ln">20 </span><span class="k">for</span> f <span class="k">in</span> <span class="k">$(</span>ls <span class="nv">$outdir</span><span class="k">)</span><span class="p">;</span> <span class="k">do</span> +<span class="ln">21 </span> <span class="nv">title</span><span class="o">=</span><span class="k">$(</span>getfattr --only-values -n user.dcterms:title <span class="nv">$outdir</span>/<span class="nv">$f</span><span class="k">)</span> +<span class="ln">22 </span> +<span class="ln">23 </span> <span class="nv">f_typ</span><span class="o">=</span><span class="k">$(</span>file -b --mime-type <span class="nv">$outdir</span>/<span class="nv">$f</span><span class="k">)</span> +<span class="ln">24 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">""</span> +<span class="ln">25 </span> <span class="k">case</span> <span class="s2">"</span><span class="nv">$f_typ</span><span class="s2">"</span> <span class="k">in</span> +<span class="ln">26 </span> <span class="s2">"application/pdf"</span><span class="o">)</span> +<span class="ln">27 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".pdf"</span> +<span class="ln">28 </span> <span class="p">;;</span> +<span class="ln">29 </span> <span class="s2">"application/epub+zip"</span><span class="o">)</span> +<span class="ln">30 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".epub"</span> +<span class="ln">31 </span> <span class="p">;;</span> +<span class="ln">32 </span> <span class="s2">"application/x-mobipocket-ebook"</span><span class="o">)</span> +<span class="ln">33 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".mobi"</span> +<span class="ln">34 </span> <span class="p">;;</span> +<span class="ln">35 </span> <span class="s2">"text/plain"</span><span class="o">)</span> +<span class="ln">36 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".txt"</span> +<span class="ln">37 </span> <span class="p">;;</span> +<span class="ln">38 </span> <span class="s2">"text/html"</span><span class="o">)</span> +<span class="ln">39 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".html"</span> +<span class="ln">40 </span> <span class="p">;;</span> +<span class="ln">41 </span> *<span class="o">)</span> +<span class="ln">42 </span> ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> unhandled mime <span class="nb">type</span> <span class="nv">$f_typ</span> +<span class="ln">43 </span> <span class="nb">exit</span> <span class="m">1</span> +<span class="ln">44 </span> <span class="k">esac</span> +<span class="ln">45 </span> mv -v <span class="nv">$outdir</span>/<span class="nv">$f</span> <span class="nv">$outdir</span>/<span class="si">${</span><span class="nv">title</span><span class="si">}${</span><span class="nv">f_ext</span><span class="si">}</span> +<span class="ln">46 </span><span class="k">done</span> +</pre> +<p>This last example will result in:</p> +<ul class="simple"> +<li>A media file named <tt class="docutils literal">$outdir/Bitcoin: A <span class="pre">Peer-to-Peer</span> Electronic Cash System.epub</tt></li> +<li>... with metadata applied as extended attributes</li> +<li>An rdf-turtle metadata entry in <tt class="docutils literal"><span class="pre">~/.local/share/kitab/idx/bcd99f1ab4155f2a2a362e5b7938a852</span></tt></li> +</ul> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>The <code>kitab</code> command in the script assumes you have built the <em>kitab binary</em> and made it available in your path.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>the script uses <cite>xmllint</cite> which on archlinux is provided by the <tt class="docutils literal">libxml2</tt> package.</td></tr> +</tbody> +</table> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/970551f6b9e85da64aa25d67d89191d0a1a3223b899604a354fc694903cbf615.asc b/970551f6b9e85da64aa25d67d89191d0a1a3223b899604a354fc694903cbf615.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NWEACgkQzcvSTdHQ +4AGwJwf/baQulBK5dAEFaXhgtOzl1kSDZ/TbQ8D1FAiQK0IyDJf+/ITxB7pJ/1A6 +6I+wN/KeMa1BrRhHxOuoYLH4JQ2Cm6UW51w+wcTfnLdaX4maZer4nUi63hjvVaYj +Pn+AeyaDJGVQvXStJRLW75xKkl1/QGcg7hBH6nbVInMJqYTyWST0RC2ERX/0+kTN +mWotHZ8z6TAHtD4AfLsKmkcFpAFVLDys5xY6gRPbIeaKertoou31ESIfKC2IvCiJ +5x1veVRyPb5HSmLqzaaQkyAbWnn+PmmtHUMeUKz1ozgwLoHVmhwwd8+tOXChn7PN +mZbAEaqI2mPfghrGDWmyd7BDmAndVA== +=sxPi +-----END PGP SIGNATURE----- diff --git a/a5006a0269c2c925630e27977d97e59941f38432eae7b684a8342b92d655ec31 b/a5006a0269c2c925630e27977d97e59941f38432eae7b684a8342b92d655ec31 @@ -1,224 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> - <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> - <title>man bytes gnu - Making sense of Ethereum log bloom filters</title> - <meta charset="utf-8" /> - <meta name="generator" content="Pelican" /> - -</head> - -<body id="index" class="home"> - - <header id="banner" class="body"> - <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> - </header> - <nav id="menu"> - <ul> - <li><a href="./category/archiving.html">Archiving</a></li> - <li class="active"><a href="./category/code.html">Code</a></li> - <li><a href="./category/hygiene.html">Hygiene</a></li> - <li><a href="./category/offlining.html">Offlining</a></li> - <li><a href="./pages/identities.html">Identities</a></li> - <li><a href="./pages/shares.html">Shares</a></li> - - <li><a href="/tags.html">tags</a></li> - </ul> - <hr/> - </nav><!-- /#menu --> - <header> - <h1 class="entry-title top-body-title"> - <a href="./eth-log-bloom.html" rel="bookmark" - title="Permalink to Making sense of Ethereum log bloom filters">Making sense of Ethereum log bloom filters</a></h1> - - - <div class="category meta"> - Posted -<time class="published" datetime="2021-06-27T11:17:00+02:00"> - Sun 27 June 2021 - </time> -in <a class="category" href="./category/code.html">code</a> - <a href="./tag/crypto.html">crypto</a> - <a href="./tag/ethereum.html">ethereum</a> - <a href="./tag/bloom-filter.html">bloom filter</a> - </div> - - <div class="neighbors meta"> - <ul> - <li> - Previous: <a href="./docker-offline-3-npm.html"> - Local npm repository - </a> - </li> - | - - <li> - Next: <a href="./internet-up-monitor.html"> - Homemade internet state monitor - </a> - </li> - </ul> - </div> - <div class="meta"> -<hr/> - </div> -</header> - -<section id="content" class="body"> - <div class="entry-content"> - <p>Ethereum blocks and transaction receipts both carry a bloom filter. They provide a shortcut to check whether a particular event of a particular contract was triggered. The filter in the transaction indexes all events that occurred in the transaction. The filter in the block carries all the filter bits of the individual transactions within the block.</p> -<p>Somewhat surprisingly, there are practically no friendly descriptions out there that tell you how to generate filters to match them with. In fact, the <a class="reference external" href="https://holbrook.no/doc/ethereum/yellowpaper.pdf">Ethereum Yellow Paper</a> was the only source I found.</p> -<p><strong>So let's make a friendly description</strong></p> -<p>Before we get to the friendliness, though, an ever so slight dose of brain melt is due:</p> -<div class="section" id="the-formalities"> -<h2>The formalities</h2> -<p>The formal definition of the filter calculation in the <a class="reference external" href="https://holbrook.no/doc/ethereum/yellowpaper.pdf">Ethereum Yellow Paper section 4.3.1</a> is (here cited in painfully inadequate math formatting):</p> -<blockquote> -<div class="formula"> -<i>M</i>(<i>O</i>) ≡ ∨<i>x</i> ∈ {<i>Oa</i>}∪<i>Ot</i>(<i>M</i><sub>3 : 2048</sub>(<i>x</i>)) -</div> -</blockquote> -<p>Where <span class="formula"><i>Oa</i></span> is the contract address and <span class="formula"><i>Ot</i></span> is all of the topics.</p> -<p>In the context of <em>Solidity</em>, "all" of the topics means the actual event signature, along with all the <em>indexed</em> parameters to the event.</p> -<p>The <span class="formula"><i>M</i></span> is the filter bit generator. The filter 2048 bits long, and for each entry that is added, 3 bits need to be set.</p> -<p>Reading further, we learn that the bits to set are the "first three" pairs of bytes of the keccak256 hash <a class="footnote-reference" href="#id4" id="id1">[1]</a>, modulo the length of the filter - 2048.</p> -<p>In the context of hashes, I usually assume "first" means from the left. That happens to be the case here, too. In other words, the "first" bytes are actually the bytes of most significance in big-endian order.</p> -<p>What's less clear, however, is which bits to set in the filter. The paper says:</p> -<blockquote> -<span class="formula">ℬ</span> is the bit reference function such that <span class="formula">ℬ<sub><i>j</i></sub>(<i>x</i>)</span> equals the bit of index <span class="formula"><i>j</i></span> (indexed from 0) in the byte array <span class="formula"><i>x</i></span>.</blockquote> -<p>Turns out after some trial and error that "index 0" means the <em>right</em> end of the byte array. So when setting the bits, we have to count from the right side.</p> -</div> -<div class="section" id="the-friendly-recipe"> -<h2>The friendly recipe</h2> -<p>Now let's try to bring this down to a mortal level:</p> -<ol class="arabic simple"> -<li>Take the keccak256 hash of the contract address</li> -<li>Take the keccak256 hash of the first topic value</li> -<li>Optionally, take the keccak256 hash of the remaining topic values (the indexed parameters of the event) <a class="footnote-reference" href="#id5" id="id2">[2]</a></li> -<li>Instantiate a byte array with length 256</li> -<li>Now, <em>for each</em> of the hashes:<ol class="arabic"> -<li>Get the <em>big-endian</em> integer of the <em>first two</em> bytes of the hash</li> -<li>Calculate the 2048-modulo of that integer <a class="footnote-reference" href="#id6" id="id3">[3]</a></li> -<li><em>Bitwise or</em> the filter bit at the index corresponding to the modulated value, counting from <em>right to left</em> (i.e. the value <span class="formula">0</span> corresponds to the <em>rightmost bit</em>).</li> -<li>Repeat 2 and 3 for the two next pairs of bytes of the hash.</li> -</ol> -</li> -</ol> -</div> -<div class="section" id="the-code"> -<h2>The code</h2> -<p>All that taken into account, we can take a stab at implementing a filter generator:</p> -<pre class="code python literal-block"> -<span class="ln"> 0 </span><span class="c1"># external imports</span> -<span class="ln"> 1 </span><span class="kn">import</span> <span class="nn">sha3</span> -<span class="ln"> 2 </span> -<span class="ln"> 3 </span> -<span class="ln"> 4 </span><span class="k">class</span> <span class="nc">LogBloom</span><span class="p">:</span> -<span class="ln"> 5 </span> -<span class="ln"> 6 </span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> -<span class="ln"> 7 </span> <span class="bp">self</span><span class="o">.</span><span class="n">content</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">(</span><span class="mi">256</span><span class="p">)</span> -<span class="ln"> 8 </span> -<span class="ln"> 9 </span> -<span class="ln">10 </span> <span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">):</span> -<span class="ln">11 </span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">element</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">):</span> -<span class="ln">12 </span> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'element must be bytes'</span><span class="p">)</span> -<span class="ln">13 </span> <span class="n">h</span> <span class="o">=</span> <span class="n">sha3</span><span class="o">.</span><span class="n">keccak_256</span><span class="p">()</span> -<span class="ln">14 </span> <span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">element</span><span class="p">)</span> -<span class="ln">15 </span> <span class="n">z</span> <span class="o">=</span> <span class="n">h</span><span class="o">.</span><span class="n">digest</span><span class="p">()</span> -<span class="ln">16 </span> -<span class="ln">17 </span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span> -<span class="ln">18 </span> <span class="n">c</span> <span class="o">=</span> <span class="n">j</span> <span class="o">*</span> <span class="mi">2</span> -<span class="ln">19 </span> <span class="n">v</span> <span class="o">=</span> <span class="nb">int</span><span class="o">.</span><span class="n">from_bytes</span><span class="p">(</span><span class="n">z</span><span class="p">[</span><span class="n">c</span><span class="p">:</span><span class="n">c</span><span class="o">+</span><span class="mi">2</span><span class="p">],</span> <span class="n">byteorder</span><span class="o">=</span><span class="s1">'big'</span><span class="p">)</span> -<span class="ln">20 </span> <span class="n">v</span> <span class="o">&=</span> <span class="mh">0x07ff</span> -<span class="ln">21 </span> <span class="n">m</span> <span class="o">=</span> <span class="mi">255</span> <span class="o">-</span> <span class="nb">int</span><span class="p">(</span><span class="n">v</span> <span class="o">/</span> <span class="mi">8</span><span class="p">)</span> -<span class="ln">22 </span> <span class="n">n</span> <span class="o">=</span> <span class="n">v</span> <span class="o">%</span> <span class="mi">8</span> -<span class="ln">23 </span> <span class="bp">self</span><span class="o">.</span><span class="n">content</span><span class="p">[</span><span class="n">m</span><span class="p">]</span> <span class="o">|=</span> <span class="p">(</span><span class="mi">1</span> <span class="o"><<</span> <span class="n">n</span><span class="p">)</span> -</pre> -<p>Let's say we'd want to check if an Solidity-defined event <code>Foo(uint256,bytes32)</code> emitted by a contract at address <code>0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee</code> exists in a filter.</p> -<p>Using the class above, we would have to do something like this:</p> -<pre class="code python literal-block"> -<span class="c1"># external imports</span> -<span class="kn">import</span> <span class="nn">sha3</span> -<span class="kn">import</span> <span class="nn">some_block_getter</span> - -<span class="c1"># local imports</span> -<span class="kn">from</span> <span class="nn">the_above</span> <span class="kn">import</span> <span class="n">LogBloom</span> - -<span class="c1"># all set bits in our filter must be set in theirs</span> -<span class="k">def</span> <span class="nf">filter_match</span><span class="p">(</span><span class="n">theirs</span><span class="p">,</span> <span class="n">ours</span><span class="p">):</span> - <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">ours</span><span class="p">)):</span> - <span class="k">if</span> <span class="n">ours</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">&</span> <span class="n">theirs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">!=</span> <span class="n">ours</span><span class="p">[</span><span class="n">i</span><span class="p">]:</span> - <span class="k">return</span> <span class="kc">False</span> - <span class="k">return</span> <span class="kc">True</span> - -<span class="n">fltr</span> <span class="o">=</span> <span class="n">LogBloom</span><span class="p">()</span> - -<span class="c1"># add the contract address to the filter</span> -<span class="n">address</span> <span class="o">=</span> <span class="nb">bytes</span><span class="o">.</span><span class="n">fromhex</span><span class="p">(</span><span class="s1">'eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee'</span><span class="p">)</span> -<span class="n">h</span> <span class="o">=</span> <span class="n">sha3</span><span class="o">.</span><span class="n">keccak_256</span><span class="p">()</span> -<span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">address</span><span class="p">)</span> -<span class="n">address_element</span> <span class="o">=</span> <span class="n">h</span><span class="o">.</span><span class="n">digest</span><span class="p">()</span> -<span class="n">fltr</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">address_element</span><span class="p">)</span> - -<span class="c1"># create the topic signature for the event</span> -<span class="n">h</span> <span class="o">=</span> <span class="n">sha3</span><span class="o">.</span><span class="n">keccak_256</span><span class="p">()</span> -<span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="sa">b</span><span class="s1">'Foo(uint256,bytes32)'</span><span class="p">)</span> -<span class="n">topic</span> <span class="o">=</span> <span class="n">h</span><span class="o">.</span><span class="n">digest</span><span class="p">()</span> - -<span class="c1"># add the topic to the filter</span> -<span class="n">h</span> <span class="o">=</span> <span class="n">sha3</span><span class="o">.</span><span class="n">keccak_256</span><span class="p">()</span> -<span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">topic</span><span class="p">)</span> -<span class="n">topic_element</span> <span class="o">=</span> <span class="n">h</span><span class="o">.</span><span class="n">digest</span><span class="p">()</span> -<span class="n">fltr</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">topic_element</span><span class="p">)</span> - -<span class="c1"># get the filter from a block</span> -<span class="n">block</span> <span class="o">=</span> <span class="n">some_block_getter</span><span class="p">()</span> -<span class="n">block_bloom</span> <span class="o">=</span> <span class="nb">bytes</span><span class="o">.</span><span class="n">fromhex</span><span class="p">(</span><span class="n">block</span><span class="p">[</span><span class="s1">'logsBloom'</span><span class="p">][</span><span class="mi">2</span><span class="p">:])</span> <span class="c1"># assumes there is a 0x-prefix</span> - -<span class="c1"># check if it matches</span> -<span class="n">match</span> <span class="o">=</span> <span class="n">filter_match</span><span class="p">(</span><span class="n">block_bloom</span><span class="p">,</span> <span class="n">fltr</span><span class="o">.</span><span class="n">contents</span><span class="p">)</span> -<span class="nb">print</span><span class="p">(</span><span class="n">match</span><span class="p">)</span> -</pre> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id4" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Remember the keccak hash used in Ethereum/Bitcoin is _not_ the official <cite>sha3</cite> hash. The padding parameter is different. More on that <a class="reference external" href="./nft-tokenid-content.html#id8">here</a></td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id5" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Solidity supports up to 3 indexed parameters, which maps to the <em>evm</em> opcodes <code>LOGn</code> where <span class="formula"><i>n</i> ∈ {1, 2, 3, 4}</span></td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id6" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>Since <span class="formula">2<sup>11</sup> = 2048</span>, this is the same as zeroing out all bits above bit number 11; <span class="formula"><i>x</i>∧2048</span>. In other words, the result is an <em>11 bit integer</em>.</td></tr> -</tbody> -</table> -</blockquote> -</div> - - </div><!-- /.entry-content --> - <footer class="meta"> - </footer> - -</section> - <footer id="contentinfo" class="body"> - <hr/> - <address id="about" class="vcard body"> - Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> - </address><!-- /#about --> - </footer><!-- /#contentinfo --> -</body> -</html> -\ No newline at end of file diff --git a/a5006a0269c2c925630e27977d97e59941f38432eae7b684a8342b92d655ec31.asc b/a5006a0269c2c925630e27977d97e59941f38432eae7b684a8342b92d655ec31.asc @@ -1,11 +0,0 @@ ------BEGIN PGP SIGNATURE----- - -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmHkZ0UACgkQzcvSTdHQ -4AF9BQf/ZHncwvTWYwi6gGdwsVOB2waJDoaq/w/KWTNzqoRf5HSYINZuqhNar3Zs -CNLR1vwx9qxWRLK13+NFMb7qU4DUnvm0EmaYWT8XDQSwzHY/LKzquTOXjROO/9q3 -d7lAxftFiPBM8Mbte1seTfsdTrPZbRcX3aD+bRQhBBVaYAEXq/jdlQHI45Fbh4iA -isrLeAR0O2uYNeSZV5X5QQUuuhFuZZR3t65neuFCrrCFCCd1A8JjkjO3viE5qTJe -Wr5zd0XToUAmN+n1teApyoXIqsCEK9nya7uCpyioEJ9CMOMvEd2wig7yKVBd2Azp -kDaebgp9ivUsznfS/yRIRsPvV/Y9kw== -=ruob ------END PGP SIGNATURE----- diff --git a/a65e1931616b12b513de414d8a220ee14e85e9cc12165dd27d77773d3e593760 b/a65e1931616b12b513de414d8a220ee14e85e9cc12165dd27d77773d3e593760 @@ -0,0 +1,228 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - A portable book metadata exercise</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li class="active"><a href="./category/archiving.html">Archiving</a></li> + <li><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./portable-book-metadata.html" rel="bookmark" + title="Permalink to A portable book metadata exercise">A portable book metadata exercise</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2022-10-01T12:40:00+02:00"> + Sat 01 October 2022 + </time> +in <a class="category" href="./category/archiving.html">archiving</a> + <a href="./tag/hash.html">hash</a> + <a href="./tag/kitab.html">kitab</a> + <a href="./tag/literature.html">literature</a> + <a href="./tag/metadata.html">metadata</a> + <a href="./tag/dublincore.html">dublincore</a> + <a href="./tag/libgen.html">libgen</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./backup-rsync-duplicity.html"> + Combining duplicity and rsync + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>One of the things I have been working on the last few weeks is a rust application I have dubbed <a class="reference external" href="https://git.defalsify.net/kitab">kitab</a>.</p> +<p>In short, the application makes it easy to extract literary metadata to a separate file structure.</p> +<p>The metadata can in turn be applied as <em>extended attributes</em> recursively on a directory for files that match.</p> +<p>The way it's accomplished it simple: The file name of the metadata is the hex representation of the digest of the file. The same digest is used to match files to metadata when applying it back to the file.</p> +<p>There are two advantages to this:</p> +<ol class="arabic simple"> +<li>The digest of the media file need not be affected by the metadata, i.e. by embedding metadata in the file itself.</li> +<li>You do not need to use the file name to keep record of what a file is.</li> +</ol> +<div class="section" id="yarr-ye-matey-data"> +<h2>Yarr, ye matey-data</h2> +<p>Let's demonstrate with an example.</p> +<p>The fabulous <a class="reference external" href="https://libgen.rs">Library Genesis</a> project has made available an endpoint to retrieve <tt class="docutils literal">bibtex</tt> entries based on the <tt class="docutils literal">md5</tt> hash of the book media file.</p> +<p>A version of the <a class="reference external" href="https://libgen.rs/book/index.php?md5=BCD99F1AB4155F2A2A362E5B7938A852">Bitcoin White Paper</a>, under the <code>md5</code> hash <code>bcd99f1ab4155f2a2a362e5b7938a852</code>, can be found there.</p> +<p>If you download this file using a synchronous download link, the browser will provide you with a filename to go with the download.</p> +<p>However, if you use the torrent alternative, the filename will be the <tt class="docutils literal">md5</tt> hash itself. If you are torrenting a bunch of those files, it quickly becomes a nuisance to distinguish them.</p> +<p>And, of course: In either case there is no guarantee the any metadata comes with the file.</p> +<div class="section" id="inside-the-book"> +<h3>Inside the book</h3> +<p>Kitab (v0.0.2) is able to read metadata from both a bibtex source and xattr entries on a file, as well as its native <a class="reference external" href="https://www.w3.org/TR/turtle/">rdf-turtle</a> format.</p> +<p>In kitab's data store, every media file entity in rdf-turtle is keyed with a <a class="reference external" href="https://www.rfc-editor.org/info/rfc8141">URN</a> specifying a digest for the file.</p> +<p>To see exactly what that looks like, let's download and import the bibtex metadata for the paper <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>:</p> +<pre class="code bash literal-block"> +<span class="nv">bibtex_file</span><span class="o">=</span><span class="sb">`</span>mktemp<span class="sb">`</span> +<span class="nv">kitab_dir</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> +curl -s -X GET https://libgen.rs/book/bibtex.php?md5<span class="o">=</span>BCD99F1AB4155F2A2A362E5B7938A852 -o <span class="nv">$bibtex_file</span> +kitab --store <span class="nv">$kitab_dir</span> import --digest md5:BCD99F1AB4155F2A2A362E5B7938A852 <span class="nv">$bibtex_file</span> +cat <span class="nv">$kitab_dir</span>/* +</pre> +<p>The output of the above should be:</p> +<pre class="code turtle literal-block"> +<span class="nv"><URN:md5:bcd99f1ab4155f2a2a362e5b7938a852></span> <span class="nv"><https://purl.org/dc/terms/title></span> <span class="s">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> +<span class="nv"><https://purl.org/dc/terms/creator></span> <span class="s">"Satoshi Nakamoto"</span> <span class="p">;</span> +<span class="nv"><https://purl.org/dc/terms/type></span> <span class="s">"book"</span> <span class="p">.</span> +</pre> +<p>Now let's say the media file itself has been downloaded to <tt class="docutils literal"><span class="pre">~/.local/share/transmission</span></tt>. We can apply this metadata as extended attributes.</p> +<p>This time we turn on logging to see what's going on:</p> +<pre class="code console literal-block"> +<span class="gp">$ </span><span class="nv">RUST_LOG</span><span class="o">=</span>info kitab --store <span class="nv">$kitab_dir</span> apply --digest md5 ~/.local/share/transmission +<span class="go">[2022-10-01T11:14:59Z INFO kitab] have index directory "/tmp/tmp.r0jBm6q4hW" +[2022-10-01T11:14:59Z INFO kitab] using digest type md5 +[2022-10-01T11:14:59Z INFO kitab] apply from path "/home/lash/.local/share/transmission/" +[2022-10-01T11:14:59Z INFO kitab] apply DirEntry("/home/lash/.local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852") -> title "Bitcoin: A Peer-to-Peer Electronic Cash System" author "Satoshi Nakamoto" digest md5:bcd99f1ab4155f2a2a362e5b7938a852 + +</span><span class="gp">$ </span>find ~/.local/share/transmission -type f -regextype sed -regex <span class="s2">".*/[a-f0-9]\{32\}</span>$<span class="s2">"</span> -exec getfattr -d <span class="o">{}</span> <span class="se">\;</span> +<span class="gp"># </span>file: .local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852 +<span class="go">user.dcterms:creator="Satoshi Nakamoto" +user.dcterms:title="Bitcoin: A Peer-to-Peer Electronic Cash System" +user.dcterms:type="book"</span> +</pre> +</div> +<div class="section" id="let-the-right-one-in"> +<h3>Let the right one in</h3> +<p>Conversely, the metadata can be re-imported directly from the extended attributes. And this time, let's store it both under the <tt class="docutils literal">md5</tt> and the <tt class="docutils literal">sha512</tt> hash:</p> +<pre class="code bash literal-block"> +$ <span class="nv">kitab_dir_new</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> +$ kitab --store <span class="nv">$kitab_dir_new</span> import --digest md5 --digest sha512 .local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852 +$ find <span class="nv">$kitab_dir_new</span> -type f -exec cat <span class="o">{}</span> <span class="se">\;</span> +/tmp/tmp.B6j41YMmEM/493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792 +<URN:sha512:493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792> <https://purl.org/dc/terms/title> <span class="s2">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> + <https://purl.org/dc/terms/creator> <span class="s2">"Satoshi Nakamoto"</span> <span class="p">;</span> + <https://purl.org/dc/terms/type> <span class="s2">"book"</span> <span class="p">;</span> + <https://purl.org/dc/terms/MediaType> <span class="s2">"application/epub+zip"</span> . +/tmp/tmp.B6j41YMmEM/bcd99f1ab4155f2a2a362e5b7938a852 +<URN:md5:bcd99f1ab4155f2a2a362e5b7938a852> <https://purl.org/dc/terms/title> <span class="s2">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> + <https://purl.org/dc/terms/creator> <span class="s2">"Satoshi Nakamoto"</span> <span class="p">;</span> + <https://purl.org/dc/terms/type> <span class="s2">"book"</span> <span class="p">;</span> + <https://purl.org/dc/terms/MediaType> <span class="s2">"application/epub+zip"</span> . +</pre> +</div> +</div> +<div class="section" id="level-up"> +<h2>Level up</h2> +<p>Finally, a bash script <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a> example that lets you retrieve and apply metadata for a batch of files found in the directory given as the <em>first positional arg</em>.</p> +<p>This script even renames the files according to the metadata applied.</p> +<pre class="code bash literal-block"> +<span class="ln"> 0 </span><span class="c1"># NOTE! this will only work if your fs supports xattr. +</span><span class="ln"> 1 </span><span class="c1"># That's why we cannot use tmpfs (mktemp) here; tmpfs does not support xattr. +</span><span class="ln"> 2 </span><span class="c1"></span><span class="nv">outdir</span><span class="o">=</span>./<span class="k">$(</span>uuidgen<span class="k">)</span> +<span class="ln"> 3 </span>mkdir -vp <span class="nv">$outdir</span> +<span class="ln"> 4 </span> +<span class="ln"> 5 </span><span class="nv">IFS</span><span class="o">=</span><span class="s1">$'\n'</span> +<span class="ln"> 6 </span> +<span class="ln"> 7 </span><span class="k">for</span> f <span class="k">in</span> <span class="k">$(</span>find <span class="nv">$1</span> -type f<span class="k">)</span><span class="p">;</span> <span class="k">do</span> +<span class="ln"> 8 </span> <span class="nv">sum</span><span class="o">=</span><span class="k">$(</span>md5sum <span class="nv">$f</span> <span class="p">|</span> awk <span class="s1">'{print $1;}'</span><span class="k">)</span> +<span class="ln"> 9 </span> <span class="nb">echo</span> <span class="s2">"downloading metadata for </span><span class="nv">$1</span><span class="s2">/</span><span class="nv">$f</span><span class="s2">"</span> +<span class="ln">10 </span> <span class="nv">srct</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span> +<span class="ln">11 </span> curl -s -X GET https://libgen.rs/book/bibtex.php?md5<span class="o">=</span><span class="nv">$sum</span> -o <span class="nv">$srct</span> +<span class="ln">12 </span> <span class="nv">dstt</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span> +<span class="ln">13 </span> xmllint --html --xpath <span class="s1">'string(/html/body/textarea[@id="bibtext"])'</span> <span class="nv">$srct</span> > <span class="nv">$dstt</span> +<span class="ln">14 </span> kitab import --digest md5:<span class="nv">$sum</span> <span class="nv">$dstt</span> +<span class="ln">15 </span> cp <span class="nv">$f</span> <span class="nv">$outdir</span>/ +<span class="ln">16 </span><span class="k">done</span> +<span class="ln">17 </span> +<span class="ln">18 </span><span class="nv">RUST_LOG</span><span class="o">=</span>info kitab apply --digest md5 <span class="nv">$outdir</span>/ +<span class="ln">19 </span> +<span class="ln">20 </span><span class="k">for</span> f <span class="k">in</span> <span class="k">$(</span>ls <span class="nv">$outdir</span><span class="k">)</span><span class="p">;</span> <span class="k">do</span> +<span class="ln">21 </span> <span class="nv">title</span><span class="o">=</span><span class="k">$(</span>getfattr --only-values -n user.dcterms:title <span class="nv">$outdir</span>/<span class="nv">$f</span><span class="k">)</span> +<span class="ln">22 </span> +<span class="ln">23 </span> <span class="nv">f_typ</span><span class="o">=</span><span class="k">$(</span>file -b --mime-type <span class="nv">$outdir</span>/<span class="nv">$f</span><span class="k">)</span> +<span class="ln">24 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">""</span> +<span class="ln">25 </span> <span class="k">case</span> <span class="s2">"</span><span class="nv">$f_typ</span><span class="s2">"</span> <span class="k">in</span> +<span class="ln">26 </span> <span class="s2">"application/pdf"</span><span class="o">)</span> +<span class="ln">27 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".pdf"</span> +<span class="ln">28 </span> <span class="p">;;</span> +<span class="ln">29 </span> <span class="s2">"application/epub+zip"</span><span class="o">)</span> +<span class="ln">30 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".epub"</span> +<span class="ln">31 </span> <span class="p">;;</span> +<span class="ln">32 </span> <span class="s2">"application/x-mobipocket-ebook"</span><span class="o">)</span> +<span class="ln">33 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".mobi"</span> +<span class="ln">34 </span> <span class="p">;;</span> +<span class="ln">35 </span> <span class="s2">"text/plain"</span><span class="o">)</span> +<span class="ln">36 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".txt"</span> +<span class="ln">37 </span> <span class="p">;;</span> +<span class="ln">38 </span> <span class="s2">"text/html"</span><span class="o">)</span> +<span class="ln">39 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".html"</span> +<span class="ln">40 </span> <span class="p">;;</span> +<span class="ln">41 </span> *<span class="o">)</span> +<span class="ln">42 </span> ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> unhandled mime <span class="nb">type</span> <span class="nv">$f_typ</span> +<span class="ln">43 </span> <span class="nb">exit</span> <span class="m">1</span> +<span class="ln">44 </span> <span class="k">esac</span> +<span class="ln">45 </span> mv -v <span class="nv">$outdir</span>/<span class="nv">$f</span> <span class="nv">$outdir</span>/<span class="si">${</span><span class="nv">title</span><span class="si">}${</span><span class="nv">f_ext</span><span class="si">}</span> +<span class="ln">46 </span><span class="k">done</span> +</pre> +<p>This last example will result in:</p> +<ul class="simple"> +<li>A media file named <tt class="docutils literal">$outdir/Bitcoin: A <span class="pre">Peer-to-Peer</span> Electronic Cash System.epub</tt></li> +<li>... with metadata applied as extended attributes</li> +<li>An rdf-turtle metadata entry in <tt class="docutils literal"><span class="pre">~/.local/share/kitab/idx/bcd99f1ab4155f2a2a362e5b7938a852</span></tt></li> +</ul> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>The <code>kitab</code> command in the script assumes you have built the <em>kitab binary</em> and made it available in your path.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>the script uses <cite>xmllint</cite> which on archlinux is provided by the <tt class="docutils literal">libxml2</tt> package.</td></tr> +</tbody> +</table> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/a65e1931616b12b513de414d8a220ee14e85e9cc12165dd27d77773d3e593760.asc b/a65e1931616b12b513de414d8a220ee14e85e9cc12165dd27d77773d3e593760.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NhoACgkQzcvSTdHQ +4AHT/gf+Oi3UdT0nu9HJTV0Bv0nthuiaqnrxXNseEVxdZsVMP5cqUvi53e6+AH/c +Od5CJsgF5YAVZS0pSCVzvCmhzzeAsrXvntDiNawsGMJn9AUNO1xumLKhaU4ZR/8n +H16lXhClcGUAIWSO3F+c92HBe+h2gQlfyDB9/HEnXl2aFvFA+EopifF/oc4zxrzm +ET2X7FTl5gqOtcuPIMvaP2+dzrFvFgRNfR5jlE9y8qfFhIDPdZKCyfiHEqQS0lzN +s4s1NiRf6i0/bje8vozX1HbALN3b1HcCNZk3nyJYjw4FuFKCVGLjjFWTZUL3YMF/ +LMwjTPqHIGwEn0mhJ7bqWlsfoSidxg== +=/rEP +-----END PGP SIGNATURE----- diff --git a/af20d83900b9bea7a7258e13dccec33be38aa849bf2387fcdd4ab6ab1e6069f1 b/af20d83900b9bea7a7258e13dccec33be38aa849bf2387fcdd4ab6ab1e6069f1 @@ -1,249 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> - <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> - <title>man bytes gnu - Homemade internet state monitor</title> - <meta charset="utf-8" /> - <meta name="generator" content="Pelican" /> - -</head> - -<body id="index" class="home"> - - <header id="banner" class="body"> - <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> - </header> - <nav id="menu"> - <ul> - <li><a href="./category/archiving.html">Archiving</a></li> - <li><a href="./category/code.html">Code</a></li> - <li class="active"><a href="./category/hygiene.html">Hygiene</a></li> - <li><a href="./category/offlining.html">Offlining</a></li> - <li><a href="./pages/identities.html">Identities</a></li> - <li><a href="./pages/shares.html">Shares</a></li> - - <li><a href="/tags.html">tags</a></li> - </ul> - <hr/> - </nav><!-- /#menu --> - <header> - <h1 class="entry-title top-body-title"> - <a href="./internet-up-monitor.html" rel="bookmark" - title="Permalink to Homemade internet state monitor">Homemade internet state monitor</a></h1> - - - <div class="category meta"> - Posted -<time class="published" datetime="2021-07-01T13:59:00+02:00"> - Thu 01 July 2021 - </time> -in <a class="category" href="./category/hygiene.html">hygiene</a> - <a href="./tag/bash.html">bash</a> - <a href="./tag/systemd.html">systemd</a> - <a href="./tag/network.html">network</a> - <a href="./tag/i3wm.html">i3wm</a> - <a href="./tag/tmpfs.html">tmpfs</a> - <a href="./tag/openvpn.html">openvpn</a> - </div> - - <div class="neighbors meta"> - <ul> - <li> - Previous: <a href="./eth-log-bloom.html"> - Making sense of Ethereum log bloom filters - </a> - </li> - | - - <li> - Next: <a href="./backup-rsync-duplicity.html"> - Combining duplicity and rsync - </a> - </li> - </ul> - </div> - <div class="meta"> -<hr/> - </div> -</header> - -<section id="content" class="body"> - <div class="entry-content"> - <p>I use <a class="reference external" href="https://i3wm.org">i3wm</a>. I use it because it has a very low resource footprint, it has great documentation, and it's simple to configure.</p> -<p>Complementing <em>i3wm</em> is the <em>i3status</em> status provider to the <em>i3bar</em>. It provides a good suite of preset components for things like temperatur measurement, disk capacity. It also provides a couple of generic ones, namely "output the contents of a file" or "yes if file exists, no if not."</p> -<div class="section" id="it-s-not-you-it-s-me"> -<h2>It's not you, it's me?</h2> -<div class="admonition warning"> -<p class="first admonition-title">Warning</p> -<p class="last"><strong>Always use a VPN! Always!!</strong></p> -</div> -<p>Now, I <em>always</em> use VPN <em>no matter what</em>, and I configure network manually. That is, I do not use a network manager. For this reason, when network applications start misbehaving, it's sometimes hard to tell whether it is because the host is gone, the local network is flaky, the internet route is flaky, or simply that the connection is gone for good.</p> -<p>Evidently, it would be practical to have <strong>an item in the status bar telling me whether I have access to internet right now or not</strong>.</p> -<p>The "yes if file exists, no if not" option seems to be the right one to go for. Basically, we need to stick an empty file in a runtime file store whenever we detect that internet is available.</p> -</div> -<div class="section" id="ping-rinse-repeat"> -<h2>Ping, rinse, repeat</h2> -<p>There's not much trickery involved here. The check is a ping to a remote location every now and then.</p> -<p>Some caveats to cover to grant it some minimum of intelligence:</p> -<ul class="simple"> -<li>We need a few alternative destinations for the ping, that can take over if others happen to be down or take too long to repond.</li> -<li>To make the script useful for general purpose use, we'd also want to be able to supply a custom list of hosts to ping.</li> -<li>We also want to make sure that when the script is interrupted, it deletes the internet indicator file. That way, when it stops running, the internet indicator will be frozen in the "no" state.</li> -<li>Lastly, it can be useful to log to syslog every time the state of the internet connection (or the script itself) changes.</li> -</ul> -<p>That can translate to a shell script like this;</p> -<pre class="code bash literal-block"> -<span class="ch">#!/bin/bash -</span> -<span class="c1"># this is our runtime file, if - which exists - tells us internet is up -</span><span class="nv">f</span><span class="o">=</span><span class="s2">"/run/user/</span><span class="nv">$UID</span><span class="s2">/probe_up"</span> -rm -f <span class="nv">$f</span> - -up<span class="o">()</span> <span class="o">{</span> - <span class="k">if</span> <span class="o">[</span> ! -f <span class="nv">$f</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - logger -p info <span class="s2">"iup up"</span> - <span class="k">fi</span> - touch <span class="nv">$f</span> -<span class="o">}</span> -down<span class="o">()</span> <span class="o">{</span> - <span class="k">if</span> <span class="o">[</span> -f <span class="nv">$f</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - logger -p info <span class="s2">"iup down"</span> - <span class="k">fi</span> - rm -f <span class="nv">$f</span> -<span class="o">}</span> - -argh<span class="o">()</span> <span class="o">{</span> - logger -p info <span class="s2">"iup die"</span> - rm -f <span class="nv">$f</span> - <span class="nb">exit</span> <span class="m">0</span> -<span class="o">}</span> - -<span class="nv">IUP_DELAY</span><span class="o">=</span><span class="si">${</span><span class="nv">IUP_DELAY</span><span class="k">:-</span><span class="nv">$1</span><span class="si">}</span> -<span class="k">if</span> <span class="o">[</span> -z <span class="nv">$IUP_DELAY</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - <span class="nv">IUP_DELAY</span><span class="o">=</span><span class="m">5</span> -<span class="k">fi</span> -logger -p debug iup started with delay <span class="nv">$IUP_DELAY</span> - -<span class="nv">hosts</span><span class="o">=()</span> -<span class="k">while</span> <span class="nb">read</span> h<span class="p">;</span> <span class="k">do</span> - <span class="nv">hosts</span><span class="o">+=(</span><span class="nv">$h</span><span class="o">)</span> -<span class="k">done</span> < <span class="nv">$HOME</span>/.config/iup <span class="m">2</span>> /dev/null -<span class="k">if</span> <span class="o">[</span> <span class="si">${#</span><span class="nv">hosts</span><span class="p">[@]</span><span class="si">}</span> -eq <span class="s2">"0"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - <span class="nv">hosts</span><span class="o">=(</span><span class="m">8</span>.8.8.8 <span class="m">151</span>.101.193.67<span class="o">)</span> <span class="c1"># google nameserver and cnn.com -</span> logger -p warn missing hosts input file, using evil default: <span class="si">${</span><span class="nv">hosts</span><span class="p">[@]</span><span class="si">}</span> -<span class="k">fi</span> - -<span class="nb">trap</span> argh SIGINT -<span class="nb">trap</span> argh SIGTERM -<span class="nb">trap</span> argh SIGQUIT - -<span class="k">while</span> true<span class="p">;</span> <span class="k">do</span> - <span class="nv">isup</span><span class="o">=</span><span class="s2">""</span> - <span class="k">for</span> h <span class="k">in</span> <span class="si">${</span><span class="nv">hosts</span><span class="p">[@]</span><span class="si">}</span><span class="p">;</span> <span class="k">do</span> - ping -c1 <span class="nv">$h</span> -w1 -q <span class="p">&</span>> /dev/null - <span class="k">if</span> <span class="o">[</span> <span class="nv">$?</span> -eq <span class="s1">'0'</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - up - <span class="nv">isup</span><span class="o">=</span><span class="m">1</span> - <span class="c1">#logger -p debug "ping success $h" -</span> <span class="nb">break</span> - <span class="k">fi</span> - <span class="k">done</span> - <span class="k">if</span> <span class="o">[</span> -z <span class="nv">$isup</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> - down - <span class="k">fi</span> - sleep <span class="nv">$IUP_DELAY</span> -<span class="k">done</span> -</pre> -<p>Since we'll want this to run when network interfaces allegedly are up, it makes sense to link it to systemd and the network target:</p> -<pre class="code ini literal-block"> -<span class="k">[Unit]</span> -<span class="na">Description</span><span class="o">=</span><span class="s">Internet connection poller</span> -<span class="na">After</span><span class="o">=</span><span class="s">multi-user.target</span> - -<span class="k">[Service]</span> -<span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/iup.sh</span> -<span class="na">Restart</span><span class="o">=</span><span class="s">always</span> - -<span class="k">[Install]</span> -<span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span> -</pre> -<p>Stick this in <code>~/.config/systemd/user/iup.service</code> and enable and start the service:</p> -<div class="highlight"><pre><span></span>$ systemctl --user <span class="nb">enable</span> iup.service -$ systemctl --user start iup.service -</pre></div> -</div> -<div class="section" id="checking-what-status-our-status-is-in"> -<h2>Checking what status our status is in</h2> -<p>Now, in the i3status configuration, assuming that your system uid is 1000 <a class="footnote-reference" href="#id3" id="id1">[1]</a>, add:</p> -<div class="highlight"><pre><span></span>order += "run_watch INET" - -kj - -run_watch INET { - pidfile = "/run/user/1000/probe_up" -} -</pre></div> -<p>To reload the config on the fly, press <code>mod+r</code> (that's <code>ctrl+r</code> on mine). The result should be a new item in your bar showing <code>INET: yes</code> (most likely yes, anyway, since you're currently reading this).</p> -</div> -<div class="section" id="safety-first-eh-second"> -<h2>Safety first, eh ... second</h2> -<div class="admonition warning"> -<p class="first admonition-title">Warning</p> -<p class="last"><strong>Did I mention that you should always use a VPN?</strong></p> -</div> -<p>The same <code>run_watch</code> trick can be used for VPN.</p> -<p>I use <code>openvpn</code>, and it defines a flag <code>--writepid</code>. If you pass a file path to this parameter, either through the <code>writepid</code> config directiry or on the <code>--writepid</code> flag on the command line, it will write the openvpn pid to that file, and similarly remove it when the openvpn service ends.</p> -<p>However, this raises another issue. Namely the fact that a location is needed for openvpn to write the file to, for which it also has access.</p> -<p>If we want to use the <code>/run</code> directory again, now we also has to make sure that this directory exists.</p> -<div class="section" id="got-the-runs"> -<h3>Got the runs?</h3> -<p>This is exactly what <code>systemd-tmpfiles</code> is for. You can add files to <code>/etc/tmpfiles.d</code> which describe what kind of temporary files or directories to add <a class="footnote-reference" href="#id4" id="id2">[2]</a>.</p> -<p>Create a file <code>/etc/tmpfiles.d/openvpn.conf</code> and add a single line to it:</p> -<div class="highlight"><pre><span></span>d /run/openvpn 0755 openvpn openvpn -</pre></div> -<p>This will ensure that the folder is created at startup.</p> -<p>Now, to create one right away for the current session, run <code>systemd-tmpfiles --create</code>.</p> -<p>Now we have everything we need to add the VPN status to <em>i3status</em> aswell. Provided that we chose <code>/run/openvpn/openvpn.pid</code> as our argument to <code>openvpn --writepid</code>, the setup is more or less the same as before:</p> -<div class="highlight"><pre><span></span>order += "run_watch VPN" - -kj - -run_watch VPN { - pidfile = "/run/openvpn/openvpn.pid" -} -</pre></div> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id3" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>The <code>/run/user/$UID</code> folder will be created on system startup, and will be writable by that user. It provides a tmpfs storage location for processes that run in early stages of boot, before the entire filesystem tree (which may include <code>/var/run</code> because <code>/var</code> may be on a separate partition) has been mounted. More in that here: <a class="reference external" href="https://lwn.net/Articles/436012/">https://lwn.net/Articles/436012/</a></td></tr> -</tbody> -</table> -</blockquote> -<!-- --> -<blockquote> -<table class="docutils footnote" frame="void" id="id4" rules="none"> -<colgroup><col class="label" /><col /></colgroup> -<tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>This service has an impressive amount of flexibility, which you can inspect yourself by visiting the <code>tmpfiles.d</code> (and <code>systemd-tmpfiles</code>) manpage.</td></tr> -</tbody> -</table> -</blockquote> -</div> -</div> - - </div><!-- /.entry-content --> - <footer class="meta"> - </footer> - -</section> - <footer id="contentinfo" class="body"> - <hr/> - <address id="about" class="vcard body"> - Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> - </address><!-- /#about --> - </footer><!-- /#contentinfo --> -</body> -</html> -\ No newline at end of file diff --git a/af20d83900b9bea7a7258e13dccec33be38aa849bf2387fcdd4ab6ab1e6069f1.asc b/af20d83900b9bea7a7258e13dccec33be38aa849bf2387fcdd4ab6ab1e6069f1.asc @@ -1,11 +0,0 @@ ------BEGIN PGP SIGNATURE----- - -iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmHkZ0UACgkQzcvSTdHQ -4AGg9Qf/RzFbohchDoQ47kuELxjZLLJW0PO3zYKRHlafSHJUSSRK0e0p363+WUgU -UbZYy6VSicVwBCPjT8pRz8192vobT+Zs8O3On87ikij3KufovJTfmdRfl6PxR4RC -kZIYCgDrktW6lEUyRM5E7FurouITEGFQahjO/X29KSXiL95Tt995SKYAMCTR/lo9 -0Wmo/ITojndNV/PrndQo55/fREiKtD1E+uElzTsMldPdYi6a0jWGdjwE50fgIkDC -VgfjHRLfIpKc3KDpywSBrqebWd1bKJbSGYDFQquuz/TcVwuH4fsHllw49xNKEmbN -LguBumV4lbxVagZtmdS26pjebsfc9A== -=tWqB ------END PGP SIGNATURE----- diff --git a/archives.html b/archives.html @@ -29,6 +29,8 @@ <h1>Archives for man bytes gnu</h1> <dl> + <dt>Sat 01 October 2022</dt> + <dd><a href="./portable-book-metadata.html">A portable book metadata exercise</a></dd> <dt>Sat 15 January 2022</dt> <dd><a href="./backup-rsync-duplicity.html">Combining duplicity and rsync</a></dd> <dt>Thu 01 July 2021</dt> diff --git a/author/louis-holbrook.html b/author/louis-holbrook.html @@ -31,6 +31,11 @@ <ol id="post-list"> <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../portable-book-metadata.html" rel="bookmark" title="Permalink to A portable book metadata exercise">A portable book metadata exercise</a></h3> </header> + <p><p class="first last">Structured approach to generate portable metadata files for bibliographies and literature files using cryptographic hash mapping.</p> +</p> + </article></li> + <li><article class="hentry"> <header> <h3 class="entry-title"><a href="../backup-rsync-duplicity.html" rel="bookmark" title="Permalink to Combining duplicity and rsync"> Organizing backups: Combining duplicity and rsync</a></h3> </header> <p><p class="first last">An exercise in combining plain and encrypted backups on local and remote hosts</p> </p> @@ -75,12 +80,12 @@ <p><p class="first last">Feeding python packages to your offline Docker setup</p> </p> </article></li> - <li><article class="hentry"> - <header> <h3 class="entry-title"><a href="../docker-offline-1-routing.html" rel="bookmark" title="Permalink to The routing to freedom"> Offline Docker: The routing to freedom</a></h3> </header> - <p><p class="first last">How to not be forced being online when forced to use docker</p> -</p> - </article></li> </ol><!-- /#posts-list --> +<p class="paginator"> + Page 1 / 2 + <a href="../author/louis-holbrook2.html">»</a> + <a href="../author/louis-holbrook2.html">⇉</a> +</p> </section><!-- /#content --> <footer id="contentinfo" class="body"> <hr/> diff --git a/author/louis-holbrook2.html b/author/louis-holbrook2.html @@ -0,0 +1,52 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="../theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="../theme/css/syntax.css" /> + <title>man bytes gnu - Articles by Louis Holbrook</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="../">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="../category/archiving.html">Archiving</a></li> + <li><a href="../category/code.html">Code</a></li> + <li><a href="../category/hygiene.html">Hygiene</a></li> + <li><a href="../category/offlining.html">Offlining</a></li> + <li><a href="../pages/identities.html">Identities</a></li> + <li><a href="../pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> +<section id="content"> +<h2>Articles by Louis Holbrook</h2> + +<ol id="post-list"> + <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../docker-offline-1-routing.html" rel="bookmark" title="Permalink to The routing to freedom"> Offline Docker: The routing to freedom</a></h3> </header> + <p><p class="first last">How to not be forced being online when forced to use docker</p> +</p> + </article></li> +</ol><!-- /#posts-list --> +<p class="paginator"> + <a href="../author/louis-holbrook.html">⇇</a> + <a href="../author/louis-holbrook.html">«</a> + Page 2 / 2 +</p> +</section><!-- /#content --> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/authors.html b/authors.html @@ -28,7 +28,7 @@ </nav><!-- /#menu --> <h1>Authors on man bytes gnu</h1> <ul> - <li><a href="./author/louis-holbrook.html">Louis Holbrook</a> (10)</li> + <li><a href="./author/louis-holbrook.html">Louis Holbrook</a> (11)</li> </ul> <footer id="contentinfo" class="body"> <hr/> diff --git a/backup-rsync-duplicity.html b/backup-rsync-duplicity.html @@ -52,6 +52,13 @@ in <a class="category" href="./category/archiving.html">archiving</a> Homemade internet state monitor </a> </li> + | + + <li> + Next: <a href="./portable-book-metadata.html"> + A portable book metadata exercise + </a> + </li> </ul> </div> <div class="entry-series meta"> @@ -66,7 +73,7 @@ Part 1 from the series "Organizing backups" <section id="content" class="body"> <div class="entry-content"> - <p>There are two awesome, weathered tools out there that are all you really need for your personal backups. <a class="footnote-reference" href="#id4" id="id1">[1]</a> One is the <a class="reference external" href="https://rsync.samba.org/">rsync cli</a>, the other is <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a>.</p> + <p>There are two awesome, weathered tools out there that are all you really need for your personal backups. <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a> One is the <a class="reference external" href="https://rsync.samba.org/">rsync cli</a>, the other is <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a>.</p> <p>The former should need no introduction.</p> <p>The latter operates more like tar. But it still works over ssh like rsync. In fact, it's based on <a class="reference external" href="http://librsync.sourcefrog.net/">librsync</a> which implements the <a class="reference external" href="https://rsync.samba.org/tech_report/">rsync protocol</a>. The special sauce however is, of course, <em>encryption</em>.</p> <div class="section" id="backup-categories"> @@ -92,13 +99,13 @@ Part 1 from the series "Organizing backups" <h2>Why not just one or the other?</h2> <p><a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">Duplicity</a> stores everything in an archive file format. That means that you must first authenticate, decrypt and unpack the archive in order to even browse the files inside.</p> <p>If there is no reason to keep the files from prying eyes, then it's much more practical to be able to browse the files where they lie, with the regular filesystem tools. In such a case, <a class="reference external" href="https://rsync.samba.org/">rsync</a> will scratch your itch.</p> -<p>For the <strong>sensitive</strong> and <strong>secret stuff</strong>, there would be no real need to use <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a> if you were only operating on your local host. You'd just use an encrypted volume <a class="footnote-reference" href="#id5" id="id2">[2]</a> and <a class="reference external" href="https://rsync.samba.org/">rsync</a> everything in there.</p> +<p>For the <strong>sensitive</strong> and <strong>secret stuff</strong>, there would be no real need to use <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a> if you were only operating on your local host. You'd just use an encrypted volume <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a> and <a class="reference external" href="https://rsync.samba.org/">rsync</a> everything in there.</p> <p>But half the point here is to keep remote copies aswell as your local ones. You know, in case of fire, hardware-eating locust swarms or some totalitarian minions nabbing all your electronics. Unless "remote" here means some box hidden in some moated leisure castle of yours, you'll want to encrypt everything <em>before</em> you ship it off. And that's where <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a> comes in.</p> </div> <div class="section" id="vive-la-difference"> <h2>Vive la difference</h2> <p>Of course, it would be too much to hope for that <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a> and <a class="reference external" href="https://rsync.samba.org/">rsync cli</a> have aligned the ways they parse their invocation parameters.</p> -<p>Here are some examples <a class="footnote-reference" href="#id6" id="id3">[3]</a> of how they do <em>not</em> match:</p> +<p>Here are some examples <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a> of how they do <em>not</em> match:</p> <div class="section" id="local-to-local"> <h3>local to local</h3> <div class="highlight"><pre><span></span>$ rsync -a src/ /path/to/dst/ @@ -142,7 +149,7 @@ $ duplicity -v debug <p>Since you will want to select up front which tool to use for which sensititivy category, you'll be writing the includes and excludes specifically for the tool anyway.</p> <p>So the only real issue with the above is the way remote host is specified.</p> <p>Let's say we choose to stick to the <a class="reference external" href="https://rsync.samba.org/">rsync cli</a> host format. That means we need to make the following translations:</p> -<table border="1" class="colwidths-given docutils"> +<table border="1" class="docutils"> <colgroup> <col width="50%" /> <col width="50%" /> @@ -276,22 +283,22 @@ duplicity -v info secret/ <span class="nv">$remote_duplicity_base</span>:secret/ </ul> <!-- --> <blockquote> -<table class="docutils footnote" frame="void" id="id4" rules="none"> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Ok, I know, I assuming that you are using <tt class="docutils literal">git</tt> in daily life, too.</td></tr> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Ok, I know, I assuming that you are using <tt class="docutils literal">git</tt> in daily life, too.</td></tr> </tbody> </table> -<table class="docutils footnote" frame="void" id="id5" rules="none"> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Provided, of course, that it's an encrypted volume that you don't keep unlocked all the time.</td></tr> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>Provided, of course, that it's an encrypted volume that you don't keep unlocked all the time.</td></tr> </tbody> </table> -<table class="docutils footnote" frame="void" id="id6" rules="none"> +<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> -<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>Duplicity needs at a minimum a password for symmetric encryption, and will prompt for it unless it's set in the environment. Simply <tt class="docutils literal">export PASSPHRASE=test</tt> for these examples to relieve you of the annoyance.</td></tr> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>Duplicity needs at a minimum a password for symmetric encryption, and will prompt for it unless it's set in the environment. Simply <tt class="docutils literal">export PASSPHRASE=test</tt> for these examples to relieve you of the annoyance.</td></tr> </tbody> </table> </blockquote> diff --git a/backup-rsync-duplicity.sha256 b/backup-rsync-duplicity.sha256 @@ -1 +1 @@ -1c19dbdeef6f5dea9dfc688b2aff4784fe2c5dbb2a8af928974d41c536f6afdd backup-rsync-duplicity.html -\ No newline at end of file +c38ce3a51eeaa98f5524cf9902e84ee8ba5d179d6fbe18c49f9707554c896b6e backup-rsync-duplicity.html +\ No newline at end of file diff --git a/c38ce3a51eeaa98f5524cf9902e84ee8ba5d179d6fbe18c49f9707554c896b6e b/c38ce3a51eeaa98f5524cf9902e84ee8ba5d179d6fbe18c49f9707554c896b6e @@ -0,0 +1,322 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - Combining duplicity and rsync</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li class="active"><a href="./category/archiving.html">Archiving</a></li> + <li><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./backup-rsync-duplicity.html" rel="bookmark" + title="Permalink to Combining duplicity and rsync">Organizing backups: Combining duplicity and rsync</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2022-01-15T16:57:00+01:00"> + Sat 15 January 2022 + </time> +in <a class="category" href="./category/archiving.html">archiving</a> + <a href="./tag/backup.html">backup</a> + <a href="./tag/rsync.html">rsync</a> + <a href="./tag/duplicity.html">duplicity</a> + <a href="./tag/bash.html">bash</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./internet-up-monitor.html"> + Homemade internet state monitor + </a> + </li> + | + + <li> + Next: <a href="./portable-book-metadata.html"> + A portable book metadata exercise + </a> + </li> + </ul> + </div> + <div class="entry-series meta"> +Part 1 from the series "Organizing backups" + <ol class="entry-series-parts"> + </ol> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>There are two awesome, weathered tools out there that are all you really need for your personal backups. <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a> One is the <a class="reference external" href="https://rsync.samba.org/">rsync cli</a>, the other is <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a>.</p> +<p>The former should need no introduction.</p> +<p>The latter operates more like tar. But it still works over ssh like rsync. In fact, it's based on <a class="reference external" href="http://librsync.sourcefrog.net/">librsync</a> which implements the <a class="reference external" href="https://rsync.samba.org/tech_report/">rsync protocol</a>. The special sauce however is, of course, <em>encryption</em>.</p> +<div class="section" id="backup-categories"> +<h2>Backup categories</h2> +<p>Let's for the sake of argument say that our personal backups can be divided in three categories:</p> +<div class="section" id="stuff-that-can-be-public"> +<h3>Stuff that can be public</h3> +<p>Code snippets, git repositories, public data store states (e.g. blockchain ledgers), copies of OS packages and any other assets assets without redistribution issues.</p> +<p>For this we will use <a class="reference external" href="https://rsync.samba.org/">rsync</a>.</p> +</div> +<div class="section" id="sensitive-stuff"> +<h3>Sensitive stuff</h3> +<p>Passwords, keys, contacts, calendars, contracts, invoices, task lists, databases, system configurations, application data.</p> +<p>For this we will use <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a>.</p> +</div> +<div class="section" id="secret-stuff"> +<h3>Secret stuff</h3> +<p>Long-lived keys, password- and volume decryption keys, cryptocurrency keys and meta-information about the backups themselves.</p> +<p>This will not be addressed now.</p> +</div> +</div> +<div class="section" id="why-not-just-one-or-the-other"> +<h2>Why not just one or the other?</h2> +<p><a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">Duplicity</a> stores everything in an archive file format. That means that you must first authenticate, decrypt and unpack the archive in order to even browse the files inside.</p> +<p>If there is no reason to keep the files from prying eyes, then it's much more practical to be able to browse the files where they lie, with the regular filesystem tools. In such a case, <a class="reference external" href="https://rsync.samba.org/">rsync</a> will scratch your itch.</p> +<p>For the <strong>sensitive</strong> and <strong>secret stuff</strong>, there would be no real need to use <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a> if you were only operating on your local host. You'd just use an encrypted volume <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a> and <a class="reference external" href="https://rsync.samba.org/">rsync</a> everything in there.</p> +<p>But half the point here is to keep remote copies aswell as your local ones. You know, in case of fire, hardware-eating locust swarms or some totalitarian minions nabbing all your electronics. Unless "remote" here means some box hidden in some moated leisure castle of yours, you'll want to encrypt everything <em>before</em> you ship it off. And that's where <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a> comes in.</p> +</div> +<div class="section" id="vive-la-difference"> +<h2>Vive la difference</h2> +<p>Of course, it would be too much to hope for that <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a> and <a class="reference external" href="https://rsync.samba.org/">rsync cli</a> have aligned the ways they parse their invocation parameters.</p> +<p>Here are some examples <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a> of how they do <em>not</em> match:</p> +<div class="section" id="local-to-local"> +<h3>local to local</h3> +<div class="highlight"><pre><span></span>$ rsync -a src/ /path/to/dst/ + +$ duplicity src/ file:///path/to/dst/ +</pre></div> +</div> +<div class="section" id="local-to-remote-relative-path"> +<h3>local to remote, relative path</h3> +<div class="highlight"><pre><span></span>$ rsync -a src/ user@remotehost:path/to/dst/ + +$ duplicity src/ scp://user@remotehost/path/to/dst +</pre></div> +</div> +<div class="section" id="toggle-dotfiles-from-current-path"> +<h3>toggle dotfiles from current path</h3> +<div class="highlight"><pre><span></span><span class="c1"># include only .foo/foo.txt given the current structure:</span> +$ tree src/ -a +src/ +├── .bar +├── baz +└── .foo + └── foo.txt + +$ rsync --exclude<span class="o">=</span><span class="s2">".b*"</span> --include<span class="o">=</span><span class="s2">".*/***"</span> --exclude<span class="o">=</span><span class="s2">"*"</span> ./ ../dst/ + +$ duplicity --exclude<span class="o">=</span><span class="s2">"./.b*"</span> --include<span class="o">=</span><span class="s2">"./.*/***"</span> --exclude<span class="o">=</span><span class="s2">"*"</span> ./ file:///home/lash/tmp/dst/ +</pre></div> +</div> +<div class="section" id="logging"> +<h3>logging</h3> +<div class="highlight"><pre><span></span><span class="c1"># spill the beans</span> +$ rsync -vv ... + +$ duplicity -v debug +</pre></div> +</div> +</div> +<div class="section" id="batchin"> +<h2>Batchin'</h2> +<p>Since you will want to select up front which tool to use for which sensititivy category, you'll be writing the includes and excludes specifically for the tool anyway.</p> +<p>So the only real issue with the above is the way remote host is specified.</p> +<p>Let's say we choose to stick to the <a class="reference external" href="https://rsync.samba.org/">rsync cli</a> host format. That means we need to make the following translations:</p> +<table border="1" class="docutils"> +<colgroup> +<col width="50%" /> +<col width="50%" /> +</colgroup> +<thead valign="bottom"> +<tr><th class="head">rsync</th> +<th class="head">duplicity</th> +</tr> +</thead> +<tbody valign="top"> +<tr><td><tt class="docutils literal">foo/bar</tt></td> +<td><tt class="docutils literal"><span class="pre">file://foo/bar</span></tt></td> +</tr> +<tr><td><tt class="docutils literal">/foo/bar</tt></td> +<td><tt class="docutils literal"><span class="pre">file:///foo/bar</span></tt></td> +</tr> +<tr><td><tt class="docutils literal">user@host:foo/bar</tt></td> +<td><tt class="docutils literal"><span class="pre">scp://user@host/foo/bar</span></tt></td> +</tr> +<tr><td><tt class="docutils literal"><span class="pre">user@host:/foo/bar</span></tt></td> +<td><tt class="docutils literal"><span class="pre">scp://user@host//foo/bar</span></tt></td> +</tr> +</tbody> +</table> +<p>Expressed in <tt class="docutils literal">bash</tt> that could look like this:</p> +<pre class="code bash literal-block"> +to_duplicity_remote<span class="o">()</span> <span class="o">{</span> + + <span class="c1"># remote host is defined in rsync format +</span> <span class="c1"># ... and we will only support scp +</span> <span class="c1"># $remote_base is the path we want to parse +</span> <span class="nv">remote_duplicity_base</span><span class="o">=</span> + <span class="nv">remote_base</span><span class="o">=</span><span class="nv">$1</span> + + <span class="c1"># substring up until the first slash +</span> <span class="nv">s_firstslash</span><span class="o">=</span><span class="si">${</span><span class="nv">remote_base</span><span class="p">%%/*</span><span class="si">}</span> + + <span class="c1"># substring up until the first colon +</span> <span class="nv">s_firstcolon</span><span class="o">=</span><span class="si">${</span><span class="nv">remote_base</span><span class="p">%%:*</span><span class="si">}</span> + + <span class="c1"># string index of the first slash +</span> <span class="nv">i_firstslash</span><span class="o">=</span><span class="k">$((</span><span class="si">${#</span><span class="nv">s_firstslash</span><span class="si">}</span><span class="k">))</span> + + <span class="c1"># string index of the first colon +</span> <span class="nv">i_firstcolon</span><span class="o">=</span><span class="k">$((</span><span class="si">${#</span><span class="nv">s_firstcolon</span><span class="si">}</span><span class="k">))</span> + + <span class="c1"># if colon is before first slash that most likely means we have a remote host +</span> <span class="c1"># (Exception is if first directory of path happens to have ":" in it. Seriously, don't use ":" in filenames) +</span> <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$i_firstcolon</span><span class="s2">"</span> -gt <span class="s2">"0"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + + <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$i_firstcolon</span><span class="s2">"</span> -lt <span class="s2">"</span><span class="nv">$i_firstslash</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + + <span class="c1"># pexpect addition due to lack of implicit private key fetch, without pexpect works only with key wo pwd +</span> <span class="c1"># https://serverfault.com/questions/982591/duplicity-backup-fails-private-key-file-is-encrypted +</span> <span class="nv">remote_duplicity_base</span><span class="o">=</span><span class="s2">"pexpect+scp://"</span> + + <span class="c1"># trim url so that colon after hostname is removed +</span> <span class="c1"># (no support here for setting an alternate port number) +</span> <span class="nv">remote_duplicity_base</span><span class="o">=</span><span class="si">${</span><span class="nv">remote_duplicity_base</span><span class="si">}${</span><span class="nv">remote_base</span><span class="p">:</span><span class="nv">0</span><span class="p">:</span><span class="nv">$i_firstcolon</span><span class="si">}</span> + <span class="nv">remote_duplicity_base</span><span class="o">=</span><span class="si">${</span><span class="nv">remote_duplicity_base</span><span class="si">}</span>/<span class="si">${</span><span class="nv">remote_base</span><span class="p">:((</span><span class="nv">$i_firstcolon</span><span class="p">+1))</span><span class="si">}</span> + + <span class="c1"># indicate that we have a remote host +</span> <span class="nv">remote</span><span class="o">=</span><span class="m">1</span> + <span class="k">fi</span> + <span class="k">fi</span> + + <span class="c1"># If it's not a remote host, treat it as a file +</span> <span class="k">if</span> <span class="o">[</span> -z <span class="nv">$remote_duplicity_base</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + <span class="nv">remote_duplicity_base</span><span class="o">=</span><span class="s2">"file://</span><span class="si">${</span><span class="nv">remote_base</span><span class="si">}</span><span class="s2">"</span> + <span class="k">fi</span> +<span class="o">}</span> + +<span class="k">if</span> <span class="o">[</span> ! -z <span class="s2">"</span><span class="nv">$BAK_TEST</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + <span class="nv">src</span><span class="o">=(</span>/foo/bar foo/bar localhost:foo/bar localhost:/foo/bar<span class="o">)</span> + <span class="nv">res</span><span class="o">=(</span>file:///foo/bar file://foo/bar pexpect+scp://localhost/foo/bar pexpect+scp://localhost//foo/bar<span class="o">)</span> + + <span class="nv">i</span><span class="o">=</span><span class="m">0</span> + <span class="k">for</span> case_src <span class="k">in</span> <span class="si">${</span><span class="nv">src</span><span class="p">[@]</span><span class="si">}</span><span class="p">;</span> <span class="k">do</span> + <span class="nv">case_res</span><span class="o">=</span><span class="si">${</span><span class="nv">res</span><span class="p">[</span><span class="nv">$i</span><span class="p">]</span><span class="si">}</span> + + to_duplicity_remote <span class="nv">$case_src</span> + <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$remote_duplicity_base</span><span class="s2">"</span> !<span class="o">=</span> <span class="s2">"</span><span class="nv">$case_res</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> <span class="s2">"expected </span><span class="nv">$case_res</span><span class="s2"> got </span><span class="nv">$remote_duplicity_base</span><span class="s2"> from </span><span class="nv">$case_src</span><span class="s2">"</span> + <span class="nb">exit</span> <span class="m">1</span> + <span class="k">elif</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$remote_base</span><span class="s2">"</span> !<span class="o">=</span> <span class="s2">"</span><span class="nv">$case_src</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$case_src</span><span class="s2"> got mangled into </span><span class="nv">$remote_base</span><span class="s2">"</span> + <span class="nb">exit</span> <span class="m">1</span> + <span class="k">fi</span> + <span class="nv">i</span><span class="o">=</span><span class="k">$((</span>i+1<span class="k">))</span> + <span class="k">done</span> +<span class="k">fi</span> +</pre> +<p>Let's behave and test our code:</p> +<pre class="code bash literal-block"> +<span class="k">if</span> <span class="o">[</span> ! -z <span class="s2">"</span><span class="nv">$BAK_TEST</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + <span class="nv">src</span><span class="o">=(</span>/foo/bar foo/bar localhost:foo/bar localhost:/foo/bar<span class="o">)</span> + <span class="nv">res</span><span class="o">=(</span>file:///foo/bar file://foo/bar pexpect+scp://localhost/foo/bar pexpect+scp://localhost//foo/bar<span class="o">)</span> + + <span class="nv">i</span><span class="o">=</span><span class="m">0</span> + <span class="k">for</span> case_src <span class="k">in</span> <span class="si">${</span><span class="nv">src</span><span class="p">[@]</span><span class="si">}</span><span class="p">;</span> <span class="k">do</span> + <span class="nv">case_res</span><span class="o">=</span><span class="si">${</span><span class="nv">res</span><span class="p">[</span><span class="nv">$i</span><span class="p">]</span><span class="si">}</span> + + to_duplicity_remote <span class="nv">$case_src</span> + <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$remote_duplicity_base</span><span class="s2">"</span> !<span class="o">=</span> <span class="s2">"</span><span class="nv">$case_res</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> <span class="s2">"expected </span><span class="nv">$case_res</span><span class="s2"> got </span><span class="nv">$remote_duplicity_base</span><span class="s2"> from </span><span class="nv">$case_src</span><span class="s2">"</span> + <span class="nb">exit</span> <span class="m">1</span> + <span class="k">elif</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$remote_base</span><span class="s2">"</span> !<span class="o">=</span> <span class="s2">"</span><span class="nv">$case_src</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> + ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$case_src</span><span class="s2"> got mangled into </span><span class="nv">$remote_base</span><span class="s2">"</span> + <span class="nb">exit</span> <span class="m">1</span> + <span class="k">fi</span> + <span class="nv">i</span><span class="o">=</span><span class="k">$((</span>i+1<span class="k">))</span> + <span class="k">done</span> +<span class="k">fi</span> +</pre> +<div class="highlight"><pre><span></span><span class="c1"># 0 == good!</span> +$ <span class="nv">BAK_TEST</span><span class="o">=</span><span class="m">1</span> bash remote.sh <span class="o">&&</span> <span class="nb">echo</span> <span class="nv">$?</span> +<span class="m">0</span> +</pre></div> +<p>Now we can use the <a class="reference external" href="https://rsync.samba.org/">rsync cli</a> path input, and use that same input to a batch of single backup steps, each which may use <a class="reference external" href="https://rsync.samba.org/">rsync cli</a> or <a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/">duplicity</a></p> +<div class="highlight"><pre><span></span>to_duplicity_remote localhost:/foo/bar + +rsync -avzP pub/ <span class="nv">$remote_base</span>:src/ + +duplicity -v info secret/ <span class="nv">$remote_duplicity_base</span>:secret/ +</pre></div> +</div> +<div class="section" id="see-also"> +<h2>See also</h2> +<ul class="simple"> +<li><a class="reference external" href="https://git.defalsify.org/rsync-duplicity-backups">https://git.defalsify.org/rsync-duplicity-backups</a></li> +</ul> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Ok, I know, I assuming that you are using <tt class="docutils literal">git</tt> in daily life, too.</td></tr> +</tbody> +</table> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>Provided, of course, that it's an encrypted volume that you don't keep unlocked all the time.</td></tr> +</tbody> +</table> +<table class="docutils footnote" frame="void" id="footnote-3" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>Duplicity needs at a minimum a password for symmetric encryption, and will prompt for it unless it's set in the environment. Simply <tt class="docutils literal">export PASSPHRASE=test</tt> for these examples to relieve you of the annoyance.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/c38ce3a51eeaa98f5524cf9902e84ee8ba5d179d6fbe18c49f9707554c896b6e.asc b/c38ce3a51eeaa98f5524cf9902e84ee8ba5d179d6fbe18c49f9707554c896b6e.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NzsACgkQzcvSTdHQ +4AEroAf/cwRW5+lTqHl8wud8DdUasDkAb7QRqImVdfMG506e2S6JtG1ImB/g/Y3x +nXxG0m+jzJTVUocG9qzcntkxvIoZ6j4BY5/FY4hia4+jcm0rstNZ+Cu69LhsVA+1 +uM4igcEKFqh8uRI8Znxu3N6HuamLtfFFCon34Jv6hboM50CDfT0FGlCkmuhu9heO +7oF6/zODYlu4giY0AC7z7Mv6SFJRMekNT1jgS2PhcSLyvzmjeveMd6hu+XISWkTL +vYFpyWrkpgwAtqr5G9u82NIzT0C6suCoYrqjq8M3t1Ys2Fqjg6cTSfHKfT3VCiYa +Q/Qa9WN1hIDLOMbsEjgsqPkv5it0XQ== +=pca1 +-----END PGP SIGNATURE----- diff --git a/ca197565fb955f2328bcc11540521adebeb2d0f95d6501901516eb76fb25707c b/ca197565fb955f2328bcc11540521adebeb2d0f95d6501901516eb76fb25707c @@ -0,0 +1,237 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - A portable book metadata exercise</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li class="active"><a href="./category/archiving.html">Archiving</a></li> + <li><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./portable-book-metadata.html" rel="bookmark" + title="Permalink to A portable book metadata exercise">A portable book metadata exercise</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2022-10-01T12:40:00+02:00"> + Sat 01 October 2022 + </time> +in <a class="category" href="./category/archiving.html">archiving</a> + <a href="./tag/hash.html">hash</a> + <a href="./tag/kitab.html">kitab</a> + <a href="./tag/literature.html">literature</a> + <a href="./tag/metadata.html">metadata</a> + <a href="./tag/dublincore.html">dublincore</a> + <a href="./tag/libgen.html">libgen</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./backup-rsync-duplicity.html"> + Combining duplicity and rsync + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>One of the things I have been working on the last few weeks is a rust application I have dubbed <a class="reference external" href="https://git.defalsify.net/kitab">kitab</a>.</p> +<p>In short, the application makes it easy to extract literary metadata to a separate file structure.</p> +<p>The metadata can in turn be applied as <em>extended attributes</em> recursively on a directory for files that match.</p> +<p>The way it's accomplished it simple: The file name of the metadata is the hex representation of the digest of the file. The same digest is used to match files to metadata when applying it back to the file.</p> +<p>There are two advantages to this:</p> +<ol class="arabic simple"> +<li>The digest of the media file need not be affected by the metadata, i.e. by embedding metadata in the file itself.</li> +<li>You do not need to use the file name to keep record of what a file is.</li> +</ol> +<div class="section" id="yarr-ye-matey-data"> +<h2>Yarr, ye matey-data</h2> +<p>Let's demonstrate with an example.</p> +<p>The fabulous <a class="reference external" href="https://libgen.rs">Library Genesis</a> project has made available an endpoint to retrieve <tt class="docutils literal">bibtex</tt> entries based on the <tt class="docutils literal">md5</tt> hash of the book media file.</p> +<p>A version of the <a class="reference external" href="https://libgen.rs/book/index.php?md5=BCD99F1AB4155F2A2A362E5B7938A852">Bitcoin White Paper</a>, under the <code>md5</code> hash <code>bcd99f1ab4155f2a2a362e5b7938a852</code>, can be found there.</p> +<p>If you download this file using a synchronous download link, the browser will provide you with a filename to go with the download.</p> +<p>However, if you use the torrent alternative, the filename will be the <tt class="docutils literal">md5</tt> hash itself. If you are torrenting a bunch of those files, it quickly becomes a nuisance to distinguish them.</p> +<p>And, of course: In either case there is no guarantee the any metadata comes with the file.</p> +<div class="section" id="inside-the-book"> +<h3>Inside the book</h3> +<p>Kitab (v0.0.2) is able to read metadata from both a bibtex source and xattr entries on a file, as well as its native <a class="reference external" href="https://www.w3.org/TR/turtle/">rdf-turtle</a> format.</p> +<p>In kitab's data store, every media file entity in rdf-turtle is keyed with a <a class="reference external" href="https://www.rfc-editor.org/info/rfc8141">URN</a> specifying a digest for the file.</p> +<p>To see exactly what that looks like, let's download and import the bibtex metadata for the paper <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>:</p> +<pre class="code bash literal-block"> +<span class="nv">bibtex_file</span><span class="o">=</span><span class="sb">`</span>mktemp<span class="sb">`</span> +<span class="nv">kitab_dir</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> +curl -s -X GET https://libgen.rs/book/bibtex.php?md5<span class="o">=</span>BCD99F1AB4155F2A2A362E5B7938A852 -o <span class="nv">$bibtex_file</span> +kitab --store <span class="nv">$kitab_dir</span> import --digest md5:BCD99F1AB4155F2A2A362E5B7938A852 <span class="nv">$bibtex_file</span> +cat <span class="nv">$kitab_dir</span>/* +</pre> +<p>The output of the above should be:</p> +<pre class="code turtle literal-block"> +<span class="nv"><URN:md5:bcd99f1ab4155f2a2a362e5b7938a852></span> <span class="nv"><https://purl.org/dc/terms/title></span> <span class="s">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> +<span class="nv"><https://purl.org/dc/terms/creator></span> <span class="s">"Satoshi Nakamoto"</span> <span class="p">;</span> +<span class="nv"><https://purl.org/dc/terms/type></span> <span class="s">"book"</span> <span class="p">.</span> +</pre> +<p>Now let's say the media file itself has been downloaded to <tt class="docutils literal"><span class="pre">~/.local/share/transmission</span></tt>. We can apply this metadata as extended attributes.</p> +<p>This time we turn on logging to see what's going on:</p> +<pre class="code console literal-block"> +<span class="gp">$ </span><span class="nv">RUST_LOG</span><span class="o">=</span>info kitab --store <span class="nv">$kitab_dir</span> apply --digest md5 ~/.local/share/transmission +<span class="go">[2022-10-01T11:14:59Z INFO kitab] have index directory "/tmp/tmp.r0jBm6q4hW" +[2022-10-01T11:14:59Z INFO kitab] using digest type md5 +[2022-10-01T11:14:59Z INFO kitab] apply from path "/home/lash/.local/share/transmission/" +[2022-10-01T11:14:59Z INFO kitab] apply DirEntry("/home/lash/.local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852") -> title "Bitcoin: A Peer-to-Peer Electronic Cash System" author "Satoshi Nakamoto" digest md5:bcd99f1ab4155f2a2a362e5b7938a852 + +</span><span class="gp">$ </span>find ~/.local/share/transmission -type f -regextype sed -regex <span class="s2">".*/[a-f0-9]\{32\}</span>$<span class="s2">"</span> -exec getfattr -d <span class="o">{}</span> <span class="se">\;</span> +<span class="gp"># </span>file: .local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852 +<span class="go">user.dcterms:creator="Satoshi Nakamoto" +user.dcterms:title="Bitcoin: A Peer-to-Peer Electronic Cash System" +user.dcterms:type="book"</span> +</pre> +</div> +<div class="section" id="let-the-right-one-in"> +<h3>Let the right one in</h3> +<p>Conversely, the metadata can be re-imported directly from the extended attributes. And this time, let's store it both under the <tt class="docutils literal">md5</tt> and the <tt class="docutils literal">sha512</tt> hash:</p> +<pre class="code bash literal-block"> +$ <span class="nv">kitab_dir_new</span><span class="o">=</span><span class="sb">`</span>mktemp -d<span class="sb">`</span> +$ kitab --store <span class="nv">$kitab_dir_new</span> import --digest md5 --digest sha512 .local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852 +$ find <span class="nv">$kitab_dir_new</span> -type f -exec cat <span class="o">{}</span> <span class="se">\;</span> +/tmp/tmp.B6j41YMmEM/493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792 +<URN:sha512:493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792> <https://purl.org/dc/terms/title> <span class="s2">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> + <https://purl.org/dc/terms/creator> <span class="s2">"Satoshi Nakamoto"</span> <span class="p">;</span> + <https://purl.org/dc/terms/type> <span class="s2">"book"</span> <span class="p">;</span> + <https://purl.org/dc/terms/MediaType> <span class="s2">"application/epub+zip"</span> . +/tmp/tmp.B6j41YMmEM/bcd99f1ab4155f2a2a362e5b7938a852 +<URN:md5:bcd99f1ab4155f2a2a362e5b7938a852> <https://purl.org/dc/terms/title> <span class="s2">"Bitcoin: A Peer-to-Peer Electronic Cash System"</span> <span class="p">;</span> + <https://purl.org/dc/terms/creator> <span class="s2">"Satoshi Nakamoto"</span> <span class="p">;</span> + <https://purl.org/dc/terms/type> <span class="s2">"book"</span> <span class="p">;</span> + <https://purl.org/dc/terms/MediaType> <span class="s2">"application/epub+zip"</span> . +</pre> +</div> +</div> +<div class="section" id="level-up"> +<h2>Level up</h2> +<p>Finally, a bash script <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a> example that lets you retrieve and apply metadata for a batch of files found in the directory given as the <em>first positional arg</em>.</p> +<p>This script even renames the files according to the metadata applied.</p> +<pre class="code bash literal-block"> +<span class="ln"> 0 </span><span class="c1"># NOTE! this will only work if your fs supports xattr. +</span><span class="ln"> 1 </span><span class="c1"># That's why we cannot use tmpfs (mktemp) here; tmpfs does not support xattr. +</span><span class="ln"> 2 </span><span class="c1"></span> +<span class="ln"> 3 </span><span class="c1"># directory to copy media files to +</span><span class="ln"> 4 </span><span class="c1"></span><span class="nv">outdir</span><span class="o">=</span>./<span class="k">$(</span>uuidgen<span class="k">)</span> +<span class="ln"> 5 </span>mkdir -vp <span class="nv">$outdir</span> +<span class="ln"> 6 </span> +<span class="ln"> 7 </span><span class="c1"># Input dir is the first positional arg. +</span><span class="ln"> 8 </span><span class="c1"></span><span class="nv">indir</span><span class="o">=</span><span class="nv">$1</span> +<span class="ln"> 9 </span> +<span class="ln">10 </span><span class="nv">IFS</span><span class="o">=</span><span class="s1">$'\n'</span> +<span class="ln">11 </span> +<span class="ln">12 </span><span class="c1"># Retrieve metadata for each file and import it into. +</span><span class="ln">13 </span><span class="c1"># Also copy the media file to the separate output directory. +</span><span class="ln">14 </span><span class="c1"></span><span class="k">for</span> f <span class="k">in</span> <span class="k">$(</span>find <span class="nv">$indir</span> -type f<span class="k">)</span><span class="p">;</span> <span class="k">do</span> +<span class="ln">15 </span> <span class="nv">sum</span><span class="o">=</span><span class="k">$(</span>md5sum <span class="nv">$f</span> <span class="p">|</span> awk <span class="s1">'{print $1;}'</span><span class="k">)</span> +<span class="ln">16 </span> <span class="nb">echo</span> <span class="s2">"downloading metadata for </span><span class="nv">$indir</span><span class="s2">/</span><span class="nv">$f</span><span class="s2">"</span> +<span class="ln">17 </span> <span class="nv">srct</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span> +<span class="ln">18 </span> curl -s -X GET https://libgen.rs/book/bibtex.php?md5<span class="o">=</span><span class="nv">$sum</span> -o <span class="nv">$srct</span> +<span class="ln">19 </span> <span class="nv">dstt</span><span class="o">=</span><span class="k">$(</span>mktemp<span class="k">)</span> +<span class="ln">20 </span> xmllint --html --xpath <span class="s1">'string(/html/body/textarea[@id="bibtext"])'</span> <span class="nv">$srct</span> > <span class="nv">$dstt</span> +<span class="ln">21 </span> kitab import --digest md5:<span class="nv">$sum</span> <span class="nv">$dstt</span> +<span class="ln">22 </span> cp <span class="nv">$f</span> <span class="nv">$outdir</span>/ +<span class="ln">23 </span><span class="k">done</span> +<span class="ln">24 </span> +<span class="ln">25 </span><span class="c1"># Apply metadata imported from bibtex as xattr for the media files. +</span><span class="ln">26 </span><span class="c1"></span><span class="nv">RUST_LOG</span><span class="o">=</span>info kitab apply --digest md5 <span class="nv">$outdir</span>/ +<span class="ln">27 </span> +<span class="ln">28 </span><span class="c1"># Rename the files according to the metadata title and media type. +</span><span class="ln">29 </span><span class="c1"></span><span class="k">for</span> f <span class="k">in</span> <span class="k">$(</span>ls <span class="nv">$outdir</span><span class="k">)</span><span class="p">;</span> <span class="k">do</span> +<span class="ln">30 </span> <span class="nv">title</span><span class="o">=</span><span class="k">$(</span>getfattr --only-values -n user.dcterms:title <span class="nv">$outdir</span>/<span class="nv">$f</span><span class="k">)</span> +<span class="ln">31 </span> +<span class="ln">32 </span> <span class="nv">f_typ</span><span class="o">=</span><span class="k">$(</span>file -b --mime-type <span class="nv">$outdir</span>/<span class="nv">$f</span><span class="k">)</span> +<span class="ln">33 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">""</span> +<span class="ln">34 </span> <span class="k">case</span> <span class="s2">"</span><span class="nv">$f_typ</span><span class="s2">"</span> <span class="k">in</span> +<span class="ln">35 </span> <span class="s2">"application/pdf"</span><span class="o">)</span> +<span class="ln">36 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".pdf"</span> +<span class="ln">37 </span> <span class="p">;;</span> +<span class="ln">38 </span> <span class="s2">"application/epub+zip"</span><span class="o">)</span> +<span class="ln">39 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".epub"</span> +<span class="ln">40 </span> <span class="p">;;</span> +<span class="ln">41 </span> <span class="s2">"application/x-mobipocket-ebook"</span><span class="o">)</span> +<span class="ln">42 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".mobi"</span> +<span class="ln">43 </span> <span class="p">;;</span> +<span class="ln">44 </span> <span class="s2">"text/plain"</span><span class="o">)</span> +<span class="ln">45 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".txt"</span> +<span class="ln">46 </span> <span class="p">;;</span> +<span class="ln">47 </span> <span class="s2">"text/html"</span><span class="o">)</span> +<span class="ln">48 </span> <span class="nv">f_ext</span><span class="o">=</span><span class="s2">".html"</span> +<span class="ln">49 </span> <span class="p">;;</span> +<span class="ln">50 </span> *<span class="o">)</span> +<span class="ln">51 </span> ><span class="p">&</span><span class="m">2</span> <span class="nb">echo</span> unhandled mime <span class="nb">type</span> <span class="nv">$f_typ</span> +<span class="ln">52 </span> <span class="nb">exit</span> <span class="m">1</span> +<span class="ln">53 </span> <span class="k">esac</span> +<span class="ln">54 </span> mv -v <span class="nv">$outdir</span>/<span class="nv">$f</span> <span class="nv">$outdir</span>/<span class="si">${</span><span class="nv">title</span><span class="si">}${</span><span class="nv">f_ext</span><span class="si">}</span> +<span class="ln">55 </span><span class="k">done</span> +</pre> +<p>This last example will result in:</p> +<ul class="simple"> +<li>A media file named <tt class="docutils literal">$outdir/Bitcoin: A <span class="pre">Peer-to-Peer</span> Electronic Cash System.epub</tt></li> +<li>... with metadata applied as extended attributes</li> +<li>An rdf-turtle metadata entry in <tt class="docutils literal"><span class="pre">~/.local/share/kitab/idx/bcd99f1ab4155f2a2a362e5b7938a852</span></tt></li> +</ul> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>The <code>kitab</code> command in the script assumes you have built the <em>kitab binary</em> and made it available in your path.</td></tr> +</tbody> +</table> +</blockquote> +<!-- --> +<blockquote> +<table class="docutils footnote" frame="void" id="footnote-2" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>the script uses <cite>xmllint</cite> which on archlinux is provided by the <tt class="docutils literal">libxml2</tt> package.</td></tr> +</tbody> +</table> +</blockquote> +</div> + + </div><!-- /.entry-content --> + <footer class="meta"> + </footer> + +</section> + <footer id="contentinfo" class="body"> + <hr/> + <address id="about" class="vcard body"> + Powered by <a title="Pelican is a static site generator" href="https://getpelican.com/">Pelican</a> | <a title="manbytesgnu.com is licensed under Creative Commons Attribution-ShareAlike 4.0 International" href="https://holbrook.no/share/licenses/cc/cc-by-sa-4.0.txt">CC-BY-SA 4.0</a> | <a title="GIT site history since jan 15th 2022" href="https://git.defalsify.org/manbytesgnu_site">Site history</a> | ETH <a href="./images/donate_eth_qr.png" title="Ethereum address for tipping">0x185Cbce7650FF7Ad3B587E26B2877d95568805e3</a> | BTC <a href="./images/donate_btc_qr.png" title="Bitcoin address for tipping">12DnRH9HpJ6cfET2LKHrURn2yZBDfDEwHv</a> + </address><!-- /#about --> + </footer><!-- /#contentinfo --> +</body> +</html> +\ No newline at end of file diff --git a/ca197565fb955f2328bcc11540521adebeb2d0f95d6501901516eb76fb25707c.asc b/ca197565fb955f2328bcc11540521adebeb2d0f95d6501901516eb76fb25707c.asc @@ -0,0 +1,11 @@ +-----BEGIN PGP SIGNATURE----- + +iQEzBAABCAAdFiEEWahEpISsESU9Oj6dzcvSTdHQ4AEFAmM4NzsACgkQzcvSTdHQ +4AGM5gf/Q5lqpvRe+V30rZSz754R8cMIXM9LWiDzp3l3h18JwzMcnKshlnSbUHea +ab0fQdNrCyg62RrCIxi346GcJVJwmGSXaZCkxRZjF5KBSe/8I7s7dE+F0jAL1g0w +TG/BzZl4Lzl7wKDdEIZiZFuS193DgsKb4+4D9sHjAKZRnAJDE82BIJCWnIQyGJmc +mx7teiXOSRzF+LaFcS4nMZJz7muQ1R6832rGEgzg2NxjuCLeNk9shWENf7hl4Kjr +KlPgzvpn1TiBHBVg1hoiBiPWK56Mb3czOvihG3bh9r+Cjzfu6Pn020WKrcS23XQf +prtJwLAxwTeLcfHGRzDAF2H9gOnLMw== +=8hlj +-----END PGP SIGNATURE----- diff --git a/categories.html b/categories.html @@ -28,7 +28,7 @@ </nav><!-- /#menu --> <h1>Categories on man bytes gnu</h1> <ul> - <li><a href="./category/archiving.html">Archiving</a> (2)</li> + <li><a href="./category/archiving.html">Archiving</a> (3)</li> <li><a href="./category/code.html">Code</a> (3)</li> <li><a href="./category/hygiene.html">Hygiene</a> (2)</li> <li><a href="./category/offlining.html">Offlining</a> (3)</li> diff --git a/category/archiving.html b/category/archiving.html @@ -31,6 +31,11 @@ <ol id="post-list"> <li><article class="hentry"> + <header> <h3 class="entry-title"><a href="../portable-book-metadata.html" rel="bookmark" title="Permalink to A portable book metadata exercise">A portable book metadata exercise</a></h3> </header> + <p><p class="first last">Structured approach to generate portable metadata files for bibliographies and literature files using cryptographic hash mapping.</p> +</p> + </article></li> + <li><article class="hentry"> <header> <h3 class="entry-title"><a href="../backup-rsync-duplicity.html" rel="bookmark" title="Permalink to Combining duplicity and rsync"> Organizing backups: Combining duplicity and rsync</a></h3> </header> <p><p class="first last">An exercise in combining plain and encrypted backups on local and remote hosts</p> </p> diff --git a/dab643d2a647cfd037878b4fd6bd447457b8b568c3992f411350fc88977d4d88 b/dab643d2a647cfd037878b4fd6bd447457b8b568c3992f411350fc88977d4d88 @@ -0,0 +1,363 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <link rel="stylesheet" type="text/css" href="./theme/css/style.css" /> + <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css" /> + <title>man bytes gnu - wasm and C - The bare necessities</title> + <meta charset="utf-8" /> + <meta name="generator" content="Pelican" /> + +</head> + +<body id="index" class="home"> + + <header id="banner" class="body"> + <h1>> <a href="./">man bytes gnu</a><span id="cursor">_</span></h1> + </header> + <nav id="menu"> + <ul> + <li><a href="./category/archiving.html">Archiving</a></li> + <li class="active"><a href="./category/code.html">Code</a></li> + <li><a href="./category/hygiene.html">Hygiene</a></li> + <li><a href="./category/offlining.html">Offlining</a></li> + <li><a href="./pages/identities.html">Identities</a></li> + <li><a href="./pages/shares.html">Shares</a></li> + + <li><a href="/tags.html">tags</a></li> + </ul> + <hr/> + </nav><!-- /#menu --> + <header> + <h1 class="entry-title top-body-title"> + <a href="./wasm-c-bare.html" rel="bookmark" + title="Permalink to wasm and C - The bare necessities">wasm and C - The bare necessities</a></h1> + + + <div class="category meta"> + Posted +<time class="published" datetime="2021-05-05T06:15:00+02:00"> + Wed 05 May 2021 + </time> +in <a class="category" href="./category/code.html">code</a> + <a href="./tag/wasm.html">wasm</a> + <a href="./tag/c.html">c</a> + <a href="./tag/clang.html">clang</a> + <a href="./tag/llvm.html">llvm</a> + </div> + + <div class="neighbors meta"> + <ul> + <li> + Previous: <a href="./web-snapshot.html"> + Proving what you link to + </a> + </li> + | + + <li> + Next: <a href="./nft-tokenid-content.html"> + The NFT token id as URI + </a> + </li> + </ul> + </div> + <div class="meta"> +<hr/> + </div> +</header> + +<section id="content" class="body"> + <div class="entry-content"> + <p>I am currently resuming my self-improvement task of learning Webassembly. Or, I should rather say, <em>using</em> Webassembly.</p> +<p>Since I have some optimized small code chunks that I want to use in an embedded environment, it seemed sensible that the first step would be to establish two-way communication with C.</p> +<p>In my last outing around three years ago, I was using <a class="reference external" href="https://emscripten.org/">Emscripten</a> to bridge the gap. That tool adds quite a few bells and whistles, and doesn't quite yield that warm, fuzzy bare-metal rush. <a class="reference external" href="https://emscripten.org/">Emscripten</a> relies on <a class="reference external" href="http://clang.org/">Clang</a> and <a class="reference external" href="https://llvm.org/">LLVM</a>, all of which seem to have gotten their <code>wasm</code> support built-in in the meantime (at least on my archlinux system). This it integrates nicely with <a class="reference external" href="https://github.com/WebAssembly/wabt">wabt</a> - the swiss-army knife of Webassembly.</p> +<p>So how far do we get with just <a class="reference external" href="http://clang.org/">clang</a>, <a class="reference external" href="https://llvm.org/">LLVM</a> and <a class="reference external" href="https://github.com/WebAssembly/wabt">wabt</a> ? Let's see if we at least can set up a code snippet which simply writes <em>"foobar"</em> to memory. The host will write <em>"foo"</em>, and <code>wasm</code> will write <em>"bar"</em>.</p> +<div class="section" id="without-libc"> +<h2>Without libc</h2> +<p>This excellent <a class="reference external" href="https://surma.dev/things/c-to-webassembly/">tutorial by Surma</a> provides a good starting point. Go ahead and <strong>read that first</strong>. This text is not a Webassembly primer, so the following will make a lot more sense if you do.</p> +<p>That setup still adds some magic. Namely, the <em>memory</em> and <em>symbol table</em> are here added by the <em>wasm linker</em>. It would be even more fun to pass this from the host system instead.</p> +<p>And so we start: <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a></p> +<pre class="code c literal-block"> +<span class="ln"> 0 </span><span class="cp">#include</span><span class="w"> </span><span class="cpf">"string.h"</span><span class="cp"> +</span><span class="ln"> 1 </span><span class="cp"></span><span class="w"> +</span><span class="ln"> 2 </span><span class="w"></span><span class="k">extern</span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">__heap_base</span><span class="p">;</span><span class="w"> +</span><span class="ln"> 3 </span><span class="w"></span><span class="k">extern</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">call_me_sometime</span><span class="p">(</span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">b</span><span class="p">);</span><span class="w"> +</span><span class="ln"> 4 </span><span class="w"> +</span><span class="ln"> 5 </span><span class="w"></span><span class="kt">void</span><span class="w"> </span><span class="nf">foo</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"> +</span><span class="ln"> 6 </span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="o">*</span><span class="p">)</span><span class="o">&</span><span class="n">__heap_base</span><span class="p">;</span><span class="w"> +</span><span class="ln"> 7 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">3</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'b'</span><span class="p">;</span><span class="w"> +</span><span class="ln"> 8 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">4</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'a'</span><span class="p">;</span><span class="w"> +</span><span class="ln"> 9 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">5</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'r'</span><span class="p">;</span><span class="w"> +</span><span class="ln">10 </span><span class="w"> </span><span class="n">call_me_sometime</span><span class="p">(</span><span class="n">buf</span><span class="p">);</span><span class="w"> +</span><span class="ln">11 </span><span class="w"></span><span class="p">}</span> +</pre> +<p>Compiling this without linking gives us a hint on what needs to be defined.</p> +<div class="highlight"><pre><span></span>$ clang --target<span class="o">=</span>wasm32 -nostdlib -nostartfiles -o bare.wasm -c bare.c +$ wasm-objdump -x bare.wasm +bare.wasm: file format wasm 0x1 + +Section Details: + +Type<span class="o">[</span><span class="m">2</span><span class="o">]</span>: + - type<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="o">()</span> -> nil + - type<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="o">(</span>i32<span class="o">)</span> -> nil +Import<span class="o">[</span><span class="m">4</span><span class="o">]</span>: + - memory<span class="o">[</span><span class="m">0</span><span class="o">]</span> pages: <span class="nv">initial</span><span class="o">=</span><span class="m">0</span> <- env.__linear_memory + - table<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="nv">type</span><span class="o">=</span>funcref <span class="nv">initial</span><span class="o">=</span><span class="m">0</span> <- env.__indirect_function_table + - global<span class="o">[</span><span class="m">0</span><span class="o">]</span> i32 <span class="nv">mutable</span><span class="o">=</span><span class="m">1</span> <- env.__stack_pointer + - func<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">1</span> <env.call_me_sometime> <- env.call_me_sometime +Function<span class="o">[</span><span class="m">1</span><span class="o">]</span>: + - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">0</span> <foo> +Code<span class="o">[</span><span class="m">1</span><span class="o">]</span>: + - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">138</span> <foo> +Custom: + - name: <span class="s2">"linking"</span> + - symbol table <span class="o">[</span><span class="nv">count</span><span class="o">=</span><span class="m">4</span><span class="o">]</span> + - <span class="m">0</span>: F <foo> <span class="nv">func</span><span class="o">=</span><span class="m">1</span> <span class="nv">binding</span><span class="o">=</span>global <span class="nv">vis</span><span class="o">=</span>hidden + - <span class="m">1</span>: G <env.__stack_pointer> <span class="nv">global</span><span class="o">=</span><span class="m">0</span> undefined <span class="nv">binding</span><span class="o">=</span>global <span class="nv">vis</span><span class="o">=</span>default + - <span class="m">2</span>: D <__heap_base> undefined <span class="nv">binding</span><span class="o">=</span>global <span class="nv">vis</span><span class="o">=</span>default + - <span class="m">3</span>: F <env.call_me_sometime> <span class="nv">func</span><span class="o">=</span><span class="m">0</span> undefined <span class="nv">binding</span><span class="o">=</span>global <span class="nv">vis</span><span class="o">=</span>default +Custom: + - name: <span class="s2">"reloc.CODE"</span> + - relocations <span class="k">for</span> section: <span class="m">3</span> <span class="o">(</span>Code<span class="o">)</span> <span class="o">[</span><span class="m">5</span><span class="o">]</span> + - R_wasm_GLOBAL_INDEX_LEB <span class="nv">offset</span><span class="o">=</span>0x000007<span class="o">(</span><span class="nv">file</span><span class="o">=</span>0x000099<span class="o">)</span> <span class="nv">symbol</span><span class="o">=</span><span class="m">1</span> <env.__stack_pointer> + - R_wasm_GLOBAL_INDEX_LEB <span class="nv">offset</span><span class="o">=</span>0x00001c<span class="o">(</span><span class="nv">file</span><span class="o">=</span>0x0000ae<span class="o">)</span> <span class="nv">symbol</span><span class="o">=</span><span class="m">1</span> <env.__stack_pointer> + - R_wasm_MEMORY_ADDR_SLEB <span class="nv">offset</span><span class="o">=</span>0x000031<span class="o">(</span><span class="nv">file</span><span class="o">=</span>0x0000c3<span class="o">)</span> <span class="nv">symbol</span><span class="o">=</span><span class="m">2</span> <__heap_base> + - R_wasm_FUNCTION_INDEX_LEB <span class="nv">offset</span><span class="o">=</span>0x000073<span class="o">(</span><span class="nv">file</span><span class="o">=</span>0x000105<span class="o">)</span> <span class="nv">symbol</span><span class="o">=</span><span class="m">3</span> <env.call_me_sometime> + - R_wasm_GLOBAL_INDEX_LEB <span class="nv">offset</span><span class="o">=</span>0x000086<span class="o">(</span><span class="nv">file</span><span class="o">=</span>0x000118<span class="o">)</span> <span class="nv">symbol</span><span class="o">=</span><span class="m">1</span> <env.__stack_pointer> +Custom: + - name: <span class="s2">"producers"</span> +</pre></div> +<p>Using nodejs as the host, we check if we can instantiate a <code>WebAssembly</code> object</p> +<pre class="code javascript literal-block"> +<span class="ln"> 0 </span><span class="kd">const</span><span class="w"> </span><span class="nx">fs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">require</span><span class="p">(</span><span class="s1">'fs'</span><span class="p">);</span><span class="w"> +</span><span class="ln"> 1 </span><span class="w"> +</span><span class="ln"> 2 </span><span class="w"></span><span class="kd">const</span><span class="w"> </span><span class="nx">imports</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{}</span><span class="w"> +</span><span class="ln"> 3 </span><span class="w"> +</span><span class="ln"> 4 </span><span class="w"></span><span class="k">async</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="nx">init</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"> +</span><span class="ln"> 5 </span><span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">code</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="s1">'./bare.wasm'</span><span class="p">);</span><span class="w"> +</span><span class="ln"> 6 </span><span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">m</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Module</span><span class="p">(</span><span class="nx">code</span><span class="p">);</span><span class="w"> +</span><span class="ln"> 7 </span><span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Instance</span><span class="p">(</span><span class="nx">m</span><span class="p">,</span><span class="w"> </span><span class="nx">imports</span><span class="p">);</span><span class="w"> +</span><span class="ln"> 8 </span><span class="w"></span><span class="p">}</span><span class="w"> +</span><span class="ln"> 9 </span><span class="w"></span><span class="nx">init</span><span class="p">();</span> +</pre> +<p>Running this tells us we are apparently missing a property <code>env</code> in the imports object.</p> +<div class="highlight"><pre><span></span>$ node bare_naive.js +/home/lash/src/tests/wasm/bare/bare_naive.js:8 +const <span class="nv">i</span> <span class="o">=</span> new WebAssembly.Instance<span class="o">(</span>m, imports<span class="o">)</span><span class="p">;</span> + ^ + +TypeError: WebAssembly.Instance<span class="o">()</span>: Import <span class="c1">#0 module="env" error: module is not an object or function</span> +</pre></div> +<p>That seems to match with the <code>Import</code> section in the <code>objdump</code> output above. Let's stick the <em>memory</em> and <em>table</em> in there. <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a></p> +<p>And let's make a bold guess that the callback function <code>call_me_sometime</code> needs to go in there aswell.</p> +<pre class="code javascript literal-block"> +<span class="ln"> 0 </span><span class="kd">const</span><span class="w"> </span><span class="nx">fs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">require</span><span class="p">(</span><span class="s1">'fs'</span><span class="p">);</span><span class="w"> +</span><span class="ln"> 1 </span><span class="w"> +</span><span class="ln"> 2 </span><span class="w"></span><span class="kd">const</span><span class="w"> </span><span class="nx">memory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Memory</span><span class="p">({</span><span class="nx">initial</span><span class="o">:</span><span class="w"> </span><span class="mf">2</span><span class="p">});</span><span class="w"> +</span><span class="ln"> 3 </span><span class="w"></span><span class="kd">const</span><span class="w"> </span><span class="nx">table</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Table</span><span class="p">({</span><span class="nx">initial</span><span class="o">:</span><span class="w"> </span><span class="mf">3</span><span class="p">,</span><span class="w"> </span><span class="nx">element</span><span class="o">:</span><span class="w"> </span><span class="s1">'anyfunc'</span><span class="p">});</span><span class="w"> +</span><span class="ln"> 4 </span><span class="w"></span><span class="kd">const</span><span class="w"> </span><span class="nx">importsObj</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"> +</span><span class="ln"> 5 </span><span class="w"> </span><span class="nx">env</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="w"> +</span><span class="ln"> 6 </span><span class="w"> </span><span class="nx">memory</span><span class="o">:</span><span class="w"> </span><span class="nx">memory</span><span class="p">,</span><span class="w"> +</span><span class="ln"> 7 </span><span class="w"> </span><span class="nx">__linear_memory</span><span class="o">:</span><span class="w"> </span><span class="nx">memory</span><span class="p">,</span><span class="w"> +</span><span class="ln"> 8 </span><span class="w"> </span><span class="nx">__indirect_function_table</span><span class="o">:</span><span class="w"> </span><span class="nx">table</span><span class="p">,</span><span class="w"> +</span><span class="ln"> 9 </span><span class="w"> </span><span class="nx">call_me_sometime</span><span class="o">:</span><span class="w"> </span><span class="p">(</span><span class="nx">n</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"> +</span><span class="ln">10 </span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">a</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nb">Uint8Array</span><span class="p">(</span><span class="nx">memory</span><span class="p">.</span><span class="nx">buffer</span><span class="p">,</span><span class="w"> </span><span class="nx">n</span><span class="p">,</span><span class="w"> </span><span class="mf">9</span><span class="p">)</span><span class="w"> +</span><span class="ln">11 </span><span class="w"> </span><span class="nx">a</span><span class="p">.</span><span class="nx">set</span><span class="p">([</span><span class="mh">0x66</span><span class="p">,</span><span class="w"> </span><span class="mh">0x6f</span><span class="p">,</span><span class="w"> </span><span class="mh">0x6f</span><span class="p">],</span><span class="w"> </span><span class="mf">0</span><span class="p">);</span><span class="w"> +</span><span class="ln">12 </span><span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">debug</span><span class="p">(</span><span class="s1">'heap is at: '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">n</span><span class="p">);</span><span class="w"> +</span><span class="ln">13 </span><span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'heap contains: '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">TextDecoder</span><span class="p">().</span><span class="nx">decode</span><span class="p">(</span><span class="nx">a</span><span class="p">));</span><span class="w"> +</span><span class="ln">14 </span><span class="w"> </span><span class="p">},</span><span class="w"> +</span><span class="ln">15 </span><span class="w"> </span><span class="p">},</span><span class="w"> +</span><span class="ln">16 </span><span class="w"></span><span class="p">}</span><span class="w"> +</span><span class="ln">17 </span><span class="w"> +</span><span class="ln">18 </span><span class="w"></span><span class="k">async</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="nx">init</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"> +</span><span class="ln">19 </span><span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">code</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="s1">'./bare.wasm'</span><span class="p">);</span><span class="w"> +</span><span class="ln">20 </span><span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">m</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Module</span><span class="p">(</span><span class="nx">code</span><span class="p">);</span><span class="w"> +</span><span class="ln">21 </span><span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">WebAssembly</span><span class="p">.</span><span class="nx">Instance</span><span class="p">(</span><span class="nx">m</span><span class="p">,</span><span class="w"> </span><span class="nx">importsObj</span><span class="p">);</span><span class="w"> +</span><span class="ln">22 </span><span class="w"> </span><span class="nx">i</span><span class="p">.</span><span class="nx">exports</span><span class="p">.</span><span class="nx">foo</span><span class="p">();</span><span class="w"> +</span><span class="ln">23 </span><span class="w"></span><span class="p">}</span><span class="w"> +</span><span class="ln">24 </span><span class="w"></span><span class="nx">init</span><span class="p">();</span> +</pre> +<p>The linker needs a little help from us for this:</p> +<ul class="simple"> +<li>Our callback function will not be available at link time, so we have to <code>--allow-undefined</code> to promise that the host has got this covered.</li> +<li><code>--import-memory</code> and <code>--import-table</code> to enable us to get memory and symbol table from the host.</li> +<li><code>--export="foo"</code> to make sure we only export exactly what we intend to from our <code>wasm</code>.</li> +</ul> +<div class="highlight"><pre><span></span>$ clang --target<span class="o">=</span>wasm32 -nostdlib -nostartfiles -Wl,--no-entry -Wl,--export<span class="o">=</span><span class="s2">"foo"</span> -Wl,--import-memory -Wl,--import-table -Wl,--allow-undefined -o bare.wasm bare.c +</pre></div> +<p>And that should give us:</p> +<div class="highlight"><pre><span></span>$ node bare.js +heap is at: <span class="m">66560</span> +heap contains: foobar +</pre></div> +<p>This way of pointing to memory is of course grossly inadequate <em>and</em> unsafe <em>and</em> ridiculous for any purpose more advanced that this one. So some proper memory management would not be a bad thing.</p> +</div> +<div class="section" id="adding-libc"> +<h2>Adding libc</h2> +<p>And what do you know. In other news since last time I looked at this is the addition of "a libc for WebAssembly programs built on top of WASI system calls." <a class="citation-reference" href="#wasi-libc" id="citation-reference-1">[wasi-libc]</a>. Let's see if we can add a slightly less manual way of handling memory with <code>malloc</code> and <code>memcpy</code></p> +<pre class="code c literal-block"> +<span class="ln"> 0 </span><span class="cp">#ifdef HAVE_LIBC +</span><span class="ln"> 1 </span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><string.h></span><span class="cp"> +</span><span class="ln"> 2 </span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdlib.h></span><span class="cp"> +</span><span class="ln"> 3 </span><span class="cp">#endif +</span><span class="ln"> 4 </span><span class="cp"></span><span class="w"> +</span><span class="ln"> 5 </span><span class="w"></span><span class="k">extern</span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">__heap_base</span><span class="p">;</span><span class="w"> +</span><span class="ln"> 6 </span><span class="w"></span><span class="k">extern</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">call_me_sometime</span><span class="p">(</span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">b</span><span class="p">);</span><span class="w"> +</span><span class="ln"> 7 </span><span class="w"> +</span><span class="ln"> 8 </span><span class="w"></span><span class="kt">void</span><span class="w"> </span><span class="nf">foo</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"> +</span><span class="ln"> 9 </span><span class="w"> +</span><span class="ln">10 </span><span class="w"></span><span class="cp">#ifdef HAVE_LIBC +</span><span class="ln">11 </span><span class="cp"></span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">buf</span><span class="p">;</span><span class="w"> +</span><span class="ln">12 </span><span class="w"> </span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">malloc</span><span class="p">(</span><span class="mi">9</span><span class="p">);</span><span class="w"> +</span><span class="ln">13 </span><span class="w"> </span><span class="n">memcpy</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="s">"bazbar"</span><span class="p">,</span><span class="w"> </span><span class="mi">6</span><span class="p">);</span><span class="w"> +</span><span class="ln">14 </span><span class="w"></span><span class="cp">#else +</span><span class="ln">15 </span><span class="cp"></span><span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="o">*</span><span class="p">)</span><span class="o">&</span><span class="n">__heap_base</span><span class="p">;</span><span class="w"> +</span><span class="ln">16 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">3</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'b'</span><span class="p">;</span><span class="w"> +</span><span class="ln">17 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">4</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'a'</span><span class="p">;</span><span class="w"> +</span><span class="ln">18 </span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="n">buf</span><span class="o">+</span><span class="mi">5</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'r'</span><span class="p">;</span><span class="w"> +</span><span class="ln">19 </span><span class="w"></span><span class="cp">#endif +</span><span class="ln">20 </span><span class="cp"></span><span class="w"> </span><span class="n">call_me_sometime</span><span class="p">(</span><span class="n">buf</span><span class="p">);</span><span class="w"> +</span><span class="ln">21 </span><span class="w"> +</span><span class="ln">22 </span><span class="w"></span><span class="cp">#ifdef HAVE_LIBC +</span><span class="ln">23 </span><span class="cp"></span><span class="w"> </span><span class="n">free</span><span class="p">(</span><span class="n">buf</span><span class="p">);</span><span class="w"> +</span><span class="ln">24 </span><span class="w"></span><span class="cp">#endif +</span><span class="ln">25 </span><span class="cp"></span><span class="w"> +</span><span class="ln">26 </span><span class="w"></span><span class="p">}</span> +</pre> +<p>As you see, we need a few more parameters for the compiler and linker at this point. The <code>--target=wasm32-unknown-wasi --sysroot /opt/wasi-libc .. /opt/wasi-libc/lib/wasm32-wasi/libc.a</code> is needed to hook us up with headers and symbols for the libc.</p> +<p>My archlinux puts that sysroot in <code>/opt/wasi-libc</code>, that may of course not be the case elsewhere.</p> +<div class="highlight"><pre><span></span>$ clang -DHAVE_LIBC<span class="o">=</span><span class="m">1</span> --target<span class="o">=</span>wasm32-unknown-wasi --sysroot /opt/wasi-libc -nostdlib -nostartfiles -Wl,--no-entry -Wl,--export<span class="o">=</span><span class="s2">"foo"</span> -Wl,--import-memory -Wl,--import-table -Wl,--allow-undefined -o bare.wasm bare.c /opt/wasi-libc/lib/wasm32-wasi/libc.a +$ wasm-objdump -x bare.wasm + +bare.wasm: file format wasm 0x1 + +Section Details: + +Type<span class="o">[</span><span class="m">3</span><span class="o">]</span>: + - type<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="o">(</span>i32<span class="o">)</span> -> nil + - type<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="o">()</span> -> nil + - type<span class="o">[</span><span class="m">2</span><span class="o">]</span> <span class="o">(</span>i32<span class="o">)</span> -> i32 +Import<span class="o">[</span><span class="m">3</span><span class="o">]</span>: + - memory<span class="o">[</span><span class="m">0</span><span class="o">]</span> pages: <span class="nv">initial</span><span class="o">=</span><span class="m">2</span> <- env.memory + - table<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="nv">type</span><span class="o">=</span>funcref <span class="nv">initial</span><span class="o">=</span><span class="m">1</span> <- env.__indirect_function_table + - func<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">0</span> <call_me_sometime> <- env.call_me_sometime +Function<span class="o">[</span><span class="m">7</span><span class="o">]</span>: + - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">1</span> <foo> + - func<span class="o">[</span><span class="m">2</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">2</span> <malloc> + - func<span class="o">[</span><span class="m">3</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">2</span> <dlmalloc> + - func<span class="o">[</span><span class="m">4</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">0</span> <free> + - func<span class="o">[</span><span class="m">5</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">0</span> <dlfree> + - func<span class="o">[</span><span class="m">6</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">1</span> <abort> + - func<span class="o">[</span><span class="m">7</span><span class="o">]</span> <span class="nv">sig</span><span class="o">=</span><span class="m">2</span> <sbrk> +Global<span class="o">[</span><span class="m">2</span><span class="o">]</span>: + - global<span class="o">[</span><span class="m">0</span><span class="o">]</span> i32 <span class="nv">mutable</span><span class="o">=</span><span class="m">1</span> - init <span class="nv">i32</span><span class="o">=</span><span class="m">67072</span> + - global<span class="o">[</span><span class="m">1</span><span class="o">]</span> i32 <span class="nv">mutable</span><span class="o">=</span><span class="m">0</span> <__heap_base> - init <span class="nv">i32</span><span class="o">=</span><span class="m">67072</span> +Export<span class="o">[</span><span class="m">2</span><span class="o">]</span>: + - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <foo> -> <span class="s2">"foo"</span> + - global<span class="o">[</span><span class="m">1</span><span class="o">]</span> -> <span class="s2">"__heap_base"</span> +Code<span class="o">[</span><span class="m">7</span><span class="o">]</span>: + - func<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">171</span> <foo> + - func<span class="o">[</span><span class="m">2</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">10</span> <malloc> + - func<span class="o">[</span><span class="m">3</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">6984</span> <dlmalloc> + - func<span class="o">[</span><span class="m">4</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">10</span> <free> + - func<span class="o">[</span><span class="m">5</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">1908</span> <dlfree> + - func<span class="o">[</span><span class="m">6</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">4</span> <abort> + - func<span class="o">[</span><span class="m">7</span><span class="o">]</span> <span class="nv">size</span><span class="o">=</span><span class="m">78</span> <sbrk> +Data<span class="o">[</span><span class="m">2</span><span class="o">]</span>: + - segment<span class="o">[</span><span class="m">0</span><span class="o">]</span> <span class="nv">memory</span><span class="o">=</span><span class="m">0</span> <span class="nv">size</span><span class="o">=</span><span class="m">7</span> - init <span class="nv">i32</span><span class="o">=</span><span class="m">1024</span> + - <span class="m">0000400</span>: <span class="m">6261</span> 7a62 <span class="m">6172</span> <span class="m">00</span> bazbar. + - segment<span class="o">[</span><span class="m">1</span><span class="o">]</span> <span class="nv">memory</span><span class="o">=</span><span class="m">0</span> <span class="nv">size</span><span class="o">=</span><span class="m">500</span> - init <span class="nv">i32</span><span class="o">=</span><span class="m">1032</span> + - <span class="m">0000408</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - <span class="m">0000418</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - <span class="m">0000428</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - <span class="m">0000438</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - <span class="m">0000448</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - <span class="m">0000458</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - <span class="m">0000468</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - <span class="m">0000478</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - <span class="m">0000488</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - <span class="m">0000498</span>: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - 00004a8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - 00004b8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - 00004c8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - 00004d8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> ................ + - 00004e8: <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m">0000</span> <span class="m&