commit b6ee7392111a8b468f03c9ce37388253a5087e2f
parent 3572da179b871c90ae31b706bfcd44bd3c76cf95
Author: nolash <dev@holbrook.no>
Date: Thu, 1 Jul 2021 20:07:29 +0200
State at 1.7.2021
Diffstat:
10 files changed, 406 insertions(+), 29 deletions(-)
diff --git a/content/20210508_nft_tokenid_content.rst b/content/20210508_nft_tokenid_content.rst
@@ -58,7 +58,7 @@ Mirror, mirror
Since the asset reference shouldn't change, we can refer to it by its fingerprint or `content address <https://en.wikipedia.org/wiki/Content-addressable_storage>`_. If we define that the resource can be looked up over HTTP by that fingerprint as its basename, then we are free to define and modify whatever list of mirrors for that resource that's valid for any point in time. The application layer would simply try the endpoints one after another.
-We take the :code:`sha2-256` of the asset reference (the json file above, free of evil whitespace and newlines):
+We take the :code:`sha2-256` [2]_ of the asset reference (the json file above, free of evil whitespace and newlines):
.. code-block:: bash
diff --git a/content/20210609_iup.rst b/content/20210609_iup.rst
@@ -1,20 +1,138 @@
-Internet up monitor
-###################
+Homemade internet state monitor
+###############################
-:date: 2021-06-09 22:26
+:date: 2021-07-01 13:59
:category: Hygiene
:author: Louis Holbrook
-:tags: bash,systemd,network
+:tags: bash,systemd,network,i3wm,tmpfs
:slug: internet-up-monitor
:summary: A small script to detect internet status
:lang: en
-:status: draft
+:status: published
+I use `i3wm <https://i3wm.org>`_. I use it because it has a very low resource footprint, it has great documentation, and it's simple to configure.
+
+Complementing *i3wm* is the *i3status* status provider to the *i3bar*. It provides a good suite of preset components for things like temperatur measurement, disk capacity. It also provides a couple of generic ones, namely "output the contents of a file" or "yes if file exists, no if not."
+
+
+It's not you, it's me?
+======================
+
+.. WARNING::
+
+ **Always use a VPN! Always!!**
+
+
+Now, I *always* use VPN *no matter what*, and I configure network manually. That is, I do not use a network manager. For this reason, when network applications start misbehaving, it's sometimes hard to tell whether it is because the host is gone, the local network is flaky, the internet route is flaky, or simply that the connection is gone for good.
+
+Evidently, it would be practical to have **an item in the status bar telling me whether I have access to internet right now or not**.
+
+The "yes if file exists, no if not" option seems to be the right one to go for. Basically, we need to stick an empty file in a runtime file store whenever we detect that internet is available.
+
+
+Ping, rinse, repeat
+===================
+
+There's not much trickery involved here. The check is a ping to a remote location every now and then.
+
+Some caveats to cover to grant it some minimum of intelligence:
+
+- We need a few alternative destinations for the ping, that can take over if others happen to be down or take too long to repond.
+
+- To make the script useful for general purpose use, we'd also want to be able to supply a custom list of hosts to ping.
+
+- We also want to make sure that when the script is interrupted, it deletes the internet indicator file. That way, when it stops running, the internet indicator will be frozen in the "no" state.
+
+- Lastly, it can be useful to log to syslog every time the state of the internet connection (or the script itself) changes.
+
+That can translate to a shell script like this;
+
.. include:: code/iup/iup.sh
- :code: python
+ :code: bash
+Since we'll want this to run when network interfaces allegedly are up, it makes sense to link it to systemd and the network target:
.. include:: code/iup/iup.service
- :code: systemd
+ :code: ini
+
+Stick this in :code:`~/.config/systemd/user/iup.service` and enable and start the service:
+
+.. code-block:: bash
+
+ $ systemctl --user enable iup.service
+ $ systemctl --user start iup.service
+
+
+Checking what status our status is in
+=====================================
+
+Now, in the i3status configuration, assuming that your system uid is 1000 [1]_, add:
+
+.. code-block:: code
+
+ order += "run_watch INET"
+
+ kj
+
+ run_watch INET {
+ pidfile = "/run/user/1000/probe_up"
+ }
+
+To reload the config on the fly, press :code:`mod+r` (that's :code:`ctrl+r` on mine). The result should be a new item in your bar showing :code:`INET: yes` (most likely yes, anyway, since you're currently reading this).
+
+
+Safety first, eh ... second
+===========================
+
+.. WARNING::
+
+ **Did I mention that you should always use a VPN?**
+
+
+The same :code:`run_watch` trick can be used for VPN.
+
+I use :code:`openvpn`, and it defines a flag :code:`--writepid`. If you pass a file path to this parameter, either through the :code:`writepid` config directiry or on the :code:`--writepid` flag on the command line, it will write the openvpn pid to that file, and similarly remove it when the openvpn service ends.
+
+However, this raises another issue. Namely the fact that a location is needed for openvpn to write the file to, for which it also has access.
+
+If we want to use the :code:`/run` directory again, now we also has to make sure that this directory exists.
+
+
+Got the runs?
+-------------
+
+This is exactly what :code:`systemd-tmpfiles` is for. You can add files to :code:`/etc/tmpfiles.d` which describe what kind of temporary files or directories to add [2]_.
+
+Create a file :code:`/etc/tmpfiles.d/openvpn.conf` and add a single line to it:
+
+.. code-block:: code
+
+ d /run/openvpn 0755 openvpn openvpn
+
+This will ensure that the folder is created at startup.
+
+Now, to create one right away for the current session, run :code:`systemd-tmpfiles --create`.
+
+Now we have everything we need to add the VPN status to *i3status* aswell. Provided that we chose :code:`/run/openvpn/openvpn.pid` as our argument to :code:`openvpn --writepid`, the setup is more or less the same as before:
+
+
+.. code-block:: code
+
+ order += "run_watch VPN"
+
+ kj
+
+ run_watch VPN {
+ pidfile = "/run/openvpn/openvpn.pid"
+ }
+
+
+..
+
+ .. [1] The :code:`/run/user/$UID` folder will be created on system startup, and will be writable by that user. It provides a tmpfs storage location for processes that run in early stages of boot, before the entire filesystem tree (which may include :code:`/var/run` because :code:`/var` may be on a separate partition) has been mounted. More in that here: `<https://lwn.net/Articles/436012/>`_
+
+
+..
+ .. [2] This service has an impressive amount of flexibility, which you can inspect yourself by visiting the :code:`tmpfiles.d` (and :code:`systemd-tmpfiles`) manpage.
diff --git a/content/20210609_python_pipe_args.rst b/content/20210609_python_pipe_args.rst
@@ -11,5 +11,5 @@ Python pipe args
:status: draft
-.. include:: code/python-pipe-atgs
- :code: python
+.. include:: code/python-pipe-args/stdinargs.py
+ :code: ini
diff --git a/content/20210623_docker_debian.rst b/content/20210623_docker_debian.rst
@@ -0,0 +1,19 @@
+Debian repository
+#################
+
+:date: 2021-06-23 13:14
+:category: Offlining
+:author: Louis Holbrook
+:tags: docker,networking,debian
+:slug: docker-offline-4-debian
+:summary: How to not be forced being online when forced to use docker
+:series: Offline Docker
+:seriesprefix: docker-offline
+:seriespart: 4
+:lang: en
+:status: draft
+
+
+..
+
+ .. _Mirroring a debian repository with rsync: https://chrisgilmerproj.github.io/debian/mirror/rsync/2013/08/29/mirror-debian.html
diff --git a/content/20210623_swarm_qemu.rst b/content/20210623_swarm_qemu.rst
@@ -0,0 +1,59 @@
+Self-sufficient swarm on qemu
+#############################
+
+:date: 2021-06-23 12:49
+:modified: 2021-06-23 12:49
+:category: Hygiene
+:author: Louis Holbrook
+:tags: swarm,golang,qemu
+:slug: swarm-qemu
+:summary: How do document complex task chains in Python Celery using graphviz
+:lang: en
+:status: draft
+:series: Self-sufficient Swarm
+:seriesprefix: swarm-self-sufficient
+:seriespart: 1
+
+
+base debian qemu install
+========================
+
+
+.. code-block:: bash
+
+ dd if=/dev/zero of=debootstrap.bin count=10485760 bs=512
+ mkfs.ext4 debootstrap.bin
+ mount -o loop debootstrap.bin /mnt
+ debootstrap --arch=amd64 buster /mnt
+
+.. code-block:: bash
+
+ qemu-system-x86_64 -cdrom /mnt/iso/debian-10.9.0-amd64-xfce-CD-1.iso -hda debian.bin -hdb clef.bin -m 4g --enable-kvm -virtfs local,path=/mnt,id=foo,readonly=on,mount_tag=bar,security_model=passthrough
+
+
+
+.. code-block:: bash
+
+ mount -t 9p -o trans=virtio mount_tag /mnt/resource -oversion=9p2000.L,posixacl,msize=104857600,cache=loose
+
+clef/swarm provisions
+=====================
+
+
+ethereum node provisions
+========================
+
+
+xdai requisites
+===============
+
+
+mainchain requisites
+====================
+
+
+orchestration
+=============
+
+
+
diff --git a/content/20210627_eth_log_filters.rst b/content/20210627_eth_log_filters.rst
@@ -0,0 +1,140 @@
+Making sense of Ethereum log bloom filters
+##########################################
+
+:date: 2021-06-27 11:17
+:modified: 2021-06-27 15:56
+:category: Code
+:author: Louis Holbrook
+:tags: crypto,ethereum,bloom filter
+:slug: eth-log-bloom
+:summary: How to generate values to match the log bloom filters in Ethereum
+:lang: en
+:status: published
+
+
+Ethereum blocks and transaction receipts both carry a bloom filter. They provide a shortcut to check whether a particular event of a particular contract was triggered. The filter in the transaction indexes all events that occurred in the transaction. The filter in the block carries all the filter bits of the individual transactions within the block.
+
+Somewhat surprisingly, there are practically no friendly descriptions out there that tell you how to generate filters to match them with. In fact, the `Ethereum Yellow Paper <https://holbrook.no/doc/ethereum/yellowpaper.pdf>`_ was the only source I found.
+
+**So let's make a friendly description**
+
+Before we get to the friendliness, though, an ever so slight dose of brain melt is due:
+
+The formalities
+===============
+
+The formal definition of the filter calculation in the `Ethereum Yellow Paper section 4.3.1 <https://holbrook.no/doc/ethereum/yellowpaper.pdf>`_ is (here cited in painfully inadequate math formatting):
+
+ .. math::
+
+ M(O) \equiv \bigvee x \in \{Oa\} \cup Ot (M_{3:2048}(x))
+
+Where :math:`Oa` is the contract address and :math:`Ot` is all of the topics.
+
+In the context of *Solidity*, "all" of the topics means the actual event signature, along with all the *indexed* parameters to the event.
+
+The :math:`M` is the filter bit generator. The filter 2048 bits long, and for each entry that is added, 3 bits need to be set.
+
+Reading further, we learn that the bits to set are the "first three" pairs of bytes of the keccak256 hash [1]_, modulo the length of the filter - 2048.
+
+In the context of hashes, I usually assume "first" means from the left. That happens to be the case here, too. In other words, the "first" bytes are actually the bytes of most significance in big-endian order.
+
+What's less clear, however, is which bits to set in the filter. The paper says:
+
+ :math:`\mathcal{B}` is the bit reference function such that :math:`\mathcal{B}_j(x)` equals the bit of index :math:`j` (indexed from 0) in the byte array :math:`x`.
+
+Turns out after some trial and error that "index 0" means the *right* end of the byte array. So when setting the bits, we have to count from the right side.
+
+
+The friendly recipe
+===================
+
+Now let's try to bring this down to a mortal level:
+
+1. Take the keccak256 hash of the contract address
+
+2. Take the keccak256 hash of the first topic value
+
+3. Optionally, take the keccak256 hash of the remaining topic values (the indexed parameters of the event) [2]_
+
+4. Instantiate a byte array with length 256
+
+5. Now, *for each* of the hashes:
+
+ 1. Get the *big-endian* integer of the *first two* bytes of the hash
+
+ 2. Calculate the 2048-modulo of that integer [3]_
+
+ 3. *Bitwise or* the filter bit at the index corresponding to the modulated value, counting from *right to left* (i.e. the value :math:`0` corresponds to the *rightmost bit*).
+
+ 4. Repeat 2 and 3 for the two next pairs of bytes of the hash.
+
+
+The code
+========
+
+All that taken into account, we can take a stab at implementing a filter generator:
+
+.. include:: code/eth-log-bloom/generate.py
+ :code: python
+ :number-lines: 0
+
+Let's say we'd want to check if an Solidity-defined event :code:`Foo(uint256,bytes32)` emitted by a contract at address :code:`0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee` exists in a filter.
+
+Using the class above, we would have to do something like this:
+
+.. code:: python
+
+ # external imports
+ import sha3
+ import some_block_getter
+
+ # local imports
+ from the_above import LogBloom
+
+ # all set bits in our filter must be set in theirs
+ def filter_match(theirs, ours):
+ for i in range(len(ours)):
+ if ours[i] & theirs[i] != ours[i]:
+ return False
+ return True
+
+ fltr = LogBloom()
+
+ # add the contract address to the filter
+ address = bytes.fromhex('eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee')
+ h = sha3.keccak_256()
+ h.update(address)
+ address_element = h.digest()
+ fltr.add(address_element)
+
+ # create the topic signature for the event
+ h = sha3.keccak_256()
+ h.update(b'Foo(uint256,bytes32)')
+ topic = h.digest()
+
+ # add the topic to the filter
+ h = sha3.keccak_256()
+ h.update(topic)
+ topic_element = h.digest()
+ fltr.add(topic_element)
+
+ # get the filter from a block
+ block = some_block_getter()
+ block_bloom = bytes.fromhex(block['logsBloom'][2:]) # assumes there is a 0x-prefix
+
+ # check if it matches
+ match = filter_match(block_bloom, fltr.contents)
+ print(match)
+..
+
+ .. [1] Remember the keccak hash used in Ethereum/Bitcoin is _not_ the official `sha3` hash. The padding parameter is different. More on that `here <{filename}/20210508_nft_tokenid_content.rst#id8>`_
+
+..
+
+ .. [2] Solidity supports up to 3 indexed parameters, which maps to the *evm* opcodes :code:`LOGn` where :math:`n \in \{1,2,3,4\}`
+
+
+..
+
+ .. [3] Since :math:`2^11 = 2048`, this is the same as zeroing out all bits above bit number 11; :math:`x \wedge 2048`. In other words, the result is an *11 bit integer*.
diff --git a/content/code/docker-offline-4-debian/download.sh b/content/code/docker-offline-4-debian/download.sh
@@ -0,0 +1,20 @@
+set -e
+set +o noglob
+FLAGS="-H -a --no-motd"
+EXCLUDE_ARCH="alpha arm source i386 mips ppc s390"
+EXCLUDE=""
+for arch in $EXCLUDE_ARCH; do
+ EXCLUDE="--exclude arch-${arch}* \
+ --exclude binary-${arch}/ \
+ --exclude *_${arch}.deb \
+ --exclude *_${arch}.udeb \
+ --exclude installer-${arch}/ \
+ --exclude Contents-${arch}* \
+ --exclude Contents-udeb-${arch}* \
+ $EXCLUDE"
+done
+OPTIONS1=$EXCLUDE
+OPTIONS2="--delete-after"
+/usr/bin/rsync $FLAGS $OPTIONS1 ftp.de.debian.org::debian/ ./debian &&
+/usr/bin/rsync $FLAGS $OPTIONS2 ftp.de.debian.org::debian/ ./debian &&
+/bin/date -u > /srv/debian/project/trace/`hostname`
diff --git a/content/code/eth-log-bloom/generate.py b/content/code/eth-log-bloom/generate.py
@@ -0,0 +1,24 @@
+# external imports
+import sha3
+
+
+class LogBloom:
+
+ def __init__(self):
+ self.content = bytearray(256)
+
+
+ def add(self, element):
+ if not isinstance(element, bytes):
+ raise ValueError('element must be bytes')
+ h = sha3.keccak_256()
+ h.update(element)
+ z = h.digest()
+
+ for j in range(3):
+ c = j * 2
+ v = int.from_bytes(z[c:c+2], byteorder='big')
+ v &= 0x07ff
+ m = 255 - int(v / 8)
+ n = v % 8
+ self.content[m] |= (1 << n)
diff --git a/content/code/iup/iup.sh b/content/code/iup/iup.sh
@@ -1,28 +1,16 @@
#!/bin/bash
+# this is our runtime file, if - which exists - tells us internet is up
f="/run/user/$UID/probe_up"
rm -f $f
up() {
- #ip route | awk '{ if (/^default/) c=$5; if (c ~ /^tun/ ) { o="VPN"; c=0; } if ( c ~ /^wlp/ ) { o="CLEAR"; c=0; } } END { print o }'
- #ip route | awk 'BEGIN { o="CLEAR";} { if ( /^0.0.0.0/ ) { if ($5 ~ /^tun/) { o="VPN"; c=""; } } } END { print o; }' >&3
- #truncate -s0 $f
- #exec 3<>$f
- #r=`ip route | awk 'BEGIN { o="CLEAR";} { if ( $2 ~ dev && $3 ~ /^tun/ ) o="VPN" } END { print o; }'`
- #echo -e "\033[32m$r\033[39m" >&3
- #exec 3>&-
- #exit 0
if [ ! -f $f ]; then
logger -p info "iup up"
fi
touch $f
}
down() {
- #truncate -s0 $f
- #exec 3<>$f
- #echo "DOWN" >&3
- #exec 3>&-
- #exit 0
if [ -f $f ]; then
logger -p info "iup down"
fi
@@ -46,9 +34,8 @@ while read h; do
hosts+=($h)
done < $HOME/.config/iup 2> /dev/null
if [ ${#hosts[@]} -eq "0" ]; then
- # use cnn, google as default
- hosts=(8.8.8.8 151.101.193.67)
- logger -p warn missing hosts input file, using default: ${hosts[@]}
+ hosts=(8.8.8.8 151.101.193.67) # google nameserver and cnn.com
+ logger -p warn missing hosts input file, using evil default: ${hosts[@]}
fi
trap argh SIGINT
diff --git a/content/pages/projects.rst b/content/pages/projects.rst
@@ -27,15 +27,22 @@ Taint_
Taint tags crypto addresses, and merge tags for crypto addresses that trade with each other. It can be used as a simple forensic tool to check whether cryptographic identities are isolated from each other.
+
`Eth Statsyncer`_
-----------------
Statsyncer collects and aggregates blockchain state, like gas prices, over time. It in turns serves this data on a (JSON-RPC) API.
-Ecuth_
-------
+Leveldir_
+---------
-Leverages the `HTTP HOBA challenge-response authentication scheme`_ to authenticate with PGP_ and Ethereum_ wallets. It is supported by the dependencies `python-http-hoba-auth`_ and `python-yaml-acl`_, which provide parsing of HOBA authorization strings and a simple YAML-based ACL structure respectively.
+Automagical subdirectory structures for integer and hex ranges.
+
+..
+ Ecuth_
+ ------
+
+ Leverages the `HTTP HOBA challenge-response authentication scheme`_ to authenticate with PGP_ and Ethereum_ wallets. It is supported by the dependencies `python-http-hoba-auth`_ and `python-yaml-acl`_, which provide parsing of HOBA authorization strings and a simple YAML-based ACL structure respectively.
..
Promisc
@@ -88,6 +95,9 @@ A library that aims to simplify mutually signing generic serializable items offl
..
Project links
+..
+
+ .. _Leveldir: https://gitlab.com/nolash/leveldir
..
