manbytesgnu_site

Source files for manbytesgnu.org
git clone git://holbrook.no/manbytesgnu_site.git
Info | Log | Files | Refs

20210524_docker_npm.rst (10542B)


      1 Local npm repository
      2 ####################
      3 
      4 :date: 2021-05-25 11:47
      5 :category: Offlining
      6 :author: Louis Holbrook
      7 :tags: docker,npm,nodejs,javascript,devops
      8 :slug: docker-offline-3-npm
      9 :summary: Feeding npm packages to your offline Docker setup
     10 :series: Offline Docker
     11 :seriesprefix: docker-offline
     12 :seriespart: 3
     13 :lang: en
     14 :status: published
     15 
     16 
     17 As expected, serving a local ``npm`` repository is a *lot* less straightforward than that for ``pip`` and ``python``.
     18 
     19 The npm registry uses the :code:`CouchDB` nosql server as backend to resolve dependencies and serve resource metadata. The :code:`npm` CLI tool expects a corresponding json response to requests that cite the *name* (and not the path) of the package.
     20 
     21 Let's ask the `registry <https://registry.npmjs.org>`_ [1]_ for the package ``ftp``, and have a look at the response (excerpt):
     22 
     23 
     24 .. code-block:: console
     25 
     26         $ curl -X GET https://registry.npmjs.org/ftp | jq
     27         {
     28           "_id": "ftp",
     29           "_rev": "113-89fe76508a7ece41b4c9a157114f966f",
     30           "name": "ftp",
     31           "description": "An FTP client module for node.js",
     32           "dist-tags": {
     33             "latest": "0.3.10"
     34           },
     35           "versions": {
     36 
     37 
     38 
     39             "0.3.10": {
     40               "name": "ftp",
     41               "version": "0.3.10",
     42               "author": {
     43                 "name": "Brian White",
     44                 "email": "mscdex@mscdex.net"
     45               },
     46               "description": "An FTP client module for node.js",
     47               "main": "./lib/connection",
     48               "engines": {
     49                 "node": ">=0.8.0"
     50               },
     51               "dependencies": {
     52                 "xregexp": "2.0.0",
     53                 "readable-stream": "1.1.x"
     54               },
     55               "scripts": {
     56                 "test": "node test/test.js"
     57               },
     58               "keywords": [
     59                 "ftp",
     60                 "client",
     61                 "transfer"
     62               ],
     63               "licenses": [
     64                 {
     65                   "type": "MIT",
     66                   "url": "http://github.com/mscdex/node-ftp/raw/master/LICENSE"
     67                 }
     68               ],
     69               "repository": {
     70                 "type": "git",
     71                 "url": "http://github.com/mscdex/node-ftp.git"
     72               },
     73               "bugs": {
     74                 "url": "https://github.com/mscdex/node-ftp/issues"
     75               },
     76               "homepage": "https://github.com/mscdex/node-ftp",
     77               "_id": "ftp@0.3.10",
     78               "_shasum": "9197d861ad8142f3e63d5a83bfe4c59f7330885d",
     79               "_from": "https://github.com/mscdex/node-ftp/tarball/v0.3.10",
     80               "_resolved": "https://github.com/mscdex/node-ftp/tarball/v0.3.10",
     81               "_npmVersion": "1.4.28",
     82               "_npmUser": {
     83                 "name": "mscdex",
     84                 "email": "mscdex@mscdex.net"
     85               },
     86               "maintainers": [
     87                 {
     88                   "name": "mscdex",
     89                   "email": "mscdex@mscdex.net"
     90                 }
     91               ],
     92               "dist": {
     93                 "shasum": "9197d861ad8142f3e63d5a83bfe4c59f7330885d",
     94                 "tarball": "https://registry.npmjs.org/ftp/-/ftp-0.3.10.tgz"
     95               },
     96               "directories": {}
     97             }
     98           },
     99 
    100 
    101 That looks a lot like embellished contents of the basic package.json set up by :code:`npm init`, except it's wrapped in a versions array. It also explicitly defines an *absolute* url to a `tarball` under the `dist` key. [2]_
    102 
    103 
    104 How low can you go
    105 ==================
    106 
    107 It seems that all that's needed is a json index page served for the package name subpath of repository path, together with the tarball itself. We check this by putting together an utterly pointless package ``foobarbarbar``:
    108 
    109 **foobarbarbar/index.js**
    110 
    111 .. include:: code/docker-offline-3-npm/foobarbarbar/index.js
    112    :code: javascript
    113    :name: foobarbarbar/index.js
    114 
    115 **foobarbarbar/package.json**
    116 
    117 .. include:: code/docker-offline-3-npm/foobarbarbar/package.json
    118    :code: json
    119    :name: foobarbarbar/package.json
    120 
    121 Then make a tarball from those two files named ``foobarbarbar-1.0.0.tgz``, take the sha1 sum of it,
    122 
    123 Remembering our `virtual interface setup from the pip example <{filename}/20210419_docker_python.rst>`_ we stick it behind our webserver (here with npm sub-path) and add our minimal version json wrapper:
    124 
    125 **package.json**
    126 
    127 .. include:: code/docker-offline-3-npm/package.json
    128    :code: json
    129    :name: package.json
    130 
    131 
    132 Making introductions
    133 ====================
    134 
    135 The central trick here is to serve a json document as the "directory index" of the HTTP server. It's useful to remind ourselves at this point that we are *not* setting up a *registry* but a *repository* that contains provisions for a *locked* or *frozen* dependency graph. The former would surely need some of the ``CouchDB`` magic to resolve dependencies. Our assumption is that the latter can theoretically be realized using static files.
    136 
    137 In other words; just as with the previous `python repository example <{filename}/20210419_docker_python.rst>`_, we don't try to handle dependency resolutions for pip, but merely serve the actual package files after dependences have been resolved.
    138 
    139 With Apache Web Server, using the ``package.json`` as the directory index is as easy as:
    140 
    141 .. code-block:: markup
    142 
    143         <Directory "/srv/http/npm">
    144                 DirectoryIndex package.json
    145         </Directory>
    146 
    147 Of course adjusting the `Directory` path as needed to match the local setup.
    148 
    149 Make sure that the tarball and the latter `package.json` can be found in the `foobarbarbar` virtual subfolder of the above `Directory` directive path:
    150 
    151 .. code-block:: console
    152                 
    153         $ ls /srv/http/npm/foobarbarbar/
    154         -rw-r--r-- 1 root root  351 May 24 18:38 foobarbarbar-1.0.0.tgz
    155         -rw-r--r-- 1 root root  380 May 25 09:36 package.json
    156 
    157 Set the registry entry in your ``~./npmrc`` as follows:
    158 
    159 .. code-block:: ini
    160 
    161         registry=http://10.1.2.1/npm
    162 
    163 Then restart the Apache server and give it a go:
    164 
    165 .. code-block:: console
    166 
    167         $ npm install --verbose foobarbarbar
    168         npm verb cli [ '/usr/bin/node', '/usr/bin/npm', 'install', '--verbose', 'foobarbarbar' ]
    169         npm info using npm@7.13.0
    170         npm info using node@v16.1.0
    171         [...]
    172         npm http fetch GET 200 http://10.1.2.1/npm/foobarbarbar/ 6ms
    173         [...]
    174         npm http fetch GET 200 http://10.1.2.1/npm/foobarbarbar/foobarbarbar-1.0.0.tgz 25ms
    175         [...]
    176         added 2 packages in 545ms
    177         npm timing command:install Completed in 70ms
    178         [...]
    179 
    180 
    181 Cache dodging
    182 =============
    183 
    184 Seriously, ``npm --help`` leaves a lot to be desired. However, the online ``npm install`` documentation does not yield any more clues as to whether it gives you an option of ignoring local cache like you can with ``pip --no-cache``. This is a major pain in the ass, as you have to remember to keep your ``~/.npm`` folder clean between each install attempt. Otherwise, changes you make to the repository won't be used by ``npm install``.
    185 
    186 While testing, it pays to use a folder fairly close to the fs root, like a subfolder in `tmp`. That way, you won't be thrown off by some stray `node_modules` folder somewhere down the tree.
    187 
    188 
    189 Think locally, act locally
    190 ==========================
    191 
    192 Serving packages globally apparently comes down to this:
    193 
    194 1. Pull the ``package.json`` versions list from a proper registry.
    195 2. Get the tarball
    196 3. Transform the absolute package url to our host instead. (sigh)
    197 4. (optional) Prune all the versions and metadata which is not needed (anything that's not in our minimal ``foobarbarbar`` example above).
    198 
    199 Obviously, this is an annoyingly large amount of work to code up parsing and retrieval for.
    200 
    201 Incidentally, there is a very nice tool called `verdaccio <https://verdaccio.org>`_ which gets us most of the way there. [3]_ Its *storage* directory [4]_ uses exactly the directory structure we need. After retrieving a package collection using it as a proxy, getting the files in place is merely a case of copying [5]_ the files from the *storage* directory to the corresponding webserver directory.
    202 
    203 Looking at the ``package.json`` versions wrapper saved by ``verdaccio``, however, we see that it still preserves the absolute path of the upstream registry. Sadly we are still left with doing the third task ourselves. 
    204 
    205 As parsing json is one of my least favorite things in the world, I won't include the code for this last step here. For now it's sufficient that we understand the minimums required for manually setting up and serving a static, local, offline npm repository. Regardless of the pain involved.
    206 
    207 
    208 Making it personal
    209 ==================
    210 
    211 Whether we massage jsons ourselves or lazily resort to ``verdaccio``, the final step we need to take is the same: Setting the registry url in the npm configuration of the docker image.
    212 
    213 This is merely a case of setting the registry url in the `npmrc <https://docs.npmjs.com/cli/v6/configuring-npm/npmrc#files>`_ inside the container.
    214 
    215 .. code-block:: console
    216       
    217         $ docker run -it [...] npm config ls -l | grep etc/npmrc
    218         globalconfig = "/usr/etc/npmrc" 
    219 
    220 To get at our manually provided ``foobarbarbar`` package from before, the dockerfile will be (using archlinux):
    221 
    222 .. code-block:: docker
    223 
    224         [...]
    225         RUN pacman -S nodejs npm
    226         RUN mkdir -vp /usr/etc
    227         RUN echo "registry=http:/10.1.2.1/npm" > /usr/etc/npmrc
    228         WORKDIR /root
    229         RUN npm install foobarbarbar
    230 
    231 ..
    232 
    233         .. [1] This was the registry address I had in my `~/.npmrc` or whether I put it in myself.
    234 
    235 ..
    236 
    237         .. [2] I have tried using a relative path instead, both with and without a leading ``/``, but in either case the install then errors our complaining that the lock file is "corrupt." Whether the schema allows a base-url setting I do not know, as the schema documentation doesn't seem to be readily available.
    238               
    239 ..
    240 
    241         .. [3] In fact, ``verdaccio`` by itself solves the particular problem that we're trying to solve; providing an offline repository alternative. The task at hand, however, is to understand how to cope *without* using other tools than what can be considered base infrastructure (i.e. a web server). 
    242 
    243 ..
    244 
    245         .. [4] Should be ``/etc/verdaccio/storage``; see ``/etc/verdaccio/config.yaml``
    246 
    247 ..
    248 
    249         .. [5] An alternative approach would be to set the storage path to point to the same folder as the web server is serving. However, since we need to mess with the tarball paths