20210524_docker_npm.rst (10542B)
1 Local npm repository 2 #################### 3 4 :date: 2021-05-25 11:47 5 :category: Offlining 6 :author: Louis Holbrook 7 :tags: docker,npm,nodejs,javascript,devops 8 :slug: docker-offline-3-npm 9 :summary: Feeding npm packages to your offline Docker setup 10 :series: Offline Docker 11 :seriesprefix: docker-offline 12 :seriespart: 3 13 :lang: en 14 :status: published 15 16 17 As expected, serving a local ``npm`` repository is a *lot* less straightforward than that for ``pip`` and ``python``. 18 19 The npm registry uses the :code:`CouchDB` nosql server as backend to resolve dependencies and serve resource metadata. The :code:`npm` CLI tool expects a corresponding json response to requests that cite the *name* (and not the path) of the package. 20 21 Let's ask the `registry <https://registry.npmjs.org>`_ [1]_ for the package ``ftp``, and have a look at the response (excerpt): 22 23 24 .. code-block:: console 25 26 $ curl -X GET https://registry.npmjs.org/ftp | jq 27 { 28 "_id": "ftp", 29 "_rev": "113-89fe76508a7ece41b4c9a157114f966f", 30 "name": "ftp", 31 "description": "An FTP client module for node.js", 32 "dist-tags": { 33 "latest": "0.3.10" 34 }, 35 "versions": { 36 37 38 39 "0.3.10": { 40 "name": "ftp", 41 "version": "0.3.10", 42 "author": { 43 "name": "Brian White", 44 "email": "mscdex@mscdex.net" 45 }, 46 "description": "An FTP client module for node.js", 47 "main": "./lib/connection", 48 "engines": { 49 "node": ">=0.8.0" 50 }, 51 "dependencies": { 52 "xregexp": "2.0.0", 53 "readable-stream": "1.1.x" 54 }, 55 "scripts": { 56 "test": "node test/test.js" 57 }, 58 "keywords": [ 59 "ftp", 60 "client", 61 "transfer" 62 ], 63 "licenses": [ 64 { 65 "type": "MIT", 66 "url": "http://github.com/mscdex/node-ftp/raw/master/LICENSE" 67 } 68 ], 69 "repository": { 70 "type": "git", 71 "url": "http://github.com/mscdex/node-ftp.git" 72 }, 73 "bugs": { 74 "url": "https://github.com/mscdex/node-ftp/issues" 75 }, 76 "homepage": "https://github.com/mscdex/node-ftp", 77 "_id": "ftp@0.3.10", 78 "_shasum": "9197d861ad8142f3e63d5a83bfe4c59f7330885d", 79 "_from": "https://github.com/mscdex/node-ftp/tarball/v0.3.10", 80 "_resolved": "https://github.com/mscdex/node-ftp/tarball/v0.3.10", 81 "_npmVersion": "1.4.28", 82 "_npmUser": { 83 "name": "mscdex", 84 "email": "mscdex@mscdex.net" 85 }, 86 "maintainers": [ 87 { 88 "name": "mscdex", 89 "email": "mscdex@mscdex.net" 90 } 91 ], 92 "dist": { 93 "shasum": "9197d861ad8142f3e63d5a83bfe4c59f7330885d", 94 "tarball": "https://registry.npmjs.org/ftp/-/ftp-0.3.10.tgz" 95 }, 96 "directories": {} 97 } 98 }, 99 100 101 That looks a lot like embellished contents of the basic package.json set up by :code:`npm init`, except it's wrapped in a versions array. It also explicitly defines an *absolute* url to a `tarball` under the `dist` key. [2]_ 102 103 104 How low can you go 105 ================== 106 107 It seems that all that's needed is a json index page served for the package name subpath of repository path, together with the tarball itself. We check this by putting together an utterly pointless package ``foobarbarbar``: 108 109 **foobarbarbar/index.js** 110 111 .. include:: code/docker-offline-3-npm/foobarbarbar/index.js 112 :code: javascript 113 :name: foobarbarbar/index.js 114 115 **foobarbarbar/package.json** 116 117 .. include:: code/docker-offline-3-npm/foobarbarbar/package.json 118 :code: json 119 :name: foobarbarbar/package.json 120 121 Then make a tarball from those two files named ``foobarbarbar-1.0.0.tgz``, take the sha1 sum of it, 122 123 Remembering our `virtual interface setup from the pip example <{filename}/20210419_docker_python.rst>`_ we stick it behind our webserver (here with npm sub-path) and add our minimal version json wrapper: 124 125 **package.json** 126 127 .. include:: code/docker-offline-3-npm/package.json 128 :code: json 129 :name: package.json 130 131 132 Making introductions 133 ==================== 134 135 The central trick here is to serve a json document as the "directory index" of the HTTP server. It's useful to remind ourselves at this point that we are *not* setting up a *registry* but a *repository* that contains provisions for a *locked* or *frozen* dependency graph. The former would surely need some of the ``CouchDB`` magic to resolve dependencies. Our assumption is that the latter can theoretically be realized using static files. 136 137 In other words; just as with the previous `python repository example <{filename}/20210419_docker_python.rst>`_, we don't try to handle dependency resolutions for pip, but merely serve the actual package files after dependences have been resolved. 138 139 With Apache Web Server, using the ``package.json`` as the directory index is as easy as: 140 141 .. code-block:: markup 142 143 <Directory "/srv/http/npm"> 144 DirectoryIndex package.json 145 </Directory> 146 147 Of course adjusting the `Directory` path as needed to match the local setup. 148 149 Make sure that the tarball and the latter `package.json` can be found in the `foobarbarbar` virtual subfolder of the above `Directory` directive path: 150 151 .. code-block:: console 152 153 $ ls /srv/http/npm/foobarbarbar/ 154 -rw-r--r-- 1 root root 351 May 24 18:38 foobarbarbar-1.0.0.tgz 155 -rw-r--r-- 1 root root 380 May 25 09:36 package.json 156 157 Set the registry entry in your ``~./npmrc`` as follows: 158 159 .. code-block:: ini 160 161 registry=http://10.1.2.1/npm 162 163 Then restart the Apache server and give it a go: 164 165 .. code-block:: console 166 167 $ npm install --verbose foobarbarbar 168 npm verb cli [ '/usr/bin/node', '/usr/bin/npm', 'install', '--verbose', 'foobarbarbar' ] 169 npm info using npm@7.13.0 170 npm info using node@v16.1.0 171 [...] 172 npm http fetch GET 200 http://10.1.2.1/npm/foobarbarbar/ 6ms 173 [...] 174 npm http fetch GET 200 http://10.1.2.1/npm/foobarbarbar/foobarbarbar-1.0.0.tgz 25ms 175 [...] 176 added 2 packages in 545ms 177 npm timing command:install Completed in 70ms 178 [...] 179 180 181 Cache dodging 182 ============= 183 184 Seriously, ``npm --help`` leaves a lot to be desired. However, the online ``npm install`` documentation does not yield any more clues as to whether it gives you an option of ignoring local cache like you can with ``pip --no-cache``. This is a major pain in the ass, as you have to remember to keep your ``~/.npm`` folder clean between each install attempt. Otherwise, changes you make to the repository won't be used by ``npm install``. 185 186 While testing, it pays to use a folder fairly close to the fs root, like a subfolder in `tmp`. That way, you won't be thrown off by some stray `node_modules` folder somewhere down the tree. 187 188 189 Think locally, act locally 190 ========================== 191 192 Serving packages globally apparently comes down to this: 193 194 1. Pull the ``package.json`` versions list from a proper registry. 195 2. Get the tarball 196 3. Transform the absolute package url to our host instead. (sigh) 197 4. (optional) Prune all the versions and metadata which is not needed (anything that's not in our minimal ``foobarbarbar`` example above). 198 199 Obviously, this is an annoyingly large amount of work to code up parsing and retrieval for. 200 201 Incidentally, there is a very nice tool called `verdaccio <https://verdaccio.org>`_ which gets us most of the way there. [3]_ Its *storage* directory [4]_ uses exactly the directory structure we need. After retrieving a package collection using it as a proxy, getting the files in place is merely a case of copying [5]_ the files from the *storage* directory to the corresponding webserver directory. 202 203 Looking at the ``package.json`` versions wrapper saved by ``verdaccio``, however, we see that it still preserves the absolute path of the upstream registry. Sadly we are still left with doing the third task ourselves. 204 205 As parsing json is one of my least favorite things in the world, I won't include the code for this last step here. For now it's sufficient that we understand the minimums required for manually setting up and serving a static, local, offline npm repository. Regardless of the pain involved. 206 207 208 Making it personal 209 ================== 210 211 Whether we massage jsons ourselves or lazily resort to ``verdaccio``, the final step we need to take is the same: Setting the registry url in the npm configuration of the docker image. 212 213 This is merely a case of setting the registry url in the `npmrc <https://docs.npmjs.com/cli/v6/configuring-npm/npmrc#files>`_ inside the container. 214 215 .. code-block:: console 216 217 $ docker run -it [...] npm config ls -l | grep etc/npmrc 218 globalconfig = "/usr/etc/npmrc" 219 220 To get at our manually provided ``foobarbarbar`` package from before, the dockerfile will be (using archlinux): 221 222 .. code-block:: docker 223 224 [...] 225 RUN pacman -S nodejs npm 226 RUN mkdir -vp /usr/etc 227 RUN echo "registry=http:/10.1.2.1/npm" > /usr/etc/npmrc 228 WORKDIR /root 229 RUN npm install foobarbarbar 230 231 .. 232 233 .. [1] This was the registry address I had in my `~/.npmrc` or whether I put it in myself. 234 235 .. 236 237 .. [2] I have tried using a relative path instead, both with and without a leading ``/``, but in either case the install then errors our complaining that the lock file is "corrupt." Whether the schema allows a base-url setting I do not know, as the schema documentation doesn't seem to be readily available. 238 239 .. 240 241 .. [3] In fact, ``verdaccio`` by itself solves the particular problem that we're trying to solve; providing an offline repository alternative. The task at hand, however, is to understand how to cope *without* using other tools than what can be considered base infrastructure (i.e. a web server). 242 243 .. 244 245 .. [4] Should be ``/etc/verdaccio/storage``; see ``/etc/verdaccio/config.yaml`` 246 247 .. 248 249 .. [5] An alternative approach would be to set the storage path to point to the same folder as the web server is serving. However, since we need to mess with the tarball paths