manbytesgnu_site

Source files for manbytesgnu.org
git clone git://holbrook.no/manbytesgnu_site.git
Info | Log | Files | Refs

20210419_docker_python.rst (5967B)


      1 Local python repository
      2 #######################
      3 
      4 :date: 2021-04-26 07:55
      5 :category: Offlining
      6 :author: Louis Holbrook
      7 :tags: docker,python,devops
      8 :slug: docker-offline-2-python
      9 :summary: Feeding python packages to your offline Docker setup
     10 :series: Offline Docker
     11 :seriesprefix: docker-offline
     12 :seriespart: 2
     13 :lang: en
     14 :status: published
     15 
     16 
     17 ..
     18         .. CAUTION::
     19 
     20            This is a purely technical article. You should probably be `this geeky <https://g33k.holbrook.no/c18beedd>`_ to continue reading.
     21 
     22 In the previous part of this series we were able to connect a Docker network to a virtual interface on our host, neither of which have access to the internet. That means we are ready to host content for the container locally. And we will start out with creating a local Python repository.
     23 
     24 Packaging the packages
     25 ======================
     26 
     27 I'll be so bold as to assume that you are using ``pip`` to manage your packages. It gives you not only the option to *install* packages, but merely *download* them to storage aswell. So let's do that and try to serve the packages.
     28 
     29 
     30 .. code-block:: bash
     31 
     32         $ pip download faker
     33         [...]
     34         Successfully downloaded faker python-dateutil six text-unidecode
     35         $ ls
     36         Faker-8.1.0-py3-none-any.whl  python_dateutil-2.8.1-py2.py3-none-any.whl  six-1.15.0-py2.py3-none-any.whl  text_unidecode-1.3-py2.py3-none-any.whl
     37         $ python -m RangeHTTPServer 
     38 
     39 .. code-block:: bash
     40 
     41         $ python -m venv .venv
     42         $ . .venv/bin/activate
     43         (.venv) $ pip install --index http://localhost:8000 faker
     44         Looking in indexes: http://localhost:8000
     45         ERROR: Could not find a version that satisfies the requirement faker (from versions: none)
     46         ERROR: No matching distribution found for faker
     47 
     48 Dag nabbit. Apparently not that simple. And indeed, if we read up on the spec for `PEP 503 spec for Simple Repository API`_, we learn that we are going to stick those package files under directories named after the packages.
     49 
     50 Well, nothing that a bit of bash scripting can't sort out:
     51 
     52 .. include:: code/docker-offline-python/pep503.sh
     53    :code: bash
     54 
     55 
     56 Armed with this, let's try again:
     57 
     58 .. code-block:: bash
     59 
     60         $ sh /home/lash/bin/shell/pep503.sh . packages
     61         $ mkdir packages
     62         $ bash /home/lash/bin/shell/pep503.sh . packages
     63         mkdir: created directory 'packages/text-unidecode'
     64         './text_unidecode-1.3-py2.py3-none-any.whl' -> 'packages/text-unidecode/text_unidecode-1.3-py2.py3-none-any.whl'
     65         mkdir: created directory 'packages/six'
     66         './six-1.15.0-py2.py3-none-any.whl' -> 'packages/six/six-1.15.0-py2.py3-none-any.whl'
     67         mkdir: created directory 'packages/python-dateutil'
     68         './python_dateutil-2.8.1-py2.py3-none-any.whl' -> 'packages/python-dateutil/python_dateutil-2.8.1-py2.py3-none-any.whl'
     69         mkdir: created directory 'packages/faker'
     70         './Faker-8.1.0-py3-none-any.whl' -> 'packages/faker/Faker-8.1.0-py3-none-any.whl'
     71         $ find packages/ -type f
     72         packages/faker/Faker-8.1.0-py3-none-any.whl
     73         packages/python-dateutil/python_dateutil-2.8.1-py2.py3-none-any.whl
     74         packages/six/six-1.15.0-py2.py3-none-any.whl
     75         packages/text-unidecode/text_unidecode-1.3-py2.py3-none-any.whl
     76         k $ python -m RangeHTTPServer 
     77 
     78 .. code-block:: bash
     79 
     80         $ pip install --index http://localhost:8000/packages faker
     81         Looking in indexes: http://localhost:8000/packages
     82         Collecting faker
     83         [...]
     84         Successfully installed faker-8.1.0 python-dateutil-2.8.1 six-1.15.0 text-unidecode-1.3
     85 
     86 
     87 Extra prepping
     88 ==============
     89 
     90 There are some basic packages you will most always need, and which ``pip`` often will expect to find in at least one of its available repositories, even regardless of whether its installed or not. If you don't have this on your local offline repository when the internet goes *poof*, then that will block any build you are trying to do.
     91 
     92 Let's make sure we have the packages around all the time:
     93 
     94 .. code-block:: bash
     95 
     96         $ pip download pip setuptools setuptools-markdown wheel
     97         [...]
     98         $ bash /home/lash/bin/shell/pep503.sh . packages
     99         [...]
    100 
    101 
    102 Choosing a server
    103 =================
    104 
    105 As I tend to favor the classics, I still use ``Apache Web Server`` to host stuff to my local environment. One practical (if not all too safe) thing about it is that it will automatically bind to all interfaces. So to make the repository available, we simply link or add the ``packages`` directory to the virtual root, and restart the server e.g. with
    106 
    107 .. code-block:: bash
    108 
    109         systemctl restart httpd
    110 
    111 Of course you can use any HTTP server you like, as long as you know how to bind it to the virtual interface.
    112 
    113 To verify that the service is providing what's needed, simply point your web browser to the location, e.g.
    114 
    115 .. code-block:: bash
    116 
    117         lynx http://10.1.2.1/packages/
    118 
    119 Serving the packages
    120 ====================
    121 
    122 Now that we are all prepped, the next step is to do install packages from within a Docker container.
    123 
    124 Let's start with an Archlinux base image with basic python provisions.
    125 
    126 .. include:: code/docker-offline-python/Dockerfile.pythonarch
    127    :code: docker
    128 
    129 Build and tag it with ``pythonbase``, and then let's add the Dockerfile to test the repository:
    130 
    131 .. code-block:: docker
    132 
    133         [...]
    134 
    135         RUN pip install --index http://10.1.2.1/packages --trusted-host 10.1.2.1 faker
    136 
    137 
    138 Now, turn internet off (and lights, too, if you'd like some extra suspense), and build the second file.
    139 
    140 .. code-block:: bash
    141 
    142         $ docker build . 
    143         Sending build context to Docker daemon  3.072kB
    144         [...]
    145         Successfully installed faker-8.1.0 python-dateutil-2.8.1 six-1.15.0 text-unidecode-1.3
    146         Removing intermediate container 2c10ffcdf3ad
    147         ---> 1ba83bb8e111
    148         Successfully built 1ba83bb8e111
    149 
    150 
    151 ..
    152 
    153         .. _PEP 503 spec for Simple Repository API: https://www.python.org/dev/peps/pep-0503