The ironic state of dependency management in Python (II)

December 11, 2023

◾In the previous post of this series we talked about environments and interpreters and how to adjust them to every need. Now let's talk about virtual worlds.

Python virtual worlds

Let's assume that we already have the right interpreter running. Now it's time to install the project dependencies. One of them is the tech.py library version 2.0. Since it is a system that hosts several projects, it so happens that Python already has that package installed because another project makes use of that library.

But there is a problem. The library that is installed is version 1.2 of tech.py and our project uses many functions and classes defined in version 2.0. Even if we force the installation, when we run the project, it will give error. What do we do? Because we only have one Python interpreter with that version.

The Python project itself tried to provide a solution to this problem in pep-582. It aimed to share interpreter but at the same time, make the libraries of each project independent of each other by putting them in a folder: __pypackages__ local to the project.

In this way, there would be no collision and the same interpreter would use some libraries or others when starting from a different project. A good idea... that ended up being rejected by the committee itself.

How can we have multiple projects and separate their dependencies so that they don't get in each other's way?

The solution is to have a copy of the interpreter, its associated libraries and binaries and change the paths pointing to the python binary. In other words, duplicate the environment and isolate it from the outside by making the shell "believe" that Python is installed in a different and alternate path to the system installation (and even to that of pyenv). It is what is called, with no further mystery: virtual environment.

And that is exactly what a virtual environment management tool or library does: it copies or binds the same executable and provides an environment of associated libraries, as long as the appropriate environment variables are manipulated to activate it.

So now, when we install Python packages or project dependencies, they will be isolated to that virtual environment, without affecting any other environment at all. This solves the problem of having one interpreter serving multiple projects.

How do we use it? We have several options, the most direct one is using the venv module included in all Python distributions from version 3.3 onwards (by the way, its respective pep is here).

As easy as this:

That creates an environment called 'mientornovirtual' in the same directory of the project.

However, we must activate it before working with it in a similar way to how we activate the Python versions with pyenv:

From there, everything we do will be on the environment that is in that folder.

When we finish working, we only have to deactivate it to return to the original interpreter:

What are the alternatives to the management of virtual environments?

We have the project that originated the creation of virtual environments in Python, virtualenv. In fact, it is the tool to be used instead of the module (which we would use only if we do not have virtualenv). The usage is exactly the same, but we would invoke virtualenv instead of the module.

The bonus if you are using pyenv

In case you are using pyenv to install and activate interpreters, there is a plugin or pyenv that allows us to manage virtual environments.

Everything is integrated in a single tool with this plugin: installation and selection of the interpreter and also the creation and management of virtual environments.

Another advantage is that it does not create a folder in the project, but in a directory as a cache of virtual environments so that in projects that we consider common (that do not have problems of dependencies) we can reuse the environments already created only with its name.

In addition, if we indicate the name of the virtual environment in the .python-version file, when we enter the project directory, it will be activated automagically. Top.

Are we sure that we already have everything that technology gives us to have a reproducible Python environment?

Well, no. There is still a "small" detail that we will see in the next post.

—CONTINUING THIS SERIES