Python Virtual Environments#
This page describes the recommended setup for using Python virtual environments (or virtualenv) on the clusters.
A virtualenv is a way of isolating a particular python environment from the default one. One can install Python packages from public registries without affecting the default environment. Each environment resides in a self-contained directory, so multiple virtualenvs can co-exist side by side with different versions of tools or dependencies installed.
Before you start#
The current default Python version is Python 3.11.7 on Jed and Kuma and Python 3.10.4 on Helvetios and Izar.
Loading the default Python version#
[user@cluster ~]$ module load gcc python py-virtualenv
[user@cluster ~]$ python --version
Python 3.11.7
Choosing a Python version#
[user@cluster:~]$ module spider python
-----------------------------------------------------------------------------------------
python:
-----------------------------------------------------------------------------------------
Versions:
python/3.11.7-bcjr33z
python/3.11.7
Other possible modules matches:
py-biopython py-gitpython py-ipython py-ipython-genutils py-meson-python ...
-----------------------------------------------------------------------------------------
To find other possible module matches execute:
$ module -r spider '.*python.*'
-----------------------------------------------------------------------------------------
For detailed information about a specific "python" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
[user@cluster:~]$ module spider python/3.11.7
-----------------------------------------------------------------------------------------
module spider python/3.11.7
-----------------------------------------------------------------------------------------
python: python/3.11.7
-----------------------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "python/3.11.7" module is available to load.
gcc/13.2.0
intel-oneapi-compilers/2024.1.0
llvm/18.1.8
Help:
The Python programming language.
[user@cluster:~]$ module spider py-virtualenv
-----------------------------------------------------------------------------------------
py-virtualenv:
-----------------------------------------------------------------------------------------
Versions:
py-virtualenv/20.24.5
-----------------------------------------------------------------------------------------
For detailed information about a specific "py-virtualenv" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider py-virtualenv/20.24.5
-----------------------------------------------------------------------------------------
[user@cluster:~]$ module load gcc/13.2.0 python/3.11.7 py-virtualenv/20.24.5
[user@cluster:~]$ python --version
Python 3.11.7
Creating a virtualenv#
We can now create a virtualenv, the name can be whatever one wants, but must not exist yet:
[user@cluster:~]$ virtualenv --system-site-packages venvs/venv-for-demo
created virtual environment CPython3.11.7.final.0-64 in 4798ms
creator CPython3Posix(dest=/home/user/venvs/venv-for-demo, clear=False, no_vcs_ignore=False, global=True)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/user/.local/share/virtualenv)
added seed packages: pip==23.2.1, setuptools==68.2.0, wheel==0.41.2
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
We strongly recommend that, as in the example above, one uses the
--system-site-packages option. This ensures that the optimized packages
already installed in the python modules are used.
An alternative method to create the virtual environment that produces the same results, but works differently (and thus may work in some cases where the method above doesn't) is presented here:
This command will silently produce the virtual environment. All necessary components mentioned above (e.g. `pip``) are installed as well.
Using a virtualenv#
To use a virtualenv it is necessary to activate it:
Once activated the prompt changes and any Python related commands that follow will refer to the Python installation and packages contained/linked in the virtualenv.
If all the packages in the virtualenv are pure Python packages it might not be necessary to load the packages used during the creation of the virtualenv, but when in doubt, it's recommended to always load the same modules used at creation time.
Installing new packages#
To install packages in the virtualenv it needs to be activated, and while it is
active any packages installed with pip will be actually installed inside the
virtualenv itself:
[user@cluster:~]$ source venvs/venv-for-demo/bin/activate
(venv-for-demo) [user@cluster:~]$ module list
Currently Loaded Modules:
1) gcc/13.2.0 2) python/3.11.7 3) python-venv/1.0-odeiohf 4) py-distlib/0.3.7 5) py-filelock/3.12.4-glnrr5z 6) py-platformdirs/3.10.0 7) py-virtualenv/20.24.5
(venv-for-demo) [user@cluster:~]$ pip install --no-cache-dir biopython
When installing packages it is particularly important to have loaded the same modules that were used during the virtualenv creation. This is because if any compilation is necessary during module installation the same compiler and libraries are used.
It is also important to use the --no-cache-dir option to ensure that any
existing pre-compiled copy of the python package is not used. Otherwise we could
be mixing different compilers and architectures which could lead to hard to
debug issues.
Stop using a virtualenv#
To stop using a virtualenv one needs to deactivate it:
Removing a virtualenv#
To permanently remove a virtualenv one simply deletes the directory which contains it.
pip3 environment#
The pip3 command is available in this way: