Skip to content

Python Virtual Environments#

This page describes the recommended setup for using Python virtual environments (or virtualenv) on the clusters.

A virtualenv is a way of isolating a particular python environment from the default one. One can install Python packages from public registries without affecting the default environment. Each environment resides in a self-contained directory, so multiple virtualenvs can co-exist side by side with different versions of tools or dependencies installed.

Before you start#

The default Python version is Python 3.10.4. Python 2.7.18 is still available for backwards compatibility (only virtualenv and pip packages), but should not be used as much as possible.

Loading default Python version#

[user@jed ~]$ module load gcc python
[user@jed ~]$ python --version
Python 3.10.4

Choosing a Python version#

[user@ucluster:~]$ module spider python

--------------------------------------------------------------------------------------------------------------------------
  python:
--------------------------------------------------------------------------------------------------------------------------
     Versions:
        python/2.7.18
        python/3.10.4
     Other possible modules matches:
        py-biopython

--------------------------------------------------------------------------------------------------------------------------
  To find other possible module matches execute:

      $ module -r spider '.*python.*'

--------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "python" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider python/3.10.4
--------------------------------------------------------------------------------------------------------------------------

[user@cluster:~]$ module load gcc/11.3.0 python/3.10.4
[user@cluster:~]$ python --version
Python 3.10.4

Creating a virtualenv#

We can now create a virtualenv, the name can be whatever one wants, but must not exist yet: 

[user@cluster:~]$ virtualenv --system-site-packages venvs/venv-for-demo
created virtual environment CPython3.10.4.final.0-64 in 3557ms
creator CPython3Posix(dest=/home/user/venvs/venv-for-demo, clear=False, no_vcs_ignore=False, global=True)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/user/.local/share/virtualenv)
added seed packages: pip==22.3.1, setuptools==66.1.1, wheel==0.38.4
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

We strongly recommend that, as in the example above, one uses the --system-site-packages option. This ensures that the optimized packages already installed in the python modules are used.

An alternative method to create the virtual environment that produces the same results, but works differently (and thus may work in some cases where the method above doesn't) is presented here: 

[user@cluster:~]$ python -m venv --system-site-packages venvs/venv-for-demo

This command will silently produce the virtual environment. All necessary components mentioned above (e.g. `pip``) are installed as well.

Using a virtualenv#

To use a virtualenv it is necessary to activate it: 

[user@cluster:~]$ source venvs/venv-for-demo/bin/activate
(venv-for-demo) [user@cluster:~]$
Once activated the prompt changes and any Python related commands that follow will refer to the Python installation and packages contained/linked in the virtualenv.

If all the packages in the virtualenv are pure Python packages it might not be necessary to load the packages used during the creation of the virtualenv, but when in doubt, it's recommended to always load the same modules used at creation time.

Installing new packages#

To install packages in the virtualenv it needs to be activated, and while it is active any packages installed with pip will be actually installed inside the virtualenv itself: 

[user@cluster:~]$ source venvs/venv-for-demo/bin/activate
(venv-for-demo) [user@cluster:~]$ module list
Currently Loaded Modules:
  1) gcc/11.3.0   2) bzip2/1.0.8   3) zlib/1.2.12   4) tcl/8.6.12   5) python/3.10.4
(venv-for-demo) [user@cluster:~]$ pip install --no-cache-dir biopython
When installing packages it is particularly important to have loaded the same modules that were used during the virtualenv creation. This is because if any compilation is necessary during module installation the same compiler and libraries are used.

It is also important to use the --no-cache-dir option to ensure that any existing pre-compiled copy of the python package is not used. Otherwise we could be mixing different compilers and architectures which could lead to hard to debug issues.

Stop using a virtualenv#

To stop using a virtualenv one needs to deactivate it: 

(venv-for-demo) [user@cluster:~]$ deactivate
[user@cluster:~]$

Removing a virtualenv#

To permanently remove a virtualenv one simply deletes the directory which contains it. 

[user@cluster:~]$ rm -rf venvs/venv-for-demo/

pip3 environnment#

The pip3 command is available in this way: 

[user@jed ~]$ module load gcc python
[user@jed ~]$ which pip3

/ssoft/spack/syrah/v1/opt/spack/linux-rhel8-icelake/gcc-11.3.0/python-3.10.4-mcloxmxhlovfwwkxzyafm35bq545ntqx/bin/pip3