Running Docker images using Apptainer/Singularity#

Before you start#

Apptainer/Singularity is the most widely used container system for HPC. It is designed to execute applications at bare-metal performance while being secure, portable, and 100% reproducible. Apptainer is an open-source project with a friendly community of developers and users. The user base continues to expand, with Apptainer/Singularity now used across industry and academia in many areas of work.

Authentication#

You can use public images without being authenticated in the following cases:

Registry	Image Type	Login required
Dockerhub	Public	No
Dockerhub	Private	Yes, Dockerhub credentials
C4science registry	Public	No
C4science registry	Private	Yes, c4science registry credentials

When using authentication, it's relative to the source you want to use (Dockerhub and c4science registry supported) and the authentication is saved per cluster.

C4science registry account

You need to ask for an account on the c4science registry. Please change your password as soon as you receive the temporary password.

Upload an image to Apptainer/Singularity from Dockerhub using the registry.c4science.ch (preferred)#

Prerequisite

You need to have Docker installed on your machine

Get a docker image from Dockerhub for instance (From local computer)
```
$ docker pull alpine:latest
$ docker images
```
Set up your machine (From local computer)

Login on the C4Science registry from your local Docker installation (you need an account)
```
$ docker login registry.c4science.ch
Username (username): username
Password:
Login Succeeded
```
Upload a Docker image to the registry (From local computer)
1. On the web interface, create a Project on the registry (private or public)
2. Tag the image you want to upload on your local machine and push it to the registry
Tag name

Do not use the - character in the tag name, only letters, numbers and underscore
```
$ docker tag alpine:latest registry.c4science.ch/<yourproject>/alpine:latest
$ docker push registry.c4science.ch/<yourproject>/alpine:latest
```

Pull an image on Apptainer/Singularity and specify a user or group ACL (From cluster login node)

From each cluster frontend (i.e.: jed.hpc.epfl.ch), login to the registry, pull the image and check it's was pulled with success.

$ mkdir ~/myimages
$ cd ~/myimages
$ apptainer remote login --username <username> docker://registry.c4science.ch
Password / Token: 
INFO:    Token stored in /home/<username>/.apptainer/remote.yaml
$ apptainer pull docker://registry.c4science.ch/<yourproject>/alpine:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 63b65145d645 done  
Copying config 6a2bcc1c7b done  
Writing manifest to image destination
Storing signatures
2023/03/10 09:50:14  info unpack layer: sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c
INFO:    Creating SIF file...
$ ls
alpine_latest.sif

Upload an image directly from Dockerhub to Apptainer/Singularity#

Optional (needed for private images only)

$ apptainer remote login --username <username> docker://registry-1.docker.i

Get the image from Dockerhub by specifying the image account (optional for official image) and name

$ mkdir ~/myimages
$ cd ~/myimages
$ apptainer pull docker://registry-1.docker.io/library/alpine:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 63b65145d645 skipped: already exists  
Copying config 6a2bcc1c7b done  
Writing manifest to image destination
Storing signatures
2023/03/10 10:25:14  info unpack layer: sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c
INFO:    Creating SIF file...
$ ls
alpine_latest.sif 

$ srun --pty --qos debug -N 1 apptainer shell alpine_latest.sif 
Apptainer> cat /etc/alpine-release 
3.17.2

Run the image on the clusters#

You can submit the following Slurm script with the sbatch command

#!/bin/bash -l

#SBATCH --nodes 1
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --mem 1024
#SBATCH --qos debug

srun apptainer exec alpine_latest.sif ls /etc

You can check the Slurm output file:

sbatch apptainer.run 
Submitted batch job 2649847
$ ls
alpine_latest.sif  apptainer.run  slurm-2649847.out
$ cat slurm-2649847.out 
alpine-release
apk
conf.d
crontabs
...

Interactive Shell#

To have an interactive shell within your image, simply use this:

$ srun --pty --qos debug -N 1 apptainer shell <filename>.sif