CryoSPARC on Izar#

Step-by-step guide#

Installation:#

Prior to the installation you will have to request a license_id on the obtaining a license ID page and the base port number from SCITAS by email to 1234@epfl.ch with the title "HPC: CryoSPARC base port number"
To install CryoSPARC, log in to the front node izar3.hpc.epfl.ch and run the script as follows:
```
/ssoft/spack/external/cryosparc-scripts/cryosparc_install.sh -v
```
When prompted, enter <license_id> and <port_number> obtained in step 1.

Use:#

Method 1#

Once installed, you can start the master on the front node of the cluster izar3.hpc.epfl.ch (this is under test and might change depending on the load generated by the CryoSPARC masters)
```
/ssoft/spack/external/cryosparc-scripts/cryosparcm.sh start
```
It should give you the address at which you can connect yourself to the web page. Since this is on the front node you should be able to connect from anywhere on the EPFL network, on sites or through VPN

!!! note The cryosparcm.sh script is just a wrapper that sources the config.sh file and runs cryosparcm In addition it changes the variables mentionning the hostname.

Create your CryoSPARC user account (you only need to do this once)
```
/ssoft/spack/external/cryosparc-scripts/create_user.sh
```
When prompted, enter the required information.

Warning

If you asked for the free license, do not create user accounts for your colleagues since this would be in breach of the license terms. If a lab/group wants to have one installation/instance for multiple users, please contact SCITAS for an agreement on security.
Create the lanes for job submission inside CryoSPARC (you only need to do this once)
```
/ssoft/spack/external/cryosparc-scripts/create_lanes.sh
```
Warning

For jobs submitted using the SSDExclusive lanes, the maximum number of GPUs available on a node will be charged instead of the number of GPUs selected.

On your local machine, generate an SSH key and copy the public key from your computer to any of the clusters. Please, have a look at the dedicated documentation to learn how to do it.
Set up a multi-hop connection from your local machine to the remote host. To do this, open the file ~/.ssh/config (or create it if it doesn't exist) and add the following lines:
```
Host *
  ServerAliveCountMax 4
  ServerAliveInterval 15

Host izar
  Hostname izar.hpc.epfl.ch
  User <username>

Host i*.izar
  User <username>
  ProxyCommand ssh -q izar -W $(echo "%h" | sed -e 's/.izar//'):%p
```
Steps 1 and 2 only need to be done once. You need to replace <username> with your user name at SCITAS.

If you're using Windows, you can first install and open Windows Subsystem for Linux (WSL), and then perform the steps above. WSL can be installed by entering the command wsl --install in an administrator PowerShell (restarting and creating linux user are required afterwards). Then, WSL can be started by entering the command wsl in a PowerShell.
On the front node of izar, start an interactive session on a compute node allocated through Sinteract and start the master there.
```
Sinteract -n 10 -g gpu:1 -t 24:00:00 -m 90GB
/ssoft/spack/external/cryosparc-scripts/cryosparcm.sh start
```
After, type the command hostname to get the name (<node name>) of the allocated compute node.
On your local machine, start an SSH tunnel to expose port <port_number> from the allocated compute node to your local machine.
```
ssh -N -f -L localhost:<port_number>:localhost:<port_number> <node name>.izar
```
Now, open your browser and navigate to http://localhost:<port_number>. You should be presented with the CryoSPARC login page.
Once you complete the jobs, stop the master and quit the allocated compute node (this is important because the compute node allocation is charged)
```
/ssoft/spack/external/cryosparc-scripts/cryosparcm.sh stop
exit
```

Update CryoSPARC#

Complete or kill running jobs. It is also highly recommended to make a backup of your database

$HOME/cryosparc/cryosparc_master/bin/cryosparcm backup --dir=<backup_path> --file=<backup_file>

Update the master (automatic)

$HOME/cryosparc/cryosparc_master/bin/cryosparcm update

Once the automatic update above is complete, navigate to the directory $HOME/cryosparc/cryosparc_master via command-line. Look for the latest downloaded worker release, named cryosparc_worker.tar.gz. Copy this file to the directory $HOME/cryosparc/cryosparc_worker/. Navigate to that directory and run
```
bin/cryosparcm update
```

CryoSPARC live on reserved nodes (draft)#

Purchase work storage at SCITAS using the request storage form. Note that raw data generated by CryoEM is about 3TB per day and the live mode can take up to 7 days, so a quota of 20TB is expected (double check the numbers with DCI). Disk space on /work is likely to be measured every day and billed per month.
Reserve izar nodes (4 GPUs in total are needed. It can be 1 node of 4 GPUs or 2 nodes of 2 GPUs each) at SCITAS using the request a reservation form and reserve the CryoEM at DCI over the scheduled period of live experiment (typically one week). Note that reservations can take some days, and can be cancelled quickly.
When the reservations are done, launch the script
```
/ssoft/spack/external/cryosparc-scripts/create_reserved_lanes.sh
```
and enter your SCITAS account and the reservation name. This will create the lane with node reservation inside CryoSPARC. Use this lane for all jobs in the live mode and assign the number of GPUs for each job type according to this CryoSPARC guide.
When the reservation period starts, ask DCI to transfer data to your work directory at SCITAS and submit jobs in CryoSPARC live interface.
Once the live experiment is done or interrupted due to unexpected problems, cancel the node reservation by sending a request to 1234@epfl.ch with the title "HPC: Cancel node reservation"