Running TeraChem at SCITAS#

TeraChem is a general purpose quantum chemistry code developed to run on NVIDIA GPUs. It supports several methods as described here.

Steps to run TeraChem#

Since TeraChem only runs on GPUs, you need to log in to Izar.

Loading the module#

Once on the cluster you can:

$ module load terachem

Terachem is now on your path. Given the license limitations discussed in a later section, for running long jobs, you need to either set the TeraChemLicense variable in the Slurm script of each job or have it in your .bashrc. This second option is more robust – since you don't risk forgetting it, or mistyping it – so we advise you to run a variation of this command:

$ echo "export TeraChemLicense=$HOME/.terachem" >> ~/.bashrc

Obviously, what you input as a directory here has to match what you define according to the license section.

Slurm script for TeraChem#

You're now ready to run your first TeraChem job. As of today, TeraChem cannot use more than one CPU core. Which means both --ntasks and --cpus-per-task can be set to 1 – or not defined, since these are the default values. You can use this Slurm script can be used as a template for your jobs:

#!/bin/bash -l
#SBATCH --ntasks=1
#SBATCH --time=1:00:00
#SBATCH --mem=60000
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
module load terachem
cd $TMPDIR/
terachem $OLDPWD/molecule_opt.inp > $OLDPWD/molecule_opt.out

By going into $TMPDIR, any temporary files are created in the local storage of the node where the job is running. This improves the I/O and reduces the time the calculation takes to run.

Please note that the Slurm script does not include the export TeraChemLicense command. This script assumes that command has been set in $HOME/.bashrc and is available when the job runs.

License limitations#

TeraChem is a licensed software with either node-locked licenses (license files named license.dat) and floating licenses (files named license.key).

PetaChem generously provides access to TeraChem for unlicensed users for short runs of up to 15 minutes. If you want to test the code you can ask us, since we limit the access to licensed users and other interested parties.

At EPFL, a few labs have TeraChem licenses and these labs can load a valid license by setting an environment variable. With a valid license and provided you have enough tokens available your runs are not limited in time.

Loading a valid license#

Since version 1.96 you can tell TeraChem where to look for the license file by exporting an environment variable:

$ export TeraChemLicense=/path/to/license/dir

Location and permissions of the license file#

While any directory will work for placing the license file, we suggest two options:

• if you're the only one using TeraChem in your lab, place it in $HOME/.terachem. You can chmod 700 $HOME/.terachem to prevent others from accessing it.
• if several of your colleagues use TeraChem, place it in a directory in the /work partition (e.g. /work/<lab>/terachem-license). This way all of you can share one license file and it can more easily be updated in case you renew it or buy more licenses. Since this license is meant to be shared with others in your group you should chmod 750 /work/<lab>/terachem-license.

Note that neither of these is the actual directory where the job is running. Not having the license on the job directory prevents having to move the file each time you launch a new job from a different directory, or having multiple copies of the file in different places.

After you create the directory above, don't forget to put the license file inside the directory.

Checking that the license file is found#

If you follow the steps above, you should be able to run a job with your license. Nevertheless, the first time you run such a job you may want to check on your output to see whether the license was found.

Successful#

If you open the output file and all went well, you should find something similar to what is displayed below (although not necessarily in one continuous block):

| Cannot find license.dat file in the directory /home/user/.terachem

| TRYING THE NETWORK LICENSE...
| Connecting to license server 'license.petachem.com' port '8877'...
| Connected!
| Checking your license...

| **************************************************************
|  Greetings, John Smith! You have 9 licenses in total
|  IN USE: 0
|  AVAILABLE: 9
| **************************************************************

The program correctly found the license and contacted the server. It tells you that of the 9 license tokens you have, 0 are currently in use (i.e. you just launched your first job).

Unsuccessful#

If, on the other hand, the license cannot be located you'll find instead:

| Cannot find license.dat file in the directory /home/user/.terachem

| TRYING THE NETWORK LICENSE...
| Cannot find license.key file in the directory /home/user/.terachem

| *************************** WARNING ******************************
| *                 Running in Demo mode.                          *
| *                                                                *
| * You are using the demo version of TeraChem. Commercial users   *
| * may only use this software for evaluation purposes and are     *
| * not licensed to use the results for any other business purpose.*
| * Use of TeraChem in demo mode is only permitted for             *
| * noncommercial purposes.                                        *
| ******************************************************************

If this is the case and you think you've done all the steps correctly, contact us and detail what you did.

Available versions#

We currently have version 1.96-beta installed. Prior to version 1.96 there was no way for individual users to manage their licenses separately from other users, which makes these versions unsuitable for our shared installation.