Running COMSOL on our clusters#

Target audience#

This how-to is meant for people with a working knowledge of COMSOL (e.g. on their laptop) wanting to use the SCITAS clusters to run larger simulations than feasible on standard machines. It covers details like connecting to clusters and launching your simulations.

This how-to is not meant for a total beginner, since it does not cover any of the basics of COMSOL and does not address creating models.

Before you start#

Access to COMSOL is restricted and the number of licenses fairly limited. If you do need to use COMSOL you have to go through this page and choose the relevant version of COMSOL (for research or for teaching). This access is not managed by SCITAS.

Launching COMSOL jobs on the clusters#

There are two main ways of launching COMSOL on the clusters:

in batch mode, i.e. independently from an active COMSOL session;
from the COMSOL desktop environment on your own computer.

The first option is ideal for a cluster, since you can define and fine tune many parameters directly on the command line, which are hard or impossible through the GUI. As such, we will dedicate most of the document to this method. With this first method you save the model or models you want to study, send them to the cluster and run them there, retrieving the files at the end.

A later section briefly goes through the steps to connect from the GUI directly to the cluster. As of the writing of this tutorial, many options are hard to change while submitting jobs from the GUI. You cannot easily change the number of cores used per job, for instance. With this method you may find the COMSOL window to be blocked for the duration of the job, even though the job is running on a different machine.

Launching COMSOL jobs in batch mode#

Once you save your model (let's assume it's called my_model.mph) you need to upload it to the cluster:

If you using Linux or MacOS X your easiest option is to use a terminal and the scp command to transfer the files.
If you are using Windows you may need to install a program or two (e.g. PuTTY to connect to the cluster and WinSCP to transfer files).

You'll also need a script to submit the jobs to the scheduler on the cluster (let's call it comsol.slurm.sh). The script should be something along these lines:

#!/bin/bash -l
#SBATCH -J comsol-project
#SBATCH --time=24:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=30000
module load comsol/6.0
comsol batch slurm -np ${SLURM_CPUS_PER_TASK} -inputfile my_model.mph

You're now ready to run your first job. You'll need to copy your files to the cluster first. On my computer I'm using Linux so I type the following on the terminal (adjust as needed):

$ scp my_model.mph comsol.slurm.sh fidis:/scratch/ddossant/comsol_project/

All you need to run your COMSOL job is now on the cluster. You then need to log in to the cluster and submit the script above to the scheduler:

$ ssh ddossant@fidis
$ cd /scratch/ddossant/comsol_project/
$ sbatch comsol.slurm.sh
Submitted batch job 16143086

If you did everything right that last command should return a message with the batch job ID (16143086 in my example). The progress of the calculation will be written in a Slurm output file which is specific to the job you are running. In the case of the job above you can take a quick look at the progress with a command like:

$ tail slurm-16143086.out
---------  Current Progress:  95 % - Constraint handling
Memory: 4465/4708 11477/11816
---------  Current Progress:  95 % - Creating multigrid hierarchy
Memory: 4142/4708 11151/11816
Iter      SolEst     Damping    Stepsize #Res #Jac #Sol LinIt   LinErr   LinRes
---------  Current Progress:  95 % - Solving linear system
Memory: 4067/4708 11076/11816
   1      0.0013   0.5000000      0.0013   22   11   11   259  0.00065        -
---------  Current Progress:  95 % - Assembling matrices
Memory: 3749/4708 10759/11816

This is similar to the output you get within the GUI, but with some more information.

Optimizing your job script#

While the example script above works fine, you may want to change some options. Adjusting the number of cores is trivial (just change the corresponding value on the #SBATCH --cpus-per-task line).

But there are other options that may be useful to change. By default COMSOL will store a lot of data in the $HOME/.comsol directory. This may be a problem since at SCITAS your home directory is limited in size. Several options allow you to control this behavior. The most important one concerns the recovery files which are the biggest files stored in that directory. A script that stores those files directly on the node could be like this:

#!/bin/bash -l
#SBATCH -J comsol-project
#SBATCH --time=24:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=30000
module load comsol/6.0
comsol batch slurm -np ${SLURM_CPUS_PER_TASK} -recoverydir $TMPDIR/recovery -inputfile my_model.mph

Besides avoiding the size limit of your home, storing these files on the node, also improves the performance as writing to the $TMPDIR is faster than writing to your home.

Keep in mind that $TMPDIR is a directory that Slurm deletes at the end of the job. But if the job finishes properly the files we are storing there are not needed, so we don't have to worry about them. In case you'd rather keep the recovery files an easier option is perhaps:

#!/bin/bash -l
#SBATCH -J comsol-project
#SBATCH --time=24:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=30000
module load comsol/6.0
comsol batch slurm -np ${SLURM_CPUS_PER_TASK} -recoverydir $PWD/recovery.@process_id -inputfile my_model.mph

The @process_id is replaced by a number specific to the job, so you can have multiple jobs running at the same time, without risking overwriting files from different simulations.

And since we want to maximize the performance, we can store every temporary file related to the job on the node. We can also turn off the autosave feature of the recovery files, since it's unlikely the nodes of the cluster will fail. You can do that with something like:

#!/bin/bash -l
#SBATCH -J comsol-project
#SBATCH --time=24:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=30000
module load comsol/6.0
comsol batch slurm -np ${SLURM_CPUS_PER_TASK} -autosave off -data $TMPDIR/data -configuration $TMPDIR/configuration -prefsdir $TMPDIR/prefs -tmpdir $TMPDIR/tmp -inputfile my_model.mph

By doing this we can get a few percentage points of improvement on the performance, relative to the standard calculation.

By default COMSOL will update the model file in place, but you may also find it interesting to separate the input model from the output model. In the example below, the input file is read, but nothing is written to it:

#!/bin/bash -l
#SBATCH -J comsol-project
#SBATCH --time=24:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=30000
module load comsol/6.0
comsol batch slurm -np ${SLURM_CPUS_PER_TASK} -inputfile my_model.mph -outputfile my_model_solved.mph

You're all set. You can now run any COMSOL job you want. A word of caution about performance, though...

Performance and scaling#

You are perhaps tempted to think that with 40 cores per node your jobs will run 10 times faster than on your 4 core laptop. Unfortunately things don't work that way. Parallelization is never perfect and COMSOL is not the most efficient code out there.

As such, before you start submitting your jobs to full nodes on our clusters, you should perhaps test the scalability of your calculations. Some methods parallelize better than others and for those that do you can scale to more cores without a big penalty. Other methods, or parts of calculations, will run large segments on only one core or a few cores and this can significantly impact the performance. Part of being a proficient user of a code is knowing its strengths and limitations.

Submit a job that is representative of what you'll be doing (perhaps close to the final model, but with a coarser mesh) with --cpus-per-task equaling 2, 4, 8 and perhaps even 16. If at each step the time reduces to close to one half of the previous test, then the scaling is still good. If at some point the performance degrades significantly (e.g. rather than 50% of the time, it takes 75% of the time of the previous test) then perhaps it makes less sense to use that many cores.

And as mentioned earlier, temporary files stored on the node (and not in your home), as well as deactivating unnecessary options (e.g. the turning off the recovery files with --autosave off) helps reduce the time to solution. And if you're planning on doing many simulations, the gains add up.

Launching COMSOL jobs from the desktop environment#

If you're not ready not deal with the details described above and are willing to accept the limitations imposed by running from the GUI, then COMSOL Desktop can be configured to submit jobs to the clusters. Just to make it clear, this is COMSOL running on your computer, but launching jobs that run on the cluster. The jobs run independently of your computer, but as long as the connection to the cluster is working COMSOL will update some information about the job.

As of August 2022, this page on COMSOL's website describes in detail how to configure COMSOL in order to connect to a cluster. That page assumes you are using a Windows machine and have done a full installation of PuTTY with one of the Windows installers. If you need help getting started with PuTTY, the first couple sections of this documentation should be enough for COMSOL. Since we use the same scheduler as mentioned in the COMSOL page (i.e. Slurm), most of the instructions are valid. The section Settings for the Cluster Computing Node has a few details that are specific to each cluster. You'll want to edit most of these directly in the Preferences menu, so that they are defaults for all jobs and not just the current one.

After you open the Preferences, go to the Multicore and Cluster Computing section and change the following:

Number of cores: 4
Additional scheduler arguments: --cpus-per-task=4 --time=72:00:00 --mem=30000 (choose values that fit your needs)
User: leave this blank
Queue name: standard
Batch directory: A directory on your own computer where the files will be stored (e.g. a COMSOL directory within your Documents)
External COMSOL batch directory path: A directory within your scratch, in my case it could be /scratch/ddossant/comsol/outputfiles
External COMSOL installation directory path: For COMSOL 6.0 use /ssoft/spack/external/comsol/6.0

And then still within Preferences you need to edit Remote Computing:

Activate Run Remote
Remote invoke command: SSH
SSH command: Putty
SSH directory: path to your PuTTY installation (probably C:\Program Files\PuTTY)
SSH key file: path to the SSH key you created on your own computer (probably stored in your Documents as a file with a .ppk extension). You need to add the public part of this key to the cluster!
SSH user: your GASPAR user name
File transfer command: SCP
SCP command: Putty
SCP directory: likely the same as the SSH directory above
SCP key file: the same as the SSH key file above
Remote hosts list: Table
Remote hosts: jed or, if you know have access to it Helvetios
Remote OS: Linux

With these options set and after adding a Cluster Computing node to your study, you should be able to run your jobs directly on the cluster when you hit Compute.

There is no easy way to adjust the number of cores, the amount of RAM, or the total time asked to Slurm. You'll need to change the values of Additional scheduler arguments in Preferences and launch the job again.

Troubleshooting#

This sections is meant to be a work in progress. If you think you have solved an issue that may be relevant to others, contact us so we can add that information here.

Job fails with an out-of-memory error#

One of the lines of your batch script is of the form #SBATCH --mem=10000. If the value you set was too small for the job you tried to run you may have an error like:

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 4819 RUNNING AT f104
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
   Intel(R) MPI Library troubleshooting guide:
      https://software.intel.com/node/561764
===================================================================================
slurmstepd: error: Detected 1 oom-kill event(s) in step 16148135.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

There is no way of telling COMSOL how much RAM to use. It will see all the RAM available in the node and feel free to use it. However, we configure our clusters with control groups (or cgroups, as in the output above) and we limit the amount of RAM the program can use to what you asked on the #SBATCH --mem=10000 line. If you see this kind of message you'll need to increase the amount of RAM you asked for in your Slurm script. If you launch your jobs through the GUI you need to change the memory value of the Additional scheduler parameters in Preferences.

SSH path is not a directory#

If you've configured the COMSOL Desktop installation on your computer to be able to send jobs to a cluster (as explained above) and you then opt to submit jobs directly to the batch scheduler, some of the options on the GUI may interfere with the correct functioning of the batch job. If you encounter this error:

$ tail slurm-16147807.out
----- Time-Dependent Solver 1 in fpt time/Solution 2 (sol2) ------------------->
Running: Study 2

/******************/
/*****Error********/
/******************/
The SSH path is not a directory
Saving model: /scratch/ddossant/comsol_project/my_model_solved.mph
Save time: 2 s.
Total time: 681 s.

You need to open your model in the GUI and delete the Cluster Computing section from the study. Cluster Computing is meant for submiting directly from the GUI to the cluster, but not when launching jobs directly from the cluster.

License errors#

The school has a relatively small number of COMSOL licenses. Your job may fail with an error like:

/******************/
/*****Error********/
/******************/
Could not obtain license for COMSOL ...
License error: -5.
No such product exists.
Feature:       COMSOL
License path:  ...
FlexNet Licensing error:-5,414

One common issue is caused by having COMSOL running on your laptop/workstation using the tokens for the same features you need for your jobs. Please close any COMSOL sessions you are not actively using and try again.

The exact number of licenses depends on the specific feature you're trying to use. If the number of jobs you can run seems a bit erratic (some times you can launch more, some times less) remember that you are also sharing the licenses with other users. More importantly, the license server doesn't check the licenses in real time, but at regular intervals. This means that you may be able to launch a few jobs if they all start in a narrow time window in between the checks by the server. But if they happen to start at different moments they could fail due to the lack of licenses.

A possible workaround for this is to submit multiple jobs in the same session. This is explained in a bit more detail in this COMSOL webpage but suffice it to say that several calculations within one job should work. So, an example script for this could be:

#!/bin/bash -l
#SBATCH -J comsol-project
#SBATCH --time=24:00:00
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=8
#SBATCH --mem=120000
module load comsol/6.0
comsol batch slurm -np ${SLURM_CPUS_PER_TASK} -inputfile model_1.mph -outputfile model_1_solved.mph &
comsol batch slurm -np ${SLURM_CPUS_PER_TASK} -inputfile model_2.mph -outputfile model_2_solved.mph &
comsol batch slurm -np ${SLURM_CPUS_PER_TASK} -inputfile model_3.mph -outputfile model_3_solved.mph &
comsol batch slurm -np ${SLURM_CPUS_PER_TASK} -inputfile model_4.mph -outputfile model_4_solved.mph &
wait

Each of these jobs would use 8 cores and, according to COMSOL, this should allow you to run multiple jobs at once.

Submitting multiple jobs#

If it seems cumbersome to launch only one job at a time you can try setting up Slurm dependencies between the jobs. Supposing you typically launch a job with:

$ sbatch submit_comsol.sh
Submitted batch job 123456

When you submit your next job you could tell it to depend on this job being finished before the new job is actually eligible to run. You can do this with:

$ sbatch --dependency=afterany:123456 submit_comsol.sh
Submitted batch job 123457

This won't avoid issues if you're competing for licenses with other users, but at least you avoid the situation where your own jobs are killing your next calculations.

Last update: July 24, 2023