Compiling LAMMPS with Kokkos+ML-PACE#

Before you start#

At SCITAS we have all the tools needed to compile a standard LAMMPS version. To include Kokkos support you need:

• to load the kokkos module;
• to download the source code of a recent version of LAMMPS, at least the Update 2 for Stable release 23 June 2022, since older versions have an issue compiling ML-PACE. (Note: for this tutorial I used the 2Aug2023_update1 version.)

For an optimal compilation of LAMMPS with Kokkos support you need to provide some information about the cluster hardware as CMake keywords:

• the CPUs on the nodes are Skylake Xeon server CPUs (SKX);
• the GPUs are of the Volta architecture with compute capability 7.0 (VOLTA70).

Compiling LAMMPS with Kokkos#

You're now ready to build LAMMPS. On my tests I used the Intel compiler, Intel MPI and the corresponding Kokkos module. (More on the GCC later on.)

Kokkos provides a program called nvcc_wrapper which will be used to compile LAMMPS. There is a minor issue with this nvcc_wrapper so we recommend starting the process with:

$ module load intel intel-oneapi-mpi intel-oneapi-mkl python cmake cuda kokkos
$ export NVCC_WRAPPER_DEFAULT_COMPILER=$CC

This will make sure the right compiler is chosen, rather than the which would be a system compiler and not the stack one.

You can then use a variation of this procedure that fits your needs:

$ mkdir lammps/build
$ cd lammps/build/
cmake ../cmake \
  -DBUILD_SHARED_LIBS=ON -DLAMMPS_EXCEPTIONS=OFF -DBUILD_MPI=ON \
  -DBUILD_OMP=ON -DCMAKE_BUILD_TYPE=Release -DFFT=MKL \
  -DPKG_COMPRESS=ON -DPKG_CORESHELL=ON -DPKG_DIPOLE=ON -DPKG_GRANULAR=ON \
  -DPKG_KSPACE=ON -DPKG_MANYBODY=ON -DPKG_MC=ON -DPKG_EXTRA-PAIR=ON \
  -DPKG_MOLECULE=ON -DPKG_PERI=ON -DPKG_PYTHON=ON -DPKG_QEQ=ON \
  -DPKG_REPLICA=ON -DPKG_RIGID=ON -DPKG_SHOCK=ON -DPKG_ML-SNAP=ON \
  -DPKG_SRD=ON -DPKG_ATC=ON -DPKG_DIFFRACTION=ON -DPKG_EXTRA-DUMP=ON \
  -DPKG_ML-PACE=ON \
  -DPKG_KOKKOS=ON \
  -DPKG_VORONOI=OFF \
  -DPKG_MISC=OFF \
  -DKokkos_ARCH_SKX=ON \
  -DKokkos_ARCH_VOLTA70=ON \
  -DKokkos_ENABLE_CUDA=ON \
  -DKokkos_ENABLE_OPENMP=ON \
  -DCMAKE_CXX_COMPILER=$(which nvcc_wrapper) \
  -DMPI_CXX_COMPILER=$MPICXX \
  -DCMAKE_INSTALL_PREFIX=/path/to/the/installation/dir/lammps
$ make
$ make install

At this point you have your own LAMMPS installed on /path/to/the/installation/dir/lammps.

The choice of PKG_SOMETHING depends on your needs and you can opt not to include the ones shown above, or to include others. Please note:

• we're explicitly disabling both the Voronoi package and the MISC package (i.e. -DPKG_VORONOI=OFF -DPKG_MISC=OFF) since at present the installation fails otherwise. If you need PKG_MISC take a look at the GCC section below;
• we're adding -DKokkos_ARCH_SKX=ON and -DKokkos_ARCH_VOLTA70=ON explicitly since, unlike Kokkos itself, LAMMPS will not recognize the architecture of the hardware.

Compiling with GCC#

If you are trying to compile with GCC here are a few things worth noting:

• you can use the intel-mkl module with -DFFT=MKL or;
• you can use the openblas fftw modules with -DFFT=FFTW3;
• PKG_MISC compiles with GCC. So, if you need that package and your using GCC you can -DPKG_MISC=ON;
• from my tests, Lammps will not compile with Kokkos 3.6 from our stack. You'll need to install Kokkos 3.7.01 following the instructions below;
• if compiling with GCC you may see errors like error: redefinition of 'constexpr const size_t .... This is due to an old C++ standard being defined. To change the C++ standard used run the command:
  $ sed -i 's/-std=c++14/-std=c++17/g' CMakeFiles/*.dir/flags.make and type make again.
• you can try using the -DCMAKE_EXE_LINKER_FLAGS to hard-code the search paths to the libraries. (This currently does not work for Intel.) For instance: -DCMAKE_EXE_LINKER_FLAGS="-Wl,-rpath,$CUDA_ROOT/lib64 -Wl,-rpath,${OPENBLAS_ROOT}/lib"

Extra: Compiling Kokkos#

In case you want to compile your own version of Kokkos (e.g. to have newer features, or to work around a bug) you need to start by getting a recent version from their website. You can either clone it with git, or download a recent version from the releases page. The installation at SCITAS uses Kokkos 3.6.0, so we recommend that as the minimum version. For Lammps compiled with GCC you may need 3.7.01 at the very minimum.

Here's a minimal example of how to install Kokkos:

$ cd kokkos-3.7.01
$ mkdir build
$ cd build/
$ Sinteract -p build -c 20 -m 20G -t 20:00 -g gpu:1
$ module load gcc cuda hwloc cmake
$ cmake .. \
    -DKokkos_ENABLE_CUDA=yes \
    -DKokkos_ENABLE_OPENMP=yes \
    -DKokkos_ENABLE_HWLOC=ON \
    -DKokkos_ARCH_SKX=ON \
    -DKokkos_ARCH_VOLTA70=ON \
    -DCMAKE_INSTALL_PREFIX=/path/to/the/installation/dir/kokkos
$ make
$ make install

Once this is done, you're ready to use your own version of Kokkos. Following the structure shown above you could use it by doing:

$ module load gcc cuda
$ export KOKKOS_ROOT=/path/to/the/installation/dir/kokkos
$ export PATH=$KOKKOS_ROOT/bin:$PATH