Raad2: LAMMPS

Sample Job File

#!/bin/bash
#SBATCH -J 61k-kokkos
#SBATCH -p l_long
#SBATCH --qos=ll
#SBATCH -n 9
#SBATCH -c 8
#SBATCH --hint=nomultithread
#SBATCH --time=06:00:00

# Set OMP_NUM_THREADS to value passed 
# in with the "#SBATCH -c " directive

if [ -n "$SLURM_CPUS_PER_TASK" ]; then
  omp_threads=$SLURM_CPUS_PER_TASK
else
  omp_threads=1
fi
export OMP_NUM_THREADS=$omp_threads
echo "OMP_NUM_THREADS=" $omp_threads

export OMP_PROC_BIND=spread
export OMP_PLACES=threads

module load lammps/12Dec18

INPUT=benchmark.in
OUTPUT=benchmark.out

echo "Simulation starting..."

srun lmp_mpi -k on t ${SLURM_CPUS_PER_TASK} -sf kk -in ${INPUT} -log ${OUTPUT}

echo "Simulation Ending..."
echo "Ending at "`date`

How to run this sample job file

1. Collect all required files for job execution within a single directory (e.g. slurm job file, input data, lammps input commands, etc.).

2. Fetch some sample LAMMPS job files and associated data from http://www.hecbiosim.ac.uk/lammps-benchmarks/send/2-software/10-lammps-bench

3. Un-tar the downloaded tarball and copy files from the lammps/61k-atoms folder to the job submission directory.

3. In same directory, copy the job script file above into a text file "run1.job" (could be called anything though)

4. Issue the command "sbatch run1.job"

5. Inside the batch job, note the use of "srun" instead of "mpirun" to launch the parallel job. Also note the lammps executable is called "lmp_mpi".

6. Output files will be generated in same directory.

How to compile you own instance of LAMMPS

Compiling LAMMPS can be a complex and daunting affair especially if you need to include many of its packages/modules in your build. We provide a rather simple compilation example below for a hypothetical user who requires only the packages KOKKOS, MC, MOLECULE, MISC, and USER-REAXC. This compilation supports only MPI for parallelization, and forgoes OpenMP. Naturally, you would be expected use your own installation path as a value for $MY_PREFIX in the instructions below.

Version 12Dec2018

cd $HOME
MY_PREFIX=/ddn/home/fachaud74/lammps
cd $MY_PREFIX
mkdir -p 12Dec18-mpi-only/{install,src}
cd 12Dec18-mpi-only/src 
wget https://github.com/lammps/lammps/archive/stable_12Dec2018.tar.gz
tar xzf stable_12Dec2018.tar.gz
cd  lammps-stable_12Dec2018/
mkdir build
cd build
module sw PrgEnv-cray PrgEnv-intel
module load cray-fftw
cmake \
-D CMAKE_C_COMPILER=mpiicc \
-D CMAKE_CXX_COMPILER=mpiicpc \
-D CMAKE_Fortran_COMPILER=mpiifort \
-D BUILD_MPI=yes \
-D BUILD_OMP=no \
-D LAMMPS_MACHINE=mpi \
-D CMAKE_INSTALL_PREFIX=$MY_PREFIX/12Dec18-mpi-only/install \
-C ../cmake/presets/std_nolib.cmake \
-D PKG_KOKKOS=on \
-D KOKKOS_ARCH=HSW \
-D PKG_MC=on \
-D PKG_MISC=yes \
-D PKG_MOLECULE=yes \
-D PKG_USER-REAXC=on \
-D FFT=FFTW3 \
-D FFTW3_INCLUDE_DIR=/opt/cray/pe/fftw/3.3.8.10/x86_64/include \
-D FFTW3_LIBRARY=/opt/cray/pe/fftw/3.3.8.10/x86_64/lib \
../cmake
make -j 4
make install

Version 23Jun2022

Here is another set of instructions for a more recent version of LAMMPS. This build supports both MPI and OpenMP parallelization, and includes the packages KSPACE, MANYBODY, MC, MEAM, MISC, MOLECULE, RIGID, SMTBQ, and YAFF.

cd $HOME
MY_PREFIX=/ddn/home/$USER/lammps
cd $MY_PREFIX
mkdir -p 23Jun22-hybrid/{install,src}
cd 23Jun22-hybrid/src 
wget https://github.com/lammps/lammps/archive/stable_23Jun2022.tar.gz
tar xzf stable_23Jun2022.tar.gz
cd  lammps-stable_23Jun2022/
mkdir build
cd build
module sw PrgEnv-cray PrgEnv-gnu
module load craype-haswell
module load cray-fftw
cmake \
-D CMAKE_C_COMPILER=cc \
-D CMAKE_CXX_COMPILER=CC \
-D CMAKE_Fortran_COMPILER=ftn \
-D BUILD_MPI=yes \
-D BUILD_OMP=yes \
-D LAMMPS_MACHINE=hybrid \
-D CMAKE_INSTALL_PREFIX=$MY_PREFIX/23Jun22-hybrid/install \
-C ../cmake/presets/basic.cmake \
-D Kokkos_ARCH_HSW=yes \
-D Kokkos_ENABLE_OPENMP=yes \
-D PKG_MEAM=yes \
-D PKG_SMTBQ=yes \
-D PKG_MC=yes \
-D PKG_MISC=yes \
-D PKG_MOLECULE=yes \
-D PKG_YAFF=yes \
-D PKG_USER-REAXC=yes \
-D FFT=FFTW3 \
-D FFT_FFTW_THREADS=on \
-D FFTW3_INCLUDE_DIR=/opt/cray/pe/fftw/3.3.8.10/haswell/include \
-D FFTW3_LIBRARY=/opt/cray/pe/fftw/3.3.8.10/haswell/lib/libfftw3_omp.so \
-D FFTW3_OMP_LIBRARY=/opt/cray/pe/fftw/3.3.8.10/haswell/lib/libfftw3_omp.so \
../cmake
make -j 4
make install