Types of Jobs

Interactive Job

Use of interactive jobs should be kept to a minimum, and used only for testing and development needs.
Interactive jobs should use the least amount of resources sufficient for successful development or testing:

• Do not request more than 2 nodes with an interactive job
• Do not request more than an hour of walltime when using the production queues
• Whenever possible, submit only to the s_debug or express partitions

Use of interactive jobs to launch programs with graphical interfaces is not supported by RC

• Technically you can do it, but we will not spend time fixing issues or problems with such use if and when you encounter problems
  

To launch an interactive job, you much first ssh into one of the "internal login nodes” (raad2-login1 or raad2-login2).
Here is a sample sequence of commands:

ssh raad2-login2
srun --pty --qos sd -p s_debug --gres=craynetwork:0 --time=00:30:00 /bin/bash
echo hello world!

Serial Jobs

Jobs which doesn't span over multiple nodes or serial jobs, should be submitted with parameter --gres=craynetwork:0 to avoid resource wastage.
Cray has 4 network resources available on each node, in order to run parallel jobs.
If you don't at all specify this parameter, cray submit plugin by default adds a request for craynetwork resource considering that job is going to need a network resource.
But for serial jobs, such parameter is not required and only 4 jobs per node gets scheduled.
To avoid this scenario, where other cores on node are wasted, all users submitting to serial queues i.e. s_long, s_short, s_debug partitions should specify --gres=craynetwork:0, telling Cray submit plugin that this job doesn't require any network resource for node to node communication.

Parallele Jobs

To create parallel jobs we use the concept so called "Message Passing Interface" (MPI).
Hence, in our context, a parallele job is also refered to as an "MPI Job".

MPI standard

MPI is a standard that defines a parallel computing model in which each parallel process has its own local memory, and needs to explicitly send messages to other processes in order to share data with them.
Two major specifications of this standard exist: MPICH and OpenMPI.
Both have their pros and cons, but to sum-up, MPICH is known to be more performant and portable while OpenMPI seems more structured and modular.
In Raad2, we have two implementations of MPICH available (mutually exclusive): Cray MPI and Intel MPI.

Intel MPI

Intel MPI Library is a multifabric message-passing library that implements the open source MPICH specification.
It produces efficient MPI applications, espcially on HPC systems using Intel processors.
Intel MPI relies on the Intel Fortran/C/C++ compiler suite.
The Intel MPI Library has the following features:

• Scalability up to 340k processes
• Low overhead enables analysis of large amounts of data
• MPI tuning utility for accelerating your applications
• Interconnect independence and flexible runtime fabric selection

Cray MPI

Cray MPI uses MPICH3 distribution from Argonne:

• It provides a good, robust and feature rich MPI
• Cray provides enhancements on top of it:
  • low level communication libraries
  • Point to point tuning
  • Collective tuning
  • Shared memory device is built on top of Cray XPMEM

For more information about Cray MPI : https://www.hpc.kaust.edu.sa/sites/default/files/files/public//KSL/150607-Cray_training/3.05_cray_mpi.pdf

Demo with Cray Compiler & Cray MPI

demo_mpi.c

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
    // Initialize the MPI environment
    MPI_Init(NULL, NULL);

    // Get the number of processes
    int world_size;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    // Get the rank of the process
    int world_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

    // Get the name of the processor
    char processor_name[MPI_MAX_PROCESSOR_NAME];
    int name_len;
    MPI_Get_processor_name(processor_name, &name_len);

    // Print off a hello world message
    printf("Hello world from processor %s, rank %d"
           " out of %d processors\n",
           processor_name, world_rank, world_size);

    // Finalize the MPI environment.
    MPI_Finalize();
}

slurm.job

#!/bin/bash
#SBATCH -J mpi_tester
#SBATCH -p express
#SBATCH --qos ex
#SBATCH --time=00:05:00
#SBATCH -N 2
#SBATCH --ntasks-per-node=24
#SBATCH --hint=nomultithread

srun -n 48 ./demo_mpi.out

Compilation and Submission Process

Step 1. Compile your C code

Make sure that modules PrgEnv-cray and cray-mpich are loaded by using `module list` command.

muarif092@raad2a:~> cc -o demo_mpi.o demo_mpi.c

Step 2. Make a sample job file and submit to raad2

Copy contents of `slurm.job` header above in a file named `slurm.job` and then submit job.

muarif092@raad2a:~> sbatch slurm.job