Intranet Informatique

Introduction & overview using HPC Slurm Cluster scheduler

A workload manager has the ability to reserve resources for your jobs for being executed.

Several commands to interact with the workload manager :

sbatch : submit a job script inside a queue (called partitions in Slurm) ;

scancel : cancel job ;

squeue : reports state of jobs ;

sinfo : reports the state of partitions and nodes ;

srun : initiate job steps in real time ;

salloc : allocate resources for a job in real time. Typically this is used to allocate resources and spawn a shell ;

sprio : display a detailed view of the components affecting a job’s priority.

Available documentation for slurm command can be found here: http://slurm.schedmd.com/man_index.html

sview graphical application is a user interface to get and update state information for jobs, partitions, and nodes managed by Slurm

Reservation creation for a job

The ability to reserve resources for jobs is being executed by slurm directives

ex:

#SBATCH -N 4

#SBATCH –tasks-per-node 8

will reserve 4 nodes and 8 cores per node (default, 1 task per core )

Submission script could be written in the language of your choice (bash, tcsh, python, perl...

Some useful options:

• -p : partition name for your job
• -N : number of nodes (min – max)
• -n : number of tasks (1 per node by default)
• –cpus-per-task : number of cores per task
• –tasks-per-node= :tasks per nodes
• -t J-HH:MM:SS : allocated time for a job, before it will be stopped
• –gres=gpu:N : number of GPU per node
• –mail-type=END : send an email at the end of a job (BEGIN for receiving at the beginning . ALL for receiving step by step)
• –mail-user=my@address : your email address
• –mem=0000M : memory per node in Mo (could be in Go, replace M by G)
• -o output_file : the output file

Start a job

To start a job you have to submit to the cluster scheduler by sbatch command

ex:

sbatch my_script

Job status

It is possible to display the job status with scontrol or squeue command

Current job list

squeue

Jobs list of a user

squeue -u <login>

State of the job

scontrol show job <jobid>

My job is pending….When it will start?

when a job is pending Slurm will make a periodic calculation of probable start-up time.

2 ways to show your job status:

scontrol show job <jobid> | grep StartTime=
squeue -o "%S" -j <jobid>

SRUN

▶ srun is the standard SLURM command to start an MPI program.
▶ It is well integrated with SLURM.
▶ It automatically uses the allocated job resources: nodelist, tasks, logical cores
per task.
▶ It chooses an optimal CPU binding for the tasks on an allocated host.

MPIRUN
▶ mpirun is a wrapper script that mimics srun and automatically copies settings
from the SLURM batch job, if available.
▶ by default, mpirun takes affinity from SLURM
– export SLURM_CPU_BIND=none
– alternatively, use export I_MPI_PIN_RESPECT_CPUSET=no to override

▶ unset I_MPI_PMI_LIBRARY
▶ do NOT use #SBATCH –export=none, it causes confusing errors.

Why srun instead of mpirun?

srun minimizes the risk of a mismatch between the resources allocated by Slurm and those used by Open MPI

srun as the SLURM all-around parallel-tasks distributor; once a particular set of resources is allocated, the nature of your application doesn’t matter (MPI, OpenMP, hybrid, serial farming, pipelining, multiprogram, etc.), you just have to srun it

Somme examples of batch script submission…..

1. Threaded/OpenMP job script

#!/bin/bash
# Job name:
#SBATCH --job-name=test
#
# Partition:
#SBATCH --partition=partition_name
#
# Request one node:
#SBATCH --nodes=1
#
# Specify one task:
#SBATCH --ntasks-per-node=1
#
# Number of processors for single task needed for use case (example):
#SBATCH --cpus-per-task=4
#
# Wall clock limit:
#SBATCH --time=00:00:30
#
## Command(s) to run (example):
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
./a.out

2. Simple multi-core job script (multiple processes on one node)

#!/bin/bash
# Job name:
#SBATCH --job-name=test
#
#SBATCH --account=account_name
#
# Partition:
#SBATCH --partition=partition_name
#
# Request one node:
#SBATCH --nodes=1
#
# Specify number of tasks for use case (example):
#SBATCH --ntasks-per-node=20
#
# Processors per task:
#SBATCH --cpus-per-task=1
#
# Wall clock limit:
#SBATCH --time=00:00:30
#
## Command(s) to run (example):
./a.out

3. Alternative MPI job script

#!/bin/bash
# Job name:
#SBATCH --job-name=test
#
# Partition:
#SBATCH --partition=partition_name
#
# Number of nodes needed for use case:
#SBATCH --nodes=2
#
# Tasks per node based on number of cores per node (example):
#SBATCH --ntasks-per-node=20
#
# Processors per task:
#SBATCH --cpus-per-task=1
#
# Wall clock limit:
#SBATCH --time=00:00:30
#
## Command(s) to run (example):
module load gcc openmpi
mpirun ./a.out

4. Hybrid OpenMP+MPI job script

#!/bin/bash
# Job name:
#SBATCH --job-name=test
#
# Partition:
#SBATCH --partition=partition_name
#
# Number of nodes needed for use case (example):
#SBATCH --nodes=2
#
# Tasks per node based on --cpus-per-task below and number of cores
# per node (example):
#SBATCH --ntasks-per-node=4
#
# Processors per task needed for use case (example):
#SBATCH --cpus-per-task=5
#
# Wall clock limit:
#SBATCH --time=00:00:30
#
## Command(s) to run (example):
module load gcc openmpi
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

mpirun ./a.out

5.Multiple Parallel Jobs Simultaneously

#!/bin/bash# for all of the jobs to complete, even the longest 
#SBATCH -t 02:00:00
# Ask for the total number of cores the jobs need
#SBATCH -n 56 

# Clear the environment from any previously loaded modules
module purge > /dev/null 2>&1

# Load the module environment suitable for the job
module load openmpi/10.4-ucx

# And finally run the job​s
srun -n 8 --cpu_bind=cores ./a.out > a.out 2>&1 &
srun -n 38 --cpu_bind=cores ./b.out > b.out 2>&1 &
srun -n 10 --cpu_bind=cores ./c.out > c.out 2>&1 & 

wait

6.Disabling Hyperthreading

#!/bin/bash
#SBATCH --job-name nohyperthreading-test
#SBATCH --partition=partion_name
#SBATCH --time 01:00:00
#SBATCH --ntasks 10
#SBATCH --ntasks-per-node 10

srun -n $SLURM_NTASKS --hint=nomultithread ./a.out

1 2 3 4