OpenMP

OpenMP (Open Multi-Processing) is a standard for building parallel applications on shared-memory computers (multiprocessors). It consists primarily of a set of compiler directives, with some library routines besides.

The maximum number of threads available for OpenMP jobs on our clusters is either 4 or 16 depending on the execution host upon which the job is running. The number of threads is communicated to an OpenMP program at execution time with the environment variable OMP_NUM_THREADS via the Grid Engine $NSLOTS variable. Here is the line you need to add to your submissing script for an OpenMP program if your execution shell is bash

 export OMP_NUM_THREADS=$NSLOTS

or csh:

 setenv OMP_NUM_THREADS $NSLOTS

In order to force the compiler to interpret the OpenMP directives in the source code, you need to specify appropriate flags during the compilation, otherwise a serial code will be generated.

Compiler	Option
PGI	-mp
GCC	-fopenmp
Intel	-openmp

The good introduction to OpenMP can be found here.

Thread stack size

See: Memory Management

Hello world program

Here is an example of the simple OpenMP program written in Fortran 90, along with the instructions on how to compile, run interactively and submit this program to the cluster.

program hello

implicit none
include 'omp_lib.h'
integer :: my_rank = 0, p = 1

!$omp parallel private( my_rank )

!$omp single
write(*,'("Hello parallel world!")')
!$ p = omp_get_num_threads()
write(*,'("Number of threads is ", i3)') p
!$omp end single

!$ my_rank = omp_get_thread_num()
write(*,'("Hello world from thread", i3)') my_rank

!$omp end parallel

write(*,'("Back to the sequential world.")')

end program hello

And this is how you compile the code and run it interactively on the head node in csh.

 $ pgf90 -mp -o hello hello.f90
 $ export OMP_NUM_THREADS=4
 $ ./hello
 Hello parallel world!
 Number of threads is   4
 Hello world from thread  3
 Hello world from thread  0
 Hello world from thread  1
 Hello world from thread  2
 Back to the sequential world.
 $

To submit the code hello to the scheduler, you need to create a small submission script. With this script you instruct the scheduler where to execute the code, where to write the output, and with how many threads you want you code to be run. Here is an example of such a script called submit_hello.sh.

 #$ -cwd
 #$ -l h_rt=01:00:00
 #$ -pe openmp 4

 export OMP_NUM_THREADS=$NSLOTS
 ./hello

Finally, to submit the job, type in the command line

 $ qsub submit_hello.sh