Parallel Programming — OpenMP, MPI and OpenACC

Three HPC lab assignments from Parallel Programming (Applied Mathematics, second year, third trimester, UPF, spring 2023), together with the real job output recorded on the UPF HPC-DTIC cluster and a Python script that mines those outputs into timing and speedup tables.

The skeleton code in every exercise was provided by the course. The student's contribution is the parallelisation — the OpenMP directives, the MPI decomposition and communication, the OpenACC data clauses, the bug fixes, and the experiment design behind the measurements. Wherever an unmodified handout copy of a file survives on disk, this repository points at the exact lines that changed; where it does not, that is stated rather than guessed.

Two files carry an explicit third-party byline that was left untouched:

The five bug*.c programs in labs/lab2-mpi/3_bugs/ are the well-known deliberately-broken MPI teaching examples; the exercise is to find and fix the defect, so the diff against the broken original is the work, not the file.

The technology per lab was determined by reading the sources, not assumed. Note in particular that lab 3 is OpenACC, not OpenMP: delaunay.c includes <omp.h>, but only to call omp_get_wtime() for timing — every parallel construct in it is #pragma acc.

labs/          sources + the SLURM .cmd job script actually submitted, one dir per lab
results/       a curated sample of the recorded .out/.err files (49 of 271)
analysis/      parse_results.py — reads .out files, prints Markdown timing tables
docs/          the three lab reports the students submitted (PDF)

These are the commands the job scripts record.

# Lab 1 — OpenMP, one node
cd labs/lab1-openmp/1_cholesky && make && OMP_NUM_THREADS=8 ./cholesky 3000
cd labs/lab1-openmp/2_sudoku   && make && OMP_NUM_THREADS=4 ./sudoku
cd labs/lab1-openmp/3_histogram && make && OMP_NUM_THREADS=16 ./histogram
cd labs/lab1-openmp/4_product  && make && OMP_NUM_THREADS=2 ./product   # requires exactly 2

# Lab 2 — MPI
cd labs/lab2-mpi/1_rates   && make && mpirun -np 2 ./sendrecvp          # 2 nodes, 1 rank each
cd labs/lab2-mpi/2_polling && make && mpirun -np 4 ./count              # needs votations.dat in cwd
cd labs/lab2-mpi/3_bugs    && make && mpirun -np 2 ./bug1               # bug4 needs exactly 4
cd labs/lab2-mpi/4_stencil && make stencil_alltoallw \
    && mpirun -np 256 ./stencil_alltoallw 4096 10 1000 16 16            # n energy iters px py

# Lab 3 — OpenACC, two targets from the same source
cd labs/lab3-openacc/1_triangulation
make cpu && ./delaunay 5000 2000 1500     # pgcc -ta=multicore
make gpu && ./delaunay 5000 2000 1500     # pgcc -ta=tesla:managed
python txt2img.py image                   # renders the image.txt the run wrote

On the cluster each of these went through SLURM. The job.cmd next to every exercise is the script as last submitted, e.g.:

sbatch labs/lab2-mpi/4_stencil/job.cmd   # 16 nodes, 256 tasks, 32 tasks/node, 3 h

Requirements: an OpenMP-capable C compiler for lab 1; an MPI implementation for lab 2; the NVIDIA HPC SDK (pgcc/nvc) plus a CUDA-capable GPU for lab 3's -ta=tesla target. txt2img.py needs matplotlib and numpy.

Every number below was read out of a .out file recorded on the cluster in May–June 2023. Regenerate them with:

python analysis/parse_results.py           # parses ./results
python analysis/parse_results.py <dir>     # parses any tree of .out files

The tables in this README come from running that script over the full original job archive (271 .out files); results/ ships a 49-file sample so the script still has something to chew on.

job_new.cmd builds the same source twice, -ta=multicore then -ta=tesla:managed, and runs both back to back, so each .out file is a matched pair.

The single largest measured saving: the triangulation kernel at 5000 points drops from 1736.24 s to 51.87 s. The < 1× entries are honest — at 100 points the triangulation is so short (~16 ms) that host-to-device transfer dominates and the GPU loses. Triangle counts are identical on both targets (189 / 1982 / 9978 for 100 / 1000 / 5000 points), which is the correctness check.

The collective MPI_Alltoallw halo exchange is statistically indistinguishable from hand-rolled point-to-point at every size measured — the report's own conclusion, and the numbers bear it out. The lab 2 report also quotes 13.23 s / 13.13 s at 256 processes (speedup 26.29× / 26.40×); no surviving .out file backs those two figures — the only 256-process job that left artefacts here (out_stencil_336323) segfaulted and was killed at the wall-time limit.

Aggregated over the 6 recorded runs (job.cmd requests --cpus-per-task=16; the two oldest job ids may predate that setting):

A deliberately negative result, and the point of the exercise: with 20 000 increments per bucket and essentially no arithmetic per iteration, any mutual-exclusion scheme is 10–200× slower than doing nothing in parallel. Only the array reduction — which gives each thread a private histogram and merges once — beats the serial loop, and only modestly. (The lock variant is additionally penalised by a code defect; see the lab 1 README.)

All six runs print Matrices are equal. These six files are exactly the static row of the scheduler sweep in docs/lab1-report.pdf (chunk sizes 1/5/10/25/50). The recorded files contain no sequential baseline — see Limitations — so no speedup can be computed from them. The report's own strong-scaling table, measured separately, gives:

and finds schedule(dynamic) with chunk size 1 best for both phases (13.36 s / 23.26 s at 8 cores).

Best sweep (out_sendrecv_331559, the run reproduced in the report):

Peak ≈ 2 992 MB/s ≈ 23.4 Gbps, against the cluster's documented 56 Gbps interconnect — under half of nominal, as the report notes.

4 ranks, votations.dat (800 000 bytes = 50 000 × 16-byte records), read collectively with MPI_File_set_view + MPI_File_read_all:

The local counts sum exactly to the MPI_Allreduce total, in all 11 recorded runs, which are byte-identical.

• Sudoku (OpenMP tasks, --cpus-per-task=4): solved in 3.32–4.07 s across 7 runs (mean 3.65 s). No sequential baseline was recorded. The report is candid that the workload is too small for the task overhead to pay off.
• Producer/consumer (omp sections, 2 threads, N = 10 000): 0.000174 s, sum 191.945870. One recorded run.

analysis/parse_results.py classifies each of the 182 job .out files in 3_bugs/ (183 files, one of which is the make.out build log) by reading its content — the job script was edited in place, so file names only carry the SLURM job id and say nothing about which bug was being chased:

That distribution is the story of the exercise: bug1 is a two-line tag mismatch and cost 7 runs; bug3 and bug4 are subtler and cost 73 and 39. The wrong result verdicts are decided arithmetically — bug3 must end with Final sum exactly twice the initialised sum, and 24 runs did not.

• José Mª Pérez Clar — repository owner
• Martí Oms Graells — lab partner, co-author of all three labs and all three reports

Skeleton code, exercise statements and the deliberately-buggy MPI programs are the property of the Parallel Programming course, Universitat Pompeu Fabra. product.c is by Tim Mattson (2007); random.c follows Numerical Recipes.