OpenMP

OpenMP adds shared-memory parallelism to C, C++, and Fortran through #pragma omp directives, compiled in with -fopenmp on GCC and Clang. The runtime (libgomp on GCC 16.2, the grounding compiler) reads OMP_* variables at startup. Every command and directive below compiled and ran on GCC 16.

Contract

Field	Bound contract
Trigger	The user parallelizes loops on a multicore CPU, needs a reduction or task graph, wants simd hints, offloads to a GPU with target, tunes thread count and binding, or has a program that scales badly or gives wrong answers under threads.
Authority	Reversible local: writes only source files and compiled binaries in the working directory; rollback is deleting them. Environment variables are set for the run, not persisted. No remote mutation.
Side effect	Source and binaries on disk; runs start OMP_NUM_THREADS threads.
Done	The parallel program gives the same result as the serial version, the timing with omp_get_wtime shows the speedup, every shared variable written by more than one thread is under a reduction, atomic, or critical, and thread placement is confirmed by OMP_DISPLAY_ENV.

Inputs

• Compiler with OpenMP support: gcc -fopenmp, clang -fopenmp. -fopenmp-simd enables only the simd directives without the threading runtime.
• Core count and topology (lscpu, nproc) to size threads and places.
• For offload: a GCC built with an offload target (gcc -v lists OFFLOAD_TARGET_NAMES), or a vendor compiler; -foffload=nvptx-none or -foffload=amdgcn-amdhsa selects it and -foffload-options=<target>=<opts> passes target flags such as -march=sm_80.
• The serial baseline result and timing, to check correctness and speedup.

Procedure

1. Parallelize the hot loop with a reduction for the accumulator:

   #include <omp.h>
   #include <stdio.h>
   
   int main(void) {
       const int n = 1000000;
       double sum = 0.0;
       double t0 = omp_get_wtime();
       #pragma omp parallel for reduction(+:sum)
       for (int i = 0; i < n; i++)
           sum += i * 0.001;
       printf("sum = %f threads = %d elapsed = %f s\n", sum, omp_get_max_threads(), omp_get_wtime() - t0);
       return 0;
   }
   

   gcc -fopenmp -O2 -o omp_sum omp_sum.c && OMP_NUM_THREADS=8 ./omp_sum. Done when: the result matches the serial run and the time falls with threads.

2. Choose the schedule from the iteration cost profile:

   Clause	Use when
   schedule(static)	Every iteration costs the same; lowest overhead
   schedule(static, 1)	Interleave iterations across threads for cache blocking
   schedule(dynamic, 64)	Iteration cost varies; chunks of 64 amortize the queue
   schedule(guided)	Cost decreases along the loop; chunk size shrinks
   schedule(runtime)	Read OMP_SCHEDULE at run time to compare without recompiling

   Done when: the schedule matches the measured per-iteration cost and the timing confirms it.

3. Declare data sharing explicitly. private(x) gives each thread an uninitialized copy; firstprivate(x) copies the value in; lastprivate(x) copies the last iteration's value out; shared(x) is one variable for all threads; reduction(op:x) combines per-thread copies with +, *, -, max, min, &, |, ^, &&, ||. Add default(none) to force every variable to be listed and let the compiler catch the omission. A shared variable written by several threads outside a reduction needs #pragma omp atomic (one scalar update) or #pragma omp critical (a block). Done when: default(none) compiles and no unprotected shared write remains.

4. Vectorize with simd. #pragma omp simd before an inner loop asks the compiler to vectorize it; #pragma omp parallel for simd combines threads and vectors. Confirm with -fopt-info-vec, which prints loop vectorized with the vector width for each loop it handled. Done when: -fopt-info-vec reports the loop vectorized.

5. Use tasks for irregular or recursive work:

   #pragma omp parallel
   {
       #pragma omp single
       {
           for (int i = 0; i < 10; i++) {
               #pragma omp task firstprivate(i)
               process_subtree(i);
           }
           #pragma omp taskwait
       }
   }
   

   single lets one thread create the tasks while the team executes them; taskwait joins the children. Done when: the task version produces the serial result.

6. Offload to a GPU when the loop is large enough to pay for the transfer:

   #pragma omp target teams distribute parallel for map(to: a[0:n]) map(from: c[0:n])
   for (int i = 0; i < n; i++)
       c[i] = a[i] * 2.0f;
   

   Build with gcc -fopenmp -foffload=nvptx-none -foffload-options=nvptx-none=-march=sm_80 -o offload offload.c on a GCC with the nvptx offload compiler installed; OMP_TARGET_OFFLOAD=MANDATORY makes a missing device an error instead of a silent host fallback, and OMP_DEFAULT_DEVICE=<n> picks the device. Without an offload-enabled compiler the region runs on the host. cuda covers writing the kernel directly. Done when: OMP_TARGET_OFFLOAD=MANDATORY runs without error and the result matches.

7. Tune the runtime through the environment and print what it saw:

   export OMP_NUM_THREADS=16
   export OMP_PROC_BIND=close      # keep threads near the master
   export OMP_PLACES=cores         # one place per core
   export OMP_WAIT_POLICY=active   # spin instead of sleeping between regions
   export GOMP_SPINCOUNT=300000    # libgomp only: spins before a waiting thread sleeps
   export OMP_MAX_ACTIVE_LEVELS=2  # allow one level of nested parallelism (OMP_NESTED is deprecated)
   export OMP_DISPLAY_ENV=true     # print the effective settings at startup
   

   Done when: OMP_DISPLAY_ENV=true shows the intended values and the timing improves or the change is reverted.

8. Remove false sharing. Threads that write neighboring elements of one array (sum_array[tid] += x) bounce the same cache line between cores. Prefer a reduction; when per-thread slots are needed, pad each to a cache line (double slots[MAX_THREADS][8], 64 bytes per row) or use alignas(64) on a per-thread struct. cpu-cache-opt covers measuring it. Done when: the per-thread accumulator version scales with threads.

9. Profile the parallel regions with Score-P: scorep gcc -fopenmp -O2 -g -o app app.c, run ./app, and read scorep-*/profile.cubex with scorep-score -r scorep-*/profile.cubex (per-region time and visits) or the CUBE GUI; SCOREP_ENABLE_TRACING=true adds an OTF2 trace. TAU wraps the compiler as tau_cc.sh -fopenmp -o app app.c with TAU_MAKEFILE pointing at the configuration; the run writes profile.* files that pprof summarizes and paraprof displays. intel-vtune-amd-uprof shows OpenMP region efficiency in VTune's threading analysis. Done when: the region with the most time and the least parallel efficiency is named.

Failure and recovery

Failure	Cause	Fix
No speedup	Loop too small, or the parallel region is entered too often	Move the parallel outside the hot loop; add if(n > threshold) to the directive
Wrong result	Race on a shared variable	default(none) to find it; then reduction, atomic, or critical
Slower with more threads	False sharing or memory bandwidth ceiling	Pad per-thread data; check bandwidth with linux-perf
Threads migrate or share cores	No binding	OMP_PROC_BIND=close OMP_PLACES=cores; confirm with OMP_DISPLAY_ENV=true
Offload silently runs on the host	Compiler lacks the offload target	OMP_TARGET_OFFLOAD=MANDATORY to make it fail loudly; install the offload compiler
Nested regions oversubscribe	Inner regions spawn full teams	OMP_MAX_ACTIVE_LEVELS=1, or size inner teams with num_threads
simd loop not vectorized	Loop-carried dependence or aliasing	Read the -fopt-info-vec-missed reason; add restrict or a safelen clause

Output

An OpenMP program whose parallel result matches the serial baseline, with the measured speedup from omp_get_wtime, the chosen schedule and data-sharing clauses, the OMP_DISPLAY_ENV output confirming placement, and, when profiled, the region with the most time and its parallel efficiency.