Pthreads: Barriers and Working in Phases

Module 11

When everyone must reach the same stage

A mutex protects shared access. A condition variable lets a thread wait for a condition such as “the buffer has an item.” A barrier handles a different situation: all participating threads must finish one phase before any of them starts the next.

This returns to Chapter 4’s barrier discussion (§4.8), including the condition-variable implementation in §4.8.3. We will use the Pthreads barrier interface directly to focus on the work around the barrier. Its behavior is specified in the POSIX barrier documentation.

Our two programs are:

  • pth_phases.c — Publish one value per worker, then read a neighbor’s value.
  • pth_average.c — Repeat a neighbor-averaging calculation in two phases.
NoteRun these examples on Linux

Use PicoCluster for these programs. Its Linux Pthreads implementation supplies pthread_barrier_t and the barrier functions used here. Apple’s macOS Pthreads implementation does not provide this interface. The mutex and condition-variable programs from earlier modules work on both platforms.

First example: publish, wait, then read

Four workers each write one array element. Worker 0 writes 1, worker 1 writes 2, and so on. Then each worker adds its own value to its next neighbor’s value, wrapping around from worker 3 to worker 0.

Predict: What should the four sums be? Why is it insufficient to know that each worker writes a different element?

Create pth_phases.c with the usual headers and these shared declarations:

#define THREAD_COUNT 4

int values[THREAD_COUNT];
int neighbor_sums[THREAD_COUNT];
pthread_barrier_t phase_barrier;

The worker is short:

void *work(void *argument) {
    long id = *(long *)argument;
    values[id] = (int)id + 1;

    pthread_barrier_wait(&phase_barrier);

    int neighbor = ((int)id + 1) % THREAD_COUNT;
    neighbor_sums[id] = values[id] + values[neighbor];
    return NULL;
}

Before the barrier, each worker writes its own values element. After the barrier, all four elements are ready to read. The worker then writes its own neighbor_sums element. There are no competing writers to those result slots, and no worker changes values after the barrier.

Without the barrier, a worker could read its neighbor’s element while that neighbor is still writing it. That is a data race, even if some runs happen to give the expected result. Joining in main comes too late to coordinate those reads inside the workers.

Initialize, wait, and destroy

In main, initialize the barrier before creating workers:

pthread_barrier_init(&phase_barrier, NULL, THREAD_COUNT);

NULL requests default attributes. The last argument is the number of participants in each barrier round. Here it is four workers. Main does not call this barrier, so main is not included in the count.

Create the workers using the stable ID array from Module 8, then join every worker. Destroy the barrier after the joins:

pthread_barrier_destroy(&phase_barrier);

Use the complete source to check the surrounding creation and join loops. As in our earlier examples, the teaching code assumes successful thread and synchronization calls.

gcc -Wall -O2 -pthread -o pth_phases pth_phases.c
./pth_phases

The results, printed by main in worker-ID order, should be:

Worker Value Neighbor sum
0 1 3
1 2 5
2 3 7
3 4 5

A barrier coordinates arrival, not the precise order of instructions after it. This is the same basic idea as MPI_Barrier, but the participants here are threads sharing one process’s memory.

Trace: Suppose workers 2, 0, and 3 reach the barrier first. Where do they wait? What becomes possible when worker 1 arrives? Can we conclude that worker 1 will calculate its sum first?

For a visible experiment, add a greeting just before and just after the barrier. Every “before” greeting should appear before any “after” greeting, but the order within either group can vary.

Second example: repeat the phases

Many parallel algorithms repeat a sequence of stages. We will revisit the neighbor-averaging idea from Module 5, using shared arrays instead of MPI messages. This is a small model for seeing a localized value spread to nearby positions.

Start with twelve values: all zero except position 5, which contains 12. At each step, replace an interior value with the average of its old left neighbor, itself, and its old right neighbor. Keep the two endpoint values fixed.

For the first step, positions 4, 5, and 6 each become 4. For every other interior position, all three old inputs were zero. What should happen in the second step?

We need two arrays, current and next. Reading old values and overwriting that same array in place would let some calculations see a mixture of old and new values. We will instead:

  1. Read current and calculate each worker’s part of next.
  2. Wait until every part of next is complete.
  3. Copy each worker’s part of next into current.
  4. Wait until every copy is complete before starting the next step.

Divide the array indices

With four workers and twelve entries, a cyclic loop gives these assignments:

Worker Array indices
0 0, 4, 8
1 1, 5, 9
2 2, 6, 10
3 3, 7, 11

The same worker owns an index in both phases. Every worker can read the whole current array. No scatter or neighbor-message exchange is needed, because the arrays are already shared.

The core of pth_average.c is:

for (int step = 0; step < STEPS; step++) {
    for (int i = (int)id; i < N; i += THREAD_COUNT) {
        if (i == 0 || i == N - 1) {
            next[i] = current[i];
        } else {
            next[i] = (current[i - 1] + current[i] + current[i + 1]) / 3.0;
        }
    }
    pthread_barrier_wait(&phase_barrier);

    for (int i = (int)id; i < N; i += THREAD_COUNT) {
        current[i] = next[i];
    }
    pthread_barrier_wait(&phase_barrier);
}

The barrier is reusable: after all workers arrive, it resets for the next round. These two waits use the same barrier, with every worker making the calls in the same order.

Discuss the two boundaries:

  • Without the first wait, a fast worker could overwrite current while another worker still reads it for this step’s calculation.
  • Without the second wait, a fast worker could start the next step before other workers have finished updating current.

The barrier does not perform the averaging or copy any values. Our loops do that work. We use it to make the transitions between phases safe.

Build and check it

Create the program using the complete source. Initialize the arrays before launching workers. Main prints the initial values, launches and joins the workers, then prints the final values.

gcc -Wall -O2 -pthread -o pth_average pth_average.c
./pth_average

With N = 12 and STEPS = 3, the final values rounded to three decimal places should be:

0.000 0.000 0.444 1.333 2.667 3.111 2.667 1.333 0.444 0.000 0.000 0.000

Try: Set STEPS to 1 and compare the result with your hand calculation. Then try worker counts 1, 2, and 4. The answers should agree. If a worker has no indices to process, it must still reach both barriers; it is still one of the barrier’s participants.

These arrays are deliberately small so we can trace the output. They are not a useful speed test. Larger workloads can benefit from dividing the work, but waiting and thread management also have costs.

Experiment: What happens without the barriers?

Start with four workers and the original programs. Run each a few times and save the correct output for comparison. For every change below, predict what could go wrong before running it, then recompile and run several times.

Neighbor sums: remove the one wait

In pth_phases.c, comment out the worker’s pthread_barrier_wait call. Leave the joins in main.

  • Could a worker calculate its sum before its neighbor has written a value?
  • Do the joins in main prevent that early read?
  • Does the output always differ from the original?

Restore the wait and recompile before moving on.

Averaging: remove one wait at a time

Use STEPS = 3 in pth_average.c so there is a next iteration to consider. Change the calls inside the worker function, leaving barrier initialization, destruction, and the joins in place.

  1. Remove only the first wait, between calculating next and copying into current. Keep the second wait. Could one worker start copying while another is still calculating? Which array would they access at the same time?
  2. Restore the first wait, then remove only the second wait, after copying into current. Could one worker begin the next iteration while another is still copying? Would it necessarily read a complete set of values from the previous step?
  3. Restore both waits, recompile, and compare with the original output.

Comment out a call for every worker, not just for one worker ID. Otherwise, the remaining workers could wait forever for a participant that never arrives.

ImportantA correct-looking run is not proof

Removing these waits introduces data races: one worker can read a location while another writes it without the necessary synchronization. The program’s behavior is undefined in C. A run might look correct, especially with these small arrays, but repeated correct answers do not make the program safe. Use the experiment to reason about possible overlaps; do not expect a wrong answer on every run.

In the neighbor example, the join happens after the worker has already read its neighbor’s element, so it cannot make that read safe.

In averaging, the first wait prevents changing values that someone is still reading. The second wait prevents reading values that someone is still changing. Each worker owns different output positions, but its calculation reads positions owned by other workers.

Choose the right synchronization

Situation Suitable approach
Workers update one shared total Protect the update with a common mutex, or keep separate partial results.
A consumer needs an item from an empty queue Wait on a condition variable and recheck queue state under the mutex.
Workers need all pieces of an array before the next phase Have all participants reach a barrier.
Main needs workers’ final results Join those workers before reading their results.

The bounded buffer is driven by “empty” and “full” conditions, rather than fixed whole-group phases. A barrier is not a replacement for those conditions. Likewise, Project 3’s separate-result-slot version can simply join the workers before main combines their histograms; it does not need an extra barrier.

A barrier also does not replace a mutex for arbitrary shared updates. Avoid arriving at a barrier while holding a mutex that another participant needs to reach it: the waiting threads could prevent one another from making progress.

Check your understanding

  1. What happens if a four-participant barrier is reached by only three workers?
  2. Why is main excluded from the barrier count in these programs?
  3. Why can the averaging workers write next without locking each assignment?
  4. Why do we need both waits inside the repeated calculation?
  5. Does passing a barrier mean all threads execute their next instruction simultaneously?
  1. The barrier cannot complete; the missing arrival leaves the others waiting.
  2. Only the workers call pthread_barrier_wait. Main waits through joins instead.
  3. Each worker writes distinct elements, while all workers read the unchanged current array during that phase.
  4. The first prevents copying over values that workers still read. The second makes the completed copies available before the next iteration reads them.
  5. No. Everyone must arrive before any participant returns, but scheduling after release can differ.
Back to top