In Module 4, we scattered an array into blocks. Each process then worked on its own values. But what if calculating one value also needs the value just across a block boundary? Today we will exchange those edge values without gathering the whole array first.
We will build a small ring message and then take one step in a model of heat spreading along a bar. The programs are mpi_ring.c and mpi_smooth_once.c. The book’s Chapter 3 develops point-to-point communication and uses MPI_Sendrecv in its parallel sorting discussion (§3.7); our heat example is a new application of that same idea.
A message around a ring
Imagine four ranks arranged in a circle. Each sends its rank number to the rank on its right and receives from the rank on its left:
0 → 1 → 2 → 3 → 0Predict: What number will rank 0 receive? What about rank 2?
The neighboring ranks are:
int right = (rank + 1) % size;
int left = (rank - 1 + size) % size;The % size wraps the last rank back to 0. Adding size before the second % keeps left nonnegative for rank 0.
One call sends and receives:
MPI_Sendrecv(&rank, 1, MPI_INT, right, 0,
&received, 1, MPI_INT, left, 0,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);Read the first line as “send my rank to right with tag 0.” Read the second as “receive one integer from left with tag 0 and store it in received.” The tag helps match a receive with the intended message. MPI_STATUS_IGNORE means we do not need extra information about this receive.
Here is the complete program:
#include <stdio.h>
#include <mpi.h>
int main(void) {
int rank, size;
MPI_Init(NULL, NULL);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
int right = (rank + 1) % size;
int left = (rank - 1 + size) % size;
int received = -1;
MPI_Sendrecv(&rank, 1, MPI_INT, right, 0,
&received, 1, MPI_INT, left, 0,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Rank %d received %d from rank %d\n", rank, received, left);
MPI_Finalize();
return 0;
}mpicc -g -Wall -o mpi_ring mpi_ring.c
mpirun -n 4 ./mpi_ringThe lines may print in any order, but the relationships should be the same: rank 0 receives 3, rank 1 receives 0, rank 2 receives 1, and rank 3 receives 2. Try one process too. It sends a message to itself and receives its own rank.
For the full list of arguments, see the MPICH MPI_Sendrecv reference.
If every process first made a blocking MPI_Recv call, each would wait for a message that no process had sent yet. MPI_Sendrecv performs the paired exchange safely, without us having to choreograph which rank sends first. This is why the book uses it when processes exchange data during sorting.
A line of values instead of a ring
Consider twelve positions along a bar. Position 6 starts hot (100); all others start at 0:
index: 0 1 2 3 4 5 6 7 8 9 10 11
temperature: 0 0 0 0 0 0 100 0 0 0 0 0For each interior position, calculate a new value from its old value and its two old neighbors:
\[ \text{new}[i] = \frac{\text{old}[i-1] + 2\,\text{old}[i] + \text{old}[i+1]}{4}. \]
Keep positions 0 and 11 fixed at 0. This is a simple smoothing model of heat flow, not a detailed physical simulation. We must finish reading all old values before replacing them, or the answer will depend on the order of the loop.
Predict: After one step, what are positions 5, 6, and 7? All other positions should remain 0.
With three processes, MPI_Scatter gives each rank four consecutive values:
| Rank | Global positions | Needed from outside its block |
|---|---|---|
| 0 | 0–3 | Old value at 4 |
| 1 | 4–7 | Old values at 3 and 8 |
| 2 | 8–11 | Old value at 7 |
The copied edge value from a neighboring block is often called a ghost value or halo value. It is only a copy for this step. We exchange edges again before the next step, after values change.
The downloadable example keeps this small problem fixed with #define N 12, so its array sizes and loops always agree. Passing a size at runtime is a separate step; we will try command-line arguments below.
Exchange both edges
Unlike a ring, the bar does not wrap around. Rank 0 has no left neighbor; the last rank has no right neighbor. MPI’s MPI_PROC_NULL gives us a safe “no process” endpoint. Sending to it or receiving from it has no effect, so we initialize missing edge values to 0.
int left_rank = rank == 0 ? MPI_PROC_NULL : rank - 1;
int right_rank = rank == size - 1 ? MPI_PROC_NULL : rank + 1;
double left_edge = 0.0, right_edge = 0.0;
MPI_Sendrecv(&local[0], 1, MPI_DOUBLE, left_rank, 0,
&right_edge, 1, MPI_DOUBLE, right_rank, 0,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Sendrecv(&local[local_n - 1], 1, MPI_DOUBLE, right_rank, 1,
&left_edge, 1, MPI_DOUBLE, left_rank, 1,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);In the first call, first values travel left, so each rank receives its right neighbor’s first value. In the second, last values travel right, so each rank receives its left neighbor’s last value. The different tags identify the two exchanges. Everyone calls them in the same order.
Trace it: With three ranks, which two old values does rank 1 receive? Which local element should it send in each direction?
Calculate one step
Each process chooses a left and right value for each element. Most neighbors are already in its local array; only the elements at a block edge need the received values.
for (int i = 0; i < local_n; i++) {
int global_i = rank * local_n + i;
if (global_i == 0 || global_i == N - 1) {
next[i] = 0.0;
} else {
double left = i == 0 ? left_edge : local[i - 1];
double right = i == local_n - 1 ? right_edge : local[i + 1];
next[i] = (left + 2.0 * local[i] + right) / 4.0;
}
}global_i matters: each process has a local index 0, but only rank 0 owns the global endpoint 0. We write results to next and leave local unchanged while calculating. Afterward, MPI_Gather puts the twelve new values in rank order on rank 0.
The complete program includes initialization, scatter, the edge exchange, the calculation, gather, and finalization. Compile and run it:
mpicc -g -Wall -o mpi_smooth_once mpi_smooth_once.c
mpirun -n 1 ./mpi_smooth_once
mpirun -n 3 ./mpi_smooth_once
mpirun -n 4 ./mpi_smooth_onceAll three runs should print:
After one step: 0 0 0 0 0 25 50 25 0 0 0 0The program uses #define N 12; its arrays, loop bounds, and process-count check all use N. Change that one definition to use another small size, and choose a process count that divides it. The initial hot spot is at N / 2. As in Module 4, handling unequal blocks is a later step (MPI_Scatterv/MPI_Gatherv). These small arrays help us check correctness; they are not a useful speed benchmark.
We have seen intermittent multi-node runs that print the correct answer and then wait inside MPI_Finalize. That is a cluster/runtime issue still under investigation, not a reason to remove MPI_Finalize from the program. For an in-class fallback that stays on pc0, use /usr/bin/mpirun -hosts pc0 -genv HWLOC_COMPONENTS -gl -n 4 ./mpi_smooth_once. Keep your ordinary multi-node result if it completes.
A number from the command line
Our small heat example keeps N fixed. Project 2 needs a different input size without editing and recompiling the program each time. In C, main can receive arguments from the shell:
int main(int argc, char *argv[])argc counts the words given to the program, including its own name. argv holds those words as strings: argv[0] is the program name, and argv[1] is the first value after it. For ./mpi_argument 120, argc is 2 and argv[1] is the string "120". atoi converts numeric text to an integer. This small example rejects a missing argument or a result less than 1; atoi does not detect every malformed string, so use a positive whole number here.
With MPI, let rank 0 interpret the input, then broadcast it so every rank uses the same value:
if (rank == 0 && argc == 2) {
n = atoi(argv[1]);
}
MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);All ranks must call MPI_Bcast. A zero or negative n makes all ranks take the same exit path after the broadcast. Here is the complete mpi_argument.c:
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
int rank, n = 0;
MPI_Init(NULL, NULL);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (rank == 0 && argc == 2) {
n = atoi(argv[1]);
}
MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
if (n < 1) {
if (rank == 0) fprintf(stderr, "Usage: mpi_argument N (positive integer)\n");
MPI_Finalize();
return 1;
}
printf("Rank %d knows n = %d\n", rank, n);
MPI_Finalize();
return 0;
}Try it:
mpicc -g -Wall -o mpi_argument mpi_argument.c
mpirun -n 4 ./mpi_argument 120
mpirun -n 4 ./mpi_argumentIn the first command, -n 4 belongs to mpirun and means four processes; 120 comes after the executable and is an argument to our C program. Every rank should report n = 120, though the lines may appear in any order. The second run should print the usage message and exit cleanly. On PicoCluster, the normal mpirun alias still selects the hosts as before.
For the fixed classroom demos, #define N 12 gives us a compile-time size, so double values[N] is simple. A value read from argv is known only at runtime. For a larger variable-sized problem, use arrays with a checked maximum size (as Project 2 permits) or allocate memory after validating the input. Avoid putting a very large runtime-sized array on the stack. We do not need a new compiler flag for the examples here.
Questions to discuss
- Why is a line different from a ring at its two ends?
- When rank 1 owns positions 4–7, why can’t it read position 8 from its local array?
- Why do we need two arrays (
localandnext) for one step? - What would happen if we exchanged edges only once but took ten steps?
- Why do the one-, three-, and four-process runs have the same numerical answer even though they divide the data differently?
- In
mpirun -n 4 ./mpi_argument 120, which number controls MPI processes and which is read by our program?
If we have time
- Move the initial hot position to 3. Predict which block boundaries matter with three processes, then run the program.
- Try two processes. Where is the block boundary, and what values cross it?
- Read about
MPI_Allgatherin §3.4.9 of the book. How would its result differ from our finalMPI_Gather? Would giving every rank the entire new array remove the need for neighbor messages in the next step, and what extra communication would that involve?
For next time
Read the MPI_Sendrecv discussion in Chapter 3, §3.7, and revisit §§3.4.6–3.4.9 on distributing arrays and collecting results. Try the command-line example above, then begin Project 2: Prime Number Census with MPI, where you will combine broadcast, scatter, reduce, and gather.