Module 2 – Beau Christ

From one process to several

So far, our programs have done their work in one process. Now we will run several processes and have them work together.

MPI stands for Message Passing Interface. It gives us functions for communicating between processes. We will still write C; MPI adds a library that our programs can use.

We will start with three questions:

Which process am I?
How do I send something to another process?
How do we divide a calculation and combine the answers?

This follows the beginning of Chapter 3. The book’s first program sends greeting strings to one process for printing. We will start with each process printing its own greeting, then send a single integer before combining partial sums.

Source files

Create these files as we go, or download the complete examples:

mpi_hello.c — Identify the processes.
mpi_message.c — Send one integer between two processes.
mpi_sum.c — Divide a sum among several processes.

Keep the files in the same working directory on PicoCluster. If you need a reminder about logging in or editing files, see Module 0.

Processes and memory

A process is a running instance of a program. If we launch four MPI processes, each one starts running the same program and has its own variables.

Process 0          Process 1          Process 2          Process 3
own variables      own variables      own variables      own variables
       ↔ messages between processes ↔

Changing a variable in process 0 does not change a variable with the same name in process 1. To share a value in these examples, we send a message.

Several processes can run on the same board, or they can run on different boards. A node is a computer in the cluster; a process is a running program. They are not the same thing. One node can run several processes.

Think about it: If every process runs the same program, how can we give them different jobs?

Hello from every process

Create mpi_hello.c:

#include <stdio.h>
#include <mpi.h>

int main(void) {
    int my_rank;
    int comm_sz;

    MPI_Init(NULL, NULL);
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
    MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);

    printf("Hello from process %d of %d!\n", my_rank, comm_sz);

    MPI_Finalize();
    return 0;
}

What are the new pieces?

Piece	Meaning
`#include <mpi.h>`	Makes MPI’s function declarations and constants available
`MPI_Init(NULL, NULL)`	Starts MPI for this process; we are not passing command-line arguments here
`MPI_COMM_WORLD`	The communicator containing all processes in this launched program
`MPI_Comm_rank`	Stores this process’s rank in `my_rank`
`MPI_Comm_size`	Stores the number of processes in `comm_sz`
`MPI_Finalize()`	Finishes this process’s use of MPI

A communicator identifies a group of processes and a context for their messages. We will use MPI_COMM_WORLD throughout this module.

A rank is a process’s number within that communicator. With four processes, the ranks are 0, 1, 2, and 3. Each process gets a different my_rank, but all four get the same comm_sz.

The & passes the address of a variable so MPI can fill it in. This is the same idea as passing &start to our timer in Module 1.

Keep the MPI work between MPI_Init and MPI_Finalize. Each process calls both. As in the book’s introductory examples, we leave detailed MPI error handling for later.

Compile and run on PicoCluster

Compile with mpicc:

mpicc -g -Wall -o mpi_hello mpi_hello.c

mpicc is a wrapper around the C compiler. It supplies the header and library settings needed for MPI. Our familiar -g, -Wall, and -o flags still work.

Run with one process, then two, then four:

mpirun -n 1 ./mpi_hello
mpirun -n 2 ./mpi_hello
mpirun -n 4 ./mpi_hello

-n 4 requests four processes, not four nodes. mpirun launches them; MPI_Init initializes MPI inside each one. We do not need four source files or four different executables.

On PicoCluster, mpirun is an alias that supplies the host file automatically:

alias mpirun='/usr/bin/mpirun -hostfile /etc/mpi_hosts'

This alias is already configured; you do not need to type it. You can inspect it with type mpirun. Thus, mpirun -n 4 ./mpi_hello uses /etc/mpi_hosts. The equivalent explicit command is:

/usr/bin/mpirun -hostfile /etc/mpi_hosts -n 4 ./mpi_hello

The host file tells the launcher which machines are available; it does not mean that a four-process run will necessarily use four different machines. The executable must be accessible on the nodes where it runs, at the path the launcher expects. Run from the course working directory used on the cluster.

One cluster, several users

Start with the small process counts shown here and use the nodes assigned in class. If the launcher reports an unavailable host or missing executable, check the cluster setup before changing the program. Do not add extra processes just to make a short example look faster.

Read the output

One possible four-process run is:

Hello from process 2 of 4!
Hello from process 0 of 4!
Hello from process 3 of 4!
Hello from process 1 of 4!

The order may change from run to run. The processes run independently, and the launcher collects their output. Rank 0 is not guaranteed to print first.

Run it again. Do you see each rank exactly once? Does changing the number of processes require recompiling?

Where are the processes running?

Add these lines after the rank and size calls, replacing the original greeting:

char processor[MPI_MAX_PROCESSOR_NAME];
int name_length;
MPI_Get_processor_name(processor, &name_length);

printf("Process %d of %d is running on %s\n",
       my_rank, comm_sz, processor);

processor holds the machine name. MPI fills it in and stores the length in name_length. MPI_MAX_PROCESSOR_NAME gives us enough room for the name.

Recompile and run with four processes:

mpicc -g -Wall -o mpi_hello mpi_hello.c
mpirun -n 4 ./mpi_hello

Each output line now identifies both the process’s rank and the node running it. For example, Process 2 of 4 is running on pc1 means rank 2 is running on node pc1; your actual node names and placement may differ.

Compare the names rather than assuming one process per node. A rank is a label within this run, not a permanent board number or a physical CPU-core number.

Different processes, different jobs

Every process executes the program, but an if statement lets us choose work by rank. For example, put this after the rank and size calls:

if (my_rank == 0) {
    printf("There are %d processes in this run.\n", comm_sz);
}

Only process 0 prints that line. All processes still call MPI_Finalize afterward.

This pattern is called SPMD: single program, multiple data. We write one program; different processes can work on different data or take different branches. Rank 0 has no automatic authority over the others—we give it a special job in our code.

Send one integer

Create mpi_message.c. This example uses exactly two processes. Process 0 sends a number, and process 1 receives it.

#include <stdio.h>
#include <mpi.h>

int main(void) {
    int my_rank;
    int comm_sz;

    MPI_Init(NULL, NULL);
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
    MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);

    if (comm_sz != 2) {
        if (my_rank == 0) {
            printf("Run this example with exactly 2 processes.\n");
        }
        MPI_Finalize();
        return 1;
    }

    int number = -1;

    if (my_rank == 0) {
        number = 42;
        MPI_Send(&number, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
        printf("Process 0 sent %d.\n", number);
    } else {
        MPI_Recv(&number, 1, MPI_INT, 0, 0, MPI_COMM_WORLD,
                 MPI_STATUS_IGNORE);
        printf("Process 1 received %d.\n", number);
    }

    MPI_Finalize();
    return 0;
}

mpicc -g -Wall -o mpi_message mpi_message.c
mpirun -n 2 ./mpi_message

Expected output, with either line possibly appearing first:

Process 0 sent 42.
Process 1 received 42.

Both processes start with their own number equal to -1. Assigning 42 in process 0 changes only process 0’s copy. The receive call fills process 1’s copy with the value from the message.

Read the send call

MPI_Send(&number, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);

Argument	Meaning in this example
`&number`	Send the data stored at this address
`1`	Send one element
`MPI_INT`	That element is a C `int`
`1`	Send to process 1
`0`	Use message tag 0
`MPI_COMM_WORLD`	Communicate within this group

The count is the number of elements, not the number of bytes. A tag is an integer label for a message; it is separate from the destination rank. We use tag 0 because there is only one kind of message here.

Read the receive call

MPI_Recv(&number, 1, MPI_INT, 0, 0, MPI_COMM_WORLD,
         MPI_STATUS_IGNORE);

The receive arguments have a similar structure. Here they mean: store one integer in number, coming from process 0, with tag 0, in MPI_COMM_WORLD.

MPI_STATUS_IGNORE says we do not need the additional information MPI can return about the message. It does not mean “ignore the message” or “ignore errors.”

The sender’s destination and the receiver’s source must describe the same exchange. Their tags and communicators must match too. Use compatible data types and enough room in the receive buffer for the message.

Waiting for a message

These are blocking calls. MPI_Recv waits until the matching message has arrived and the received data is ready to use. MPI_Send returns when its send buffer can safely be reused; this does not necessarily mean the receiver has finished receiving. MPI may buffer a message, but we should not depend on that to make a program work.

If a receive has no matching send, the program can wait indefinitely. Two processes can also get stuck if both wait to receive before either sends. This is a deadlock: each is waiting for something the other cannot yet do.

Think about it: What would happen if we changed only the receiver’s tag from 0 to 1?

Trace the expected calls before trying changes. If an experiment gets stuck, press Ctrl+C in the launching terminal to stop the run, then check the source, destination, tag, and communicator. Restore matching calls before continuing.

Divide a calculation

Now let us add the integers from 1 through 10. We could do this with one loop. Instead, we will give each process a share of the terms and send the partial sums to process 0.

With four processes, the work is:

Rank	Terms	Partial sum
0	1, 5, 9	15
1	2, 6, 10	18
2	3, 7	10
3	4, 8	12

Each process starts at my_rank + 1 and steps forward by comm_sz. This is a cyclic distribution: the processes take turns receiving terms. It also handles cases where the number of terms is not divisible by the number of processes.

Before coding, check that every integer appears exactly once and that the partial sums add to 55.

Complete `mpi_sum.c`

#include <stdio.h>
#include <mpi.h>

int main(void) {
    int my_rank;
    int comm_sz;

    MPI_Init(NULL, NULL);
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
    MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);

    const int n = 10;
    int local_sum = 0;

    for (int i = my_rank + 1; i <= n; i += comm_sz) {
        local_sum += i;
    }

    if (my_rank == 0) {
        int total = local_sum;
        printf("Process 0 partial sum: %d\n", local_sum);

        for (int source = 1; source < comm_sz; source++) {
            int received_sum;
            MPI_Recv(&received_sum, 1, MPI_INT, source, 0,
                     MPI_COMM_WORLD, MPI_STATUS_IGNORE);
            printf("Process %d partial sum: %d\n", source, received_sum);
            total += received_sum;
        }

        printf("Total: %d\n", total);
    } else {
        MPI_Send(&local_sum, 1, MPI_INT, 0, 0, MPI_COMM_WORLD);
    }

    MPI_Finalize();
    return 0;
}

mpicc -g -Wall -o mpi_sum mpi_sum.c
mpirun -n 4 ./mpi_sum

Expected output:

Process 0 partial sum: 15
Process 1 partial sum: 18
Process 2 partial sum: 10
Process 3 partial sum: 12
Total: 55

The partial sums are printed in rank order because process 0 receives and prints them in that order. This does not tell us which process finished computing first.

Follow the work

Every process computes its own local_sum.
Process 0 starts total with its own contribution.
The other processes each send one partial sum to process 0.
Process 0 receives those contributions and adds them to total.

Process 0 computes a share of the work too. It does not send a message to itself. With one process, its receive loop runs zero times and it already has the entire sum.

We generate the terms from their indices, so there is no input array to distribute. With real data stored on one process, distributing the input would be another part of the problem.

Try it

Run with one, two, three, and four processes. The final total should always be 55, even though the partial sums change.
Change n to 3 and use four processes. Which process gets no terms? What should it contribute?
Change n to 100 and check the total against \(n(n+1)/2\).

Keep n small in this example. Large sums can exceed the range of int; increasing the workload substantially would require revisiting the data types as well as the algorithm.

Have we made it faster?

We have divided work among processes, but that is not enough to establish a speedup. Adding ten integers is extremely quick. Starting MPI processes, moving messages, and printing output can take far longer than the arithmetic.

To compare performance fairly, we would need a substantial computation, a comparable serial version, repeated measurements, and a timer around the relevant parallel work—including the communication needed to produce the answer. Timing only one process’s local loop would miss part of the cost.

For this module, focus on who does the work, which data each process owns, and how the results move. We will return to timing once we have more work to divide.

Check your understanding

What is the difference between a process’s rank and the node where it runs?
If process 0 changes number, why does process 1 need a receive call to get that value?
In the integer-message example, why does the send use destination 1 while the receive uses source 0?
Why can the greeting lines appear in different orders, while the partial sums in mpi_sum.c appear in rank order?
How does the sum program avoid leaving out terms when n is not divisible by comm_sz?
Why might using more processes make a small program slower?

If something goes wrong

Symptom	Check
`mpi.h` is missing when compiling	Use `mpicc`, not plain `gcc`
The executable cannot be found	Save and compile successfully; check the directory and executable path on the selected nodes
Only one greeting appears	Launch with `mpirun -n 4 ./mpi_hello`; check the printed process count
Several ranks report the same machine	Multiple processes can run on one node; check the printed names rather than assuming placement
The message example asks for two processes	Use `mpirun -n 2 ./mpi_message`
A program hangs	Stop it with Ctrl+C and check that sends and receives match
The total is too large	Check that processes are working on different terms and that process 0 adds each contribution once
The launcher reports host, slot, or connection errors	Check the assigned nodes and host-file setup with me

Next steps

The same pattern—divide the work, compute locally, combine the results—will let us tackle a more substantial problem. The book next develops the trapezoidal rule in §3.2. Later, MPI’s collective operations will give us a simpler way to combine partial sums without writing the receive loop ourselves.