What control MPI_Barrier time to execute - mpi

Question

This code:
#include <mpi.h>
int main(int argc, char* argv[])
{
MPI_Init(&argc, &argv);
for (unsigned int iter = 0 ; iter < 1000 ; iter++)
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();
return 0;
}
is very long to run with MPICH 3.1.4. Here are the wall clock (in seconds) for different MPI implementations.
On a laptop with 4 processors of 2 cpu cores:
| MPI size | MPICH 1.4.1p1 | openmpi 1.8.4 | MPICH 3.1.4 |
|----------|---------------|---------------|-------------|
| 2 | 0.01 | 0.39 | 0.01 |
| 4 | 0.02 | 0.39 | 0.01 |
| 8 | 0.14 | 0.45 | 27.28 |
| 16 | 0.34 | 0.53 | 71.56 |
On a desktop with 8 processors of 4 cpu cores:
| MPI size | MPICH 1.4.1p1 | openmpi 1.8.4 | MPICH 3.1.4 |
|----------|---------------|---------------|-------------|
| 2 | 0.00 | 0.41 | 0.00 |
| 4 | 0.01 | 0.41 | 0.01 |
| 8 | 0.07 | 0.45 | 2.57 |
| 16 | 0.36 | 0.54 | 61.76 |
What explain such a difference, and how to control it?

Answer 1

You are using MPI size > number of processors available. As MPI programs spawn in such a way that each process is handled by a single processor, what this means is that, for example when you run MPI size == 16 on your 8 core machine, each processor will be responsible for two processes; this will not make the program any faster, and, in fact, will make it slower as you have seen. The way to get around it is to either get a machine with more processors available, or to ensure that you run your code with MPI size <= number of processors available.