MPI datatype for structure of arrays - mpi

Question

In an MPI application I have a distributed array of floats and two "parallel" arrays of integers: for each float value there are two associated integers that describe the corresponding value. For the sake of cache-efficiency I want to treat them as three different arrays, i.e. as a structure of arrays, rather than an array of structures.
Now, I have to gather all these values into the first node. I can do this in just one communication instruction, by defining an MPI type, corresponding to a structure, with one float and two integers. But this would force me to use the array of structures pattern instead of the structure of arrays one.
So, I can choose between:
Performing three different communications, one for each array and keep the efficient structure of arrays arrangement
Defining an MPI type, perform a single communication, and deal with the resulting array of structures by adjusting my algorithm or rearranging the data
Do you know a third option that would allow me do have the best of both worlds, i.e. having a single communication and keeping the cache-efficient configuration?

Answer 1

You take take a look at Packing and Unpacking.
http://www.mpi-forum.org/docs/mpi-11-html/node62.html
However, I think if you want to pass a same "structure" often you should define you own MPI derivate type.

Answer 2

E.g. by using the *array_of_blocklength* parameter of MPI_Type_create_struct
// #file mpi_compound.cpp
#include <iterator>
#include <cstdlib> // for rng
#include <ctime> // for rng inits
#include <iostream>
#include <algorithm>
#include <mpi.h>
const std::size_t N = 10;
struct Asset {
float f[N];
int m[N], n[N];
void randomize() {
srand(time(NULL));
srand48(time(NULL));
std::generate(&f[0], &f[0] + N, drand48);
std::generate(&n[0], &n[0] + N, rand);
std::generate(&m[0], &m[0] + N, rand);
}
};
int main(int argc, char* argv[]) {
MPI_Init(&argc,&argv);
int rank,comm_size;
MPI_Status stat;
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&comm_size);
Asset a;
MPI_Datatype types[3] = { MPI_FLOAT, MPI_INT, MPI_INT };
int bls[3] = { N, N, N };
MPI_Aint disps[3];
disps[0] = 0;
disps[1] = int(&(a.m[0]) - (int*)&a)*sizeof(int);
disps[2] = int(&(a.n[0]) - (int*)&a)*sizeof(int);
MPI_Datatype MPI_USER_ASSET;
MPI_Type_create_struct(3, bls, disps, types, &MPI_USER_ASSET);
MPI_Type_commit(&MPI_USER_ASSET);
if(rank==0) {
a.randomize();
std::copy(&a.f[0], &a.f[0] + N, std::ostream_iterator<float>(std::cout, " "));
std::cout << std::endl;
std::copy(&a.m[0], &a.m[0] + N, std::ostream_iterator<int>(std::cout, " "));
std::cout << std::endl;
std::copy(&a.n[0], &a.n[0] + N, std::ostream_iterator<int>(std::cout, " "));
std::cout << std::endl;
MPI_Send(&a.f[0],1,MPI_USER_ASSET,1,0,MPI_COMM_WORLD);
} else {
MPI_Recv(&a.f[0],1,MPI_USER_ASSET,0,0,MPI_COMM_WORLD, &stat);
std::cout << "\t=> ";
std::copy(&a.f[0], &a.f[0] + N, std::ostream_iterator<float>(std::cout, " "));
std::cout << std::endl << "\t=> ";
std::copy(&a.m[0], &a.m[0] + N, std::ostream_iterator<int>(std::cout, " "));
std::cout << std::endl << "\t=> ";
std::copy(&a.n[0], &a.n[0] + N, std::ostream_iterator<int>(std::cout, " "));
std::cout << std::endl;
}
MPI_Type_free(&MPI_USER_ASSET);
MPI_Finalize();
return 0;
}
worked with
mpirun -n 2 ./mpi_compound
with mpich2 v1.5 (HYDRA) on x86_86 linux and g++ 4.4.5-8 - based mpic++