message synchronization with boost::mpi send/recv? - mpi

Question

I call mpirun with "-np 2". I'm referring to the process with rank 0 as "master" and the process with rank 1 as "slave".
Goal:
master occasionally sends a message to slave such as mpi::send(1, UPDATE, data);. Other message types include DIE, COMPUTE ...etc. Those message types are constant integers with unique values.
slave runs an infinite loop, "listening" to any message from the master. When it receives a message, it sends an acknowledgement back to the master.
Implementation:
slave runs:
...
int updateData, computeData;
mpi::request updateRequest = world.irecv(0,UPDATE, updateData);
mpi::request computeRequest = world.irecv(0,COMPUTE, computeData);
do {
cerr << "slave ready to take a command" << endl;
if(updateRequest.test()) {
cerr << "slave ireceived UPDATE" << endl;
world.send(0, UPDATE_ACK, 0);
cerr << "slave sent UPDATE_ACK" << endl;
/* do something useful
...
...
*/
updateRequest = world.irecv(0, UPDATE, updateData);
} else if (computeRequest.test()) {
...
} else {
boost::this_thread::sleep( boost::posix_time::seconds(1) );
}
}
while the master runs:
...
world.send(1, UPDATE, 10);
cerr << "master sent UPDATE" << endl;
int dummy;
world.recv(1, UPDATE_ACK, dummy);
cerr << "master received UPDATE_ACK" << endl;
...
more context for the master's code:
...
// update1
world.send(1, UPDATE, params);
cerr << "master sent UPDATE" << endl;
int dummy;
world.recv(1, UPDATE_ACK, dummy);
cerr << "master received UPDATE_ACK" << endl;
// update2
world.send(1, UPDATE2, params2);
cerr << "master sent UPDATE2" << endl;
world.recv(1, UPDATE2_ACK, dummy);
cerr << "master received UPDATE2_ACK" << endl;
// update3
world.send(1, UPDATE3, params3);
cerr << "master sent UPDATE3" << endl;
world.recv(1, UPDATE3_ACK, dummy);
cerr << "master received UPDATE3_ACK" << endl;
...
// training iterations
do {
mpi::request computeRecvReq1, computeRecvReq2;
std::map<int, int> result1, result2;
// for each line in a text file, the master asks the slave(s)
// to compute two things and aggregates the results
for(unsigned sentId = 0; sentId != data.size(); sentId++) {
// these two functions won't return until at least one slave is "idle"
CollectSlavesWork1(computeRecvReq1, result1);
CollectSlavesWork2(computeRecvReq2, result2);
// async ask the slave to compute and async get the results
world.isend(1, COMPUTE, sentId);
computeRecvReq1 = world.irecv(1, RESULT1, result1);
computeRecvReq2 = world.irecv(1, RESULT2, result2);
}
// based on the slave(s) work, the master updates params1
// and send them again to the slave(s)
world.send(1, UPDATE, params);
cerr << "master sent UPDATE" << endl;
world.recv(1, UPDATE_ACK, dummy); // PROBLEM HAPPENS HERE
cerr << "master received UPDATE_ACK" << endl;
} while(!ModelIsConverged())
...
Output:
...
slave ready to take a command
master sent UPDATE
slave ireceived UPDATE
slave sent UPDATE_ACK
master received UPDATE_ACK
slave ready to take a command
...
slave ready to take a command
master sent UPDATE
slave ireceived UPDATE
slave sent UPDATE_ACK
slave ready to take a command
...
Problem:
the first time the master sends an UPDATE message everything seems to be alright. However, in the second time, the master doesn't receive the UPDATE_ACK.