Unexpected result from MPI isend and irecv - asynchronous

Question

My goal was to send a vector from process 0, to process 1. Then, send it back from process 1 to process 0.
I have two questions from my implementation,
1- Why does the sending back from process 1 to process 0 takes longer than the vice versa?
The first send-recv takes ~1e-4 seconds in total and the second send-recv takes ~1 second.
2- When I increase size of the vector, I get the following error. What is the reason for that issue?
mpirun noticed that process rank 0 with PID 11248 on node server1 exited on signal 11 (Segmentation fault).
My Updated C++ code is as follows
#include <mpi.h>
#include <stdio.h>
#include <iostream>
#include <vector>
#include <boost/timer/timer.hpp>
#include <math.h>
using namespace std;
int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(NULL, NULL);
MPI_Request request, request2,request3,request4;
MPI_Status status;
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
srand( world_rank );
int n = 1e3;
double *myvector = new double[n];
if (world_rank==0){
myvector[n-1] = 1;
}
MPI_Barrier (MPI_COMM_WORLD);
if (world_rank==0){
boost::timer::cpu_timer timer;
MPI_Isend(myvector, n, MPI_DOUBLE , 1, 0, MPI_COMM_WORLD, &request);
boost::timer::cpu_times elapsedTime1 = timer.elapsed();
cout << " Wallclock time on Process 1:"
<< elapsedTime1.wall / 1e9 << " (sec)" << endl;
MPI_Irecv(myvector, n, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD, &request4);
MPI_Wait(&request4, &status);
printf("Test if data is recieved from node 1: %1.0f\n",myvector[n-1]);
boost::timer::cpu_times elapsedTime2 = timer.elapsed();
cout <<" Wallclock time on Process 1:"
<< elapsedTime2.wall / 1e9 << " (sec)" << endl;
}else{
boost::timer::cpu_timer timer;
MPI_Irecv(myvector, n, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, &request2);
MPI_Wait(&request2, &status);
boost::timer::cpu_times elapsedTime1 = timer.elapsed();
cout << " Wallclock time on Process 2:"
<< elapsedTime1.wall / 1e9 << " (sec)" << endl;
printf("Test if data is recieved from node 0: %1.0f\n",myvector[n-1]);
myvector[n-1] = 2;
MPI_Isend(myvector, n, MPI_DOUBLE , 0, 0, MPI_COMM_WORLD, &request3);
boost::timer::cpu_times elapsedTime2 = timer.elapsed();
cout<< " Wallclock time on Process 2:"
<< elapsedTime1.wall / 1e9 << " (sec)" << endl;
}
MPI_Finalize();
}
The output is:
Wallclock time on Process 1:2.484e-05 (sec)
Wallclock time on Process 2:0.000125325 (sec)
Test if data is recieved from node 0: 1
Wallclock time on Process 2:0.000125325 (sec)
Test if data is recieved from node 1: 2
Wallclock time on Process 1:1.00133 (sec)

Answer 1

Timing discrepancies
First of all, you don't measure the time to send the message. This is why posting the actual code you use for timing is essential.
You measure four times, for the two sends, you only time the call to MPI_Isend. This is the Immediate version of the API call. As the name suggests, it completes immediately. The timing has nothing to do with the actual time for sending the message.
For the receive operations, you measure MPI_Irecv and a corresponding MPI_Wait. This is the time between initiating the receive and the the local availability of the message. This is again different from the message transfer time, as it does not consider the time difference between posting the receive and the corresponding send. In general, you have to consider the late sender and late receiver cases. Further even for blocking send operations, a local completion does not imply a completed transfer, remote completion, or even initiation.
Timing MPI transfers is difficult.
Checking for completion
There is still the question as to why anything in this code could take an entire second. That is certainly not a sensible time unless your network uses IPoAC. The likely reason is that you do not check for completion of all messages. MPI implementations are often single threaded and can only make progress on communication during the respective API calls. To use immediate messages, you must either periodically call MPI_Test* until the request is finished or complete the request using MPI_Wait*.
I don't know why you chose to use immediate MPI functions in the first place. If you call MPI_Wait right after starting an MPI_Isend/MPI_Irecv, you might as well just call MPI_Send/MPI_Recv. You need immediate functions for concurrent communication and computation, to allow concurrent irregular communication patterns, and to avoid deadlocks in certain situations. If you don't need the immediate functions, use the blocking ones instead.
Segfault
While I cannot reproduce, I suspect this is caused by using the same buffer (myvector) for two concurrently running MPI operations. Don't do that. Either use a separate buffer, or make sure the first operation completes. Generally - you are not allowed to touch a buffer in any way after passing it to MPI_Isend/MPI_Irecv until you know the request is completed via MPI_Test*/MPI_Wait*.
P.S.
If you think you need immediate operations to avoid deadlocks while sending and receiving, consider MPI_Sendrecv instead.