Rearranging a vector in parallel for fast performance - vector

Question

I have a vector whose length can go up to about a few million or more.
If I say the vector is vec = [a1 a2 ...b1 b2 ... c1 c2 ...d1 d2 ...]
I need to rearrange vec to new_vec = [a1 b1 c1 d1 a2 b2 c2 d2 ...] .
If viewed as a matrix of column vectors, then this could be viewed as a transpose, but I do not have a two dimensional vector. I understand how to do it on a sequential computer. That is very simple.
But I am not sure how to do this on a multiple processor cluster or on a GPU, or even if this would be feasible on parallel machines. Memory seems to be the obvious bottleneck. If there are any algorithms or any architecture specific optimizations that I can use, please let me know.
Edit: More information below.
The code structure is:
subroutine reorder(vec,parameter)
real(kind = 8),dimension(parameter%length), intent(inout) :: vec
real(kind = 8),dimension(parameter%length) :: temp
type(param) :: parameter !just a struct holding certain constant parameters
integer :: i,j,k,q1,q2,q3,nn1,n1,n2,nn2
i1 = parameter%len1 !lengths of sub-vectors in each direction
i2 = parameter%len2 !the multiplication of the 3 gives the
!overall length of vec
i3 = parameter%len3
temp = vec
n1 = i2*i1
n2 = i2*i3
do k = 1, i3
q1 = n1*(k-1)
q2 = i2*(k-1)
do j = 1, i2
q3 = i1*(j-1)
do i = 1, i1
nn1 = q1+q3+i
nn2 = q2+j+n2*(i-1)
vec(nn2) = temp(nn1)
end do
end do
end do
end subroutine reorder
Therefore the code aims to reorder the elements of the vector according to the rule. As you can see as the length of the vector becomes very large, a significant time is spent in this routine.
This routine is called in a main routine multiple times. A cartesian decomposition produces a cartesian 3D arrangement of ranks in the beginning and each rank calls this subroutine when a reordering of the elements is required for its own next subroutine call.The cartesian communicator subroutine is shown in the skeleton below:
subroutine cartesian_comm(ndim,comm_cart,comm_one_d,coord_cart)
use mpi
implicit none
integer, dimension(:), intent(in) :: ndim
integer, intent(out) :: comm_cart
integer, dimension(:), pointer :: comm_one_d, coord_cart
logical, dimension(size(ndim)) :: period, remain
integer :: dim,code, i, rank
!creating the cartesian communicator
dim = 3
allocate(comm_one_d(dim),coord_cart(dim))
period = .FALSE.
call MPI_CART_CREATE(MPI_COMM_WORLD, dim, ndim, period, .FALSE., comm_cart, code)
call MPI_COMM_RANK(comm_cart, rank, code)
call MPI_CART_COORDS(comm_cart, rank, dim, coord_cart, code)
!Creating sub-communicator for each direction
do i = 1, dim
remain = .FALSE.
remain(i) = .TRUE.
call MPI_CART_SUB(comm_cart, remain, comm_one_d(i), code)
end do
end subroutine cartesian_comm
And it is called in the main function as follows:
Program main
!initialize some stuff and intialize all the required variables
! ndim is the number of processes the program is called
! with "mpirun -np 8 ./exec" would mean ndim is cuberoot of 8,
! and therefore 2 for the 3D case. It is always made sure that
! np is a cube(or square for 2D) while calling the program
call cartesian_comm(ndim,comm_cart,comm_one_d,coord_cart)
do while (t<tend-1D-8) !start time loop
t = t + dt
!do some computations get the vector "vec" for
!each rank separately (different and independent in each rank)
call reorder(vec,parameter) ! all ranks call this subroutine
!do some computations here with the new reordered vec
end do !end time loop
!do other stuff (unrelated to reorder but use the "vec" vector) here
end Program main
I would like to know if there is a better way to do this in a multiprocessor cluster, or how I could proceed in the case of a GPU.