What is an efficient way to send/receive bits from bitset class without conversion. Is it possible to use MPI_BYTE?
If so, what to define as container for the array that is going to hold the bits?
If this is not possible, which conversion is more efficient, conversion to ulong or to strings?
Convert it to array of "bool" and send by MPI_BYTE, this method works on cluster.
Related
I'm writing a golang application which interacts with Windows Services using the windows/svc package.
When I'm looking at the package source code how syscalls are being done I see interesting cast construct:
name := syscall.UTF16ToString((*[1 << 20]uint16)(unsafe.Pointer(s.ServiceName))[:]
Extracted from mgr.go
This is a common patttern when dealing with Win32 API when one needs to pass a pre-allocated buffer to receive a value from Win32 API function, usually an array or a structure.
I understand that Win API returns a unicode string represented by its pointer and it is passed to the syscall.UTF16ToString(s []uint16) function to convert it to the go string in this case.
I'm confused from the part when an unsafe pointer is cast to the pointer to 1M array, *[1<<20]uint16.
Why the size if 1M [1<<20]?
Buffer for a value is allocated dynamically, not with fixed size of 1M.
You need to choose a static size for the array type, so 1<<20 is chosen to be large enough to allow for any reasonable buffer returned by the call.
There is nothing special about this size, sometimes you'll see 1<<31-1 since it's the largest array for 32bit platforms, or 1<<30 since it looks nicer. It really doesn't matter as long as the type can contain the returned data.
QByteArray holds individual bytes, and if we make a QList of unsigned chars that will also hold the individual bytes.
What is the reason QByteArray exists when there can made a QList<unsigned char>?
What is the difference between QByteArray and QList<unsigned char> or QVector<unsigned char>?
What points am I missing?
QByteArray is a usefull wrapper around char*. Suitable for streaming with QDataStream, string operations and other data management.
From the docs you also can find that:
Behind the scenes, it always ensures that the data is followed by a
'\0' terminator, and uses implicit sharing (copy-on-write) to reduce
memory usage and avoid needless copying of data.
QList, at first is not linear(subsequent) in memory (you should use QVector), and have no such usefull API
QList allocates memory on heap and does not guarantee any data locality, so there can be a performance difference between using QList and QByteArray.
From: http://doc.qt.io/qt-5/qlist.html#details
QVector should be your default first choice. QVector will usually give better performance than QList, because QVector always stores its items sequentially in memory, where QList will allocate its items on the heap unless sizeof(T) <= sizeof(void*) and T has been declared to be either a Q_MOVABLE_TYPE or a Q_PRIMITIVE_TYPE using Q_DECLARE_TYPEINFO. See the Pros and Cons of Using QList for an explanation.
QByteArray also provides convenience methods for working with bytes (and strings in general).
How can I transmit instances of my class or a std::vector using MPI_Send() or MPI_Bcast() in C++?
You cannot simply transmit instances of random classes since being C calls neither MPI_Send() nor MPI_Bcast() understand the structure of those classes. You can send instances of std::vector (since it uses contiguous memory storage) by providing &vector[0] to MPI_Send() but the receive operation should then be implemented in several steps: MPI_Probe() -> get the number of elements in the message -> resize the vector instance -> MPI_Recv() into the resized instance. For all other cases, you should use something like Boost.MPI or you should use MPI_Pack() and MPI_Unpack() to serialise and deserialise your class instances to and from MPI messages.
MPI doesn't operate on objects, it operates on memory locations. So to send an object from your own class, you will need to know the memory layout from your class. You can then use this to build an MPI datatype. There is an entire chapter of the MPI specification (chapter 4) devoted to how to do this. The basic premise is that you build a datatype based on the standard MPI types, arranged in a specified memory layout. Once this type is built and committed, you can then use it in MPI operations.
Ok, I've already found a solution:
http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node83.htm#Node83
in 5th Example
You could try sending it as a byte array like this:
MPI_Send(vec.data(), vec.size() * sizeof(VectorElement), MPI_CHAR, 0, 0, MPI_COMM_WORLD);
And recieve it like this:
MPI_Recv(&vec.front(), vec.size() * sizeof(VectorElement), MPI_CHAR, 0, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
Where vec is an instanse of std::vector<VectorElement>
If you think that the receiving vector size could change, you can simply send it before the vector.
I can't guarantee that this method is safe but i believe it should be since both padding and the order of fields of a certain class should be the same on every subprocess.
I'm new to Qt, so if this is a completely stupid question...
I'm trying to use QTcpSocket.
If I do:
...
QTcpSocket * socket
socket = new QTcpSocket(this);
socket.write(1);
...
It complains about write() not working with integers (not a const char *).
If I do:
...
QTcpSocket * socket
socket = new QTcpSocket(this);
socket.write("1");
...
the other side sees it as the integer 49 (ASCII for 1).
On a similar but different issue, is it possible to send structs or unions over QTcpSocket?
==================================================
EDIT:
The server already accepts integers, and is expecting an integer - I have no control over that.
The problem you have is not really related to Qt, the same issue would arise with any other Socket or Streaming interface.
The provider of the Server needs to give you the protocol description. This description usually contains the ports used (TCP, UDP, numbers), other TCP parameters and the coding of the transmitted data. Sometimes, this protocol (or its implementation) is called a (protocol-) stack.
The coding not only contains the byte ordering, but also a description of how complex structures are transmitted.
The latter information is often coded in something that is called "ASN.1" - Abstract Syntax Notation.
In case your server is really simple and just accepts Integers one after the other without any meta-information and is on the same platform, than you could do something like this:
foreach (int i in my set of integers)
{
ioDevice->write((const char*) &i, sizeof(i));
}
You take the address of your integer as a data buffer start and transmit as many bytes as your integer has.
But note well, this will fail if you transmit data from an Intel architecture to a 16-bit architecture or a motorola PPC.
I suggest using QDataStream with sockets. This will protect you from little endian/big endian conversion problem.
So something as below :
qint32 myData = 1;
QDataStream os( &mySocket );
os << myData;
When you write in a string representation, you also have to interpret the string on the other side. There is, for example, QString::toInt().
When you want to write the integer as integer, you will have more throughput, as it takes less bytes to transmit. However you should read about the topic of network byte order.
In principal it is possible to copy structs, etc. into a buffer and also over the network. However things get complicated again when you transmit data between different architectures or even only different builds of your software. So you shouldn't send the raw data, but use serialization! See this question:
Serialization with Qt
It provides answers on how to generate streams out of objects and objects out of streams. These streams is what you then use to transmit over the network. Then you don't have to deal with the integers themselves anymore!
Different overloads, you are looking for:
qint64 QIODevice::write ( const char * data, qint64 maxSize );
and
qint64 QIODevice::write ( const QByteArray & byteArray );
I am trying to send data(forces) across 2 processes, using MPI_SendRecv. Usually the data will be over written in the received buffer, I do not want to overwrite the data in the received buffer instead I want to add the data it received.
I can do the following. Store the data in the previous time step to a different array and then add it after receiving. But I have huge number of nodes and I do not want to have memory allocated for its storage every time step. (or overwrite the same)
My question is there a way to add the received data directly to the buffer and store it in the received memory using MPI?
Any help in this direction would be really thankful.
I am sure collective communication calls (MPI Reduce)cannot be worked out here. Are there any other commands that can do this?
In short: no, but you should be able to do this.
In long: Your suggestion makes a great deal of sense and the MPI Forum is currently considering new features that would enable essentially what you want.
It is incorrect to suggest that the data must be received before it can be accumulated. MPI_Accumulate does a remote accumulation in a one-sided fashion. You want MPI_Sendrecv_accumulate rather than MPI_Sendrecv_replace. This makes perfect sense and an implementation can internally do much better than you can because it can buffer on a per-packet basis, for example.
For suszterpatt, MPI internally buffers in the eager protocol and in the rendezvous protocol can setup a pipeline to minimize buffering.
The implementation of MPI_Recv_accumulate (for simplicity, as the MPI_Send part need not be considered) looks like this:
int MPI_Recv(void *buf, int count, MPI_Datatype datatype, MPI_Op op, int source, int tag, MPI_Comm comm, MPI_Status *status)
{
if (eager)
MPI_Reduce_local(_eager_buffer, buf, count, datatype, op);
else /* rendezvous */
{
malloc _buffer
while (mycount<count)
{
receive part of the incoming data into _buffer
reduce_local from _buffer into buf
}
}
In short: no.
In long: your suggestion doesn't really make sense. The machine can't perform any operations on your received value without first putting it into local memory somewhere. You'll need a buffer to receive the newest value, and a separate sum that you will increment by the buffer's content after every receive.