I have a program that in outline processes binary data from afile.
Code outline is the following:
QFile fileIn ("the_file");
fileIn.open(QIODevice::ReadOnly);
The file has a mix of binary and text data.
The file contents are read using QDataStream:
QDataStream stream(&fileIn);
stream.setByteOrder(QDataStream::LittleEndian);
stream.setVersion(QDataStream::Qt_5_0);
I can read the data from the QDataStream into various data types. e.g.
QString the_value; // String
stream >> the_value;
qint32 the_num;
stream >> the_numm;
Nice and easy. Overall I read the file data byte by byte until I hit certain values that represent delimiters, e.g. 0x68 0x48. At this point I then next the next couple of bytes that tell me what type of data is next (floats, Strings, ints, etc) and extract as appropriate.
So, the data is orocessed (outline) like:
while ( ! stream.atEnd() )
{
qint8 byte1 = getInt8(stream);
qint8 byte2 = getInt8(stream);
if ( byte1 == 0x68 && byte2 == 0x48 )
{
qint8 byte3 = getInt8(stream);
qint8 byte4 = getInt8(stream);
if ( byte3 == 0x1 && byte4 == 0x7 )
{
do_this(stream);
}
else if ( byte3 == 0x2 && byte4 == 0x8 )
{
do_that(stream);
}
}
}
Some of this embedded data may be compressed, so we use
long dSize = 1024;
QByteArray dS = qUncompress( stream.device()->read(dSize));
QBuffer buffer;
buffer.setData(dS);
if (!buffer.open(QBuffer::ReadOnly)) {
qFatal("Buffer could not be opened. Something is very wrong!");
}
QDataStream stream2(&buffer);
stream2.setByteOrder(QDataStream::LittleEndian);
stream2.setVersion(QDataStream::Qt_5_0);
The convenience of QDataStream makes it easy to read the data, in terms of mapping to particular types but also in handling endianess easily, but it seems to be at the expense of speed. The issues is compounded by the fact that the processing is recursive - data being read could itself contain embedded file data, which needs to be read and processed in the same way.
Is there an alternative that is faster, and so if, how then to handle Endianess the same way?
Your code looks straight forward .. recursion should not be the show stopper ...
Do you have lots of strings ? Thousands ?
stream >> string allocates memory using new what is really slow. And needs to be freed manually afterwards. Refer to the Qt Docs for operator>>(char *&s) method. This is used when reading into QStrings.
Same is true for readBytes(char *&s, uint &l) which may be called internally slowing everything down !
The QString itself will also allocate memory (twice as much as it uses 16bit encoding) what slows down further.
If you use one of these functions often, consider rewriting that code parts for directly reading into a preallocated buffer using readRawData(char *s, int len) before further processing.
Overall, if you need high performance QDataStream itself may well be the show stopper.
Related
I am trying using the arduino IDE to write a sketch. I have data in progmem and want to move the data with a function to a memory address allocated using malloc. My code is below:
const uint8_t Data_6 [256] PROGMEM = { 0x11, 0x39......};
void setup() {
Serial.begin(57600);
oddBallData (Data_6, 0x00, 256);
}
void main() {
}
void oddBallData(const uint8_t *data, uint8_t mem, uint16_t bytes) {
uint8_t *buff1 = (uint8_t*)malloc(sizeof(bytes));
if (buff1 = 0) {
Serial.println(F("FATAL ERROR - NO MEMORY"));
}
else {
for (uint16_t x = 0; x < 6; x++ ) {
buff1[x] = data[x]; //edited from data[0] to [x] made a mistake in post
Serial.println(buff1[x],HEX);
}
}
buff1[0] = data[0];
Serial.println(buff1[0],HEX);
free(buff1);
}
I have some data saved in progmem and want to write that data to a second device using i2c protocol. I have multiple constant arrays of data saved to my progmem, with different sizes. So I have used malloc to reserve some memory from the heap, inside of the function.
I have not been able to write the data from the progmem so I have stripped things back to so that I am just trying to point to the progmem data using malloc and then print it.
This is where I found a the problem. If I print a single array entry from the data constant. It prints the correct value. If I use a loop I get mixed results, the loop works as long as the condition check value is below 3 or sometimes below 6!!!...?
If above this value the entire print is just garbage. Can anyone explain what I am seeing?
The culprit is probably
uint8_t *buff1 = (uint8_t*)malloc(sizeof(bytes));
sizeof(bytes) returns the size of the variable (which is probably 2 bytes) so you are just allocating 2 bytes of memory. You should use the value directly, eg:
uint8_t* buff1 = malloc(bytes);
Mind that the cast is not required in C since a void* is convertible to any other pointer type directly.
Again - AVR PROGMEM is not directly accessible from memory space, it needs different instruction than access into the RAM. If you are using it like this, you'll get RAM content on passed address, not the FLASH one. You have to use special functions for this. For example memcpy_P(ram_buff,flash_ptr); makes a copy from flash into the ram. Or you can read one byte by pgm_read_byte(flash_ptr + offset)
BTW: If you are using Data_6[0] and it's working, it's just because compiler sees it as a constant and constant can be replaced by its value compile time.
I Guess you just forgot to flush()
try to do Serial.flushI() after Serial.println(buff1[x],HEX);
you can also check flush documentation
I know Qt has a lot of good methods for reading file.But in my work , I have to open a file by using Qt and get a pointer pointing to the start address of the file.So how to do that?
Here is my codes:
char *buffer;
if(file.open(QIODevice::ReadOnly))
{
QByteArray dataArray=file.readAll();
buffer=dataArray.data();
}
char test[1024];
for(int i=0;i<1024;i++)
{
test[i]=*buffer;
buffer++;
}
I use QByteArray QIODevice::readAll()to read all available data from the device, and returns it as a byte array.
Then I use char * QByteArray::data() to set my pointer buffer.
But when I try to read the data by a for loop,my Qt debugger throw me an error:read access violation,so how would this happen?
BTW , the file to be read is very big so I can't use a buffer to read them all once. Instead , I have to do as what I did here. When I read a 7kb-sizes-file ,my codes runs well. When I read a 700kb-sizes-file , here comes the problem.
The reason you are getting access violation is that dataArray is declared inside if code block and goes out of scope. You want to declare the dataArray outside of that block, eg:
QByteArray dataArray;
if(file.open(QIODevice::ReadOnly))
{
dataArray = file.readAll();
}
else
{
// give error
}
char *buffer = dataArray.data();
char test[1024];
for(int i = 0; i < 1024; i++)
{
test[i] = *buffer;
buffer++;
}
There are other potential problems in your code though:
First, what if the size of data read from the file is less than 1024? Then you will be reading past the end of the buffer (more access violations).
Second, what if the file is really big? Reading all that data at once may cause swapping/paging.
We have two Qt applications. App1 accepts a connection from App2 through QTcpServer and stores it in an instance of QTcpSocket* tcpSocket. App1 runs a simulation with 30 Hz. For each simulation run, a QByteArray consisting of a few kilobytes is sent using the following code (from the main/GUI thread):
QByteArray block;
/* lines omitted which write data into block */
tcpSocket->write(block, block.size());
tcpSocket->waitForBytesWritten(1);
The receiver socket listens to the QTcpSocket::readDataBlock signal (in main/GUI thread) and prints the corresponding time stamp to the GUI.
When both App1 and App2 run on the same system, the packages are perfectly in sync. However when App1 and App2 are run on different systems connected through a network, App2 is no longer in sync with the simulation in App2. The packages come in much slower. Even more surprising (and indicating our implementation is wrong) is the fact that when we stop the simulation loop, no more packages are received. This surprises us, because we expect from the TCP protocol that all packages will arrive eventually.
We built the TCP logic based on Qt's fortune example. The fortune server, however, is different, because it only sends one package per incoming client. Could someone identify what we have done wrong?
Note: we use MSVC2012 (App1), MSVC2010 (App2) and Qt 5.2.
Edit: With a package I mean the result of a single simulation experiment, which is a bunch of numbers, written into QByteArray block. The first bits, however, contain the length of the QByteArray, so that the client can check whether all data has been received. This is the code which is called when the signal QTcpSocket::readDataBlock is emitted:
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_5_2);
if (blockSize == 0) {
if (tcpSocket->bytesAvailable() < (int)sizeof(quint16))
return; // cannot yet read size from data block
in >> blockSize; // read data size for data block
}
// if the whole data block is not yet received, ignore it
if (tcpSocket->bytesAvailable() < blockSize)
return;
// if we get here, the whole object is available to parse
QByteArray object;
in >> object;
blockSize = 0; // reset blockSize for handling the next package
return;
The problem in our implementation was caused by data packages being piled up and incorrect handling of packages which had only arrived partially.
The answer goes in the direction of Tcp packets using QTcpSocket. However this answer could not be applied in a straightforward manner, because we rely on QDataStream instead of plain QByteArray.
The following code (run each time QTcpSocket::readDataBlock is emitted) works for us and shows how a raw series of bytes can be read from QDataStream. Unfortunately it seems that it is not possible to process the data in a clearer way (using operator>>).
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_5_2);
while (tcpSocket->bytesAvailable())
{
if (tcpSocket->bytesAvailable() < (int)(sizeof(quint16) + sizeof(quint8)+ sizeof(quint32)))
return; // cannot yet read size and type info from data block
in >> blockSize;
in >> dataType;
char* temp = new char[4]; // read and ignore quint32 value for serialization of QByteArray in QDataStream
int bufferSize = in.readRawData(temp, 4);
delete temp;
temp = NULL;
QByteArray buffer;
int objectSize = blockSize - (sizeof(quint16) + sizeof(quint8)+ sizeof(quint32));
temp = new char[objectSize];
bufferSize = in.readRawData(temp, objectSize);
buffer.append(temp, bufferSize);
delete temp;
temp = NULL;
if (buffer.size() == objectSize)
{
//ready for parsing
}
else if (buffer.size() > objectSize)
{
//buffer size larger than expected object size, but still ready for parsing
}
else
{
// buffer size smaller than expected object size
while (buffer.size() < objectSize)
{
tcpSocket->waitForReadyRead();
char* temp = new char[objectSize - buffer.size()];
int bufferSize = in.readRawData(temp, objectSize - buffer.size());
buffer.append(temp, bufferSize);
delete temp;
temp = NULL;
}
// now ready for parsing
}
if (dataType == 0)
{
// deserialize object
}
}
Please not that the first three bytes of the expected QDataStream are part of our own procotol: blockSize indicates the number of bytes for a complete single package, dataType helps deserializing the binary chunk.
Edit
For reducing the latency of sending objects through the TCP connection, disabling packet bunching was very usefull:
// disable Nagle's algorithm to avoid delay and bunching of small packages
tcpSocketPosData->setSocketOption(QAbstractSocket::LowDelayOption,1);
The documentation says that QDataStream writes data in system independent way, but it says nothing about QBuffer. I develop a program that saves data in a file like this:
QByteArray a;
QBuffer b(&a);
b.open(QIODevide::WriteOnly);
quint32 x = 1;
b.write((char*)&x, sizeof(x));
b.close();
QFile f(...);
f.open(QIODevide::WriteOnly);
f.write(a.constData(), a.size());
f.close();
, and i want this file can be read in any other OS (win, linux, Mac OS). Will this code work or i must use QDataStream instead?
The QBuffer documentation says :
The QBuffer class provides a QIODevice interface for a QByteArray.
ie it is only a QByteArray underneath. On the other hand a QByteArray is portable because as long as you see the data as an array of byte and write one byte at a time you are fine. Your code will work:
When you say
I want this file to be read in any other OS
Is your file used by your program only or will it be used by other applications in the system? QDataStream provides nicer functions for I\O and you may be still able to take advantage of it.
It will be platform specific. x representation in memory depend on the endianess.It doesn't occur in the QBuffer, but when you do :
b.write((char*)&x, sizeof(x));
If you are on machines of different endianess, you will obtain different values for the resulting array by doing
char* data = &x;
qDebug()<< data[0];
qDebug()<< data[1];
qDebug()<< data[2];
qDebug()<< data[3];
Take a look at the source code of QDataStream operator
QDataStream &QDataStream::operator<<(qint32 i){
CHECK_STREAM_WRITE_PRECOND(*this)
if (!noswap) {
i = qbswap(i);
}
if (dev->write((char *)&i, sizeof(qint32)) != sizeof(qint32))
q_status = WriteFailed;
return *this;
}
I need to send floating point numbers using a UDP connection to a Qt application. Now in Qt the only function available is
qint64 readDatagram ( char * data, qint64 maxSize, QHostAddress * address = 0, quint16 * port = 0 )
which accepts data in the form of signed character buffer. I can convert my float into a string and send it but it will obviously not be very efficient converting a 4 byte float into a much longer sized character buffer.
I got hold of these 2 functions to convert a 4 byte float into an unsinged 32 bit integer to transfer over network which works fine for a simple C++ UDP program but for Qt I need to receive the data as unsigned char.
Is it possible to avoid converting the floatinf point data into a string and then sending it?
uint32_t htonf(float f)
{
uint32_t p;
uint32_t sign;
if (f < 0) { sign = 1; f = -f; }
else { sign = 0; }
p = ((((uint32_t)f)&0x7fff)<<16) | (sign<<31); // Whole part and sign.
p |= (uint32_t)(((f - (int)f) * 65536.0f))&0xffff; // Fraction.
return p;
}
float ntohf(uint32_t p)
{
float f = ((p>>16)&0x7fff); // Whole part.
f += (p&0xffff) / 65536.0f; // Fraction.
if (((p>>31)&0x1) == 0x1) { f = -f; } // Sign bit set.
return f;
}
Have you tried using readDatagram? Or converting the data to a QByteArray after reading? In many cases a char* is really just a byte array. This is one of those cases. Note that the writeDatagram can take a QByteArray.
Generally every thing sent across sockets is in bytes not strings, layers on either end do the conversions. Take a look here, especially the Broadcaster examples. They show how to create a QByteArray for broadcast and receive.
Not sure why the downvote, since the question is vague in requirements.
A 4-byte float is simply a 4 character buffer, if cast as one. If the systems are homogenous, the float can be sent as a signed char *, and bit for bit it'll be the same read into the signed char * on the receiver directly, no conversion needed. If the systems are heterogenous, then this won't work and you need to convert it to a portable format, anyway. IEEE format is often used, but my question is still, what are the requirements, is the float format the same between systems?
If I read it correctly, your primary question seems to be how to receive data of type unsigned char with QT's readDatagram function which uses a pointer to a buffer of type char.
The short answer is use a cast along these lines:
const size_t MAXSIZE = 1024;
unsigned char* data = malloc(MAXSIZE);
readDatagram ( (unsigned char *)data, MAXSIZE, address, port )
I'm going to assume you have multiple machines which use the same IEEE floating point format but some of which are big endian and some of which are little endian. See this SO post for a good discussion of this issue.
In that case you could do something a bit simpler like this:
const size_t FCOUNT = 256;
float* data = malloc(FCOUNT * sizeof(*data));
readDatagram ( (char *)data, FCOUNT * sizeof(*data), address, port )
for (int i = 0; i != FCOUNT; ++i)
data[i] = ntohf(*((uint32_t*)&data[i]));
The thing to remember is that as far as networking functions like readDatagram are concerned, the data is just a bunch of bits and it doesn't care what type those bits are interpreted as.
If both ends of your UDP connection use Qt, I would suggest looking at QDataStream. You can create this from a QByteArray each time you read a datagram, and then read whatever values you require - floats, maps, lists, QVariants, and of course string.
Similarly, on the sending side, you'd create a data stream, push data into it, then send the resulting QByteArray over writeDatagram.
Obviously this only works if both ends use Qt - the data encoding is well-defined, but non-trivial to generate by hand.
(If you want stream orientated behaviour, you could use the fact that QUDPSocket is a QIODevice with a data-stream, but it sounds as if you want per-datagram behaviour)