QTextStream - What exactly does the position method return - qt

I have a question about what a QTextStream is calculating with the pos() method. I assumed it was the number of bytes, but it seems that this might not be the case.
The reason I ask, is that I am processing rows in a file, and once the number of rows read reached some arbitrary number or stream.atEnd() is true, I break out of the loop and save stream.pos() to a qint64* variable. Once the processing is complete, I go back to the file and seek(*filePosition) to get back to my last position and grab more data until stream.atEnd() is true. This works in the sense that can keep track of where I am, but it is very slow calling stream.pos() as is noted in the Qt docs.
What I am attempting is to update the file position after each line is read in an efficient manner. However, it is not working and when the program goes back to read the file again, the position is not correct as the first line it reads starts in the middle of line previously read on the last iteration.
Here is what is have so far:
QTextStream stream(this);
stream.seek(*filePosition);
int ROW_COUNT = 1;
while (!stream.atEnd()) {
QString row = stream.readLine();
QStringList rowData = row.split(QRegExp(delimiter));
*filePosition += row.toUtf8().size();
/*
processing rowData...
*/
if (ROW_COUNT == ROW_UPLOAD_LIMIT) {
break;
}
ROW_COUNT++;
}
/*
close files, flush stream, more processing, etc...
*/

QTextStream::pos returns position in bytes. I see the following problems:
You are not accounting for the line ending character (or 2 characters)
In UTF-8, a single character might take more than 1 byte
Also, why save buffer position after reading each line? This might be faster:
if (ROW_COUNT == ROW_UPLOAD_LIMIT) {
*filePosition = stream.pos();
break;
}

The solution was to create the QTextStream outside of the function and pass it in as a parameter. Doing this allows me to not have to worry about tracking the position on each iteration because I keep the stream in scope until I have completely finished processing the file.
class FeedProcessor {
...
void processFeedFile() {
IntegrationFile file("big_file.txt");
file.open(QIODevice::ReadOnly | QIODevice::Text);
QTextStream stream(&file);
while(!stream.atEnd()) {
file.extractFileContents(&stream);
/*
do more processing with file
*/
}
}
...
}
class IntegrationFile : public QFile {
...
void extractFileContents(QTextStream* stream) {
int ROW_COUNT = 1;
while (!stream.atEnd()) {
QString row = stream.readLine();
QStringList rowData = row.split(QRegExp(delimiter));
/*
processing rowData...
*/
if (ROW_COUNT == ROW_UPLOAD_LIMIT) {
break;
}
ROW_COUNT++;
}
/*
close files, flush stream, more processing, etc...
*/
}
...
}

Related

arduino, setup ethernet & network using data from SD config file

Im try to add to my sketch a dynamic way to setup the ethernet info (mac, ip, gateway, subnet) from a configuration file (config.txt). So running a webserver and serving htm files from sd card, user can go to setting page, fill a form with these info and when posted , the webserver parse the http form and save (update) the config.txt file. After that system do a restart, in order to start with the new settings (by read the config.txt file)
I have create succesfully all the parts (sd, ethernet, webserver, webclient, create the config file from posted form data) except the get params by reading the config.txt file.
I can read line by line the config, I can split the line to param & value, and now I need to fill some byte variables with the readed data. I can (after a month of google searching) to read IPs (decimal values) to byte array. Im stack to read the MAC ADDRESS hex into byte array. The config file contains the:
mac=8f:2c:2b:19:e0:b7;
ip=192.168.1.200;
netmask=255.255.255.0;
gateway=192.168.1.254;
dns=8.8.8.8;
posturl=192.168.1.157;
postport=8080;
postscript=/itherm/update.php;
interval=60000;
and the code that I use to read is:
byte myMAC[6];
byte myIP[4];
File fset;
fset = SD.open("config.txt");
if (fset){
char ff[40];
while (fset.available()>1){
bool eol=false;
for (int i=0; !eol;i++){
ff[i]=fset.read();
if (ff[i]=='\n'){
eol=true;
}
}
String par="";
bool DONE=false;
for (int i=0; !DONE;i++){
par+=ff[i];
if (ff[i]== '='){DONE=true;}
}
String pval="";
DONE=false;
//------------------------
if (par=="ip=" ){
int x=0;
while(!DONE){
for(int i=3;i<=i+21;i++){
if(ff[i]=='.'){
myIP[x]=pval.toInt();
x++;
i++;
pval="";
}
else if(ff[i]==';' || i>20){
myIP[x]=pval.toInt();
DONE=true;
break;
}
pval+=ff[i];
}
}
}
} //while (fset.available()>1)
} //if (fset)
I will appreciate any help. Please no answers with simple use of Serial.print(). I have found hundreds of suggestions but none, that work properly to read all the parameters (dec, hex, strings). After a month of effort & searching, I wonder why something so necessary and useful does not exist as an example in the community, completely functional !!
Best regards
Okay so here is a complete set of routines to do what you want -I think you misunderstood the concept of char arrays vs a single char[0] The routines are documented and self explanatory. I recomend not to finish lines with ; but with '\n' which in your example is there anyway (also you can not see the new line terminator) To get the mac address I need three lines:
if (strncmp(cfgLine, "mac=", 4) == 0) {
strcpy (macAddr, cfgLine + 4);
}
line one compares the first 4 characters and if it is 0 (meaning its a fit)
line two copies the chars from the fifth to the last char from the lineBuffer to the target array, which can actually be used as param for functions.
The file structure should be with no ; as you would have to parse ; and \n
mac=8f:2c:2b:19:e0:b7
ip=192.168.1.200
....
postport=8080
To convert a char array to eg int we use atoi(), to convert a single char[0] to a single number we use int singleDigit = char[0]-48;
const char configurationFilePath [] = "/someconfig.txt";
char cfgLine[128] = {'\0'}; // this is a global temp char array to hold the read lines (lenght= chars longest line +1)
char numBuffer[16] = {'\0'}; // this is a global temo char array to help to convert char to number
char macAddr [18] = {'\0'}; // this is a global char array to hold the mac address
char ipAddr [16] = {'\0'}; // this is a global char array to hold the IP address - max xxx.xxx.xxx.xxx
int postport=0;
// .... you can easyly implement for all other data you want to store/retrieve
// Counts the lines of a file
uint16_t countLines() {
uint16_t currentLineCount = 0;
File cfgFile = SD.open(configurationFilePath, "r");
if (!cfgFile) {
Serial.println(F("Config file open failed on read"));
} else {
while (cfgFile.available()) {
/** Lets read line by line from the file */
if (cfgFile.read() == '\n') currentLineCount ++; // Lines are delimited by '\n'
}
cfgFile.close();
}
return currentLineCount;
}
//Load the config file from SD/SPIFFS/LittleFS
bool loadConfigFile() {
uint16_t lineCounter = countLines();
if (lineCounter <= 0) {
Serial.print(F("No config data stored in file ")); Serial.println(configurationFilePath);
return false;
}
else {
File cfgFile = SD.open(configurationFilePath, "r");
while (cfgFile.available()) {
strcpy (cfgLine, (cfgFile.readStringUntil('\n').c_str())); // normaly you use new line, we copy one line at a time
// Serial.println(cfgLine); /** Printing for debuging purpose */
while (cfgLine[0] != '\0') { /* Block refilling of cfgLine till processed */
loadSingleCfgLine();
}
}
cfgFile.close();
Serial.println(F("[Success] Loaded config !"));
return true;
}
}
//Load the data of a single line into a char array
void loadSingleCfgLine() {
if (strncmp(cfgLine, "mac=", 4) == 0) {
strcpy (macAddr, cfgLine + 4);
}
if (strncmp(cfgLine, "ip=", 3) == 0) {
strcpy (ipAddr, cfgLine + 3);
}
if (strncmp(cfgLine, "postport=", 9) == 0) {
strcpy (numBuffer, cfgLine + 9);
postport = atoi(numBuffer); // One extra step to convert to int
}
// ... easy to implement for all other data
}
I divided the routines into small independend functions, so its easy adaptable for different uses. I'm sorry for not digging into your code as it is hard to follow and unclear what you want todo.As an added bonus we do not use the String class. These Strings tend to fragment heap - causing resets/crashes while the global char arrays are compiled to flash and don't show this behavior.

Add String from textEdit to a QStack

I am trying to capture the contents from textEdit and add it to a QStack. The content I am splitting so to be able to reverse the order of the sentence captured. I already have that part covered, but I want to be able to convert from QStringList to be pushed to the QStack. This is what I have:
void notepad::on_actionReversed_Text_triggered()
{
QString unreversed = ui->textEdit->toPlainText();
QStringList ready = unreversed.split(QRegExp("(\\s|\\n|\\r)+"), QString::SkipEmptyParts);
QStack<QString> stack;
stack.push(ready);
QString result;
while (!stack.isEmpty())
{
result += stack.pop();
ui->textEdit->setText(result);
}
}
QStack::push only takes objects of it's template type.
i.e. In your case you must push QString's onto the QStack<QString>, and not a list.
So, iterate the list of strings pushing each one in turn.
foreach (const QString &str, ready) {
stack.push(str);
}
Something else that is wrong is that you are updating textEdit inside the for loop. You actually want to build up the string in the loop and afterwards update the text.
QString result;
while (!stack.isEmpty()) {
result += stack.pop();
}
ui->textEdit->setText(result);
An alternate answer could be to do away with the stack and just iterate the QStringList ready in reverse order to build up result string.

How to report progress without going through the entire file before I process

I have to parse a big file with several thousand records. I want to display a progress bar, but I do not know the number of recordings in advance. Do I have to roll over this file twice?
The first to know the number of recording.
The second to perform the treatment
Or is there a more simple way to get the number of records without going through the entire file before I process ?
My code snippet :
void readFromCSV(const QString &filename){
int line_count=0;
QString line;
QFile file(filename);
if(!file.open(QFile::ReadOnly | QFile::Text))
return;
QTextStream in(&file);
while(!in.atEnd()){ //First loop
line = in.readLine();
line_count++;
}
while (!in.atEnd()) { //Second loop
...
line = in.readLine();
process();
...
}
}
Thank you for help
This question is different from the one here : counting the number of lines in a text file
1) The loop process is already done. In this case it is prevention of double shooting.
2) The code is a QT one, not a C++ fonction to add as a redundancy
I think there is no way to count lines without reading the file at least once. To avoid it I wouldn't rely on the number of lines, but rather on how much data I have been read. Here is how I would solve this problem:
QFile file(fileName);
file.open(QFile::ReadOnly | QFile::Text);
const auto size = file.size();
QTextStream in(&file);
while (!in.atEnd()) {
auto line = in.readLine();
// Process line.
// [..]
// Calculate the progress in percent.
auto remains = file.bytesAvailable();
auto progress = ((size - remains) * 100) / size;
printf("progress: %d%%\r", progress);
}

Strange behavior of program on Qt 5

I'm writing two programs on Qt 5. The first ("Debug") program sends requests to the second ("Simulator") through a PCI-E device. The "Simulator" must respond as fast as possible. This is a separate issue, but now I want to ask about another strange effect. The "Debug" program writes 10 bytes from QLineEdit to a PCI-E device and then waits for an answer from the "Simulator". When the answer came fast enough, I see the right data bytes in the "Debug" program window, otherwise a PCI-E device returns a copy of a sended data and I see it too. The issue is that data can be sended by two ways: by clicking the Send button on the form and by clicking the Return button on a keyboard. In both cases the data is sent from the following slot:
void MyWin::on_pushButton_Send_clicked()
{
if(data_matched)
{
QString exp = ui.lineEdit_Data->text();
QStringList list = exp.split(QRegExp("\\s"),
QString::SkipEmptyParts);
for(int i=0; i<list.size(); i++)
{
quint8 a = list[i].toUInt(0, 16);
data[i] = a;
}
write_insys(bHandle, data, DataSize);
ui.textEdit->append( /* show sended bytes */ );
read_insys(bHandle, data, DataSize);
ui.textEdit->append( /* show received bytes */ );
}
}
But in the second case (on Return key press) the only difference is that the above slot is invoked inside the following:
void MyWin::on_lineEdit_Data_returnPressed()
{
on_pushButton_Send_clicked();
}
But the results:
1st case: 90% wrong answers
2st case: 90% right answers
The code of write_insys and read_insys is absolutely trivial, simply call the library functions:
bool write_insys(BRD_Handle handle, void* data, int size)
{
S32 res = BRD_putMsg(handle, NODE0, data, (U32*)&size, BRDtim_FOREVER);
return (res >= 0);
}
bool read_insys(BRD_Handle handle, void* data, int size)
{
S32 res = BRD_getMsg(handle, NODE0, data, (U32*)&size, BRDtim_FOREVER);
return (res >= 0);
}
Does anyone know why this might happen?
Windows 7, Qt 5.4.2, Msvc 2010.
edit: Most likely it is a Qt bug...

ifstream sets failbit on declaration

I don't really understand why this is happening at all. I'm trying to open a file to read some data into my program but failbit gets set instantly, having moved error messages around it seems that failbit actually gets set before I even attempt input.open(). The undeclared variables in the code are globals which live elsewhere (messy but will refine later) Here's the offending function from a larger project:
int read_input()
{
ifstream input;
string value;
int file_complete=0;
int count=0;
if(input.failbit)
printf("Really?\n");
input.clear();
input.open("Input.txt");
while(!file_complete)
{
if(input.failbit)
{
printf("Reading in set-up value number %d failed.", count+1);
getchar();
return 1;
}
else if(input.eofbit)
{
file_complete=1;
}
if(input.get()=='=')
{
getline(input, value);
switch(count)
{
case 0:
n_loci=atoi(value.c_str());
count++;
break;
case 1:
n_founders=atoi(value.c_str());
count++;
break;
case 2:
n_self=atoi(value.c_str());
count++;
break;
// Add more cases later
}
}
}
input.close();
return 0;
}
This program for me turns out:
Really?
Reading in set-up value number 1 failed.
I'm assuming I've done something very stupid but have been at this quite a while now.
P.S I'm compiling with the latest version of g++ on cygwin on top of Windows 7.
OK fixed this myself now:
Failbit seems to be set by EOFs in some implementations so instead I switched to using input.good() and consequently all was good.
There was also some logical error in my program with the checking for "=" part too as ifstream.get() returns it's value as an integer so I needed to cast them back into chars for comparison.

Resources