In my Console Project I have 4 functions: Response, AddDataToList, Request and main.
In main function chars is a Vector containing characters a-z. manager, doc, array and list are variables to be used in Request function along with the elements of chars. In the first for loop a single character (eg. a,b,c,…,z) from chars is passed to Request and in the following nested for loop two characters (eg. aa,ab,ac,…,zz) are passed to Request.
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
QVector<char> chars;
for (int i = 97; i < 123; i++) chars.push_back((char)i);
QNetworkAccessManager manager;
QJsonDocument doc;
QJsonArray array;
QVector<QString> list;
int count = chars.count();
for (int i = 0; i < count; i++) {
Request(manager, QString(chars[i]), doc, array, list);
for (int j = 0; j < count; j++)
Request(manager, QString("%1%2").arg(chars[i]).arg(chars[j]), doc, array, list);
}
qDebug() << "Total: " + QString::number(list.count());
return a.exec();
}
In Request those characters are used as querystring in the url that is passed to Response function to get json array and each element of array is then added to the Vector named list in AddDataToList function.
void Request(QNetworkAccessManager &manager, QString &queryString,
QJsonDocument &doc, QJsonArray &array, QVector<QString> &list)
{
QUrl url =QString("http://www.icab.org.bd/icabweb/firmCompanyAudited/geJsonAuditedFirm?term=%1").arg(queryString);
Response(manager.get(QNetworkRequest(url)), doc, array);
AddDataToList(list, array);
}
void Response(QNetworkReply *reply, QJsonDocument &doc, QJsonArray& array)
{
QEventLoop loop;
QObject::connect(reply, SIGNAL(finished()), &loop, SLOT(quit()));
loop.exec();
doc = QJsonDocument::fromJson(reply->readAll());
delete reply;
array = doc.array();
}
void AddDataToList(QVector<QString> &list, QJsonArray &array)
{
int count = array.count();
for (int n = 0; n < count; n++) list.append(array[n].toString());
}
Last line of the main writes the number of elements in the list and I expected to get same number each time it runs, but the number was:
19793, 19703, 19791, etc.
reply->error() always gives NoError.
Where is the problem?
EDIT:
I also have written equivalent code in C# where I had to use Newtonsoft.Json's DeserializeObject method in a try-catch in order to complete the loop. Result of 5 consecutive test sessions was:
Test Session: 0
19793 item found
0 item in error List
====================================================
Test Session: 1
ev : Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
yy : Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
19743 item found
2 item in error List
====================================================
Test Session: 2
cm : Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
jv : Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
nl : Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
19737 item found
3 item in error List
====================================================
Test Session: 3
uk : Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
19743 item found
1 item in error List
====================================================
Test Session: 4
lk : Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
19768 item found
1 item in error List
====================================================
May I get somewhat similar error in my Qt Response function when I call:
QJsonDocument::fromJson(reply->readAll()) or doc.array()?
I guess the problem is in doc = QJsonDocument::fromJson(reply->readAll()). It assigns <html> in doc in some cases.
Why does it send <html> instead of json?
I got curious about your problem and added your code to my test application.
I needed to do one change to get you code compiled with gcc. You were passing temporary object as a query string. Add const to QString &queryString in Request function. I guess you are using MS compiler which is not as strict with this...
error: invalid initialization of non-const reference of
type ‘QString&’ from an rvalue of type ‘QString’
Request(manager, QString(chars[i]), doc, array, list);
I also added logging to Request
qDebug() << "queryString:" << queryString;
and also to AddDataToList
void AddDataToList(QVector<QString> &list, QJsonArray &array)
{
int count = array.count();
qDebug() << "count:" << count;
for (int n = 0; n < count; n++)
{
QString appendString = array[n].toString();
qDebug() << "index:" << n << "appendString:" << appendString;
list.append(appendString);
}
}
Then I started testing. I used shorter array of chars by changing integers to 110 -> 119 which lead to query strings n,nn, -> vu,vv. I directed output to a file on each test round so I could analyze the results afterwards.
I got result 2984 over 10 times. During the testing session of about half an hour I got two results that differed. I compared the log files and noticed that sometimes the server returned empty response for some query strings in differing results. The count of result 2984 in parenthesis.
I got result 2816 once:
queryString: "nr"
count: 0 (18)
queryString: "ns"
count: 0 (50)
queryString: "nt"
count: 0 (50)
queryString: "nu"
count: 0 (50)
I got result 2934 once:
queryString: "or"
count: 0 (50)
I would say it's a server problem.
Addition to EDIT part of the question:
You don't get as detailed error output but anyway it's caught like this:
QJsonParseError e;
doc = QJsonDocument::fromJson(reply->readAll(), &e);
if ( e.error != QJsonParseError::NoError )
{
qWarning() << "Json parse error:" << e.errorString().toLatin1().data();
}
Here is what I got from parsing:
queryString: "uy"
Json parse error: illegal value
count: 0
See more details about QJsonParseError Class from here.
Note: There is no point of calling QCoreApplication::exec() and entering the main event loop in your current implementation.
Related
I am trying to reverse the words within a QStringList. Below is the code up to now, but I keep on getting a "Index out of Range" error. From the error it would seem that I am trying to use data that is out of scope, but not able to figure out my problem.
QString reversed;
QStringList reversedlist;
QStringlist list = input.split(" ");
for (int i=0;i<list.length(); ++1)
reversedlist[i] = list[list.length() -1 -i];
reversed = reversedlist.join(" ");`
Thanks
As pointed out by #ThorngardSO, the reversedlist is initially empty, and you are trying to access the invalid index in your loop code. You should add values to the list using one of the following functions:
STL-compatible function push_back() (inserts value at the end of the list)
STL-compatible function push_front() (inserts value at the beginning of the list)
Qt function append() (Qt alternative for push_back)
Qt function prepend() (Qt alternative for push_front)
As you see, prepend() inserts the element at the beginning of the list, that is why it makes the reversing of the list very simple:
for (int i = 0; i < list.length(); ++i) {
reversedlist.prepend(list[i]);
}
Also, note that there is a typo in your loop: it should be ++i instead of ++1.
Your reversedList is initially empty. You have to actually append the items, like this:
reversedlist.push_back (list[list.length () - 1 - i]);
Naturally, trying to access non-existing items via reversedList[i] does not work and throws the index-out-out-range error.
You got the index out of range as there is no sting inside the QStringList reversedlist. So when your code reach the line reversedlist[0], it throw the "index out of range" error. And you can read the value using [0] and cant assign.
if you want to assign the value to a particular index of the QStringList.
QString reversed;
QStringList reversedlist;
QString input="123 456 789 000";
QStringList list = input.split(" ");
for (int i=0;i<list.length(); ++i){
//value to particular index
reversedlist.insert(i,list[list.length() -1 -i]);
}
reversed = reversedlist.join(" ");
qDebug() << reversed;
I'm writing a unix cp program, but I'm unclear about checking for EOF. The code I have is:
int main(int argc, const char * argv[]) {
int in, out;
char buf[BUFFER_SIZE];
if (argc != 3)
cout << "Error: incorrect number of params" << endl;
if ((in = open(argv[1], O_RDONLY, 0666)) == -1)
cout << "Error: cannot open input file" << endl;
if ((out = open(argv[2], O_WRONLY | O_CREAT, 0666)) == -1)
cout << "Cannot create output file" << endl;
else
while ((read(in, buf, BUFFER_SIZE)) != -1)
write(out, buf, BUFFER_SIZE);
return 0;
}
It reads and writes fine, but writes past EOF when writing the output file. So I get a couple lines of gibberish past the end of the file. Am I just not checking for EOF correctly? I appreciate the input.
You should read the man page for the read function.
On end-of-file, read returns 0. It returns -1 only if there's an error.
read can read fewer bytes than you asked to (and it must do so if there aren't that many bytes remaining to be read). Your write call assumes that read actually read BUFFER_SIZE bytes.
You need to save the result returned by read and write only that many bytes -- and you need to terminate the loop when read returns 0 (indicating end-of-file) or -1 (indicating an error). In the latter case, you should probably do something to handle the error, or at least inform the user.
Incidentally, you don't need the 0666 mode argument when calling open to open the file for reading; that applies only with O_CREAT. Since open is actually a variadic function (like printf), you don't have to supply all the arguments.
The man page is not clear on this point; it pretends that there are two different forms of the open function:
int open(const char *pathname, int flags);
int open(const char *pathname, int flags, mode_t mode);
but in fact that's not legal in C. The POSIX description correctly shows the declaration as:
int open(const char *path, int oflag, ...);
I'm trying to ISend() two arrays: arr1,arr2 and an integer n which is the size of arr1,arr2. I understood from this post that sending a struct that contains all three is not an option since n is only known at run time. Obviously, I need n to be received first since otherwise the receiving process wouldn't know how many elements to receive. What's the most efficient way to achieve this without using the blokcing Send() ?
Sending the size of the array is redundant (and inefficient) as MPI provides a way to probe for incoming messages without receiving them, which provides just enough information in order to properly allocate memory. Probing is performed with MPI_PROBE, which looks a lot like MPI_RECV, except that it takes no buffer related arguments. The probe operation returns a status object which can then be queried for the number of elements of a given MPI datatype that can be extracted from the content of the message with MPI_GET_COUNT, therefore explicitly sending the number of elements becomes redundant.
Here is a simple example with two ranks:
if (rank == 0)
{
MPI_Request req;
// Send a message to rank 1
MPI_Isend(arr1, n, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD, &req);
// Do not forget to complete the request!
MPI_Wait(&req, MPI_STATUS_IGNORE);
}
else if (rank == 1)
{
MPI_Status status;
// Wait for a message from rank 0 with tag 0
MPI_Probe(0, 0, MPI_COMM_WORLD, &status);
// Find out the number of elements in the message -> size goes to "n"
MPI_Get_count(&status, MPI_DOUBLE, &n);
// Allocate memory
arr1 = malloc(n*sizeof(double));
// Receive the message. ignore the status
MPI_Recv(arr1, n, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}
MPI_PROBE also accepts the wildcard rank MPI_ANY_SOURCE and the wildcard tag MPI_ANY_TAG. One can then consult the corresponding entry in the status structure in order to find out the actual sender rank and the actual message tag.
Probing for the message size works as every message carries a header, called envelope. The envelope consists of the sender's rank, the receiver's rank, the message tag and the communicator. It also carries information about the total message size. Envelopes are sent as part of the initial handshake between the two communicating processes.
Firstly you need to allocate memory (full memory = n = elements) to arr1 and arr2 with rank 0. i.e. your front end processor.
Divide the array into parts depending on the no. of processors. Determine the element count for each processor.
Send this element count to the other processors from rank 0.
The second send is for the array i.e. arr1 and arr2
In other processors allocate arr1 and arr2 according to the element count received from main processor i.e. rank = 0. After receiving element count, receive the two arrays in the allocated memories.
This is a sample C++ Implementation but C will follow the same logic. Also just interchange Send with Isend.
#include <mpi.h>
#include <iostream>
using namespace std;
int main(int argc, char*argv[])
{
MPI::Init (argc, argv);
int rank = MPI::COMM_WORLD.Get_rank();
int no_of_processors = MPI::COMM_WORLD.Get_size();
MPI::Status status;
double *arr1;
if (rank == 0)
{
// Setting some Random n
int n = 10;
arr1 = new double[n];
for(int i = 0; i < n; i++)
{
arr1[i] = i;
}
int part = n / no_of_processors;
int offset = n % no_of_processors;
// cout << part << "\t" << offset << endl;
for(int i = 1; i < no_of_processors; i++)
{
int start = i*part;
int end = start + part - 1;
if (i == (no_of_processors-1))
{
end += offset;
}
// cout << i << " Start: " << start << " END: " << end;
// Element_Count
int e_count = end - start + 1;
// cout << " e_count: " << e_count << endl;
// Sending
MPI::COMM_WORLD.Send(
&e_count,
1,
MPI::INT,
i,
0
);
// Sending Arr1
MPI::COMM_WORLD.Send(
(arr1+start),
e_count,
MPI::DOUBLE,
i,
1
);
}
}
else
{
// Element Count
int e_count;
// Receiving elements count
MPI::COMM_WORLD.Recv (
&e_count,
1,
MPI::INT,
0,
0,
status
);
arr1 = new double [e_count];
// Receiving FIrst Array
MPI::COMM_WORLD.Recv (
arr1,
e_count,
MPI::DOUBLE,
0,
1,
status
);
for(int i = 0; i < e_count; i++)
{
cout << arr1[i] << endl;
}
}
// if(rank == 0)
delete [] arr1;
MPI::Finalize();
return 0;
}
#Histro The point I want to make is, that Irecv/Isend are some functions themselves manipulated by MPI lib. The question u asked depend completely on your rest of the code about what you do after the Send/Recv. There are 2 cases:
Master and Worker
You send part of the problem (say arrays) to the workers (all other ranks except 0=Master). The worker does some work (on the arrays) then returns back the results to the master. The master then adds up the result, and convey new work to the workers. Now, here you would want the master to wait for all the workers to return their result (modified arrays). So you cannot use Isend and Irecv but a multiple send as used in my code and corresponding recv. If your code is in this direction you wanna use B_cast and MPI_Reduce.
Lazy Master
The master divides the work but doesn't care of the result from his workers. Say you want to program a pattern of different kinds for same data. Like given characteristics of population of some city, you want to calculate the patterns like how many are above 18, how
many have jobs, how much of them work in some company. Now these results don't have anything to do with one another. In this case you don't have to worry about whether the data is received by the workers or not. The master can continue to execute the rest of the code. This is where it is safe to use Isend/Irecv.
hi a have a function that reads from a text file line by line each line I do some operations on it substitute a string..etc
then I push_back that line into a vector
this is my class in Parser.h
class Parser
{// start class
public:
vector<const char*> patterns;
Parser();
~Parser();
void RuleParser(const char *TextFileName); // this is the function that takes the file name
private:
};// end class
segment from function RuleParser
std::ifstream ifs(TextFileName);
while (!ifs.eof())
{
.
.modification code
.
patterns.push_back((buildString).c_str()); //buildString is the modified line
cout << buildString << endl;
}
but when I try to check out if the data in the vector is correct it output totally different data.
I even put a cout after the push_back to check it's integrty but I found buildString is correct... thats the data each time being pushed ... what I am doing wrong.
here is the loop I use to see if my data correct.
for (int i = 0;i < patterns.size() ;i++)
{
cout << patterns.at(i) << endl;
}
Well patterns is the collection of pointers so you end up push_back'ing a pointer to the same buildString in each iteration of the loop, instead of push_back'ing the string contents. Then when buildString changes in next iteration of the loop, the pointer becomes invalid but it still remains in patterns - not good
I suggest you declare patterns as:
vector<std::string> patterns;
This way when you do:
patterns.push_back(buildString.c_str())
the contents of the string will be copied instead of the pointer, and remain valid througout.
I have some problems combining the processing results I recieve from several Threads. And I'm not sure, if I use openmp correctly. The below code extract shows the openmp portion of my code.
Parameters:
thread private:
it: map iterator (timestamp, userkey)
ite: map iterator ((timestamp,userkey)/int amount)
thread_result_map: typedef map < userkey(str),timestamp(str) >
when, who: matching regex (timestamp, userkey)
shared among threads:
log: char array
size: log.size()
identifier, timestamp, userkey: boost::regex patterns
combined_result_map: typedef map < thread_result_map, hits(int) >
#pragma omp parallel shared(log, size, identifier, timestamp, userkey) private(it, ite, str_time, str_key, vec_str_result, i, id, str_current, when, who, thread_result_map)
{
#pragma omp for
for (i = 0 ; i < size ; i++){
str_current.push_back(log[i]);
if (log[i] == '\n') {
if (boost::regex_search(str_current, identifier)){
boost::regex_search(str_current, when, timestamp);
str_time = when[0];
boost::regex_search(str_current, who, userkey);
str_key = who[0];
thread_result_map.insert(make_pair(str_time, str_key));
}
str_current = ""; //reset temp string
}
}
#pragma omp critical
{
for (it=thread_result_map.begin(); it!=thread_result_map.end(); it++) {
id = omp_get_thread_num();
cout << thread_result_map[it->first] <<
thread_result_map[it->second];
cout << "tID_" << id << " reducing" << endl;
}
}
}
As you can see every thread has his own partition of the char array, it parses line by line from the array and if the current string is identified by "identifier", the timestamp and userkey are added to the thread's private result map (string/string).
Now after the loop I have several thread's private result maps. The combined_result_map is a map inside a map. The key is the combination of key/value of the threads result and the value is the amount of occurences of this combination.
I'm parsing only a portion of the timestamp so when in 1 hour the same userkey appears multiple times the hit counter will be increased.
The result should look something like this:
TIME(MMM/DD/HH/);USERKEY;HITS
May/25/13;SOMEKEY124345;3
So I have no problems combining hit amounts in the critical section (which I removed) by specifying combined+=results.
But how can I combine my result maps the same way? I know I have to iterate through threads maps, but when I put a "cout" inside the loop for testing every thread calls it only once.
A test run on my local syslog gives me the following output when I set all the regex to "error" (to make sure every identified line will have a userkey and a timestamp with the same name):
Pattern for parsing Access String:
error Pattern for parsing Timestamp:
error Pattern for parsing Userkey:
error
*** Parsing File /var/log/syslog
errortID_0 reducing errortID_1
reducing errortID_2 reducing
errortID_3 reducing
*** Ok! ________________ hits :
418 worktime: 0.0253871s
(The calculated hits come from thread private counters, that I removed in the code above)
So every of my 4 threads does a single cout and leaves the loop, although all together should have 418 hits. So what do I do wrong? How do I iterate through my results from inside my openmp area?
Found the problem myself, sorry for asking stupid questions.
I was trying to add the same key multiple times, that's why map size didn't increase and every thread looped only once.
Edit:
If anybody is interested in the solution how to combine thread results, this is how I did it. perhaps you see anything that could be improved.
I just changed the local threads result map to a vector of pairs(str,str).
This is the full working openmp code section. Pehaps it's useful for anyone:
#pragma omp parallel shared(log, size, identifier, timestamp, userkey) private(it, ite, str_time, str_key, i, id, str_current, when, who, local_res)
{
#pragma omp for
for (i = 0 ; i < size ; i++){
str_current.push_back(log[i]);
if (log[i] == '\n') { // if char is newline character
if (boost::regex_search(str_current, identifier)){ // if current line is access string
boost::regex_search(str_current, when, timestamp); // get timestamp from string
str_time = when[0];
boost::regex_search(str_current, who, userkey); // get userkey from string
str_key = who[0];
local_res.push_back((make_pair(str_time, str_key))); // append key-value-pair(timestamp/userkey)
id = omp_get_thread_num();
//cout << "tID_" << id << " - adding pair - my local result map size is now: " << local_res.size() << endl;
}
str_current = "";
}
}
#pragma omp critical
{
id = omp_get_thread_num();
hits += local_res.size();
cout << "tID_" << id << " had HITS: " << local_res.size() << endl;
for (i = 0; i < local_res.size(); i++) {
acc_key = local_res[i].second;
acc_time = local_res[i].first;
if(m_KeyDatesHits.count(acc_key) == 0) { // if there are no items for this key yet, make a new entry
m_KeyDatesHits.insert(make_pair(acc_key, str_int_MapType()));
}
if (m_KeyDatesHits[acc_key].count(acc_time) == 0) { // "acc_time" is a key value, if it doesn't exist yet, add it and set "1" as value
m_KeyDatesHits[acc_key].insert(make_pair(acc_time, 1 ));
it = m_KeyDatesHits.begin(); // iterator for userkeys/maps
ite = m_KeyDatesHits[acc_key].begin(); // iterator for times/clicks
} else m_KeyDatesHits[acc_key][acc_time]++; // if userkey already exist and timestamp already exists, count hits +1 for it
}
}
}
I did some tests and it's really running fast.
Using 4 Threads this searches a 150MB LogFile for access events, parses a custom user key and date from every event and combines the results in under 4 seconds.
At the End it creates a export list. This is the program output:
HELLO, welcome to LogMap 0.1!
C++/OpenMP Memory Map Parsing Engine
__________________ Number of processors available = 4
Number of threads = 4
Pattern for parsing Access String:
GET /_openbooknow/key/ Pattern for
parsing Timestamp: \d{2}/\w{3}/\d{4}
Pattern for parsing Userkey:
[a-zA-Z0-9]{20,32}
* Parsing File
/home/c0d31n/Desktop/access_log-test.txt
HITS: 169147 HITS: 169146 HITS: 169146
HITS: 169147
* Ok! ________ hits :
676586 worktime: 4.03816s
* new export file created: "./test.csv"
root#c0d3b0x:~/workspace/OpenBookMap/Release#
cat test.csv
"1nDh0gV6eE3MzK0517aE6VIU0";"28/Mar/2011";"18813"
"215VIU1wBN2O2Fmd63MVmv6QTZy";"28/Mar/2011";"6272"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"18/Mar/2011";"18816"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"21/Mar/2011";"12544"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"22/Mar/2011";"12544"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"23/Mar/2011";"18816"
"9E1608JFGk2GZQ4ppe1Grtv2";"28/Mar/2011";"12544"
"pachCsiog05bpK0kDA3K2lhEY";"17/Mar/2011";"18029"
"pachCsiog05bpK0kDA3K2lhEY";"18/Mar/2011";"12544"
"pachCsiog05bpK0kDA3K2lhEY";"21/Mar/2011";"18816"
"pachCsiog05bpK0kDA3K2lhEY";"22/Mar/2011";"6272"
"pachCsiog05bpK0kDA3K2lhEY";"23/Mar/2011";"18816"
"pachCsiog05bpK0kDA3K2lhEY";"28/Mar/2011";"501760"
"1nDh0gV6eE3MzK0517aE6VIU0";"28/Mar/2011";"18813"
"215VIU1wBN2O2Fmd63MVmv6QTZy";"28/Mar/2011";"6272"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"18/Mar/2011";"18816"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"21/Mar/2011";"12544"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"22/Mar/2011";"12544"
"36Pu0A2Wly3uYeIPZ4YPAuBy";"23/Mar/2011";"18816"
"9E1608JFGk2GZQ4ppe1Grtv2";"28/Mar/2011";"12544"
"pachCsiog05bpK0kDA3K2lhEY";"17/Mar/2011";"18029"
"pachCsiog05bpK0kDA3K2lhEY";"18/Mar/2011";"12544"
"pachCsiog05bpK0kDA3K2lhEY";"21/Mar/2011";"18816"
"pachCsiog05bpK0kDA3K2lhEY";"22/Mar/2011";"6272"
"pachCsiog05bpK0kDA3K2lhEY";"23/Mar/2011";"18816"
"pachCsiog05bpK0kDA3K2lhEY";"28/Mar/2011";"501760"