I have a question about MPI. I want to have processes read a binary file simultaneously and therefore I'm trying to use MPI_File_iread, but it is not working as I'd expect and I don't know what's wrong.
Sorry if the format of the question is not correct, I am new here.
Here is my code (NINTCI is an integer pointer, file_name is a char array that has the file name already):
#include "mpi.h"
[...]
MPI_Request request;
MPI_Status status;
MPI_File fp;
MPI_File_open(MPI_COMM_WORLD, file_name, MPI_MODE_RDONLY, MPI_INFO_NULL, &fp);
MPI_File_iread(fp, NINTCI, 1, MPI_INT, &request);
MPI_Wait(&request, &status);
printf("%d\n", *NINTCI);
MPI_Barrier(MPI_COMM_WORLD);
The correct number printed should be 0 for all processes, but instead I get random numbers like the following (8 processes):
-1475867408
1495223536
-219489040
-840278800
629550320
1309351152
-321049360
21273840
Funny thing is if I replace the MPI_File_iread with MPI_File_read and remove the MPI_Wait the result is correct.
Does anybody have an idea what I'm doing wrong? Thanks in advance!
You do not check any of your error codes. Does the file exist? Does it have proper permissions? All these answers and more can be yours if you check errors.
See How to use and interpret MPI-IO Error codes? for an example of how to get meaningful information out of your MPI implementation.
Related
I am working on a big (~500Mb) RAW txt file.
There are about 20,000,000 lines in the file.
Each line includes one double and one int. For example:
45782.1234852 10
Below is my simple code:
QTextStream rdStream(&qFile_Input);
while (!rdStream.atEnd())
{
//QStringList qList_data=rdStream.readLine().split(" ",QString::SkipEmptyParts);
rdStream.readLine();
}
It takes about 30 seconds just to read line QTextStream::readLine();
If I add .split(" ",QString::SkipEmptyParts) into a Qstringlist, then the total time required jumps to 5 minutes. My question is three fold:
Where does the time gap comes from?
Is there a way to get a shorter processing time?
If my file is larger than the RAM of PC, will I encounter an
error? If so, what can I do?
Thanks in advance!
Well, it seems that the splitting part adds an enormous overhead time-wise. Instead of using the Qt class QTextStream, you could probably just use the methods from the c++ standard library. You should get better performance than the 5 minutes you are seeing now.
#include <fstream>
int main()
{
std::ifstream infile("thefile.txt");
double a;
int b;
while(infile >> a >> b)
{
//Do something with a and b here, they've been read
}
return 0;
}
I am trying to use a single 4KB string in my arduino sketch but this always seems to give a whole bunch of java errors in the console and never compiles. I believe, I am using it correctly:
const char sequence[] PROGMEM = {"0F0FF0 ... 0F0F0FF"};
By trial-and-error I determined that the maximum length I can get to compile successfully is 1104 characters. This doesn't seem to make much sense. Is there some unknown limitation in the compiler or is it an issue with the IDE? I'm using 1.0.5 but I get the same results in 1.6.5 as well. I'd really rather not split the array. Reading online, the size limit should be 32KB, which is far higher, than what I need.
Any help or explanation appreciated, please and thank you.
It's a limitation of the IDE, not the compiler. If you make it a single string still, but use C's string concatenation, it will compile. eg.
const char sequence[] PROGMEM = {
"0F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF0"
"0F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF0"
...
"0F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF00F0F0FF0"
};
I am confused about the declaration of (for example) pid_t. What does __pid_t mean? Is it another type defined elsewhere? If yes, where? Why is my types.h in ubuntu 13.04 64bit defining pid_t like:
#ifndef __pid_t_defined
typedef __pid_t pid_t;
#define __pid_t_defined
#endif
and not something like
typedef int pid_t;
I saw some websites that have types.h headers with the declaration done the last way. This is one:
http://www.sde.cs.titech.ac.jp/~gondow/dwarf2-xml/HTML-rxref/app/gcc-3.3.2/lib/gcc-lib/sparc-sun-solaris2.8/3.3.2/include/sys/types.h.html
UPDATE:
Ok I found out that a pid_t is an __pid_t which is an __PID_T_TYPE which is which is an __S32_TYPE which is an int. My question now is why is this? POSIX only states that pid_t has to be a signed integer, so why make the declaration enter soo deep in header files?
If you are pulling up your types.h via 'man types' then at the top of the header file(under description in the man page) there should exist an include file that defines '__pid_t' at some point as a signed integer(if Ubuntu claims that their types are POSIX compliant; otherwise pid_t could be anything). The symbol ' __' is considered reserved(C standard, dunno about C++). If I had to take a wild guess as to why pid_t is defined as __pid_t and not some int is because __pid_t is what Debian's or the Linux Kernel's developers decided to use for the process ID variable name in all of their library functions; therefore only '__pid_t' needs to be changed to change the integer size for a process ID.
You should really look around before asking a question, similar stackoverflow questions are easily found: Size of pid_t, uid_t, gid_t on Linux .
Ok I found what the __pid_t means on the answer to this question:
Why PID of a process is represented by opaque data type?
QUOTE
typedef __pid_t pid_t;
...
# define __STD_TYPE typedef
__STD_TYPE __PID_T_TYPE __pid_t; /* Type of process identifications. */
...
#define __PID_T_TYPE __S32_TYPE
...
#define __S32_TYPE int
UNQUOTE
So pid_t is an __pid_t which is an __PID_T_TYPE which is which is an __S32_TYPE which is an int.
Here's my code for a program that does a histogram on an image then a prefix sum scan on the histogram http://codepad.org/4RMtWn1e. The problem is at line 396, which correctly outputs the value of numBins (number of bins for histogram) at 1024. I than copy the histogram from device memory back to host memory. This somehow changes the value of numBins and the next output of that is 0. That seems very strange. I didn't make any specific modifications to numBins, but it changes the value anyway. I feel like if I can figure this out, It will me understand why my program isn't working.
This is wrong:
unsigned int* h_histogram;
h_histogram = (unsigned int*) malloc(sizeof(unsigned int)* 1024);
std::cout << numBins;
checkCudaErrors(cudaMemcpy(&h_histogram, d_histogram, sizeof(unsigned int)* numBins, cudaMemcpyDeviceToHost));
^
|
h_histogram is already a pointer
Instead do this:
checkCudaErrors(cudaMemcpy(h_histogram, d_histogram, sizeof(unsigned int)* numBins, cudaMemcpyDeviceToHost));
(the only change being to remove the ampersand in front of h_histogram
I'm not sure really why numBins got corrupted exactly, but this particular errant cudaMemcpy operation would not be copying to the destination you expect, instead it would overwrite the pointer value stored at h_histogram as well as anything that came after it. And if numBins happened to be stored after it, it would overwrite that also.
I'm using the SD.h library to write onto an SD card with the Arduino Uno. I need to write out in a file a template string with some placeholder replaced by certain values, in the way of printf behaves. I would use the fprintf function, but when I tried this:
File dataFile = SD.open("myfile.txt", FILE_WRITE);
fprintf(dataFile, "mynumber: %d\n", 100);
I got this error:
cannot convert 'File*' to '__file*' for argument '1' to 'int
fprintf(__file*, const char*, ...)'
How can I manage this?
printf() makes your executable object ~1000 bytes larger, so you may not want to use it if size is a problem.
The fprintf is not intended to use with the SD.h so I think
The simple solution that comes into my mind is using sprintf to format your text then write it to the file with the println function
File dataFile = SD.open("myfile.txt", FILE_WRITE);
char text[100];
sprintf(text,"My number: %d",yournumber);
dataFile.println(text);