Reading a large structured binary file in Julia - julia

I have a large binary file containing identical records with this memory layout:
# Julia code
struct Event
ia::Int32
ig::Int32
Eg::Float64
Tg::Float64
xn::Float64
yn::Float64
zn::Float64
# uninitialized constructor
Event() = new()
end
How can I translate this C++ code in Julia?
// C++ code
struct Event
{
int32_t ia;
int32_t ig;
double Eg;
double Tg;
double xn;
double yn;
double zn;
};
// ... compute event_count
std::ifstream in(filename,std::ifstream::binary);
std::vector<Event> array(event_count);
in.read((char*)array.data(), event_count*sizeof(Event)); // <- Julia way: how to?

you can use read(filename, Event, n), where n is number of elements you want to read (size of target vector). Actually n can be e.g. a tuple giving dimensions of the output array.
You can check out help of read function for other options.

Related

two dimensional array and pointer arithmetic

I am trying to copy a 2 dimensional array to another 2 dimensional array. Since the name (srcAry) is the address of the first element of the source array, I have been able to print out all the values in the source array using pointer arithmetic in a for loop. I am using the number of rows times the number of columns as the condition to stop looping. If I try to assign the values to the new array using this method I get an error message (error: assignment to expression with array type). Is this possible to do this or am I limited to using two nested for loops with indexes?
...
void copyAry(double *pAry, int numRows, int numCols)
{
double newAry[numRows][numCols];
int end = numRows * numCols;
int ctr = 0;
for( ; ctr < end; ctr++)
// printf("*(pAry + %d) = %.1f\n", ctr, *(pAry + ctr)); //this works fine
{
*(newAry + ctr) = *(pAry + ctr); //this is where I receive error
}
return;
}
...
Thanks in advance.
I would assume that the type of newAry + ctr is not double* as your code assumes, but rather double*[numCols] i.e. a pointer to an array of numCols elements. Which also means that you would advance not one element at a time, but numCols.
Usually you would use memcpy for this kind of low level data copying. Barring that, you might start with double* pNewAry = &newAry[0][0] or some such in order to test the 2d array as a linear sequence of doubles.

Will an array of pointers be equal to an array of chars?

I have got this code:
import std.stdio;
import std.string;
void main()
{
char [] str = "aaa".dup;
char [] *str_ptr;
writeln(str_ptr);
str_ptr = &str;
*(str_ptr[0].ptr) = 'f';
writeln(*str_ptr);
writeln(str_ptr[0][1]);
}
I thought that I am creating an array of pointers char [] *str_ptr so every single pointer will point to a single char. But it looks like str_ptr points to the start of the string str. I have to make a decision because if I am trying to give access to (for example) writeln(str_ptr[1]); I am getting a lot of information on console output. That means that I am linking to an element outside the boundary.
Could anybody explain if it's an array of pointers and if yes, how an array of pointers works in this case?
What you're trying to achieve is far more easily done: just index the char array itself. No need to go through explicit pointers.
import std.stdio;
import std.string;
void main()
{
char [] str = "aaa".dup;
str[0] = 'f';
writeln(str[0]); // str[x] points to individual char
writeln(str); // faa
}
An array in D already is a pointer on the inside - it consists of a pointer to its elements, and indexing it gets you to those individual elements. str[1] leads to the second char (remember, it starts at zero), exactly the same as *(str.ptr + 1). Indeed, the compiler generates that very code (though plus range bounds checking in D by default, so it aborts instead of giving you gibberish). The only note is that the array must access sequential elements in memory. This is T[] in D.
An array of pointers might be used if they all the pointers go to various places, that are not necessarily in sequence. Maybe you want the first pointer to go to the last element, and the second pointer to to the first element. Or perhaps they are all allocated elements, like pointers to objects. The correct syntax for this in D is T*[] - read from right to left, "an array of pointers to T".
A pointer to an array is pretty rare in D, it is T[]*, but you might use it when you need to update the length of some other array held by another function. For example
int[] arr;
int[]* ptr = &arr;
(*ptr) ~= 1;
assert(arr.length == 1);
If ptr wasn't a pointer, the arr length would not be updated:
int[] arr;
int[] ptr = arr;
ptr ~= 1;
assert(arr.length == 1); // NOPE! fails, arr is still empty
But pointers to arrays are about modifying the length of the array, or maybe pointing it to something entirely new and updating the original. It isn't necessary to share individual elements inside it.

declaring and defining pointer vetors of vectors in OpenCL Kernel

I have a variable which is vector of vector, And in c++, I am easily able to define and declare it but in OpenCL Kernel, I am facing the issues. Here is an example of what I am trying to do.
std::vector<vector <double>> filter;
for (int m= 0;m<3;m++)
{
const auto& w = filters[m];
-------sum operation using w
}
Now Here, I can easily referencing the values of filters[m] in w, but I am not able to do this OpenCl kernel file. Here is what I have tried,but it is giving me wrong output.
In host code:-
filter_dev = cl::Buffer(context,CL_MEM_READ_ONLY|CL_MEM_USE_HOST_PTR,filter_size,(void*)&filters,&err);
filter_dev_buff = cl::Buffer(context,CL_MEM_READ_WRITE,filter_size,NULL,&err);
kernel.setArg(0, filter_dev);
kernel.setArg(1, filter_dev_buff);
In kernel code:
__kernel void forward_shrink(__global double* filters,__global double* weight)
{
int i = get_global_id[0]; // I have tried to use indiviadual values of i in filters j, just to check the output, but is not giving the same values as in serial c++ implementation
weight = &filters[i];
------ sum operations using weight
}
Can anyone help me? Where I am wrong or what can be the solution?
You are doing multiple things wrong with your vectors.
First of all (void*)&filters doesn't do what you want it to do. &filters doesn't return a pointer to the beginning of the actual data. For that you'll have to use filters.data().
Second you can't use an array of arrays in OpenCL (or vector of vectors even less). You'll have to flatten the array yourself to a 1D array before you pass it to a OpenCL kernel.

Adding one to a vector

I am trying to translate c code into MATLAB, and I have come across some code that I don't understand. Specifically, there is a variable defined as:
static float *lpfdata;
This gets assigned during a function call to:
envelope_old(&fdata[0], lpfdata, winlength, samprate, BW);
Which accepts input as:
void envelope_old (float *fdata, float *lpfdata, int nsamps, int samprate,
float cutoff)
Within envelope_old, lpfdata is referenced as a vector, being assigned values in a loop in the format "lpfdata[i] = ..." where i is the index variable in the loop.
Later, a function call in the format:
downsample( lpfdata+1, dwndata, winlength, downby);
is called. What does the +1 mean in this instance?
When dealing with a pointer, lpfdata[n] and lpfdata+n are the same - they both add n * sizeof(*lpfdata) to the raw pointer and access the memory at that address.
In this case, lpfdata points to elements of type float, so sizeof(*lpfdata) == sizeof(float)

QMap Memory Error

I am doing one project in which I define a data types like below
typedef QVector<double> QFilterDataMap1D;
typedef QMap<double, QFilterDataMap1D> QFilterDataMap2D;
Then there is one class with the name of mono_data in which i have define this variable
QFilterMap2D valid_filters;
mono_data Scan_data // Class
Now i am reading one variable from a .mat file and trying to save it in to above "valid_filters" QMap.
Qt Code: Switch view
for(int i=0;i<1;i++)
{
for(int j=0;j<1;j++)
{
Scan_Data.valid_filters[i][j]=valid_filters[i][j];
printf("\nValid_filters=%f",Scan_Data.valid_filters[i][j]);
}
}
The transferring is done successfully but then it gives run-time error
Windows has triggered a breakpoint in SpectralDataCollector.exe.
This may be due to a corruption of the heap, and indicates a bug in
SpectralDataCollector.exe or any of the DLLs it has loaded.
The output window may have more diagnostic information
Can anyone help in solving this problem. It will be of great help to me.
Thanks
Different issues here:
1. Using double as key type for a QMap
Using a QMap<double, Foo> is a very bad idea. the reason is that this is a container that let you access a Foo given a double. For instance:
map[0.45] = foo1;
map[15.74] = foo2;
This is problematic, because then, to retrieve the data contained in map[key], you have to test if key is either equal, smaller or greater than other keys in the maps. In your case, the key is a double, and testing if two doubles are equals is not a "safe" operation.
2. Using an int as key while you defined it was double
Here:
Scan_Data.valid_filters[i][j]=valid_filters[i][j];
i is an integer, and you said it should be a double.
3. Your loop only test for (i,j) = (0,0)
Are you aware that
for(int i=0;i<1;i++)
{
for(int j=0;j<1;j++)
{
Scan_Data.valid_filters[i][j]=valid_filters[i][j];
printf("\nValid_filters=%f",Scan_Data.valid_filters[i][j]);
}
}
is equivalent to:
Scan_Data.valid_filters[0][0]=valid_filters[0][0];
printf("\nValid_filters=%f",Scan_Data.valid_filters[0][0]);
?
4. Accessing a vector with operator[] is not safe
When you do:
Scan_Data.valid_filters[i][j]
You in fact do:
QFilterDataMap1D & v = Scan_Data.valid_filters[i]; // call QMap::operator[](double)
double d = v[j]; // call QVector::operator[](int)
The first one is safe, and create the entry if it doesn't exist. The second one is not safe, the jth element in you vector must already exist otherwise it would crash.
Solution
It seems you in fact want a 2D array of double (i.e., a matrix). To do this, use:
typedef QVector<double> QFilterDataMap1D;
typedef QVector<QFilterDataMap1D> QFilterDataMap2D;
Then, when you want to transfer one in another, simply use:
Scan_Data.valid_filters = valid_filters;
Or if you want to do it yourself:
Scan_Data.valid_filters.clear();
for(int i=0;i<n;i++)
{
Scan_Data.valid_filters << QFilterDataMap1D();
for(int j=0;j<m;j++)
{
Scan_Data.valid_filters[i] << valid_filters[i][j];
printf("\nValid_filters=%f",Scan_Data.valid_filters[i][j]);
}
}
If you want a 3D matrix, you would use:
typedef QVector<QFilterDataMap2D> QFilterDataMap3D;

Resources