My team has been given HDF5 files to read. They contain structured data with unsigned variables. I and my team were overjoyed to find the NetCDF library, which allows pure-Java reading of HDF5 files, albeit using the NetCDF data model.
No problem---we thought we'd just translate from the NetCDF data model to whatever model we wanted. As long as we get the data out. Then we tried to read an unsigned 32-bit integer from the HDF5 file. We can load up HDFView 2.9 and see that the variable is an unsigned 32-bit integer. But... it turns out that NetCDF-3 doesn't support unsigned values!
To add insult to injury, NetCDF-3 recommends that you "widen the data type" or use an _Unsigned = "true" attribute (I am not making this up) to indicate that the 32 bits should be treated as an unsigned value.
Well, maybe those kludges would be effective if I were creating NetCDF data from scratch, but how can I detect using NetCDF that a 32-bit value in an existing HDF5 file should be interpreted as unsigned?
Update: Apparently NetCDF-4 does support unsigned data types. So this begs the question: How can I determine whether a value is signed or unsigned from the NetCDF Java library?" I don't see any unsigned types in ucar.ma2.DataType.
Yes, you can look for _Unsigned = "true" attribute, or you can call Variable.isUnsigned().
Because Java doesnt support unsigned types, it was a difficult design decision. Ultimately we decided not to automatically widen the type, for efficiency. So the application must check and do the right thing. Look at ucar.nc2.DataType.unsignedXXX() helper methods.
When you read the data, you get an Array object. you can call Array.isUnsigned(). Also the extractors like Array.getDouble() will convert correctly.
The netCDF-Java library supports an extended data model called the "Common data Model" to abstract out differences in file formats. So we are not stuck with the limits of the netCDF-3 file format or data model. But we are in Java
John
Given the fact that Java doesnt have unsigned types, I think the only options are to 1) automatically widen unsigned data (turn bytes into shorts, shorts into ins, ints into longs), or 2) represent both signed and unsigned integers with the available Java data types, and let the user decide if/when it should be widened.
Arguably the main use for unsigned data is to represent bits, and in that case conversion would be a waste, since you will just mask and test the bits.
The other main use is for eg satellite data which often uses unsigned bytes, and there again I think not automatically widening is the right choice. What you end up doing is just widening right at the point you use it.
It seems that when the CDM data types are mapped to Java, NetCDF will automatically add the attribute _Unsigned = "true" to the variable. So I assume that if I check for that attribute, it will indicate if the value is unsigned or not. This may be exactly what I was looking for; I'll verify tomorrow that it works.
Update: I tried this and it works; moreover, as John Caron indicated in the accepted answer, a NetCDF array has an isUnsigned() method which checks for the _Unsigned attribute.
Related
I am trying to find a straightforward way to store negative values in EEPROM, integer values ranging from -20 to 20. I have been using EEPROM.write and EEPROM.read functions to store strings one character at a time, but I am having trouble with negative numbers. I figure I only need one byte for this value.
It's just matter of number representation. You just have to use correct data types to print or use:
Version 1: int8_t data = EEPROM.read(addr);
Version 2:
byte data = EEPROM.read(addr);
Serial.print((int8_t)data);
EEPROM.write can be used directly with int8_t: EEPROM.write(int8_value);
Or, if you wan't int, put/get methods can be used for it (even for structs containing POD types only or so)
I am parallelizing a certain dynamic programming problem using AVX2/SSE instructions.
In the main iteration of my calculation, I calculate column in matrix where each cell is a structure of AVX2 registers (_m256i). I use values from the previous matrix column as input values for calculating the current column. Columns can be big, so what I do is I have an array of structures (on stack), where each structure has two _m256i elements.
Structure:
struct Cell {
_m256i first;
_m256i second;
};
An then I have array like this: Cell prevColumn [N]. N will tipically be few hundreds.
I know that _m256i basically represents an avx2 register, so I am wondering how should I think about this array, how does it behave, since N is much larger than 16 (which is number of avx registers)? Is it a good practice to create such an array, or is there some better approach that i should use when storing a lot of _m256i values that are going to be reused real soon?
Also, is there any aligning I should be doing with this structures? I read a lot about aligning, but I am still not sure how and when to do it exactly.
It's better to structure your code to do everything it can with a value before moving on. Small buffers that fit in L1 cache aren't going to be too bad for performance, but don't do that unless you need to.
I think it's more typical to write your code with buffers of int [] type, rather than __m256i type, but I'm not sure. Either way works, and should get the compile to generate efficient code. But the int [] way means less code has to be different for the SSE, AVX2, and AVX512 version. And it might make it easier to examine things with a debugger, to have your data in an array with a type that will get the data formatted nicely.
As I understand it, the load/store intrinsics are partly there as a cast between _m256i and int [], since AVX doesn't fault on unaligned, just slows down on cacheline boundaries. Assigning to / from an array of _m256i should work fine, and generate load/store instructions where needed, otherwise generate vector instructions with memory source operands. (for more compact code and fewer fused-domain uops.)
I am doing some image processing on an embedded system (BeagleBone Black) using OpenCV and need to write some code to take advantage of NEON optimization. Specifically, I would like to write a NEON optimized thresholding function and then a NEON optimized erosion/dilation function.
This is my first time writing NEON code and I don't have experience writing assmbly code, so I have been looking at examples and resources for the C-style NEON intrinsics. I believe that I can put some working code together, but am not sure how I should structure the vectors. According to page 2 of the "ARM NEON support in the ARM compiler" white paper:
"These registers can hold "vectors" of items which are 8, 16, 32 or 64
bits. The traditional advice when optimizing or porting algorithms
written in C/C++ is to use the natural type of the machine for data
handling (in the case of ARM 32 bits). The unwanted bits can then be
discarded by casting and/or shifting before storing to memory."
What exactly does this mean? Do I need to to restrict my NEON code to using uint32x4_t vectors rather than uint8x16_t? How would I go about loading the registers? Or does this mean than I need to take some special steps when using vst1q_u8 to store the data to memory?
I did find this example, which is untested but uses the uint8x16_t type. Does it adhere to the "32-bit" advice given above?
I would really appreciate it if someone could please elaborate on the above quotation and maybe provide a very simple working example.
The next sentence from the document you linked gives your answer.
The ability of NEON to specify the data width in the instruction and
hence use the whole register width for useful information means
keeping the natural type for the algorithm is both possible and
preferable.
Note, the document is distinguishing between the natural type of the machine (32-bit) and the natural type of the algorithm (in your case uint8_t).
The document is saying that in the past you would have written your code in such a way that it used 32-bit integers so that it could use the efficient machine instructions suited for 32-bit operations.
With Neon, this is not necessary. It is more useful to use the data type you actually want to use, as Neon can efficiently operate on those data types.
It will depend on your algorithm as to the optimal choice of register width (uint8x8_t or uint8x16_t).
To give a simple example of using the Neon intrinsics to add two sets of uint8_t:
#include <arm_neon.h>
void
foo (uint8_t a, uint8_t *b, uint8_t *c)
{
uint8x16_t t1 = vld1q_u8 (a);
uint8x16_t t2 = vld1q_u8 (b);
uint8x16_t t3 = vaddq_u8 (a, b);
vst1q_u8 (c, t3);
}
Recently I started using Armadillo C++ library. Given my C++ coding skills are not that great, I found this very friendly for linear algebra. I am also using that along with my matlab to speed things up for many of reconstruction algorithm.
I do need to create a vector of boolean and I would prefer using this library rather than . However, I could not figure out how to do it. I tried using uvec; but, documentation seems to indicate that it can not be used with boolean.
Any help would be appreciated.
Regards,
Dushyant
Consider using a matrix uchar_mat which is a typdef for Mat<unsigned char>, it should consume the same amount of memory as a matrix of boolean values.
The Armadillo documentation of version 7.8 states that a matrix Mat<type>, can be of the following types:
float, double, std::complex<float>, std::complex<double>, short, int, long, and unsigned versions of short, int, and long. The code on GitHub however contains typedef Mat <unsigned char> uchar_mat; in the file include/armadillo_bits/typedef_mat.hpp so you should also be able to use uchar_mat.
You will not save any memory by creating a matrix of bool values compared to a matrix of unsigned char values (a bool type consumes 8 bits). This is because in C++ every data type must be addressable; it must be at least 1 byte long so that it is possible to create a pointer pointing to it.
As the Title says, i am trying out this last year's problem that wants me to write a program that works the same as scanf().
Ubuntu:
Here is my code:
#include<unistd.h>
#include<stdio.h>
int main()
{
int fd=0;
char buf[20];
read(0,buf,20);
printf("%s",buf);
}
Now my program does not work exactly the same.
How do i do that both the integer and character values can be stored since my given code just takes the character strings.
Also how do i make my input to take in any number of data, (only 20 characters in this case).
Doing this job thoroughly is a non-trivial exercise.
What you show does not emulate sscanf("%s", buffer); very well. There are at least two problems:
You limit the input to 20 characters.
You do not stop reading at the first white space character, leaving it and other characters behind to be read next time.
Note that the system calls cannot provide an 'unget' functionality; that has to be provided by the FILE * type. With file streams, you are guaranteed one character of pushback. I recently did some empirical research on the limits, finding values that the number of pushed back characters ranged from 1 (AIX, HP-UX) to 4 (Solaris) to 'big', meaning up to 4 KiB, possibly more, on Linux and MacOS X (BSD). Fortunately, scanf() only requires one character of pushback. (Well, that's the usual claim; I'm not sure whether that's feasible when distinguishing between "1.23e+f" and "1.23e+1"; the first needs three characters of lookahead, it seems to me, before it can tell that the e+f is not part of the number.)
If you are writing a plug-in replacement for scanf(), you are going to need to use the <stdarg.h> mechanism. Fortunately, all the arguments to scanf() after the format string are data pointers, and all data pointers are the same size. This simplifies some aspects of the code. However, you will be parsing the scan format string (a non-trivial exercise in its own right; see the recent discussion of print format string parsing) and then arranging to make the appropriate conversions and assignments.
Unless you have unusually stringent conditions imposed upon you, assume that you will use the character-level Standard I/O library functions such as getchar(), getc() and ungetc(). If you can't even use them, then write your own variants of them. Be aware that full integration with the rest of the I/O functions is tricky - things like fseek() complicate matters, and ensuring that pushed-back characters are properly consumed is also not entirely trivial.