I have train my LSTM network and deploy it on the Arduino but my problem is that as you know the input of the LSTM network is as follows (window length of the time series data),(features) in my case 256 window length and 6 axis IMU data which means 2D input however all of the examples that I have seen are only 1D input
here is the Arduino code that used to assign the model's input
input->data.f[0] =aX
and when trying to modify the code to fit my input dimension
(256,6) like this
input->data.f[0][0] =aX
I got this error
fall_detection:142:44: error: invalid types 'float[int]' for array subscript
tflInputTensor->data.f[samplesRead][5] = (gZ + 2000.0) / 4000.0;
It looks like you are setting the tensor input data "manually" by digging in to the internal structure of the TF Lite tensor type. I would suggest using the API here:
https://www.tensorflow.org/lite/guide/inference#load_and_run_a_model_in_c
In particular the code snippet here:
float* input = interpreter->typed_input_tensor<float>(0);
// Fill `input`.
gives you the proper pointer. Then you can just cast the pointer for your actual data structure. Something like this:
input = reinterpret_cast<float *>(aX);
I see the potential for confusion for the 2D vs. 1D issue, but it is OK as long as the input tensor is properly shaped. If it has shape <256, 6> what this means is that the 1D sequence of float values will be "a0,a1,a2,a3,a4,a5,b0,b1,..." where the "a" values are all six values of the first row, the "b" values are all six values of the next row and so on. The standard C convention for multi-dimension memory layout works this way. Unless there is something unusual about your aX object, it should be fine. If aX is not laid out in this fashion, you should copy the data so that it is in this layout.
Related
Imagine a object with a socket that can be filled by something, lets say a key. This is essentially a binary state that can be changed back and forth. Now imagine that I try to socket another key into a socket that already has a key in it. I cannot, obviously. I am trying to understand how to depict such behavior in a state space model.
I have a model that conforms to the follow basic form:
x' = Ax + Bu
with A being a state matrix, x being a state vector, B being a input matrix and u being the input vector. As I understand it, the input vector is a complex number comprised of values that represent the individual inputs, and the matrix provides the relationship between each of the input variables, resulting in the overall input vector Bu.
u = [1,0,0]
B = [1,0,0
0,0,1
0,1,1]
Bu = [1,1,2]
If we imagine in the example above that the first element in the u vector is the socketable key, I am confused about is how to sensibly make a situation whereby that key, the input variable, is sensibly controlled by the existing state of Ax. All of my attempts so far feel 'janky', relying on information not present in the existing example. Whats the best way to think about this?
I am using the HEVC reference software, HM Encoder Version [16.15] (including RExt) on a [Mac OS X][GCC 4.2.1][64 bit] and would like to extract at encoder side:
1) the motion vectors for each block
2) the frame partition information, i.e. the size and location of each block in the frame to which a motion vector refers.
Does anybody have hints on what are the variables where this info is stored for each coding unit? Thanks!
All you need is available in the TComDataCU class.
1) For motion information, there is the function getCUMvField() which returns the motion vector. It's not easy to work with it though.
Basically, to access almost any of the PU/CU level syntax elements, you need to be able to work with the absolute index of that PU/CU. This unique index tells you where exactly your PU/CU is located in the CTU by pointing to the up-left 4x4 block of that part.
I rememberthat that most of the times this index is stored in the variable uiAbsPartIdx.
If you get to know how to work with this index, then you will be able to get the block partitioning information in the CTU level. so for 2) my suggestion is that you go to the slice level when you have a loop over CUTs (I think this is done in the compressSlice() function). And after the compressCtu() function is called for each CTU (which means that all RDO decisions have been made and the CTU partitioning is decided), you put a loop over all uiAbsPartIdxs of the CTU and get their width and height. For example if your CTU size is 128, then you will have 32*32=1024 unique 4x4 blocks in your CTU. The function for getting width/height of the CU corresponding to a certain uiAbsPartIdx is pCtu->getWidth(uiAbsPartIdx).
I hope it was clear.
Or the question can be paraphrased like this:
Why may one need a datatype with a non-zero lower bound?
Consider the following example:
struct S {
int a;
int b;
float c;
float d;
} array[N];
If I had an array of type S[] and I wanted to send only values of fields b and
d, I would create a datatype with the type map { (4, MPI_INT), (12, MPI_FLOAT) }.
At first, it seems that such a type could be used to correctly send an array of
struct S:
MPI_Send(array, N, datatype, ...);
But this doesn't work if N > 1.
Such a type would have lb = 4, ub = 16 and extent = ub - lb = 12. That is,
MPI would consider that the second element of the array starts 12 bytes from the
first one, which is not true.
Well, that may be not a big deal. After all, generally, for such partially sent structures
we have to specify the exact size of the structure:
MPI_Type_create_resized(datatype, 0, sizeof(struct S), &resized);
But I wonder why we always need to specify a zero lower bound. Why would
someone need a non-zero lower bound? The datatypes with non-zero lower bounds looks extremely confusing to me, and I cannot make any sense of them.
If I were to design a type system for MPI, I would describe a type with a single
parameter - its size (extent), which is the stride between two adjacent elements of an array. In terms of MPI, I would always set lb = 0 and extent = ub. Such a system looks much clearer to me, and it would work correctly in the example described above.
But MPI has chosen the different way. We have two independent parameters instead: the lower
and the upper bounds. Why is it so? What's the use of this additional flexibility? When should one use datatypes with a non-zero lower bound?
You have no idea what kind of weird and complex structures one finds in scientific and engineering codes. The standard is designed to be as general as possible and to provide maximum flexibility. Section 4.1.6 Lower-Bound and Upper-Bound Markers begins like this:
It is often convenient to define explicitly the lower bound and upper bound of a type map, and override the definition given on page 105. This allows one to define a datatype that has "holes" at its beginning or its end, or a datatype with entries that extend above the upper bound or below the lower bound. Examples of such usage are provided in Section 4.1.14.
Also, the user may want to overide [sic] the alignment rules that are used to compute upper bounds and extents. E.g., a C compiler may allow the user to overide [sic] default alignment rules for some of the structures within a program. The user has to specify explicitly the bounds of the datatypes that match these structures.
The simplest example of a datatype with non-zero lower bound is a structure with absolute addresses used as offsets, useful in, e.g., sending structures with pointers to data scattered in memory. Such a datatype is used with MPI_BOTTOM specified as the buffer address, which corresponds to the bottom of the memory space (0 on most systems). If the lower bound would be fixed to 0, then you have to find the data item with the lowest address fist and compute all offsets relative to it.
Another example is the use of MPI_Type_create_subarray to create a datatype that describes a subarray of an n-dimensional array. With zero lower bounds you will have to provide a pointer to the beginning of the subarray. With non-zero lower bounds you just give a pointer to the beginning of the whole array instead. And you can also create a contiguous datatype of such subarray datatypes in order to send such n-dimensional "slices" from an n+1-dimensional array.
IMO the only reason to have both LB and UB markers is it simplifies the description of datatype construction. The MPI datatypes are described by a type map (list of offsets and types, including possible LB/UB markers) and all the datatype construction calls define the new typemap in terms of the old typemap.
When you have LB/UB markers in the old typemap and you follow the rules of construction of the new typemap from the old, you get a natural definition of the LB/UB marker in the new type which defines the extent of the new type. If extent were a separate property on the side of the typemap, you'd have to define what the new extent is for every datatype construction call.
Other than that I fundamentally I agree with you on the meaninglessness of having LB/UB as two separate pieces of data, when the only thing they're used for is to define the extent. Once you add LB/UB markers, their meaning is completely disconnected from any notion of actual data offsets.
If you wanted to put an int at displacement 4 and have its extent be 8, it would be fine to construct
[(LB,4), (int,4), (UB,12)]
but it would be equally fine to construct any of
[(LB,0),(int,4),(UB,8)]
[(LB,1000000),(int,4),(UB,1000008)]
[(LB,-1000),(int,4),(UB,-992)]
The above are all completely equivalent in behavior because they have the same extent.
When explanations of LB/UB markers talk about how you need to have datatypes where the first data displacement is non-0, I think that's misleading. It's true you need to be able to make types like that, but the LB/UB markers aren't fundamentally connected to the data displacements. I'm concerned that suggesting they are connected will lead an MPI user to write invalid code if they think the LB is intrinsically related to the data offsets.
I am trying to work with nolearn and use the ConcatLayer to combine multiple inputs. It works great as long as every input has the same type and shape. I have three different types of inputs that will eventually produce a single scalar output value.
The first input is an image of dimensions (288,1001)
The second input is a vector of length 87
The third is a single scalar value
I am using Conv2DLayer(s) on the first input.
The second input utilizes Conv1DLayer or DenseLayer (not sure which would be better since I can't get it far enough to see what happens)
I'm not even sure how the third input should be set up since it is only a single value I want to feed into the network.
The code blows up at the ConcatLayer with:
'Mismatch: input shapes must be the same except in the concatenation axis'
It would be forever grateful if someone could write out a super simple network structure that can take these types of inputs and output a single scalar value. I have been googling all day and simply cannot figure this one out.
The fit function looks like this if it is helpful to know, as you can see I am inputting a dictionary with an item for each type of input:
X = {'base_input': X_base, 'header_input': X_headers, 'time_input':X_time}
net.fit(X, y)
It is hard to properly answer the question, because - it depends.
Without having information on what you are trying to do and what data you are working on, we are playing the guessing game here and thus I have to fall back to giving general tips.
First it is totally reasonable, that ConcatLayer complains. It just does not make a lot of sense to append a scalar to the Pixel values of an Image. So you should think about what you actually want. This is most likely combining the information of the three sources.
You are right by suggesting to process the Image with 2D convolutions and the sequence data with 1D convolutions. If you want to generate a scalar value, you propably want to use dense layers later on, to condense the information.
So it would be naturally, to leave the lowlevel processing of the three branches independent and then concatenate them later on.
Something along the lines of:
Image -> conv -> ... -> conv -> dense -> ... -> dense -> imValues
Timeseries -> conv -> ... -> conv -> dense ... -> dense -> seriesValues
concatLayer([imValues, seriesValues, Scalar] -> dense -> ... -> dense with num_units=1
Another less often reasonable Option would be, to add the Information at the lowlevel processing of the Image. This might make sense, if the local processing is much easier, given the knowledge of the scalar/timeseries.
This architecture might look like:
concatLayer(seriesValues, scalar) -> dense -> ... -> reshape((-1, N, 1, 1))
-> Upscale2DLayer(Image.shape[2:3]) -> globalInformation
concatLayer([globalInformation, Image]) -> 2D conv filtersize=1 -> conv -> ... -> conv
Note that you will almost certainly want to go with the first Option.
One unrelated Thing I noticed, is the huge size of your Input Image. You should reduce it(resizing/patches). Unless you have a gigantic load of data and tons of Memory and computing power, you will otherwise either overfit or waste Hardware.
I am currently learning Modelica by trying some very simple examples. I have defined a connector Incompressible for an incompressible fluid like this:
connector Incompressible
flow Modelica.SIunits.VolumeFlowRate V_dot;
Modelica.SIunits.SpecificEnthalpy h;
Modelica.SIunits.Pressure p;
end Incompressible;
I now wish to define a mass or volume flow source:
model Source_incompressible
parameter Modelica.SIunits.VolumeFlowRate V_dot;
parameter Modelica.SIunits.Temperature T;
parameter Modelica.SIunits.Pressure p;
Incompressible outlet;
equation
outlet.V_dot = V_dot;
outlet.h = enthalpyWaterIncompressible(T); // quick'n'dirty enthalpy function
outlet.p = p;
end Source_incompressible;
However, when checking Source_incompressible, I get this:
The problem is structurally singular for the element type Real.
The number of scalar Real unknown elements are 3.
The number of scalar Real equation elements are 4.
I am at a loss here. Clearly, there are three equations in the model - where does the fourth equation come from?
Thanks a lot for any insight.
Dominic,
There are a couple of issues going on here. As Martin points out, the connector is unbalanced (you don't have matching "through" and "across" pairs in that connector). For fluid systems, this is acceptable. However, intensive fluid properties (e.g., enthalpy) have to be marked as so-called "stream" variables.
This topic is, admittedly, pretty complicated. I'm planning on adding an advanced chapter to my online Modelica book on this topic but I haven't had the time yet. In the meantime, I would suggest you have a look at the Modelica.Fluid library and/or this presentation by one of its authors, Francesco Casella.
That connector is not a physical connector. You need one flow variable for each potential variable. This is the OpenModelica error message if it helps a little:
Warning: Connector .Incompressible is not balanced: The number of potential variables (2) is not equal to the number of flow variables (1).
Error: Too many equations, over-determined system. The model has 4 equation(s) and 3 variable(s).
Error: Internal error Found Equation without time dependent variables outlet.V_dot = V_dot
This is because the unconnected connector will generate one equation for the flow:
outlet.V_dot = 0.0;
This means outlet.V_dot is replaced in:
outlet.V_dot = V_dot;
And you get:
0.0 = V_dot;
But V_dot is a parameter and can not be assigned to in an equation section (needs an initial equation if the parameter has fixed=false, or a binding equation in the default case).