I have a time series that seems to be well described by a univariate local level model (a changing bias in human visual perception, sampled at regular intervals). I have a hunch, however, that the underlying random walk is at least partly driven by the measurements themselves. In order to test this hypothesis, I measure two types of time series. The first type is measured at regular steps, every 12 minutes:
u(0), u(12), u(24), u(36), u(48), u(60), u(72), ...
The second type is measured at partly irregular intervals, alternating between every 6 or 18 minutes:
v(0), v(6), v(24), v(30), v(48), v(54), v(72), ...
Of course I could compare the 6- and 18-minute steps in the v-series: if they're no different, then nothing really happens to the random walk between the measurements. The trouble is that the measurement noise is large compared to the random walk steps.
Is there some more principled way I could test this hypothesis? For instance, could I fit a modified local-level model in which the even and odd random-walk steps have different variances, and compare the two variances?
Related
I would like to create a forecasting model with time series in R. I have a target time series 'Sales' that I would like to forecast. I also have several time series that represent, for example, GDP or advertising spend. Unfortunately I have a lot of independent time series and I don't know how to figure out the most significant ones. It would be best to find out the most important ones already before building the model.
I have already worked with classification problems, here I have always used the Pearson correlation value. This is not possible with time series, right? How can I determine the correlation for time series and use the correlation to find suitable time series that describe my target time series?
I tried to use the corr.test() function in R, but I think thats not right.
Dataset Description: I use a dataset with neuropsychological (np) tests from several subjects. Every subject has more than one tests in his/her follow up i.e one test per year. I study the cognitive decline in these subjects. The information that I have are: Individual number(identity number), Education(years), Gender(M/F as factor), Age(years), Time from Baseline (= years after the first np test).
AIM: My aim is to measure the rate of change in their np tests i.e the cognitive decline per year for each of them. To do that I use Linear Mixture Effects Models (LMEM), taking into account the above parameters and I compute the slope for each subject.
Question: When I run the possible models (combining different parameters every time), I also check their singularity and the result in almost all cases is TRUE. So my models present singularity! In the case that I would like to use these models to do predictions this is not good as it means that the model overfits the data. But now that I just want to find the slope for each individual I think that this is not a problem, or even better I think that this is an advantage, as in that case singularity offers a more precise calculation for the subjects' slopes. Do you think that this thought is correct?
I would like to detect patterns within a weather dataset of around 10'000 data points. I have around 40 possible predictors (temperature, humidity etc.) which may explain good or bad weather the next day (dependent variable). Normally, I would apply classical machine learning methods like Random Forest to build and test models for classifying the whole dataset and find reliable predictors to forecast the next day's weather.
My task though is different. I want to find predictors and their parameters which "guarantee" me good or bad weather in a subset of the overall data. I am not interested in describing the whole dataset but finding the pattern of predictors (and their parameters) that give me good or bad weather indications. So I am trying to find, for example, 100 datapoints with 100% good weather if certain predictors are set to certain levels. I am not interested in the other 9'900 datapoints.
It is kind of the task of trying all combinations and calibrations of the predictors to find a subset of the overall data points which can be predicted with very high accuracy.
How would you do this systematically?
I want to improve step-by-step, whilst unevenly-sampled data are coming, the value of the first derivative at t = 0 s. For example, if you want to find the initial velocity in a projectile's motion, but you do not know its final position and velocity, however, you are receiving (slowly) the measurements of the projectile's current position and time.
Update - 26 Aug 2018
I would like to give you more details:
"Unevenly-sampled data" means the time intervals are not regular (irregular times between successive measurements). However, data have almost the same sampling frequency, i.e., it is about 15 min. Thus, there are some measurements without changes, because of the nature of the phenomenon (heat transfer). It gives an exponential tendency and I can fit data to a known model, but an important amount of information is required. For practical purposes, I only need to know the value of the very first slope for the whole process.
I tried a progresive Weighted Least Squares (WLS) fitting procedure, with a weight matrix such as
W = diag((0.5).^(1:kk)); % where kk is the last measurement id
But it was using preprocessed data (i.e., jitter-removing, smoothing, and fitting using the theoretical functional). I gave me the following result:
This is a real example of the problem and its "current solution"
It is good for me, but I would like to know if there is an optimal manner of doing that, but employing the raw data (or smoothed data).
IMO, additional data is not relevant to improve the estimate at zero. Because perturbations come into play and the correlation between the first and last samples goes decreasing.
Also, the asymptotic behavior of the phenomenon is probably not known rigorously (is it truly a first order linear model) ? And this can introduce a bias in the measurements.
I would stick to the first points (say up to t=20) and fit a simple model, say quadratic.
If in fact what you are trying to do is to fit a first order linear model to the data, then least-squares fitting on the raw data is fine. If there are significant outliers, robust fitting is preferable.
I have a time-series regression problem with a single predictor and a real-valued output, and I would like to use LSTM recurrent neural network to model the data. How should I choose the number of time steps in my model? Is there any upper limit for the length of LSTM layer?
LSTMs can be challenging to use when you have very long input sequences and only one or a handful of outputs. A reasonable limit of 250-500 time steps is often used in practice with large LSTM models.
How should I choose the number of time steps in my model?
It entirely depends on the task at hand, in short the time series frequency determines this if the data you have is at the following interval minutes then 60 , hours then 24 ,month then 12 and so on.
Simply stating At what level you need your predictions
Is there any upper limit for the length of LSTM layer?
It depends on the quantum of data .
A reasonable limit of 250-500 time steps in good to go. Long input sequences may result in vanishing gradients, and in turn, an unlearnable model