I'm using Graphite and Grafana and I'm trying to plot a series against a time shifted version of itself for comparison.
(I.e. is the current value similar to this time last week?)
What I'd like to do is plot;
the 5 minute moving average of the series
a band consisting of the 5 minute moving average of the series timeshifted by 7 days, bounded above and below by the standard deviation of itself
That way I can see if the current moving average falls within a band limited by the standard deviation of the moving average from a week ago.
I have managed to produce a band based on the timeshifted moving average, but only by offsetting either side by a constant amount. I can't work out any way of offsetting by the standard deviation (or indeed by any dynamic value).
I've copied a screenshot of the sort of thing I'm trying to achieve. The yellow line is the current moving average, the green area is bounded by the historical moving average offset either side by the standard deviation.
Is this possible at all in Grafana using Graphite as the backend?
I'm not quite on the latest version, but can easily upgrade (and will do so shortly anyway).
Incidentally, I'm not a statistician, if what I'm doing actually makes no sense mathematically, I'd love to know! ;-) My overall goal is to explore better alternatives, instead of using static thresholds, for highlighting anomalous or problematic server performance metrics - e.g. CPU load, disk IOPS, etc.
Related
I have the following data, plotted in R, depicting the angular rotation of an object vs. time. I would like to determine the time point at which the movement begins by calculating when the y-value significantly depart from the baseline (the start of the rise phase of the curve). I would love any ideas for how I should approach this problem using R.
I want to know when a time series stabilizes after its peak.
This time series shows a peak, and then it goes down (see images below (1); (2)).
I would like to calculate the moment at which this time-series stabilizes (becomes flatter) after its peak.
In an ideal world, the data should go down to 0 and stay there. But as you see, it does not reach 0, nor stay 100% stable.
I thought of various different ways/possibilities:
-Calculate a tangent point to see the slope change. But the data has many small ups & downs even after smoothing it.
-Calculate the average at the tail (end of the series /e.g. if time = 8000; calculate last 2000 values mean average), then calculate an interval (margin +- this value), then calculate the time at which the first value appears in this interval.
-Calculate pronounced changes in the trend or slope.
*Maybe you have a better idea I did not consider. Feel free to share it if you've already dealt with this in the past.
I need to know the time at which this stabilization happened.
Ideally, you could mute/ignore all values before the peak value, but without deleting these rows (time should stay). Calculating the peak is easy (max value).
I also standardized the data so it starts at y=0 (I have various time series, I make them all start at y=0 to compare them later*).
I do not know how to provide the data because it is about 8k values.
I would really appreciate your help.
Thank you very much.
I have a time dependent heat conduction simulation and need to plot the average temperature of some area over time. However, the exported table data apparently uses only a few data points and interpolates in between.
More specifically, I have some block of material (aluminum) that is heated periodically at some surface. I am now interested in temperature peaks at exactly this surface over time. I have defined the heating function, the surface, and have calculated the average temperature of the surface under observation over time. However, when I plot the exported data
the temperature data is really, REALLY coarse. The heating data however is very fine. Comsol seems to interpolate between very few points. Calculating with a finer temporal resolution won't fix it.
How can I tell Comsol to evaluate the temperature at every step?
OK, I found the answer:
https://www.comsol.com/support/knowledgebase/1254
Turns out the timesteps the solver chooses are completely separate from the ones the user can define for the simulation. This honestly makes me question the usefulness of the initial definition of the timesteps. It really seems to be just an additional hoop for people to be dependent on support....
Solution:
Turn the maximum timestep in Solution/Time Dependent Solver to an acceptable minimal value.
I'm trying to get some information out of a couple of waveforms, which I currently have in the format of a CSV table with the columns: Time, Volume, Pressure, Flow. (The data is the flow/pressure/volume data obtained from a healthcare ventilator.)
I want to get excel / R / another-programme-that-I've-not-yet-thought-of to do the following with the waveform:
Break it up into individual breaths - starting with when the pressure starts to rise above a baseline (shortly followed by the flow rising above 0)
For each breath, extract:The highest pressure that occurs, the pressure just before the start of the next breath, the lowest pressure that occurs
Does anyone know how to do that, without doing it manually? (It'd mean trawling through a few hours-worth of ventilator data for the project I'm trying to do)
I've attached a copy of the shapes of the waves I'm dealing with, to try to help make more sense.Pressure & Volume against time
My intention is to work out the breath-to-breath variability in maximum, minimum, and baseline pressures.
Ultimately, I've come up with my own fix using excel.
For each pressure value, I've taken the mean of that value and the two either side of it. I've then compared this to the previous mean. For my data, a change of this mean of more than 1% in either direction identifies the beginning and end of the up- and down-stroke of the pressure curve, and I can identify where in the curve I am based on the absolute value for pressure (the peak should never be lower than the highest baseline for the data I've collected), and for the slopes, the first direction of change of the mean (this smooths out occasionally inconsistent slopes).
I'm then graphing the data, along with the transition points, the calculated phase of the breath, and the breath count, so I can sense-check it visually.
Ged
Let's say I have a set of 5 markers. I am trying to find the relative distances between each marker using an augmented reality framework such as ARToolkit. In my camera feed thee first 20 frames show me the first 2 markers only so I can work out the transformation between the 2 markers. The second 20 frames show me the 2nd and 3rd markers only and so on. The last 20 frames show me the 5th and 1st markers. I want to build up a 3D map of the marker positions of all 5 markers.
My question is, knowing that there will be inaccuracies with the distances due to low quality of the video feed, how do I minimise the inaccuracies given all the information I have gathered?
My naive approach would be to use the first marker as a base point, from the first 20 frames take the mean of the transformations and place the 2nd marker and so forth for the 3rd and 4th. For the 5th marker place it inbetween the 4th and 1st by placing it in the middle of the mean of the transformations between the 5th and 1st and the 4th and 5th. This approach I feel has a bias towards the first marker placement though and doesn't take into account the camera seeing more than 2 markers per frame.
Ultimately I want my system to be able to work out the map of x number of markers. In any given frame up to x markers can appear and there are non-systemic errors due to the image quality.
Any help regarding the correct approach to this problem would be greatly appreciated.
Edit:
More information regarding the problem:
Lets say the realworld map is as follows:
Lets say I get 100 readings for each of the transformations between the points as represented by the arrows in the image. The real values are written above the arrows.
The values I obtain have some error (assumed to follow a gaussian distribution about the actual value). For instance one of the readings obtained for marker 1 to 2 could be x:9.8 y:0.09. Given I have all these readings how do I estimate the map. The result should ideally be as close to the real values as possible.
My naive approach has the following problem. If the average of the transforms from 1 to 2 is slightly off the placement of 3 can be off even though the reading of 2 to 3 is very accurate. This problem is shown below:
The greens are the actual values, the blacks are the calculated values. The average transform of 1 to 2 is x:10 y:2.
You can use a least-squares method, to find the transformation that gives the best fit to all your data. If all you want is the distance between the markers, this is just the average of the distances measured.
Assuming that your marker positions are fixed (e.g., to a fixed rigid body), and you want their relative position, then you can simply record their positions and average them. If there is a potential for confusing one marker with another, you can track them from frame to frame, and use the continuity of each marker location between its two periods to confirm its identity.
If you expect your rigid body to be moving (or if the body is not rigid, and so forth), then your problem is significantly harder. Two markers at a time is not sufficient to fix the position of a rigid body (which requires three). However, note that, at each transition, you have the location of the old marker, the new marker, and the continuous marker, at almost the same time. If you already have an expected location on the body for each of your markers, this should provide a good estimate of a rigid pose every 20 frames.
In general, if your body is moving, best performance will require some kind of model for its dynamics, which should be used to track its pose over time. Given a dynamic model, you can use a Kalman filter to do the tracking; Kalman filters are well-adapted to integrating the kind of data you describe.
By including the locations of your markers as part of the Kalman state vector, you may be able to be able to deduce their relative locations from purely sensor data (which appears to be your goal), rather than requiring this information a priori. If you want to be able to handle an arbitrary number of markers efficiently, you may need to come up with some clever mutation of the usual methods; your problem seems designed to avoid solution by conventional decomposition methods such as sequential Kalman filtering.
Edit, as per the comments below:
If your markers yield a full 3D pose (instead of just a 3D position), the additional data will make it easier to maintain accurate information about the object you are tracking. However, the recommendations above still apply:
If the labeled body is fixed, use a least-squares fit of all relevant frame data.
If the labeled body is moving, model its dynamics and use a Kalman filter.
New points that come to mind:
Trying to manage a chain of relative transformations may not be the best way to approach the problem; as you note, it is prone to accumulated error. However, it is not necessarily a bad way, either, as long as you can implement the necessary math in that framework.
In particular, a least-squares fit should work perfectly well with a chain or ring of relative poses.
In any case, for either a least-squares fit or for Kalman filter tracking, a good estimate of the uncertainty of your measurements will improve performance.