How to predict when a disk runs out of space? - azure-data-explorer

Question

I collect Free disk space metrics at regular intervals and would like to predict when the disk will be full.
I thought I could use series_decompose_forecast
Here's a sample query:
let DiskSpace =
range Timestamp from ago(60d) to now() step 1d
| order by Timestamp desc
| serialize rn=row_number() + 10
| extend FreeSpace = case
(
rn % 5 == 0, rn + 5
, rn % 3 == 0, rn -4
, rn % 7 == 0, rn +3
, rn
)
| project Timestamp, FreeSpace;
DiskSpace
| make-series
FreeSpace = max(FreeSpace) default= long(null)
on Timestamp from ago(60d) to now() step 12h
| extend FreeSpace = series_fill_backward(FreeSpace)
| extend series_decompose_forecast(FreeSpace, 24)
| render timechart
And the result
The baseline seems like it could show me when it will hit zero (or some other threshold), but if I specify more Points, it excludes more points from the learning process (still unsure if it excludes them from the start or end).
I don't even care for the whole time series, just the date of running out of free space. Is this the correct approach?

Answer 1

It seems that series_fit_line() is more than enough in this scenario.
Once you got the slope and the interception you can calculate any point on the line.
range Timestamp from now() to ago(60d) step -1d
| extend rn = row_number() + 10
| extend FreeSpace = rn + case(rn % 5 == 0, 5, rn % 3 == 0, -4, rn % 7 == 0, 3, 0)
| make-series FreeSpace = max(FreeSpace) default= long(null) on Timestamp from ago(60d) to now() step 12h
| extend FreeSpace = series_fill_forward(series_fill_backward(FreeSpace))
| extend (rsquare, slope, variance, rvariance, interception, line_fit) = series_fit_line(FreeSpace)
| project slope, interception, Timestamp, FreeSpace, line_fit
| extend x_intercept = todatetime(Timestamp[0]) - 12h*(1 + interception / slope)
| project-reorder x_intercept
| render timechart with (xcolumn=Timestamp, ycolumns=FreeSpace,line_fit)
x_intercept
2022-12-06T01:56:54.0389796Z
Fiddle
P.S.
No need for serialize after order by.
No need for order by if you create the range backwards.
Null value in a time-series breaks a lot of functionality (fixed with additional series_fill_forward)

Answer 2

If you look at the example: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/series-decompose-forecastfunction
You will see that they add 0 slots into the "future" of the original series which the forecast then predicts.
This is also stated in the notes:
The dynamic array of the original input series should include a number of points slots to be forecasted. The forecast is typically done by using make-series and specifying the end time in the range that includes the timeframe to forecast.
To make your example work:
let DiskSpace =
range Timestamp from ago(60d) to now() step 1d
| order by Timestamp desc
| serialize rn=row_number() + 10
| extend FreeSpace = case
(
rn % 5 == 0, rn + 5
, rn % 3 == 0, rn -4
, rn % 7 == 0, rn +3
, rn
)
| project Timestamp, FreeSpace;
DiskSpace
// add 4 weeks of empty slots in the "future" - these slots will be forecast
| make-series FreeSpace = max(FreeSpace) default=long(null) on Timestamp from ago(60d) to now()+24h*7*4 step 12h
| extend FreeSpace = series_fill_backward(FreeSpace)
| extend forecast=series_decompose_forecast(FreeSpace, 7*4*2)
| render timechart
The documentation could be a bit clearer but I think what the points parameter does is simply to omit the last N points from training (since they are empty and you don't want to include them in your forecast model)
Output:
To get when you hit close to 0:
let DiskSpace =
range Timestamp from ago(60d) to now() step 1d
| order by Timestamp desc
| serialize rn=row_number() + 10
| extend FreeSpace = case
(
rn % 5 == 0, rn + 5
, rn % 3 == 0, rn -4
, rn % 7 == 0, rn +3
, rn
)
| project Timestamp, FreeSpace;
DiskSpace
| make-series FreeSpace = max(FreeSpace) default=long(null) on Timestamp from ago(60d) to now()+24h*7*4 step 12h
| extend FreeSpace = series_fill_backward(FreeSpace)
| extend forecast=series_decompose_forecast(FreeSpace, 7*4*2)
| mv-apply with_itemindex=idx f=forecast to typeof(double) on (
where f <= 0.5
| summarize min(idx)
)
| project AlmostOutOfDiskSpace = Timestamp[min_idx], PredictedDiskSpaceAtThatPoint = forecast[min_idx]
AlmostOutOfDiskSpace
PredictedDiskSpaceAtThatPoint
5/12/2022 13:02:24
0.32277009977544