minute wise time series forcasting? - r

Question

I've been working on R for last one week or so, this website has helped a lot in understanding the basics.
I am doing an minute wise forecast for my company,
data is something like this:
REFEE ENTRY_DATE
1.00 01-01-2011 00:00:00
2.00 01-01-2011 00:01:00
3.00 01-01-2011 00:02:00
4.00 01-01-2011 00:03:00
5.00 01-01-2011 00:04:00
6.00 01-01-2011 00:05:00
7.00 01-01-2011 00:06:00
8.00 01-01-2011 00:07:00
9.00 01-01-2011 00:08:00
10.00 01-01-2011 00:09:00
......so on for four years till 2014
thats roughly more than 133921*12 samples. I have tried all the codes for forecasting, HoltWinters(), forecast() and all other forcasting methods....
The problem is, the application hangs everytime I try these functions; doesn't R support so many data for forecasting?
Is there any other package that can help me get the forecast for such enormous amount of data?

Answer 1

This actually is quite a lot of data, at least for R. You could look at ets() in the forecast package. I like recommending this free online forecasting textbook from the same authors.
You could of course think about your data. Do you actually expect dynamics that can only be seen on this level, e.g., sub-hourly patterns? Do you actually need your forecasts on a minute-by-minute basis, e.g., for operational decisions? (From what I know, even short-term electricity forecasting is done in 15 minute buckets - and if you are actually into high frequency trading, you'd likely have shorter time periods.)
If yes, you should probably look into specific methods that can actually model multiple types of seasonality. Electricity load forecasting may be a good point to start, since these people do deal with multiple overlaid seasonal patterns.
If no, you could think about aggregating your data, say to days, then forecasting aggregates and diaggregating, e.g., using historical proportions of minutes within days. This would at least make forecasting less of a data problem.

Answer 2

For large data sets I would recommend using predict() from the R base as opposed to forecast(). While forecast() provides more information (predict() only provides the forecast and standard errors), using rbenchmark for the two functions suggests predict() is much faster.
Additionally forecast() drops the century in its dates for their forecasted ts object, which is annoying...
As Stephan Kosla stated having such granular data may be an issue. A speed up could be found by taking a daily/weekly/monthly average of your data before performing the forecast. You can do this using the one of the apply functions, lubridate and a bit of ingenuity. I've shown and example below of how I would do this:
library(lubridate)
# Create dataframe for AirPassengers dataset (frome base)
df <- data.frame(data=as.vector(AirPassengers),
date=as.Date((time(AirPassengers))),
year=year(as.Date((time(AirPassengers)))))
# Split by year, then take average
average.by.year <- unsplit(lapply(split(df$data,df$year),mean), #lapply takes the mean
df$year)