I'm running into issues while applying a moving window function to a time series dataset. I've imported daily streamflow data (date and value) into a zoo object, as approximated by the following:
library(zoo)
df <- data.frame(sf = c("2001-04-01", "2001-04-02", "2001-04-03", "2001-04-04",
"2001-04-05", "2001-04-06", "2001-04-07", "2001-06-01",
"2001-06-02", "2001-06-03", "2001-06-04", "2001-06-05",
"2001-06-06"),
cfs = abs(rnorm(13)))
zoodf <- read.zoo(df, format = "%Y-%m-%d")
Since I want to calculate the 3-day moving minimum for each month I've defined a function using rollapply:
f.3daylow <- function(x){rollapply(x, 3, FUN=min, align = "center")}
I then use aggregate:
aggregate(zoodf, by=as.yearmon, FUN=f.3daylow)
This promptly returns an error message:
Error in zoo(df, ix[!is.na(ix)]) :
“x” : attempt to define invalid zoo object
The problem appears to be that there are unequal number of data points in each month,since using the same dataframe with an additional date for June results in a correct response. Any suggestions for how to deal with this would be appreciated!
Ok, you might be thinking of something like this then. It pastes the results for each month into one data point, so that it can be returned in the aggregate function. Otherwise you may also have a look at ?aggregate.zoo for some more precise data manipulations.
f.3daylow <- function(x){paste(rollapply(x, 3, FUN=min,
align = "center"), collapse=", ")}
data <- aggregate(zoodf, by=as.yearmon, FUN=f.3daylow)
Returns, this is then a rolling window of 3 copied into 1 data point. To analyse it, eventually you will have to break it down again, so it is not recommended.
Apr 2001
0.124581285281643, 0.124581285281643, 0.124581285281643,
0.342222172241979, 0.518874882033892
June 2001
0.454158221843514, 0.454158221843514, 0.656966528249837,
0.513613009234435
Eventually you can cut it up again via strsplit(data[1],", "), but see Convert comma separated entry to columns for more details.
Related
this is my first project using a coded environment so may not phrase things accurately. I am building an ARIMA forecast.
I want to forecast for multiple sectors (business areas) at a time. Using help forums I have managed to write code that takes my time series data as input, fits the model, and sends the outputs to CSV. I am happy with this.
My problem is that I would also like capture the results from the decomposition analysis on a sector level. Currently, when I use a solution I found elsewhere it outputs to CSV in a format that is unusable, where everything is spread by row and the different lists are half in one row and another.
Thanks In advance!
My current solution (probably not super efficient but like I say cobbled together based on forum tips)
Clean data down to TS
NLDemand <- read_excel("TS Demand 2018 + Non London no lockdown.xlsx")
NLDemand <- as_tibble(NLDemand)
NLDemand <- na.omit(NLDemand)
NLDemand <- subset(NLDemand, select = -c(Month,Year))
NLDemand <- subset(NLDemand, select = -c(YearMonth))
##this gets the data to a point where each column is has a header of business sector and the time series data underneath it with no categorical columns left E.G:
Sector 1a, sector1b, sector...
500,450,300
450,500,350
...,...,...
Season capture for all sectors
tsData<-sapply(NLDemand, FUN = ts, simplify = FALSE,USE.NAMES = TRUE,start=c(2018,1),frequency=12)
tsData
timeseriescomponents <- sapply(tsData,FUN=decompose,simplify = FALSE, USE.NAMES = TRUE)
timeseriescomponents
this produces a list of lists where each sublist is the decomposed elements of the sector time series.
##Covert all season captures to the same length
TSC <- list(timeseriescomponents[1:41])
n.obs <- sapply(TSC, length)
seq.max <- seq_len(max(n.obs))
mat <- t(sapply(TSC, "[", i = seq.max ))
##Export to CSV
write.csv(mat, "Non london 2018 + S-T componants.csv", row.names=FALSE)
***What I want as an output would be a table that showed each componant as a a column in a list
Desired output format
Current output(sample)
I started learning R three days ago so pls bear with me.... if you see any flaws in my code or calculations please call them out.
I have tried this, but get a error message every time:
table.AnnualizedReturns(Apple.Monthly.Returns[, 2:3, drop = FALSE], scale = 12,
Rf = 0, geometric = TRUE, digits = 4)
Error in checkData(R) :
The data cannot be converted into a time series. If you are trying to pass in names from a data object with one column, you should use the form 'data[rows, columns, drop = FALSE]'. Rownames should have standard date formats, such as '1985-03-15'.
As you can clearly see I have no clue what I am doing.
This is every line of code I have written this far:
Dates <- Data_Task2$`Names Date`[1801:2270]
as.numeric(Dates)
Dates <- ymd(Dates)
Monthly.Return <- Data_Task2$Returns[1801:2270]
Monthly.Return <- as.numeric(Monthly.Return)
Apple.Monthly.Returns <- data.frame(Dates, Monthly.Return)
Log.return = log(Monthly.Return + 1)
Apple.Monthly.Returns$Log.return = log(Apple.Monthly.Returns$Monthly.Return + 1)
You should check out the Tidyverse and specifically dplyr (https://dplyr.tidyverse.org/).
This gets you to a good starting point:
https://www.r-bloggers.com/2014/03/using-r-quickly-calculating-summary-statistics-with-dplyr/
Can chartSeries, candleChart, or barChart be used to create an intraday chart in R?
chartSeries, candleChart, and barChart are part of the quantmod package for R.
First we need some example intraday trading data, which you can get for free from a variety of sites including Google's Undocumented Finance API.
Get some example data (hourly intervals)
query_addr <- 'https://www.google.com/finance/getprices'
stock_symb <- 'GOOG'
stock_exch <- 'NASD'
intvl_size <- 60*60 # 1 hr interval (in seconds) -- use 24*this for daily
period_len <- '90d'
output_fmt <- 'd,o,h,l,c,v' # date, open, high, low, close, volume
library(httr)
resp <-
POST(url = query_addr,
query = list(q = stock_symb,
x = stock_exch,
i = intvl_size,
p = period_len,
f = output_fmt) )
df <-
read.csv(text = content(resp),
skip = 7,
header = FALSE,
stringsAsFactors = FALSE)
# we need a function to munge the date convention used by google finance API
g_fin_date <- function(dstr, intvl_size){
unix_dates <- numeric(length(dstr))
date_is_unix <- grepl('^a',dstr)
unix_dates[date_is_unix] <- as.numeric(sub('^a','',dstr[date_is_unix]))
for(i in 2L:length(dstr)){
if(!date_is_unix[i]){
unix_dates[i] <- unix_dates[i-1] + intvl_size
}
}
return(as.POSIXct(unix_dates,origin="1970-01-01",tz="GMT" ))
}
# see header of resp text for column order
names(df) <- c('close_date','Close','High','Low','Open','Volume')
df[,'close_date'] <- g_fin_date(df[,'close_date'], intvl_size=intvl_size)
Here I have just chosen hourly open (i.e. beginning price), high, low, close (i.e. ending price)-- but you can specify a finer level of detail if you desire -- it will still roll up to a larger period with quantmod::to.period().
Make an xts
Once we have a data frame (such as you might obtain from an API or flat file) then you need to convert the data to xts. Note that for xts the timestamp must be the row name (and can be dropped from the columns).
library(xts)
rownames(df) <- df$close_date
df$close_date <- NULL
Convert to OHLC (Open, High, Low, Close) using xts
This is straightforward aggregation -- see ?to.period
GOOG <- to.hourly(as.xts(df)) # for daily use to.daily(as.xts(df))
More chart examples available at quantmod.com.
Make some charts using quantmod
There are great charts already built into quantmod, including the ones you mentioned.
library(quantmod)
chartSeries(GOOG)
barChart(GOOG, theme='white.mono',bar.type='hlc')
candleChart(GOOG,multi.col=TRUE,theme='white')
Enjoy your charts
Me: "I'll take intra-day time series charts in R for 100 Alex!" :D
Alex: "THIS popular format for financial time series can be used by quantmod functions chartSeries, candleChart, and barChart to create intraday charts in R"
Me: "What is an xts object, indexed by data/time stamp, containing prices for the Open, High, Low, and Close?
Alex: "Right you are!"
First, full disclosure. I attempted to do this strictly in MS Access with correlated subqueries, and had some help on this post 12 month moving average by person, date. I originally thought my data would be small enough to chug through, but it is awful. As an alternative, I'm going to try running this in R and then writing results to a new table in MS Access. I have data such that I have the following fields:
rep, cyc_date, amt
Following the linked example by Andrie for a rolling 5-year period (as opposed to the 5-year average) R: Calculating 5 year averages in panel data, I am trying to get rolling 12 month average for amt field by rep. Here is my code:
library(zoo)
library(plyr)
library(RODBC)
# Pull data from local MS Access database. The referenced sqlFetch is a query
# that pulls the data, ordered by `rep`, then `cyc_date`
channel <- odbcConnectAccess2007("C://MyDB.accdb")
data <- data.frame(sqlFetch(channel, "MyView"))
# Ensure coercion of `cyc_date` to date type
data$cyc_date <- as.Date(data$cyc_date)
# Function (take from post above)
rollmean12 <- function(x) {
rollmean(x, 12)
}
# Calculate rolling average by person
rollvec <- ddply(data, .(data$rep), rollmean12(data$amt))
Unfortunately, this doesn't work. I'm getting the following error:
Error in llply(.data = .data, .fun = .fun, ..., .progress = .progress, :
.fun is not a function.
I'm not sure why this is happening. Do I need to explicitly convert data to a zoo object? If so, not sure how to handle the extra dimensionality resulting from the person_id field. Any help would be very much appreciated.
I found this code on the following post: applying rolling mean by group in R
data$movavg <- ave(data$amt, data$rep, FUN = function(x) rollmean(x, k=12, align="right", na.pad=T)).
ave saves the day!
Just some hints, as I don't work at all with time series: ddply requires a data frame input, so don't convert it to a zoo object. .(data$rep) I think should be just .(rep), and rollmean12 should not be called with arguments. Rather, you should re-write the function to extract the columns you want. So, approximately something like this:
rollmean12 <- function(x) rollmean(x$amt, 12)
If you do ?ddply there is a link to a very helpful publication in JSS.
Try the tidyquant library
x %>% tq_mutate(
# tq_mutate args
select = amt,
mutate_fun = rollapply,
col_rename = "rollmean12", ####
# rollapply args
width = 12,
align = "right",
FUN = mean,
# mean args
na.rm = TRUE
)
I'm trying to automate some seasonal adjustment with the x12 package. To do this I need a ts object. However, I do not need a simple ts object, but one whose start date and frequency has been set. For any given series I could type that, but I will be feeding a mix of monthly or weekly data in. I can get the data from a quantmod as an xta object, but can't seem to figure out how to extract the frequency from the xts.
Here is some sample code that works the the whole way through, but I would like to pull the frequency info from the xts, rather than explicitly set it:
getSymbols("WILACR3URN",src="FRED", from="2000-01-01") # get data as an XTS
lax <- WILACR3URN #shorten name
laxts <- ts(lax$WILACR3URN, start=c(2000,1), frequency=12) #explicitly it works
plot.ts(laxts)
x12out <- x12(laxts,x12path="c:\\x12arima\\x12a.exe",transform="auto", automdl=TRUE)
laxadj <- as.ts(x12out$d11) # extract seasonally adjusted series
Any suggestions? Or is it not possible and I should determine/feed the frequency explicitly?
Thanks
This is untested for this specific case, but try using xts::periodicity for the frequency:
freq <- switch(periodicity(lax)$scale,
daily=365,
weekly=52,
monthly=12,
quarterly=4,
yearly=1)
And use the year and mon elements of POSIXlt objects to calculate the start year and month.
pltStart <- as.POSIXlt(start(lax))
Start <- c(pltStart$year+1900,pltStart$mon+1)
laxts <- ts(lax$WILACR3URN, start=Start, frequency=freq)
plot.ts(laxts)
The xts::periodicity suggestion was helpful to me. I've also found the following approach using xts::convertIndex works well for monthly and quarterly data. It is untested for weekly data.
require("quantmod")
require("dplyr")
getSymbols("WILACR3URN",src="FRED", from="2000-01-01") # get data as an XTS
lax <- WILACR3URN #shorten name
laxts <- lax %>%
convertIndex("yearmon") %>% # change index of xts object
as.ts(start = start(.), end = end(.)) # convert to ts
plot.ts(laxts)