I'm trying to download CPI data from FRED with tq_get from the tidyquant package.
This code retrieves the data from here: https://fred.stlouisfed.org/series/CPIAUCSL
cpi <- tq_get(x = c("CPIAUCSL"), get = "economic.data")
But the oldest date in the imported table is January 1, 2007. The data on FRED's website goes all the way back to January 1, 1947. Clicking "download" on the page I linked to downloads the entire series. Why doesn't tidyquant and is there a way to specify the desired date range?
By default tq_get() returns 10-years of data. As #HFBrowning commented, you should add from and to arguments to specify a longer time range.
c("CPIAUCSL") %>%
tq_get(get = "economic.data", from="1947-01-01", to="2017-10-02")
quantmod::getSymbols() (which tq_get calls) returns all the data by default.
cpi <- quantmod::getSymbols("CPIAUCSL", src="FRED", auto.assign=FALSE)
start(cpi)
# [1] "1947-01-01"
Related
I am using the Quandl API in R to download the historical stock market data listed in NSE India.
The below code gives me the historical data for ICICI and PNB but needs manual entries for each stock to fetch the data. How do we download the historical data for all the stocks listed in NSE without writing these manual statements for each stock.
library(Quandl)
Quandl.api_key("API_Key")
## Download the data Set
ICICI = Quandl("NSE/ICICIBANK",collapse="daily",start_date="2018-01-01",type="raw")
PNB= Quandl("NSE/PNB",collapse="daily",start_date="2018-01-01",type="raw")
## Add another ("Stock") coloumn in Datasets using cbind command
ICICI<-cbind(ICICI,Stock="")
PNB<-cbind(PNB,Stock="")
## Paste the stock name in stock column
ICICI$Stock<-paste(ICICI$Stock,"ICICI",sep="")
PNB$Stock<-paste(PNB$Stock,"PNB",sep="")
## Consolidate under one dataset
Master_Data<-rbind(ICICI,PNB)
I do have a list of all the stocks name in an excel file as follows.
NSE/20MICRONS
NSE/3IINFOTECH
NSE/3MINDIA
NSE/A2ZMES
NSE/AANJANEYA
NSE/AARTIDRUGS
NSE/AARTIIND
NSE/AARVEEDEN
NSE/ABAN
NSE/ABB
NSE/ABBOTINDIA
NSE/ABCIL
NSE/ABGSHIP
Any help would be really appreciated.
Regards,
Akash
This worked for me, I hope it will be running for you too.
I have tested it.
Make a list (lyst in the code) items of all the companies for you want the data, like below. Then use lapply to save everything in a new list, like lyst_dwnld in the code.
If you want to avoid manually typing all that name as list, then you can save your excel sheet of names as a data frame and then use the same concept as below.
Code:
lyst <- list(icici = "NSE/ICICIBANK", pnb = "NSE/PNB")
lyst_dwnld <- lapply(names(lyst),
function(x)Quandl(lyst[[x]],
collapse="daily",
start_date="2018-01-01",type="raw"))
Output:
YOu can check if the data is downloaded or not, by quickly seeing the head.
> lapply(lyst_dwnld, head,2)
[[1]]
Date Open High Low Last Close Total Trade Quantity Turnover (Lacs)
1 2018-05-25 298.4 300.95 294.6 296.20 295.65 13541580 40235.19
2 2018-05-24 293.7 299.00 291.2 298.15 297.70 11489424 33952.28
[[2]]
Date Open High Low Last Close Total Trade Quantity Turnover (Lacs)
1 2018-05-25 81.95 84.55 81.30 83.60 83.35 19102160 15875.32
2 2018-05-24 80.70 82.50 80.05 82.35 82.10 19933989 16229.67
EDITED:
In you are unable to make a list using a dataframe, here is what you can do.
1) Read your data in excel to R dataframe using (readxl).
2) I have read a sample data and called it df.
df <- readxl::read_excel('path_where_excel_resides_with_name_of_excel_and_extension')
3) Name the column, something meaningful. I just used "name" here for this single column excel.
names(df) <- "name"
4) Use strsplit, to split the column(keep the original column), fetech only the second name out of it.
df$co_name <- lapply(strsplit(df$name, "/"),`[[`,2)
Now you can create the lyst object which is used in earlier code.
lyst <- as.list(df$name)
names(lyst) <- df$co_name
Please let me know in case this doesn't work out for you or in case of any issues. Thanks
The data is available until 2018 only via Quandl.
You can use BatchGetSymbols library in R to download further data. There is one issue that I noticed while downloading from Yahoo finance was that sometimes the data was missing for a few months in a year. Advised to use with caution before implementing any analysis and check the consistency of the data.
The following code is for reference:
if (!require(BatchGetSymbols)) install.packages('BatchGetSymbols')
#Package to download yahoo finance stock prices from internet directly
library(BatchGetSymbols)
first.date <- Sys.Date() - (365*5) #5 years back
last.date <- Sys.Date()
freq.data <- 'monthly' #change it as you like. example:'daily'
tickers <- c('RELIANCE.NS','TCS.NS')
# Reliance, Tcs
l.out <- BatchGetSymbols(tickers = tickers,
first.date = first.date,
last.date = last.date,
freq.data = freq.data,
cache.folder = file.path(your_directory_path,"stock_prices"))
str(l.out)
# To check download status of all stocks
l.out$df.control
# Actual Stock prices downloaded
head(l.out$df.tickers)
Alternate Method
I found this link during my web browsing and I am yet to implement it after understanding. Hopefully, this might help you further.
https://www.r-bloggers.com/extracting-eod-data-from-nse/
Can chartSeries, candleChart, or barChart be used to create an intraday chart in R?
chartSeries, candleChart, and barChart are part of the quantmod package for R.
First we need some example intraday trading data, which you can get for free from a variety of sites including Google's Undocumented Finance API.
Get some example data (hourly intervals)
query_addr <- 'https://www.google.com/finance/getprices'
stock_symb <- 'GOOG'
stock_exch <- 'NASD'
intvl_size <- 60*60 # 1 hr interval (in seconds) -- use 24*this for daily
period_len <- '90d'
output_fmt <- 'd,o,h,l,c,v' # date, open, high, low, close, volume
library(httr)
resp <-
POST(url = query_addr,
query = list(q = stock_symb,
x = stock_exch,
i = intvl_size,
p = period_len,
f = output_fmt) )
df <-
read.csv(text = content(resp),
skip = 7,
header = FALSE,
stringsAsFactors = FALSE)
# we need a function to munge the date convention used by google finance API
g_fin_date <- function(dstr, intvl_size){
unix_dates <- numeric(length(dstr))
date_is_unix <- grepl('^a',dstr)
unix_dates[date_is_unix] <- as.numeric(sub('^a','',dstr[date_is_unix]))
for(i in 2L:length(dstr)){
if(!date_is_unix[i]){
unix_dates[i] <- unix_dates[i-1] + intvl_size
}
}
return(as.POSIXct(unix_dates,origin="1970-01-01",tz="GMT" ))
}
# see header of resp text for column order
names(df) <- c('close_date','Close','High','Low','Open','Volume')
df[,'close_date'] <- g_fin_date(df[,'close_date'], intvl_size=intvl_size)
Here I have just chosen hourly open (i.e. beginning price), high, low, close (i.e. ending price)-- but you can specify a finer level of detail if you desire -- it will still roll up to a larger period with quantmod::to.period().
Make an xts
Once we have a data frame (such as you might obtain from an API or flat file) then you need to convert the data to xts. Note that for xts the timestamp must be the row name (and can be dropped from the columns).
library(xts)
rownames(df) <- df$close_date
df$close_date <- NULL
Convert to OHLC (Open, High, Low, Close) using xts
This is straightforward aggregation -- see ?to.period
GOOG <- to.hourly(as.xts(df)) # for daily use to.daily(as.xts(df))
More chart examples available at quantmod.com.
Make some charts using quantmod
There are great charts already built into quantmod, including the ones you mentioned.
library(quantmod)
chartSeries(GOOG)
barChart(GOOG, theme='white.mono',bar.type='hlc')
candleChart(GOOG,multi.col=TRUE,theme='white')
Enjoy your charts
Me: "I'll take intra-day time series charts in R for 100 Alex!" :D
Alex: "THIS popular format for financial time series can be used by quantmod functions chartSeries, candleChart, and barChart to create intraday charts in R"
Me: "What is an xts object, indexed by data/time stamp, containing prices for the Open, High, Low, and Close?
Alex: "Right you are!"
I'm running into issues while applying a moving window function to a time series dataset. I've imported daily streamflow data (date and value) into a zoo object, as approximated by the following:
library(zoo)
df <- data.frame(sf = c("2001-04-01", "2001-04-02", "2001-04-03", "2001-04-04",
"2001-04-05", "2001-04-06", "2001-04-07", "2001-06-01",
"2001-06-02", "2001-06-03", "2001-06-04", "2001-06-05",
"2001-06-06"),
cfs = abs(rnorm(13)))
zoodf <- read.zoo(df, format = "%Y-%m-%d")
Since I want to calculate the 3-day moving minimum for each month I've defined a function using rollapply:
f.3daylow <- function(x){rollapply(x, 3, FUN=min, align = "center")}
I then use aggregate:
aggregate(zoodf, by=as.yearmon, FUN=f.3daylow)
This promptly returns an error message:
Error in zoo(df, ix[!is.na(ix)]) :
“x” : attempt to define invalid zoo object
The problem appears to be that there are unequal number of data points in each month,since using the same dataframe with an additional date for June results in a correct response. Any suggestions for how to deal with this would be appreciated!
Ok, you might be thinking of something like this then. It pastes the results for each month into one data point, so that it can be returned in the aggregate function. Otherwise you may also have a look at ?aggregate.zoo for some more precise data manipulations.
f.3daylow <- function(x){paste(rollapply(x, 3, FUN=min,
align = "center"), collapse=", ")}
data <- aggregate(zoodf, by=as.yearmon, FUN=f.3daylow)
Returns, this is then a rolling window of 3 copied into 1 data point. To analyse it, eventually you will have to break it down again, so it is not recommended.
Apr 2001
0.124581285281643, 0.124581285281643, 0.124581285281643,
0.342222172241979, 0.518874882033892
June 2001
0.454158221843514, 0.454158221843514, 0.656966528249837,
0.513613009234435
Eventually you can cut it up again via strsplit(data[1],", "), but see Convert comma separated entry to columns for more details.
I am downloading data from FRED with the quantmod library (author Jeffrey A. Ryan). With Yahoo and Google data, I am able to set start and end dates. Can the same be done for FRED data?
The help page does not list "from" and "to" as options of quantmod's getSymbols function, from which I'm inferring that it is not currently possible.
Is there a way to set a range for the data to be downloaded or do I need to download the entire dataset and discard the data I don't need?
Thanks for your help. Below the code that illustrates the context:
The dates are ignored when downloading from FRED:
# environment in which to store data
data <- new.env()
# set dates
date.start <- "2000-01-01"
date.end <- "2012-12-31"
# set tickers
tickers <- c("FEDFUNDS", "GDPPOT", "DGS10")
# import data from FRED database
library("quantmod")
getSymbols( tickers
, src = "FRED" # needed!
, from = date.start # ignored
, to = date.end # ignored
, env = data
, adjust = TRUE
)
head(data$FEDFUNDS)
head(data$FEDFUNDS)
FEDFUNDS
1954-07-01 0.80
1954-08-01 1.22
1954-09-01 1.06
1954-10-01 0.85
1954-11-01 0.83
1954-12-01 1.28
EDIT: Solution
Thanks to GSee's suggestion below, I am using the following code to subset the data to within the range of dates specified above:
# subset data to within time range
dtx <- data$FEDFUNDS
dtx[paste(date.start,date.end,sep="/")]
Here I extracted the xts data from the environment before acting upon it. My follow-up question explores alternatives.
Follow-Up Question
I have asked some follow-up questions there: get xts objects from within an environment
You have to download all the data and subset later. getSymbols.FRED does not support the from argument like getSymbols.yahoo does.
Alternatively you can download FRED data from Quandl (http://www.quandl.com/help/r) which offers more than 4 million datasets including all of the FRED data. There is an API and R package available. ("Quandl"). Data can be returned in several formats formats e.g. data frame ("raw"), ts ("ts"), zoo ("zoo") and xts ("xts").
For example to download GDPPOT10 and specify the dates and have it returned as an xts object all you have to do is:
require(Quandl)
mydata = Quandl("FRED/GDPPOT", start_date="2005-01-03",end_date="2013-04-10",type="xts")
Quandl doesn't seem to offer all data from FRED, at least in terms data frequency. Quandl most likely offers only annual data which is not useful in many circumstance.
I'm trying to automate some seasonal adjustment with the x12 package. To do this I need a ts object. However, I do not need a simple ts object, but one whose start date and frequency has been set. For any given series I could type that, but I will be feeding a mix of monthly or weekly data in. I can get the data from a quantmod as an xta object, but can't seem to figure out how to extract the frequency from the xts.
Here is some sample code that works the the whole way through, but I would like to pull the frequency info from the xts, rather than explicitly set it:
getSymbols("WILACR3URN",src="FRED", from="2000-01-01") # get data as an XTS
lax <- WILACR3URN #shorten name
laxts <- ts(lax$WILACR3URN, start=c(2000,1), frequency=12) #explicitly it works
plot.ts(laxts)
x12out <- x12(laxts,x12path="c:\\x12arima\\x12a.exe",transform="auto", automdl=TRUE)
laxadj <- as.ts(x12out$d11) # extract seasonally adjusted series
Any suggestions? Or is it not possible and I should determine/feed the frequency explicitly?
Thanks
This is untested for this specific case, but try using xts::periodicity for the frequency:
freq <- switch(periodicity(lax)$scale,
daily=365,
weekly=52,
monthly=12,
quarterly=4,
yearly=1)
And use the year and mon elements of POSIXlt objects to calculate the start year and month.
pltStart <- as.POSIXlt(start(lax))
Start <- c(pltStart$year+1900,pltStart$mon+1)
laxts <- ts(lax$WILACR3URN, start=Start, frequency=freq)
plot.ts(laxts)
The xts::periodicity suggestion was helpful to me. I've also found the following approach using xts::convertIndex works well for monthly and quarterly data. It is untested for weekly data.
require("quantmod")
require("dplyr")
getSymbols("WILACR3URN",src="FRED", from="2000-01-01") # get data as an XTS
lax <- WILACR3URN #shorten name
laxts <- lax %>%
convertIndex("yearmon") %>% # change index of xts object
as.ts(start = start(.), end = end(.)) # convert to ts
plot.ts(laxts)