I am downloading data from FRED with the quantmod library (author Jeffrey A. Ryan). With Yahoo and Google data, I am able to set start and end dates. Can the same be done for FRED data?
The help page does not list "from" and "to" as options of quantmod's getSymbols function, from which I'm inferring that it is not currently possible.
Is there a way to set a range for the data to be downloaded or do I need to download the entire dataset and discard the data I don't need?
Thanks for your help. Below the code that illustrates the context:
The dates are ignored when downloading from FRED:
# environment in which to store data
data <- new.env()
# set dates
date.start <- "2000-01-01"
date.end <- "2012-12-31"
# set tickers
tickers <- c("FEDFUNDS", "GDPPOT", "DGS10")
# import data from FRED database
library("quantmod")
getSymbols( tickers
, src = "FRED" # needed!
, from = date.start # ignored
, to = date.end # ignored
, env = data
, adjust = TRUE
)
head(data$FEDFUNDS)
head(data$FEDFUNDS)
FEDFUNDS
1954-07-01 0.80
1954-08-01 1.22
1954-09-01 1.06
1954-10-01 0.85
1954-11-01 0.83
1954-12-01 1.28
EDIT: Solution
Thanks to GSee's suggestion below, I am using the following code to subset the data to within the range of dates specified above:
# subset data to within time range
dtx <- data$FEDFUNDS
dtx[paste(date.start,date.end,sep="/")]
Here I extracted the xts data from the environment before acting upon it. My follow-up question explores alternatives.
Follow-Up Question
I have asked some follow-up questions there: get xts objects from within an environment
You have to download all the data and subset later. getSymbols.FRED does not support the from argument like getSymbols.yahoo does.
Alternatively you can download FRED data from Quandl (http://www.quandl.com/help/r) which offers more than 4 million datasets including all of the FRED data. There is an API and R package available. ("Quandl"). Data can be returned in several formats formats e.g. data frame ("raw"), ts ("ts"), zoo ("zoo") and xts ("xts").
For example to download GDPPOT10 and specify the dates and have it returned as an xts object all you have to do is:
require(Quandl)
mydata = Quandl("FRED/GDPPOT", start_date="2005-01-03",end_date="2013-04-10",type="xts")
Quandl doesn't seem to offer all data from FRED, at least in terms data frequency. Quandl most likely offers only annual data which is not useful in many circumstance.
Related
I am very new to R, I watched a youtube video to do various time series analysis, but it downloaded data from yahoo - my data is in Excel. I wanted to follow the same analysis, but with data from an excel.csv file. I spent two days finding out that the date must be in USA style. Now I am stuck again on a basic step - loading the data so it can be analysed - this seems to be the biggest hurdle with R. Please can someone give me some guidance on why the command shown below does not do the returns for the complete column set. I tried the zoo format, but it didn't work, then I tried xts and it worked partially. I suspect the original import from excel is the major problem. Can I get some guidance please
> AllPrices <- as.zoo(AllPrices)
> head(AllPrices)
Index1 Index2 Index3 Index4 Index5 Index6 Index7 Index8 Index9 Index10
> AllRets <- dailyReturn(AllPrices)
Error in NextMethod("[<-") : incorrect number of subscripts on matrix
> AllPrices<- as.xts(AllPrices)
> AllRets <- dailyReturn(AllPrices)
> head(AllRets)
daily.returns
2012-11-06 0.000000e+00
2012-11-07 -2.220249e-02
2012-11-08 1.379504e-05
2012-11-09 2.781961e-04
2012-11-12 -2.411128e-03
2012-11-13 7.932869e-03
Try to load your data using the readr package.
library(readr)
Then, look at the documentation by running ?read_csv in the console.
I recommend reading in your data this way. Specify the column types. For instance, if your first column is the date, read it in as a character "c" and if your other columns are numeric use "n".
data <- read_csv('YOUR_DATA.csv', col_types = "cnnnnn") # date in left column, 5 numeric columns
data$Dates <- as.Date(data$Dates, format = "%Y-%m-%d") # make the dates column a date class (you need to update "Dates" to be your column name for the Dates column, you may need to change the format
data <- as.data.frame(data) # turn the result into a dataframe
data <- xts(data[,-1], order.by = XAU[,1]) # then make an xts, data is everything but the date column, order.by is the date column
I am using the Quandl API in R to download the historical stock market data listed in NSE India.
The below code gives me the historical data for ICICI and PNB but needs manual entries for each stock to fetch the data. How do we download the historical data for all the stocks listed in NSE without writing these manual statements for each stock.
library(Quandl)
Quandl.api_key("API_Key")
## Download the data Set
ICICI = Quandl("NSE/ICICIBANK",collapse="daily",start_date="2018-01-01",type="raw")
PNB= Quandl("NSE/PNB",collapse="daily",start_date="2018-01-01",type="raw")
## Add another ("Stock") coloumn in Datasets using cbind command
ICICI<-cbind(ICICI,Stock="")
PNB<-cbind(PNB,Stock="")
## Paste the stock name in stock column
ICICI$Stock<-paste(ICICI$Stock,"ICICI",sep="")
PNB$Stock<-paste(PNB$Stock,"PNB",sep="")
## Consolidate under one dataset
Master_Data<-rbind(ICICI,PNB)
I do have a list of all the stocks name in an excel file as follows.
NSE/20MICRONS
NSE/3IINFOTECH
NSE/3MINDIA
NSE/A2ZMES
NSE/AANJANEYA
NSE/AARTIDRUGS
NSE/AARTIIND
NSE/AARVEEDEN
NSE/ABAN
NSE/ABB
NSE/ABBOTINDIA
NSE/ABCIL
NSE/ABGSHIP
Any help would be really appreciated.
Regards,
Akash
This worked for me, I hope it will be running for you too.
I have tested it.
Make a list (lyst in the code) items of all the companies for you want the data, like below. Then use lapply to save everything in a new list, like lyst_dwnld in the code.
If you want to avoid manually typing all that name as list, then you can save your excel sheet of names as a data frame and then use the same concept as below.
Code:
lyst <- list(icici = "NSE/ICICIBANK", pnb = "NSE/PNB")
lyst_dwnld <- lapply(names(lyst),
function(x)Quandl(lyst[[x]],
collapse="daily",
start_date="2018-01-01",type="raw"))
Output:
YOu can check if the data is downloaded or not, by quickly seeing the head.
> lapply(lyst_dwnld, head,2)
[[1]]
Date Open High Low Last Close Total Trade Quantity Turnover (Lacs)
1 2018-05-25 298.4 300.95 294.6 296.20 295.65 13541580 40235.19
2 2018-05-24 293.7 299.00 291.2 298.15 297.70 11489424 33952.28
[[2]]
Date Open High Low Last Close Total Trade Quantity Turnover (Lacs)
1 2018-05-25 81.95 84.55 81.30 83.60 83.35 19102160 15875.32
2 2018-05-24 80.70 82.50 80.05 82.35 82.10 19933989 16229.67
EDITED:
In you are unable to make a list using a dataframe, here is what you can do.
1) Read your data in excel to R dataframe using (readxl).
2) I have read a sample data and called it df.
df <- readxl::read_excel('path_where_excel_resides_with_name_of_excel_and_extension')
3) Name the column, something meaningful. I just used "name" here for this single column excel.
names(df) <- "name"
4) Use strsplit, to split the column(keep the original column), fetech only the second name out of it.
df$co_name <- lapply(strsplit(df$name, "/"),`[[`,2)
Now you can create the lyst object which is used in earlier code.
lyst <- as.list(df$name)
names(lyst) <- df$co_name
Please let me know in case this doesn't work out for you or in case of any issues. Thanks
The data is available until 2018 only via Quandl.
You can use BatchGetSymbols library in R to download further data. There is one issue that I noticed while downloading from Yahoo finance was that sometimes the data was missing for a few months in a year. Advised to use with caution before implementing any analysis and check the consistency of the data.
The following code is for reference:
if (!require(BatchGetSymbols)) install.packages('BatchGetSymbols')
#Package to download yahoo finance stock prices from internet directly
library(BatchGetSymbols)
first.date <- Sys.Date() - (365*5) #5 years back
last.date <- Sys.Date()
freq.data <- 'monthly' #change it as you like. example:'daily'
tickers <- c('RELIANCE.NS','TCS.NS')
# Reliance, Tcs
l.out <- BatchGetSymbols(tickers = tickers,
first.date = first.date,
last.date = last.date,
freq.data = freq.data,
cache.folder = file.path(your_directory_path,"stock_prices"))
str(l.out)
# To check download status of all stocks
l.out$df.control
# Actual Stock prices downloaded
head(l.out$df.tickers)
Alternate Method
I found this link during my web browsing and I am yet to implement it after understanding. Hopefully, this might help you further.
https://www.r-bloggers.com/extracting-eod-data-from-nse/
I'm trying to download CPI data from FRED with tq_get from the tidyquant package.
This code retrieves the data from here: https://fred.stlouisfed.org/series/CPIAUCSL
cpi <- tq_get(x = c("CPIAUCSL"), get = "economic.data")
But the oldest date in the imported table is January 1, 2007. The data on FRED's website goes all the way back to January 1, 1947. Clicking "download" on the page I linked to downloads the entire series. Why doesn't tidyquant and is there a way to specify the desired date range?
By default tq_get() returns 10-years of data. As #HFBrowning commented, you should add from and to arguments to specify a longer time range.
c("CPIAUCSL") %>%
tq_get(get = "economic.data", from="1947-01-01", to="2017-10-02")
quantmod::getSymbols() (which tq_get calls) returns all the data by default.
cpi <- quantmod::getSymbols("CPIAUCSL", src="FRED", auto.assign=FALSE)
start(cpi)
# [1] "1947-01-01"
I need to modify this example code for using it with intraday data which I should get from here and from here. As I understand, the code in that example works well with any historical data (or not?), so my problem then boils down to a question of loading the initial data in a necessary format (I mean daily or intraday).
As I also understand from answers on this question, it is impossible to load intraday data with getSymbols(). I tried to download that data into my hard-drive and to get it then with a read.csv() function, but this approach didn't work as well. Finally, I found few solutions of this problem in various articles (e.g. here), but all of them seem to be very complicated and "artificial".
So, my question is how to load the given intraday data into the given code elegantly and correctly from programmer's point of view, without reinventing the wheel?
P.S. I am very new to analysis of time series in R and quantstrat thus if my question seems to be obscure let me know what you need to know to answer it.
I don't know how to do this without "reinventing the wheel" because I'm not aware of any existing solutions. It's pretty easy to do with a custom function though.
intradataYahoo <- function(symbol, ...) {
# ensure xts is available
stopifnot(require(xts))
# construct URL
URL <- paste0("http://chartapi.finance.yahoo.com/instrument/1.0/",
symbol, "/chartdata;type=quote;range=1d/csv")
# read the metadata from the top of the file and put it into a usable list
metadata <- readLines(paste(URL, collapse=""), 17)[-1L]
# split into name/value pairs, set the names as the first element of the
# result and the values as the remaining elements
metadata <- strsplit(metadata, ":")
names(metadata) <- sub("-","_",sapply(metadata, `[`, 1))
metadata <- lapply(metadata, function(x) strsplit(x[-1L], ",")[[1]])
# convert GMT offset to numeric
metadata$gmtoffset <- as.numeric(metadata$gmtoffset)
# read data into an xts object; timestamps are in GMT, so we don't set it
# explicitly. I would set it explicitly, but timezones are provided in
# an ambiguous format (e.g. "CST", "EST", etc).
Data <- as.xts(read.zoo(paste(URL, collapse=""), sep=",", header=FALSE,
skip=17, FUN=function(i) .POSIXct(as.numeric(i))))
# set column names and metadata (as xts attributes)
colnames(Data) <- metadata$values[-1L]
xtsAttributes(Data) <- metadata[c("ticker","Company_Name",
"Exchange_Name","unit","timezone","gmtoffset")]
Data
}
I'd consider adding something like this to quantmod, but it would need to be tested. I wrote this in under 15 minutes, so I'm sure there will be some issues.
I am completely new to R and I am learning how to program in R to get historical stock index data. I am planning to build an daily update code to update historic index data. I use a data environment call "indexData" to store the data as a xts. But unfortunately, the rbind or merge does not support data environments. They only support objects. I am wondering whether there are any workarounds or packages that I can used to solve this. My code is the following:
indexData<-new.env()
startDate<-"2013-11-02"
getSymbols(Symbols=indexList,src='yahoo',from=startDate,to="2013-11-12",env=indexData)
startDate<-(end(indexData$FTSE)+1)
NewIndexData<-new.env()
getSymbols(Symbols=indexList,src='yahoo',from=startDate,env=NewIndexData)
rbind(indexData,NewIndexData) #it does not work for data environments
Much appreciate for any suggestions!
If you use auto.assign=FALSE, you get explicit variables -- and those you can extend as shown below.
First we get two IBM price series:
R> IBM <- getSymbols("IBM",from="2013-01-01",to="2013-12-01",auto.assign=FALSE)
R> IBM2 <- getSymbols("IBM",from="2013-12-02",to="2013-12-28",auto.assign=FALSE)
Then we check their dates:
R> c(start(IBM), end(IBM))
[1] "2013-01-02" "2013-11-29"
R> c(start(IBM2), end(IBM2))
[1] "2013-12-02" "2013-12-27"
Finally, we merge them and check the dates of the combined series:
R> allIBM <- merge(IBM, IBM2)
R> c(start(allIBM), end(allIBM))
[1] "2013-01-02" "2013-12-27"
R>
You can approximate this by pulling distinct symbols out of the two environments but I think that is harder for you as someone beginning in R. So my recommendation would be to not work with environments -- see about lists instead.