I am importing 500 csv's that have the following similar format:
"https://www.quandl.com/api/v3/datasets/WIKI/stockname/data.csv?column_index=11&transform=rdiff&api_key=keyname"
Where stockname is the ticker symbol of a single stock. I have the list of stock tickers saved in a dataframe called stocklist.
I'd like to use lapply to iterate through my list of stocks. Here's what I have so far:
lst <- lapply(stocklist, function(i){
url <- paste0("https://www.quandl.com/api/v3/datasets/WIKI/",i,"/data.csv?column_index=11&transform=rdiff&api_key=XXXXXXXXXXXXXXX")
spdata <- read.csv(url, stringsAsFactors = FALSE)
})
I get the following error:
Error in file(file, "rt") : invalid 'description' argument
What could be causing this error? I tried using a for loop as well, but was unsuccessful and I have been told lapply is a better method in R for this type of task.
Edit:
Stucture of stocklist:
> dput(droplevels(head(stocklist)))
structure(list(`Ticker symbol` = c("MMM", "ABT", "ABBV", "ABMD",
"ACN", "ATVI")), .Names = "Ticker symbol", row.names = c(NA,
6L), class = "data.frame")
Second Edit (solution):
stockdata<-lapply(paste0("https://www.quandl.com/api/v3/datasets/WIKI/",stocklist[1][[1]],"/data.csv?column_index=11&transform=rdiff&api_key=XXXXXXX"),read.csv,stringsAsFactors=FALSE)
Add names to stockdata:
names(stockdata)<-stocklist[1][[1]]
I believe your 'i' variable is a vector.
Make sure you are sub-scripting it properly and only passing one stock at a time.
This: i would look something like this: i[x]
I can't tell what i is, but I'm guessing it's a number. Can you try this and see if it works for you? I think it's pretty close. Just make sure the URL matches the pattern of the actual URL you are fetching data from. I tried to find it; I couldn't find it.
seq <- c(1:10)
for (value in seq) {
mydownload <- function (start_date, end_date) {
start_date <- as.Date(start_date) ## convert to Date object
end_date <- as.Date(end_date) ## convert to Date object
dates <- as.Date("1970/01/01") + (start_date : end_date) ## date sequence
## a loop to download data
for (i in 1:length(dates)) {
string_date <- as.character(dates[i])
myfile <- paste0("C:/Users/Excel/Desktop/", string_date, ".csv")
string_date <- gsub("-", "-", string_date) ## replace "-" with "/"
myurl <- paste("https://www.quandl.com/api/v3/datasets/WIKI/", i, "/data.csv?column_index=11&transform=rdiff&api_key=xxxxxxxxxx", sep = "")
download.file(url = myurl, destfile = myfile, quiet = TRUE)
}
}
}
Related
Assume we have a list of characters as basis for a function, which has a xts object with the same name as result:
library(zoo)
library(xts)
library(quantmod)
l<-list("AAPL","NKE")
for(i in 1:length(l)){
getSymbols(l[[i]], src = "yahoo")
write.zoo(l[[i]], paste(l[[i]],".csv", sep=''), sep = ",")
}
My code does not work, cause getSymbols creates an xts object (named AAPL / NKE). My problem is, that I cannot call them properly in the write.zoo function. Can you please help me?
Call getSymbols with auto = FALSE to get the data directly.
library(quantmod)
syms <- c("AAPL", "NKE")
for(s in syms) {
dat <- getSymbols(s, auto = FALSE)
write.zoo(dat, paste0(s, ".csv"), sep = ",")
}
Here, we need get to get the value of the object created
for(i in 1:length(l)){
getSymbols(l[[i]], src = "yahoo")
write.zoo(get(l[[i]]), paste(l[[i]],".csv", sep=''), sep = ",")
}
-checking
I am facing difficulties after running the code and trying to export the dataset to a spreadsheet or txt.file.
I am newbie to R, so maybe this question is trivial.
After running the following code:
eia_series <- function(api_key, series_id, start = NULL, end = NULL, num = NULL, tidy_data = "no", only_data = FALSE){
# max 100 series
# test if num is not null and either start or end is nut null. Not allowed
# api_key test for character.
# series_id test for character.
# if start/end not null, then check if format matches series id date format
# parse date and numerical data
# parse url
series_url <- httr::parse_url("http://api.eia.gov/series/")
series_url$query$series_id <- paste(series_id, collapse = ";")
series_url$query$api_key <- api_key
series_url$query$start <- start
series_url$query$end <- end
series_url$query$num <- num
# get data
series_data <- httr::GET(url = series_url)
series_data <- httr::content(series_data, as = "text")
series_data <- jsonlite::fromJSON(series_data)
# Move data from data.frame with nested list and NULL excisting
series_data$data <- series_data$series$data
series_data$series$data <- NULL
# parse data
series_data$data <- lapply(X = series_data$data,
FUN = function(x) data.frame(date = x[, 1],
value = as.numeric(x[, 2]),
stringsAsFactors = FALSE))
# add names to the list with data
names(series_data$data) <- series_data$data
# parse dates
series_data$data <- eia_date_parse(series_list = series_data$data, format_character = series_data$series$f)
# tidy up data
if(tidy_data == "tidy_long"){
series_data$data <- lapply(seq_along(series_data$data),
function(x) {cbind(series_data$data[[x]],
series_time_frame = series_data$series$f[x],
series_name = series_data$series$series_id[x],
stringsAsFactors = FALSE)})
series_data$data <- do.call(rbind, series_data$data)
}
# only data
if(only_data){
series_data <- series_data$data
}
return(series_data)
}
After running the function
eia_series(api_key = "XXX",series_id = c("PET.MCRFPOK1.M", "PET.MCRFPOK2.M"))
I tried to "transfer" the data in order to export it but got the following error:
No encoding supplied: defaulting to UTF-8.
I don't understand why. Could you help me out?
That doesn't look like an error, rather a statement. Probably coming from httr::content(series_data, as = "text"). Look in https://cran.r-project.org/web/packages/httr/vignettes/quickstart.html in The body section. It shouldn't be a problem, as long as your data returns what you expect. Otherwise you can try different encoding or there is a bug elsewhere.
Try:
series_data <- httr::content(series_data, as = "text", encoding = "UTF-8")
Trying to fetch the closing prices from a list of tickers listed in a csv. file using the following code:
date <- "2017-03-03"
tickers <- read.csv("us_tickerfeed.csv", header = TRUE)
for(i in 1:nrow(tickers)){
data <- getSymbols(tickers$ticker_th[i], from = date, to = date, src = "yahoo")
tickers$close_price[i] <- Cl(get(data))[[1]]
}
these codes worked before but now I'm getting the following error message:
Error in do.call(paste("getSymbols.", symbol.source, sep = ""), list(Symbols = current.symbols, :
could not find function "getSymbols.6"
Thanks!
I had this problem previously, could you try as.character(tickers$ticker_th)?
I'm new to learning R and I'm having an issue with some of my R code. I placed all the code for your convenience so that you can see the logic in what I am trying to do. My issue is renaming the header of my xts object Monthly_Quotes. I understand that when having an invalid stock symbol, the getsymbols function will not retrieve the quotes for "zzzz", which is why I am running into the issue of renaming the header. I'd like to resolve this such that if I have a much larger list of ticker symbols that do not get downloaded, that I won't have an issue renaming the header from my list of ticker_symbols.
#Environment to store stock symbols data
ETF_Data <- new.env()
#Dates needed for getSymbols function
sDate <- as.Date("2007-09-04") #Start Date
eDate <- as.Date("2014-09-02") #End Date
#Assign vector of ticker symbols
ticker_symbol <- c("zzzz","IVW","JKE","QQQ","SPYG","VUG" )
#Load functions
source("Functions.R")
#Create empty list
Temp_ETF_Data <- list()
#Variable is a connection to a specific file that appends the output to the file
connection_string <- file("Log_File.txt", open = "wt")
#Sink allows the R output to a connection file which is connection_string
#Log_File.txt is created to the directory that has been set
sink(connection_string, type="message")
#Loop to retrieve prices from Yahoo's API and assign only the Adjusted column to the list.
for(i in ticker_symbol){
tryCatch(
{
getSymbols(i, env = ETF_Data, from = sDate, to = eDate, src = "yahoo", warning = FALSE)
Temp_ETF_Data[[i]] <- Ad(ETF_Data[[i]])
},
error = function(e) {
message("-ERROR-")
message(paste("Error for ticker symbol :", i, "\n"))
message("Here's the original error message: ")
message(e)
message("")
return(i)},
finally = {
message(paste("Data was processed for symbol:", "[", i, "]" ))
message("\n","******************************************", "\n")
})
}
#Turns off message once it is on the console and reads the line.
#Then it closes the sink connection and closes the connection to the Log_File.txt
sink(type = "message")
readLines("Log_File.txt")
close(connection_string)
#Merge list into columns from list
Daily_Quotes <- do.call(merge, Temp_ETF_Data)
#Create new xts object with the 1st trading day price of each month and assign column headers
Monthly_Quotes <- Daily_Quotes[startpoints(Daily_Quotes,'months')]
#This piece of code creates this error
# "Error in `colnames<-`(`*tmp*`, value = c("zzzz", "IVW", "JKE", "QQQ", :
# length of 'dimnames' [2] not equal to array extent"
names(Monthly_Quotes) <- c(ticker_symbol)
I tried this:
ticker_symbol <- colnames(Monthly_Quotes)
names(Monthly_Quotes) <- c(ticker_symbol)
This does not directly answer your question (see the comment), but wouldn't this be easier:
library(quantmod)
ticker_symbol <- c("zzzz","IVW","JKE","QQQ","SPYG","VUG" )
sDate <- as.Date("2007-09-04") #Start Date
eDate <- as.Date("2014-09-02") #End Date
get.symbol <- function(ticker) {
tryCatch(temp <- Ad(getSymbols(ticker, auto.assign=FALSE,
from = sDate, to = eDate, warning = FALSE)),
error = function(e) {
message("-ERROR-")
message(paste("Error for ticker symbol :", ticker, "\n"))
message("Here's the original error message: ")
message(e)
message("")
return(NULL)},
finally = {
message(paste("Data was processed for symbol:", "[", ticker, "]" ))
message("\n","******************************************", "\n")
})
}
result <- do.call(cbind,lapply(ticker_symbol,get.symbol))
names(result) <- gsub("\\..+","",names(result)) # remove ".Adjusted" from names
head(result)
# IVW JKE QQQ SPYG VUG
# 2007-09-04 61.71 65.37 46.56 52.42 57.36
# 2007-09-05 61.37 64.78 46.10 51.95 56.85
# 2007-09-06 61.47 65.13 46.06 52.18 57.13
# 2007-09-07 60.58 64.06 45.21 51.29 56.31
# 2007-09-10 60.36 64.13 45.18 51.24 56.12
# 2007-09-11 61.19 65.03 45.86 51.90 56.89
This takes advantage of the auto.assign=... argument of getSymbols(...) to avoid all this environment-setting and accumulation the temporary results in a list.
The question says it all - I want to take a list object full of data.frames and write each data.frame to a separate .csv file where the name of the .csv file corresponds to the name of the list object.
Here's a reproducible example and the code I've written thus far.
df <- data.frame(
var1 = sample(1:10, 6, replace = TRUE)
, var2 = sample(LETTERS[1:2], 6, replace = TRUE)
, theday = c(1,1,2,2,3,3)
)
df.daily <- split(df, df$theday) #Split into separate days
lapply(df.daily, function(x){write.table(x, file = paste(names(x), ".csv", sep = ""), row.names = FALSE, sep = ",")})
And here is the top of the error message that R spits out
Error: Results must have one or more dimensions.
In addition: Warning messages:
1: In if (file == "") file <- stdout() else if (is.character(file)) { :
the condition has length > 1 and only the first element will be used
What am I missing here?
Try this:
sapply(names(df.daily),
function (x) write.table(df.daily[[x]], file=paste(x, "txt", sep=".") ) )
You should see the names ("1", "2", "3") spit out one by one, but the NULLs are the evidence that the side-effect of writing to disk files was done. (Edit: changed [] to [[]].)
You could use mapply:
mapply(
write.table,
x=df.daily, file=paste(names(df.daily), "txt", sep="."),
MoreArgs=list(row.names=FALSE, sep=",")
)
There is thread about similar problem on plyr mailing list.
A couple of things:
laply performs operations on a list. What you're looking for is d_ply. And you don't have to break it up by day, you can let plyr do that for you. Also, I would not use names(x) as that returns all of the column names of a data.frame.
d_ply(df, .(theday), function(x) write.csv(x, file=paste(x$theday,".csv",sep=""),row.names=F))