Calling a list of tickers in quantmod using R - r

I want to get some data from a list of Chinese stocks using quantmod.
The list is like below:
002705.SZ -- 002730.SZ (in this sequence, there are some tickers matched with Null stock, for example, there is no stock called 002720.SZ)
300357.SZ -- 300402.SZ
603188.SS
603609.SS
603288.SS
603306.SS
603369.SS
I want to write a loop to run all these stocks to get the data from each of them and save them into one data frame.

This should get you started.
library(quantmod)
library(stringr) # for str_pad
stocks <- paste(str_pad(2705:2730,width=6,side="left",pad="0"),"SZ",sep=".")
get.stock <- function(s) {
s <- try(Cl(getSymbols(s,auto.assign=FALSE)),silent=T)
if (class(s)=="xts") return(s)
return (NULL)
}
result <- do.call(cbind,lapply(stocks,get.stock))
head(result)
# X002705.SZ.Close X002706.SZ.Close X002707.SZ.Close X002708.SZ.Close X002709.SZ.Close X002711.SZ.Close X002712.SZ.Close X002713.SZ.Close
# 2014-01-21 15.25 27.79 NA 17.26 NA NA NA NA
# 2014-01-22 14.28 28.41 NA 16.56 NA NA NA NA
# 2014-01-23 13.65 27.78 33.62 15.95 19.83 NA 36.58 NA
# 2014-01-24 15.02 30.56 36.98 17.55 21.81 NA 40.24 NA
# 2014-01-27 14.43 31.26 40.68 18.70 23.99 26.34 44.26 NA
# 2014-01-28 14.18 30.01 44.75 17.66 25.57 28.97 48.69 NA
This takes advantage of the fact that getSymbols(...) returns either an xts object, or a character string with an error message if the fetch fails.
Note that cbind(...) for xts objects aligns according to the index, so it acts like merge(...).
This produces an xts object, not a data frame. To convert this to a data.frame, use:
result.df <- data.frame(date=index(result),result)

Related

log return calculation from matrix

when I have a dataframe named "Historical_Stock_Prices_R" like this
Date1 MSFT AAPL GOOGL
25-01-05 21.03 4.87 88.56
26-01-05 21.02 4.89 94.62
27-01-05 21.10 4.91 94.04
28-01-05 21.16 5.00 95.17
I use the following formulas to get a lsit of monthly max and monthly mean log return from daily price data file
return<- cbind.data.frame(date=Historical_Stock_Prices_R$Date1[2:nrow(Historical_Stock_Prices_R)],apply(Historical_Stock_Prices_R[,2:4],2,function(x) log(x[-1]/x[-length(x)])*100))
return$Date <- as.Date(return$date,format="%d-%m-%y")
RMax <- aggregate(return[,-1],
by=list(Month=format(return$Date,"%y-%m")),
FUN=max)
RMean <- aggregate(return[,-1],
by=list(Month=format(return$Date,"%y-%m")),
FUN=mean)
But now I have a matrix (not a dataframe) named "df" like this
AAPL.High ABT.High ABBV.High ACN.High ADBE.High
07-01-02 NA NA NA NA NA
03-01-07 12.37 24.74 NA 37 41.32
04-01-07 12.28 25.12 NA 37.23 41
05-01-07 12.31 25 NA 36.99 40.9
Now how can I calculate same monthly mean and monthly max using similar kind of code?

R: creating xts changes dataset, losing data

when creating an xtsobject from a data.frame I seem to lose some data (approximately 3000 data lost over 33 000).
My dataset is as follow: (with the time being day-month-year, EU format)
> head(mesdonnees)
time value
1 05-03-2006 04:07 NA
2 05-03-2006 04:17 NA
3 05-03-2006 04:27 NA
4 05-03-2006 04:37 NA
5 05-03-2006 04:47 NA
6 05-03-2006 04:57 NA
Due to the format I had to extract the different parts of the date (at least I couldn't get as.POSIXct to work with this format).
Here is how I did it:
# Extract characters and define as S....
Syear <- substr(mesdonnees$time, 7,10)
Smonth <- substr(mesdonnees$time, 4,5)
Sday <- substr(mesdonnees$time, 1, 2)
#Gather all parts and use "-" as sep
datetext <- paste(Syear, Smonth, Sday, sep="-")
#define format of each part of the string
formatdate<-as.POSIXct(datetext, format="%Y-%m-%d", tz = "GMT")
I then try to create my xtswith...
xtsdata <- xts(mesdonnees$value, order.by = formatdate, tz = "GMT")
... but when doing this I get some quite weird results: the first value is in 1900
> head(xtsdata)
[,1]
1900-01-04 NA
2006-03-05 NA
2006-03-05 NA
2006-03-05 NA
2006-03-05 NA
2006-03-05 NA
and many (3000) dates are not kept:
> xtsdata[30225:30233,]
[,1]
2006-12-31 0
2006-12-31 0
2006-12-31 0
2006-12-31 0
<NA> NA
<NA> NA
<NA> NA
<NA> NA
<NA> NA
When looking at what should be the same line in both my data.frameand my xtsI can see that the lines are offset (I had the date format changed in the xts object creation):
> mesdonnees[25617,]
time value
25617 08-11-2006 23:51 0
> xtsdata[25617,]
[,1]
2006-11-25 0.27
How is it that my data are offset? I tried changing the tz but it doesn't affect it. I removed all duplicates using the dyplr package, it doesn't affect the xts results either. Thank you for your help !
After changing my xts code to the one suggested by Joshua:
xtsdata <- xts(mesdonnees$value, order.by = as.POSIXct(mesdonnees$time, tz = "GMT", format = "%d-%m-%Y %H:%M"))
... my data show properly for the "last" part, but I now have a different problem. The first 2300 data show the following results when doing (using xtsdata[1500,] (or any row < 2300) displays the same results)
> view(xtsdata):
0206-06-30 23:08:00 NA
0206-06-30 23:18:00 NA
0206-06-30 23:28:00 NA
1900-01-04 12:00:00 NA
2006-03-05 04:07:00 NA
2006-03-05 04:17:00 NA
I noticed this error before and thought it was due to the date format; maybe it is not? Also, when looking at the xtsdata I do not get the same results for the same row (the last rows are correct thought):
> mesdonnees[2360,]
time value
2360 23-03-2006 03:09 NA
> xtsdata[2360,]
[,1]
2006-03-05 09:07:00 NA
As requested:
> str(mesdonnees)
'data.frame': 32556 obs. of 2 variables:
$ time : chr "05-03-2006 04:07" "05-03-2006 04:17" "05-03-2006 04:27" "05-03-2006 04:37" ...
$ value: num NA NA NA NA NA NA NA NA NA NA ...
And if needed:
An ‘xts’ object on 0206-06-01 00:09:00/2006-12-31 23:29:00 containing:
Data: num [1:32556, 1] NA NA NA NA NA NA NA NA NA NA ...
Indexed by objects of class: [POSIXct,POSIXt] TZ: GMT
xts Attributes:
NULL
The problem is that you only include the date portion of the timestamp in datetext and formatdate, but your data have dates and times.
You also do not need to do all the string subsetting. You can achive the same result by specifying the format argument in your as.POSIXct call.
xtsdata <- xts(mesdonnees$value,
as.POSIXct(mesdonnees$times, "GMT", format = "%d-%m-%Y %H:%M")

How to fetch 3-years historical price serie from Oanda with R?

I would like to process Bitcoin price in R but I'm unable to download time serie from Yahoo and Google.
From Yahoo the BTCUSD historical time serie is missing and the Google doesn't recognize the URL formated by getSymbols when symbol is "CURRENCY:EURUSD". I know R expect the ":" to be a list so I applied a workaround I found in Stakeoverflow to turn CURRENCY:EURUSD in CURRENCY.EURUSD but still Google cannot process the request.
Download from Oanda works like a charm but request cannot exceed 500 days. I try this workaround to bypass the limitation but it fails to populate correctly the prices object in which I have others symbols :
for some reason BTCUSD prices are missing for 2012 and part of 2013
also there are symbols from symbols's list that get NA with the wo.
tail(prices) (with the loop bellow)
UUP FXB FXE FXF FXY SLV GLD BTC
2014-08-31 NA NA NA NA NA NA NA 506.809
2014-09-30 22.87 159.33 124.48 102.26 88.80 16.35 116.21 375.386
2014-10-31 23.09 157.20 123.49 101.45 86.65 15.50 112.66 341.852
2014-11-30 NA NA NA NA NA NA NA 378.690
2014-12-31 23.97 153.06 119.14 98.16 81.21 15.06 113.58 312.642
2015-01-24 NA NA NA NA NA NA NA 229.813
Extract of print(prices) (with the loop bellow)
2013-06-28 22.56 150.17 128.93 103.92 98.63 18.97 119.11 NA
2013-07-31 22.09 150.12 131.74 105.99 99.93 19.14 127.96 NA
2013-08-30 22.19 152.93 130.84 105.45 99.63 22.60 134.62 NA
2013-09-30 21.63 159.70 133.85 108.44 99.47 20.90 128.18 133.794
2013-10-31 21.63 158.10 134.29 108.03 99.38 21.10 127.74 203.849
2013-11-30 NA NA NA NA NA NA NA 1084.800
2013-12-31 21.52 163.30 135.99 109.82 92.76 18.71 116.12 758.526
2014-01-31 21.83 161.95 133.29 108.00 95.58 18.45 120.09 812.097
tail(prices) (without the loop bellow)
UUP FXB FXE FXF FXY SLV GLD
2014-08-29 22.02 163.23 129.54 106.42 93.61 18.71 123.86
2014-09-30 22.87 159.33 124.48 102.26 88.80 16.35 116.21
2014-10-31 23.09 157.20 123.49 101.45 86.65 15.50 112.66
2014-11-28 23.47 153.46 122.46 101.00 82.01 14.83 112.11
2014-12-31 23.97 153.06 119.14 98.16 81.21 15.06 113.58
2015-01-23 25.21 147.23 110.33 110.95 82.57 17.51 124.23
What is wrong with this code ? Tx !
require(quantmod)
require(PerformanceAnalytics)
symbols <- c(
"UUP",
"FXB",
"FXE",
"FXF",
"FXY",
"SLV",
"GLD"
)
getSymbols(symbols, from="2004-01-01")
prices <- list()
for(i in 1:length(symbols)) {
prices[[i]] <- Cl(get(symbols[i]))
}
BTC <- list()
for(i in 1:2) {
BTC[[1]] <- getFX("BTC/USD",
from = Sys.Date() -499 * (i + 1),
to = Sys.Date() - 499 * i,
env = parent.frame(),
auto.assign = FALSE)
}
BTC[[1]] <- getFX("BTC/USD",
from = Sys.Date() -499,
to = Sys.Date(),
env = parent.frame(),
auto.assign = FALSE)
prices[[length(symbols)+1]] <- BTC[[1]]
prices <- do.call(cbind, prices)
colnames(prices) <- gsub("\\.[A-z]*", "", colnames(prices))
ep <- endpoints(prices, "months")
prices <- prices[ep,]
prices <- prices["1997-03::"]
Your for loop isn't using i, and then after the for loop you're overwriting the results (the list was of length 1 because BTC[[1]] was hardcoded)
Try this
btc <- do.call(rbind, lapply(0:2, function(i) {
getFX("BTC/USD",
from = Sys.Date() -499 * (i + 1),
to = Sys.Date() - 499 * i,
env=NULL)
}))
prices <- do.call(cbind, c(prices, list(btc)))
Edit: Here's a more complete example
library(quantmod)
# Use tryCatch() in case we try to get data too far in the past that
# Oanda doesn't provide. Return NULL if there is an error, and Filter
# to only include data that has at least 1 row.
btc <- do.call(rbind, Filter(NROW, lapply(0:5, function(i) {
tryCatch(getFX("BTC/USD",
from = Sys.Date() -499 * (i + 1),
to = Sys.Date() - 499 * i,
env=NULL), error=function(e) NULL)
})))
symbols <- c(
"UUP",
"FXB",
"FXE",
"FXF",
"FXY",
"SLV",
"GLD"
)
e <- new.env()
getSymbols(symbols, from=start(btc), env=e)
prices <- do.call(cbind, c(eapply(e, Cl)[symbols], list(btc)))
colnames(prices) <- gsub("\\.[A-z]*", "", colnames(prices))
head(na.locf(prices)[endpoints(prices, "months")])
# UUP FXB FXE FXF FXY SLV GLD BTC
#2010-07-31 23.74 156.15 129.88 95.38 114.60 17.58 115.49 0.06386
#2010-08-31 24.12 152.60 126.25 97.80 117.83 18.93 122.08 0.06441
#2010-09-30 22.84 156.33 135.81 101.00 118.57 21.31 127.91 0.06194
#2010-10-31 22.37 159.45 138.69 100.81 122.93 24.17 132.62 0.18530
#2010-11-30 23.50 154.72 129.30 98.87 118.16 27.44 135.42 0.27380
#2010-12-31 22.71 155.77 133.09 106.25 121.75 30.18 138.72 0.29190

Merging price time series in xts when securities show no data

I am querying a security DB. Price series are in xts and for some there might be no data (for the chosen window). Actual time series can be simulated as follows:
require(xts)
## Simulated time series
price=function(){
x=floor(runif(1,1,4))
xts(round(rnorm(x,5),3), Sys.Date()+1:x)
}
## Sample tickers
(tick1=setNames(price(), "tick1"))
# tick1
# 2014-04-20 5.829
# 2014-04-21 6.061
# 2014-04-22 5.813
(tick2=setNames(price(), "tick2"))
# tick2
# 2014-04-20 6.458
# 2014-04-21 5.373
(tick3=xts(data.frame(tick3=numeric()), as.Date(numeric()))) # Security showing no data
# tick3
## ...
## tickn
No need to mention that I don't know in advance which security will show no data.
If I merge the prices in a single xts object, merge.xts completely removes from the output the empty security(ies):
(port=merge(tick1, tick2, tick3))
# tick1 tick2
# 2014-04-20 5.829 6.458
# 2014-04-21 6.061 5.373
# 2014-04-22 5.813 NA
Instead I would like to keep trace of them, therefore printing an output similar to:
(cbind(port, tick3=NA))
# tick1 tick2 tick3
# 2014-04-20 5.829 6.458 NA
# 2014-04-21 6.061 5.373 NA
# 2014-04-22 5.813 NA NA
One possible solution is:
port=list(tick1, tick2, tick3) # ... tickn
port.m=lapply(port, function(sec){
if(nrow(sec)==0) sec= xts(matrix(NA, dimnames=dimnames(tick3)), Sys.Date())
sec
})
(port.m=do.call('merge', port.m))
# tick1 tick2 tick3
# 2014-04-19 NA NA NA
# 2014-04-20 5.829 6.458 NA
# 2014-04-21 6.061 5.373 NA
# 2014-04-22 5.813 NA NA
if(all(is.na(port.m[Sys.Date()])))
(port.m=port.m[time(port.m)!=Sys.Date()])
# tick1 tick2 tick3
# 2014-04-20 5.829 6.458 NA
# 2014-04-21 6.061 5.373 NA
# 2014-04-22 5.813 NA NA
Is it possible to find a smarter solution?
You are making two mistakes here:
First: You need to use a vector of non-zero length. See this:
length(integer())
length(NA)
Second: For merge to work, the xts object indices have to match some where.
e.g. something like this will work:
require(xts)
x=xts(1:4, Sys.Date()+1:4)
v=xts(NA, Sys.Date()+1)
(m=merge.xts(x,v))
Here the starting index matches, and remaining indices are filled up.
If you want to be very particular, you could probably try something like this:
v=xts(rep(NA,4), Sys.Date()+1:4)
Hope this helps!!

xts assignment changes column class

I have a data.frame earlyCloses defined as follows:
earlyCloses <- read.table('EarlyCloses.txt', header=T, colClasses= c(rep("character", 3)))
earlyCloses
StartDate EndDate EarlyClose
1 2012-12-24 2012-12-24 13:00
I define a xts object pricesXts as follows:
prices <- read.table('sample.txt', header=T, colClasses=c("character", "numeric"))
pricesXts = xts(prices$Close, as.POSIXct(prices$Date, tz='America/New_York'))
colnames(pricesXts) = c("Close")
pricesXts$CloseTime = NA
pricesXts
Close CloseTime
2012-12-21 13190.84 NA
2012-12-24 13139.08 NA
2012-12-26 13114.59 NA
2012-12-27 13096.31 NA
2012-12-28 12938.11 NA
Now I execute a for loop over the rows of earlyCloses and set the CloseTime of pricesXts.
for (i in 1:nrow(earlyCloses)) {
pricesXts[paste(earlyCloses[i,"StartDate"], earlyCloses[i,"EndDate"], sep='/'), 2] = earlyCloses[i,"EarlyClose"]
}
pricesXts
Close CloseTime
2012-12-21 "13190.84" NA
2012-12-24 "13139.08" "13:00"
2012-12-26 "13114.59" NA
2012-12-27 "13096.31" NA
2012-12-28 "12938.11" NA
Why has the class of the Close column in the xts object changed from numeric to character? Is this because an xts object is represented internally as a matrix? Is there a way to avoid this conversion?
xts is encoded internally as a matrix ( better performances). Since you want just to store the Early Close, you can convert it to a numeric , for example:
strptime(earlyCloses$EarlyClose,'%H:%M')$hour
Then
for (i in 1:nrow(earlyCloses))
pricesXts[paste(earlyCloses[i,"StartDate"],
earlyCloses[i,"EndDate"],
sep='/'), 2] <- strptime(earlyCloses$EarlyClose,'%H:%M')$hour
Close CloseTime
2012-12-21 13191 NA
2012-12-24 13139 13
2012-12-26 13115 NA
2012-12-27 13096 NA
2012-12-28 12938 NA

Resources