I have a data.frame earlyCloses defined as follows:
earlyCloses <- read.table('EarlyCloses.txt', header=T, colClasses= c(rep("character", 3)))
earlyCloses
StartDate EndDate EarlyClose
1 2012-12-24 2012-12-24 13:00
I define a xts object pricesXts as follows:
prices <- read.table('sample.txt', header=T, colClasses=c("character", "numeric"))
pricesXts = xts(prices$Close, as.POSIXct(prices$Date, tz='America/New_York'))
colnames(pricesXts) = c("Close")
pricesXts$CloseTime = NA
pricesXts
Close CloseTime
2012-12-21 13190.84 NA
2012-12-24 13139.08 NA
2012-12-26 13114.59 NA
2012-12-27 13096.31 NA
2012-12-28 12938.11 NA
Now I execute a for loop over the rows of earlyCloses and set the CloseTime of pricesXts.
for (i in 1:nrow(earlyCloses)) {
pricesXts[paste(earlyCloses[i,"StartDate"], earlyCloses[i,"EndDate"], sep='/'), 2] = earlyCloses[i,"EarlyClose"]
}
pricesXts
Close CloseTime
2012-12-21 "13190.84" NA
2012-12-24 "13139.08" "13:00"
2012-12-26 "13114.59" NA
2012-12-27 "13096.31" NA
2012-12-28 "12938.11" NA
Why has the class of the Close column in the xts object changed from numeric to character? Is this because an xts object is represented internally as a matrix? Is there a way to avoid this conversion?
xts is encoded internally as a matrix ( better performances). Since you want just to store the Early Close, you can convert it to a numeric , for example:
strptime(earlyCloses$EarlyClose,'%H:%M')$hour
Then
for (i in 1:nrow(earlyCloses))
pricesXts[paste(earlyCloses[i,"StartDate"],
earlyCloses[i,"EndDate"],
sep='/'), 2] <- strptime(earlyCloses$EarlyClose,'%H:%M')$hour
Close CloseTime
2012-12-21 13191 NA
2012-12-24 13139 13
2012-12-26 13115 NA
2012-12-27 13096 NA
2012-12-28 12938 NA
Related
when creating an xtsobject from a data.frame I seem to lose some data (approximately 3000 data lost over 33 000).
My dataset is as follow: (with the time being day-month-year, EU format)
> head(mesdonnees)
time value
1 05-03-2006 04:07 NA
2 05-03-2006 04:17 NA
3 05-03-2006 04:27 NA
4 05-03-2006 04:37 NA
5 05-03-2006 04:47 NA
6 05-03-2006 04:57 NA
Due to the format I had to extract the different parts of the date (at least I couldn't get as.POSIXct to work with this format).
Here is how I did it:
# Extract characters and define as S....
Syear <- substr(mesdonnees$time, 7,10)
Smonth <- substr(mesdonnees$time, 4,5)
Sday <- substr(mesdonnees$time, 1, 2)
#Gather all parts and use "-" as sep
datetext <- paste(Syear, Smonth, Sday, sep="-")
#define format of each part of the string
formatdate<-as.POSIXct(datetext, format="%Y-%m-%d", tz = "GMT")
I then try to create my xtswith...
xtsdata <- xts(mesdonnees$value, order.by = formatdate, tz = "GMT")
... but when doing this I get some quite weird results: the first value is in 1900
> head(xtsdata)
[,1]
1900-01-04 NA
2006-03-05 NA
2006-03-05 NA
2006-03-05 NA
2006-03-05 NA
2006-03-05 NA
and many (3000) dates are not kept:
> xtsdata[30225:30233,]
[,1]
2006-12-31 0
2006-12-31 0
2006-12-31 0
2006-12-31 0
<NA> NA
<NA> NA
<NA> NA
<NA> NA
<NA> NA
When looking at what should be the same line in both my data.frameand my xtsI can see that the lines are offset (I had the date format changed in the xts object creation):
> mesdonnees[25617,]
time value
25617 08-11-2006 23:51 0
> xtsdata[25617,]
[,1]
2006-11-25 0.27
How is it that my data are offset? I tried changing the tz but it doesn't affect it. I removed all duplicates using the dyplr package, it doesn't affect the xts results either. Thank you for your help !
After changing my xts code to the one suggested by Joshua:
xtsdata <- xts(mesdonnees$value, order.by = as.POSIXct(mesdonnees$time, tz = "GMT", format = "%d-%m-%Y %H:%M"))
... my data show properly for the "last" part, but I now have a different problem. The first 2300 data show the following results when doing (using xtsdata[1500,] (or any row < 2300) displays the same results)
> view(xtsdata):
0206-06-30 23:08:00 NA
0206-06-30 23:18:00 NA
0206-06-30 23:28:00 NA
1900-01-04 12:00:00 NA
2006-03-05 04:07:00 NA
2006-03-05 04:17:00 NA
I noticed this error before and thought it was due to the date format; maybe it is not? Also, when looking at the xtsdata I do not get the same results for the same row (the last rows are correct thought):
> mesdonnees[2360,]
time value
2360 23-03-2006 03:09 NA
> xtsdata[2360,]
[,1]
2006-03-05 09:07:00 NA
As requested:
> str(mesdonnees)
'data.frame': 32556 obs. of 2 variables:
$ time : chr "05-03-2006 04:07" "05-03-2006 04:17" "05-03-2006 04:27" "05-03-2006 04:37" ...
$ value: num NA NA NA NA NA NA NA NA NA NA ...
And if needed:
An ‘xts’ object on 0206-06-01 00:09:00/2006-12-31 23:29:00 containing:
Data: num [1:32556, 1] NA NA NA NA NA NA NA NA NA NA ...
Indexed by objects of class: [POSIXct,POSIXt] TZ: GMT
xts Attributes:
NULL
The problem is that you only include the date portion of the timestamp in datetext and formatdate, but your data have dates and times.
You also do not need to do all the string subsetting. You can achive the same result by specifying the format argument in your as.POSIXct call.
xtsdata <- xts(mesdonnees$value,
as.POSIXct(mesdonnees$times, "GMT", format = "%d-%m-%Y %H:%M")
I am relatively new to R and have read as much as I have been able to on the topic but can't seem to find what I am looking for on the other questions.
I am looking to calculate the rate of change (momentum) using TTR and ROC for 12 periods, using monthly data but I would like to ignore the most recent month. In other words I am looking to find the ROC for t-2 till t-12 (January 2016 12 month momentum of a stock excluding January 2016). This is the norm in calculating momentum in portfolio construction literature.
My data is all stocks that have been listed on the South African stock exchange (JSE). The date header is the 1st column (i.e. the date is the variable in the rows) and the stocks are listed in the subsequent columns.
I know my code below is pretty straightforward however I have tried a few things and they have given errors. As I have about 250 stocks (columns) over 20 years, its not advisable to create a new lagged variable for each observation.
x <- Prices.df
x$DATE <- as.Date(x$DATE, format = "%Y/%m/%d")
y <- xts(x[,-1], order.by = x$DATE) library(TTR)
roc <- ROC(y, n = 12, type = "discrete")
Any help would be much appreciated.
Just use lag of roc. The following code works with simulated data (18 periods and 2 assets):
set.seed(123) # to reproduze the same results
x <- data.frame(matrix(rnorm(18*2,100,2),ncol=2))
x$DATE <- seq.Date(as.Date("2000/01/01"),length.out = 18,by="1 month")
x <- x[,c(3,1,2)]
library(TTR)
library(xts)
y <- xts(x[,-1], order.by = x$DATE)
roc <- ROC(y, n = 12, type = "discrete")
cbind(y,lag(roc))
X1 X2 X1.1 X2.1
1999-12-31 98.87905 101.40271 NA NA
2000-01-31 99.53965 99.05442 NA NA
2000-02-29 103.11742 97.86435 NA NA
2000-03-31 100.14102 99.56405 NA NA
2000-04-30 100.25858 97.94799 NA NA
2000-05-31 103.43013 98.54222 NA NA
2000-06-30 100.92183 98.74992 NA NA
2000-07-31 97.46988 96.62661 NA NA
2000-08-31 98.62629 101.67557 NA NA
2000-09-30 99.10868 100.30675 NA NA
2000-10-31 102.44816 97.72373 NA NA
2000-11-30 100.71963 102.50763 NA NA
2000-12-31 100.80154 100.85293 NA NA
2001-01-31 100.22137 99.40986 0.019442887 -0.005421782
2001-02-28 98.88832 101.79025 0.006848733 0.003588329
2001-03-31 103.57383 101.75627 -0.041012460 0.040115718
2001-04-30 100.99570 101.64316 0.034279755 0.022018156
2001-05-31 96.06677 101.37728 0.007352244 0.037725848
I have a large time-series file that I imported from my working directory and then turn them into log returns by:
read.csv("/Volumes/3TB/ALLsince1996.csv",header=T)-> ALL
all <- xts(ALL[,2:dim(ALL)[2]], order.by= as.POSIXct(ALL[,1], format="%m/%d/%y"))
RETS <- CalculateReturns(all, method= c("log"))
RETS<- na.locf(RETS)
RETS[is.na(RETS)] <- 0
I then download the 3-Month Treaury via FRED by:
# 3-Mo Treasury
data <- new.env()
FEDs <- c( "DGS3MO") # DGS3MO : 3-Mo Treasury Constant maturity
getSymbols( FEDs
, src = "FRED"
, env = data
)
data$DGS3MO -> TB3
TB3/100/365 -> TB3
na.locf(TB3["1996-01-01::"])-> TB3
I then try to combine the log returns series with the 3-month treasury using cbind() and get the following:
both <- cbind(RETS[,1], TB3)
both:
row.names ZX.Adjusted DGS3MO
1 1995-12-31 16:00:00 NA NA
2 1996-01-01 00:00:00 0 NA
3 1996-01-01 16:00:00 NA 0.0001424658
4 1996-01-02 00:00:00 0 NA
5 1996-01-02 16:00:00 NA 0.0001424658
6 1996-01-03 00:00:00 0 NA
7 1996-01-03 16:00:00 NA 0.0001421918
8 1996-01-04 00:00:00 0 NA
9 1996-01-04 16:00:00 NA 0.0001421918
But this returns a vector with two-times per day; such as 1996-01-01 00:00:00 and 1996-01-01 16:00:00 . What I would like is to combine the two by date not by time.
REPRODUCIBLE DATA:
#Pull Data from getSymbols()
library(quantmod)
dataset<- xts()
symbols <- c( "GLD", "IWM", "SPY", "GS")
system.time(
for(i in 1:length(symbols)) {
symbols[i]-> symbol
tryit <- try(getSymbols(symbol, from="1995-12-31", src='yahoo'))
if(inherits(tryit, "try-error")){
i <- i+1
} else {
data <- getSymbols(symbol, from="1995-12-31", src='yahoo')
dataset <- merge(dataset, Ad(get(symbols[i])))
rm(symbol)
}
}
)
Because it was a large file I saved dataset and index(dataset) in two separate files as I could not save the index with the dataset
write.csv(dataset, "dataset.csv")
write.csv(index(dataset), "index.csv")
I later opened the index.csv file in Excel & manually pasted the index to dataset.csv & saved the file.I later tried to reopen the .csv unto my workspace & calculate log returns
read.csv("dataset.csv",header=T)-> ALL
all <- xts(ALL[,2:dim(ALL)[2]], order.by= as.POSIXct(ALL[,1], format="%m/%d/%y"))
RETS <- CalculateReturns(all, method= c("log"))
RETS<- na.locf(RETS)
RETS[is.na(RETS)] <- 0
Next Download the 3-Month T-Bill, same code as above...
# 3-Mo Treasury
data <- new.env()
FEDs <- c( "DGS3MO") # DGS3MO : 3-Mo Treasury Constant maturity
getSymbols( FEDs
, src = "FRED"
, env = data
)
data$DGS3MO -> TB3
TB3/100/365 -> TB3
na.locf(TB3["1996-01-01::"])-> TB3
Now try to combine the RETS1 with TB3...
both <- cbind(RETS1, TB3)
#Rime, to reformat the index without the time information use the strptime function and later merge the series as suggested above.
index(dataset) <- strptime(index(dataset),"%Y-%m-%d")
A much easier and more elegant way to accomplish what you are trying to do is to use the makeReturnFrame function using the fantastic qmao-package (https://r-forge.r-project.org/R/?group_id=1113) with a lot of utility and helper function for this kind of stuff.
library(quantmod)
library(qmao)
symbols <- c( "GLD", "IWM", "SPY", "GS")
getSymbols(symbols, from="1995-12-31", src='yahoo')
rets <- makeReturnFrame(symbols,silent = TRUE)
FEDs <- c( "DGS3MO") # DGS3MO : 3-Mo Treasury Constant maturity
data <- new.env()
getSymbols( FEDs
, src = "FRED"
, env = data
)
data$DGS3MO -> TB3
TB3/100/365 -> TB3
na.locf(TB3["1996-01-01::"])-> TB3
series.merged <- merge(rets,TB3,join = "inner")
> tail(series.merged)
GLD IWM SPY GS DGS3MO
2014-08-07 4.050035e-03 -0.004844797 -0.005429405 -0.0037775986 8.219178e-07
2014-08-08 7.924872e-05 0.009666235 0.011502456 0.0185147075 8.219178e-07
2014-08-11 -1.824311e-03 0.009485466 0.002893760 0.0011603622 1.095890e-06
2014-08-12 2.381425e-04 -0.006905738 -0.001394160 -0.0007540822 8.219178e-07
2014-08-13 1.665411e-03 0.007787650 0.006746170 0.0002320859 1.095890e-06
2014-08-14 8.712527e-04 0.001497468 0.004710710 0.0020863525 1.095890e-06
The time component has to removed from the indexes of both datasets for proper merge across date index
#Read stock returns data
require(quantmod)
data_agg<- xts()
symbols <- c( "GLD", "IWM", "SPY", "GS")
for(i in 1:length(symbols)) {
symbols[i]->symbol #assign
tryit <- try(getSymbols(symbol, from="1995-12-31", src='yahoo'))
if(inherits(tryit, "try-error")){
i <- i+1
} else {
data <- getSymbols(symbol, from="1995-12-31", src='yahoo')
data_agg <- merge(data_agg, Ad(get(symbols[i])))
rm(symbol)
}
}
head(data_agg)
# GLD.Adjusted IWM.Adjusted SPY.Adjusted GS.Adjusted
#1996-01-02 05:30:00 NA NA 44.91 NA
#1996-01-03 05:30:00 NA NA 45.03 NA
#1996-01-04 05:30:00 NA NA 44.60 NA
#1996-01-05 05:30:00 NA NA 44.51 NA
#1996-01-08 05:30:00 NA NA 44.68 NA
#1996-01-09 05:30:00 NA NA 43.91 NA
#Read interest rate data
data <- new.env()
FEDs <- c( "DGS3MO") # DGS3MO : 3-Mo Treasury Constant maturity
getSymbols( FEDs
, src = "FRED"
, env = data
)
TB3 <- data$DGS3MO
TB3 <- TB3/100/365
TB3 <- na.locf(TB3["1996-01-01::"])
head(TB3)
# DGS3MO
#1996-01-01 NA
#1996-01-02 0.0001424658
#1996-01-03 0.0001424658
#1996-01-04 0.0001421918
#1996-01-05 0.0001421918
#1996-01-08 0.0001419178
cbind is not advisable for merging purpose as it places the inputs side-by-side by coercing to same format.
merge.xts is the method for merging across xts objects
#The index of the datasets contain time component hence the doubled rows here
head(merge.xts(data_agg,TB3))
# GLD.Adjusted IWM.Adjusted SPY.Adjusted GS.Adjusted DGS3MO
#1996-01-01 NA NA NA NA NA
#1996-01-01 NA NA 44.91 NA NA
#1996-01-02 NA NA NA NA 0.0001424658
#1996-01-02 NA NA 45.03 NA NA
#1996-01-03 NA NA NA NA 0.0001424658
#1996-01-03 NA NA 44.60 NA NA
head(as.Date(index(data_agg)))
#[1] "1996-01-02" "1996-01-03" "1996-01-04" "1996-01-05" "1996-01-08" "1996-01-09"
#Only one observation per day, since there are no duplicate dates
#We can safely strip the time component from dates
any(duplicated(as.Date(index(data_agg))))
#[1] FALSE
any(duplicated(as.Date(index(TB3))))
#[1] FALSE
#Keep only the date component across both datasets
index(data_agg)<-as.Date(index(data_agg))
index(TB3)<-as.Date(index(TB3))
#Properly merged datasets across date index
head(merge.xts(data_agg,TB3))
# GLD.Adjusted IWM.Adjusted SPY.Adjusted GS.Adjusted DGS3MO
#1995-12-31 NA NA 44.91 NA NA
#1996-01-01 NA NA 45.03 NA 0.0001424658
#1996-01-02 NA NA 44.60 NA 0.0001424658
#1996-01-03 NA NA 44.51 NA 0.0001421918
#1996-01-04 NA NA NA NA 0.0001421918
#1996-01-06 NA NA 44.68 NA NA
You can use complete.cases to keep non-missing returns only.
I have data for some dates with counts; the other dates, where the event I'm counting didn't occur, do not appear in this data set. In order to do some analysis, I'd like to create a data frame that includes those missing dates but with a count of 0. Here is what some data might look like:
mydates <- c("2013-10-01", "2013-10-04", "2013-10-05", "2013-10-08")
mycounts <- c(2,4,3,1)
df <- data.frame(mydates,mycounts)
I know how to create a vector with all the dates:
alldates <- seq.Date(as.Date("2013-10-01"), as.Date("2013-10-08"), "days")
What I want to do is check whether each item in alldates exists in df$mydates; if it does, then use the corresponding count from the data frame in a new vector and if it doesn't, use 0 as the count in a new vector. But I'm not having much luck. For example, this
mycount <- ifelse(alldates %in% df$mydates, df$mycounts, 0)
gives me a vector but an inaccurate one.
Thanks for any help!
mydates <- c("2013-10-01", "2013-10-04", "2013-10-05", "2013-10-08")
mycounts <- c(2,4,3,1)
df <- data.frame(mydates,mycounts)
alldates <- data.frame(
mydates = seq.Date(as.Date("2013-10-01"), as.Date("2013-10-08"), "days")
)
merge(
alldates,
df,
all = TRUE
)
Output -
mydates mycounts
1 2013-10-01 NA
2 2013-10-01 2
3 2013-10-02 NA
4 2013-10-03 NA
5 2013-10-04 NA
6 2013-10-04 4
7 2013-10-05 NA
8 2013-10-05 3
9 2013-10-06 NA
10 2013-10-07 NA
11 2013-10-08 NA
12 2013-10-08 1
I wonder how I can make use of an already existing date field when creating a ts in R.
Sometimes you simply have a date before you have a ts object, e.g.
x <- as.Date("2008-01-01") + c(30,60,90,120,150)
# add some data to it
df = data.frame(datefield=x,test=1:length(x))
Now, is there a way to use the datefield of the df to as an index when creating a ts object? Because:
ts(df$test,start=c(2008,1,2),frequency=12)
(obviuously) completely ignores the date information I already have. Making use of ts methods like acf is the reason why I´d like to make it a ts object. I typcically use monthly an quarterly time series...
You don't necessarily need to create new types of objects from scratch; you can always coerce to other classes, including ts as you need to. zoo or xts are arguably to most useful and intuitive but there are others. Here is your example, cast as a zoo object, which we then coerce to class ts for use in acf().
## create the data
x <- as.Date("2008-01-01") + c(30,60,90,120,150)
df = data.frame(datefield=x,test=1:length(x))
## load zoo
require(zoo)
## convert to a zoo object, with order given by the `datefield`
df.zoo <- with(df, zoo(test, order.by = x))
## or to a regular zoo object
df.zoo2 <- with(df, zooreg(test, order.by = x))
Now we can easily go to a ts object using the as.ts() method:
> as.ts(df.zoo)
Time Series:
Start = 13920
End = 14040
Frequency = 0.0333333333333333
[1] 1 2 3 4 5
> ## zooreg object:
> as.ts(df.zoo2)
Time Series:
Start = 13909
End = 14029
Frequency = 1
[1] 1 NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
[21] NA NA NA NA NA NA NA NA NA NA 2 NA NA NA NA NA NA NA NA NA
[41] NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
[61] 3 NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
[81] NA NA NA NA NA NA NA NA NA NA 4 NA NA NA NA NA NA NA NA NA
[101] NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
[121] 5
Notice the two ways in which the objects are represented (although we could have made the zooreg version the same as the standard zoo object by setting the frequency argument to 0.03333333):
> as.ts(with(df, zooreg(test, order.by = datefield,
+ frequency = 0.033333333333333)))
Time Series:
Start = 13920.0000000001
End = 14040.0000000001
Frequency = 0.033333333333333
[1] 1 2 3 4 5
We can use the zoo/zooreg object in acf() and it will get the correct lags (daily observations but every 30 days):
acf(df.zoo)
Whether this is intuitive to you or not depends on how you view the time series. We can do the same thing in terms of a 30-day interval via:
acf(coredata(df.zoo))
where we use coredata() to extract the time series itself, ignoring the date information.
I don't know exactly what you're trying to do, but acf also works on simple vectors, given off course it represents a regular time series (i.e. even spaced). Otherwise the result is just bollocks.
>acf(df$test)
Regarding the ts object :
The "dates" you see are just from the print.ts function, so they're not inherent to the ts object. The ts object has no date information in it. You can set the option calender=FALSE to get the standard print out of the ts object.
> ts(df$test,start=2008,frequency=12)
Jan Feb Mar Apr May
2008 1 2 3 4 5
> print(ts(df$test,start=2008,frequency=12),calendar=F)
Time Series:
Start = c(2008, 1)
End = c(2008, 5)
Frequency = 12
[1] 1 2 3 4 5
Now, the vector you construct looks like :
> x
[1] "2008-01-31" "2008-03-01" "2008-03-31" "2008-04-30" "2008-05-30"
which is or isn't regular, depending on how you see it. If you extract the months, then you have 1 observation for january, 2 for march, 1 for april...: not regular. You have an observation every 30 days : regular. If you have an observation every 30 days, you shouldn't bother about the dates as 365 is not dividable through 30. Hence, one year you'll have 12 observations, another one you'll have 13 observations. So you can't set the frequency in ts in a consequent correct way.
So I'd refrain from using a ts all together, as James already indicated in the comments.
If you want:
Use the date information you already have
Easily set the frequency to a desired value
End up with a ts object
You can start with an xts object, add a frequency attribute, and then convert to ts:
library("xts")
my_xts <- xts(df$test, df$datefield)
attr(my_xts, 'frequency') <- 12 # Set the frequency
my_ts <- as.ts(my_xts)
The resulting ts object will have the specified period and will have the correct dates assigned to each data point.