I have data where I have the dates in YYYY-MM-DD format in one column and another column is num.
packages:
library(forecast)
library(ggplot2)
library(readr)
Running str(my_data) produces the following:
spec_tbl_df [261 x 2] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
$ date : Date[1:261], format: "2017-01-01" "2017-01-08" ...
$ popularity: num [1:261] 100 81 79 75 80 80 71 85 79 81 ...
- attr(*, "spec")=
.. cols(
.. date = col_date(format = ""),
.. popularity = col_double()
.. )
- attr(*, "problems")=<externalptr>
I would like to do some time series analysis on this. When running the first line of code for this decomp <- stl(log(my_data), s.window="periodic")
I keep running into the following error:
Error in Math.data.frame(my_data) :
non-numeric-alike variable(s) in data frame: date
Originally my date format was in MM/DD/YYYY format, so I feel like I'm... barely closer. I'm learning R again, but it's been a while since I took a formal course in it. I did a precursory search here, but could not find anything that I could identify as helpful (I'm just an amateur.)
You currently have a data.frame (or tibble variant thereof). That is not yet time aware. You can do things like
library(ggplot2)
ggplot(data=df) + aes(x=date, y=popularity) + geom_line()
to get a basic line plot properly index by date.
You will have to look more closely at package forecast and the examples of functions you want to use to predict or model. Packages like xts can help you, i.e.
library(xts)
x <- xts(df$popularity, order.by=df$date)
plot(x) # plot xts object
besides plotting you get time- and date aware lags and leads and subsetting. The rest depends more on what you want to do ... which you have not told us much about.
Lastly, if you wanted to convert your dates to numbers (since Jan 1, 1970) a quick as.numeric(df$date)) will; but using time-aware operations is often better (but has the learning curve you see now...)
Related
I'm trying to model the impact of storms on sales patterns using the CausalImpact package. When I create a zoo object and pass it to the model I receive an error. I've read through the documentation and can't figure out what I'm doing differently from the examples in the documentation.
I'm working with the following data.frame:
> head(my.data)
date sales units
1 2014-10-17 71319.85 21436.64
2 2014-10-18 88598.26 26755.79
3 2014-10-19 95768.29 29823.86
4 2014-10-20 62303.04 19417.71
5 2014-10-21 56477.57 17562.21
6 2014-10-22 54890.39 16946.43
Then I'm converting it to a zoo object:
my.data<- zoo( my.data[ ,c('sales','units')], my.data[,'date'] )
> str(my.data)
‘zoo’ series from 2014-10-17 to 2017-04-13
Data: num [1:907, 1:2] 71320 88598 95768 62303 56478 ...
- attr(*, "dimnames")=List of 2
..$ : NULL
..$ : chr [1:2] "sales" "units"
Index: Date[1:907], format: "2014-10-17" "2014-10-18" "2014-10-19" ...
Then I set the pre and post periods and run the model:
pre.period <- as.Date(c('2015-10-17','2017-03-09'))
post.period <- as.Date(c('2017-03-10','2017-04-13'))
library(CausalImpact)
impact<- CausalImpact(data = my.data, pre.period = pre.period, post.period = post.period, alpha = .01)
But I'm receiving this error:
> impact<- CausalImpact(data = my.data, pre.period = pre.period, post.period = post.period, alpha = .05)
Error in bsts(formula, data = data, state.specification = ss, expected.model.size = kStaticRegressionExpectedModelSize, :
Caught exception with the following error message:
BregVsSampler did not start with a legal configuration.
Selector vector: 11
beta: 0 0
I've used this package successfully with univariate time series data, but cant identify why this isn't working.
Thank you for your help!
I ran into the same exact issue, after applying recent package updates (including CausalImpact). Everything was working fine previously.
While I don't have the exact cause/solution, I have discovered something that may help you.
In my data, I tried simply replacing the dates in the zoo object with a test sequence. So in your case it would be something like:
time.pts <- seq.Date(as.Date("2014-10-17"), by = 1, length.out = 907)
my.data<- zoo( my.data[ ,c('sales','units')], time.pts )
After doing this, the "BregVsSampler" exception did not occur. So I figured the issue must be related to the dates, and then put my original date series back into the zoo object. I then noticed that I had a gap between pre.period and post.period, i.e. see the gap between 3/9 and 3/20 below:
pre.period <- as.Date(c('2015-10-17','2017-03-09'))
post.period <- as.Date(c('2017-03-20','2017-04-13'))
When I adjusted the pre/post periods to remove the gap in dates, the problem again went away.
While you don't seem to have such a gap in the code you show above, you may want to look at your date series for any inconsistencies and/or try a different date range. Obviously there is a bug somewhere that needs to get fixed, but perhaps the above info will help you work around the issue in the interim.
I have the data frame with prices and the ending date of some auctions. I want to check when appears, for example, sales with minimal and maximal prices (also the median) depending on the time of the day.
More precisely, I have the data frame mtest:
> str(mtest)
'data.frame': 9144 obs. of 2 variables:
$ Price : num 178 188 228 305 202 ...
$ EndDateTime: POSIXct, format: "2015-05-25 05:00:59" "2015-05-23 00:06:01" ...
I want to build the graph(plot), having 30 minutes time internals (00:00-00:30, 00:31-01:00 etc) on the X axis, and median (maximal, minimal prices) on Y axis.
Another idea is to draw a simple histogram for each time interval, like hist(mtest$Price, breaks=10, col="red")
How can I do this in the best way?
Try this:
cutt=seq(from=min(mtest$EndDateTime),to=max(mtest$EndDateTime), by=30*60)
if (max(mtest$EndDateTime)>max(cutt))cutt[length(cutt)+1]=max(cutt)+30*60
mtest$tint=cut(mtest$EndDateTime,cutt)
stats=do.call(rbind,tapply(mtest[,"Price"],mtest[,"tint"],
function(p)c(min=min(p),median=median(p),max=max(p))))
bp=boxplot(mtest[,"Price"]~mtest[,"tint"],xaxt="n",
col=1:length(levels(mtest$tint)))
axis(1,at=1:length(levels(mtest$tint)),labels=format.Date(levels(mtest$tint),"%Y-%m-%d %H:%M"),
las=2,cex.axis=.5)
stats
Or wilth plot
plot(NA,ylim=range(stats),xlim=c(1,lint),type="n",xaxt="n",xlab="",ylab="")
sapply(1:3,function(z)points(stats[,z]~c(1:lint),col=z))
axis(1,at=1:lint,labels=format.Date(levels(mtest$tint),"%Y-%m-%d %H:%M"),
las=2,cex.axis=.5)
You will have something like this:
How to select a value from time series corresponding needed date?
I create a monthly time series object with command:
producers.price <- ts(producers.price, start=2012+0/12, frequency=12)
Then I try to do next:
value <- producers.price[as.Date("01.2015", "%m.%Y")]
But this doesn't make that I want and value is equal
[1] NA
Instead of 10396.8212805739 if producers.price is:
producers.price <- structure(c(7481.52109434237, 6393.18959031561, 6416.63065650718,
5672.08354710121, 7606.24186413516, 5201.59247092013, 6488.18361474813,
8376.39182893415, 9199.50916585545, 8261.87133079494, 8293.8195347453,
8233.13630279516, 7883.17272003961, 7537.21001580393, 6566.60260432381,
7119.99345843556, 8086.40101607729, 9125.11104610046, 10134.0228610828,
10834.5732454454, 9410.35031874371, 9559.36933274129, 9952.38679679724,
10390.3628690951, 11134.8432864557, 11652.0075507499, 12626.9616107684,
12140.6698452193, 11336.8315981684, 10526.0309052316, 10632.1492109584,
8341.26367412737, 9338.95688558448, 9732.80173656971, 10724.5525831506,
11272.2273444623, 10396.8212805739, 10626.8428853062, 11701.0802817581,
NA), .Tsp = c(2012, 2015.25, 12), class = "ts")
So, I had/have a similar problem and was looking all over to solve it. My solution is not as great as I'd have wanted it to be, but it works. I tried it out with your data and it seems to give the right result.
Explanation
Turns out in R time series data is really stored as a sequence, starting at 1, and not with yout T. Eg. If you have a time series that starts in 1950 and ends in 1960 with each data at one year interval, the Y at 1950 will be ts[1] and Y at 1960 will be ts[11].
Based on this logic you will need to subtract the date from the start of the data and add 1 to get the value at that point.
This code in R gives you the result you expect.
producers.price[((as.yearmon("2015-01")- as.yearmon("2012-01"))*12)+1]
If you need help in the time calculations, check this answer
You will need the zoo and lubridate packages
Get the difference between dates in terms of weeks, months, quarters, and years
Hope it helps :)
1) window.ts
The window.ts function is used to subset a "ts" time series by a time window. The window command produces a time series with one data point and the [[1]] makes it a straight numeric value:
window(producers.price, start = 2015 + 0/12, end = 2015 + 0/12)[[1]]
## [1] 10396.82
2) zoo We can alternately convert it to zoo and subscript it by a yearmon class variable and then use [[1]] or coredata to convert it to a plain number or we can use window.zoo much as we did with window.ts :
library(zoo)
as.zoo(producers.price)[as.yearmon("2015-01")][[1]]
## [1] 10396.82
coredata(as.zoo(producers.price)[as.yearmon("2015-01")])
## [1] 10396.82
window(as.zoo(producers.price), 2015 + 0/12 )[[1]]
## [1] 10396.82
coredata(window(as.zoo(producers.price), 2015 + 0/12 ))
## [1] 10396.82
3) xts The four lines in (2) also work if library(zoo) is replaced with library(xts) and as.zoo is replaced with as.xts.
Looking for a simple command, one line and no library needed?
You might try this.
as.numeric(window(producers.price, 2015.1, 2015.2))
I tried to read my data as zoo using read.zoo function, the data include 14436 columns and the first column is in date format (yyyy-mm-dd). My code is
HFs<-read.zoo("F:/Research/Drawdown analysis/Data analysis/HF return.csv",index=1,header=TRUE,format="%Y-%m-%d")
The result is I only read the first column into R as a date index, all other values are lost.
and my str(HFs) shows
‘zoo’ series from 1990-01-31 to 2010-02-28
Data: logi[1:242, 0 ]
- attr(*, "dimnames")=List of 2
..$ : NULL
..$ : NULL
Index: Date[1:242], format: "1990-01-31" "1990-02-28" "1990-03-31" "1990-04-30" "1990-05-31" "1990-06-30" ...
Could anyone help to figure out the correct way to read table as zoo into R?
Thanks
Wei
I am working with data, 1st two columns are dates, 3rd column is symbol, and 4th and 5th columns are prices.
So, I created a subset of the data as follows:
test.sub<-subset(test,V3=="GOOG",select=c(V1,V4)
and then I try to plot a time series chart using the following
as.ts(test.sub)
plot(test.sub)
well, it gives me a scatter plot - not what I was looking for.
so, I tried plot(test.sub[1],test.sub[2])
and now I get the following error:
Error in xy.coords(x, y, xlabel, ylabel, log) :
'x' and 'y' lengths differ
To make sure the no. of rows were same, I ran nrow(test.sub[1]) and nrow(test.sub[2]) and they both return equal rows, so as a newcomer to R, I am not sure what the fix is.
I also ran plot.ts(test.sub) and that works, but it doesn't show me the dates in the x-axis, which it was doing with plot(test.sub) and which is what I would like to see.
test.sub[1]
V1
1107 2011-Aug-24
1206 2011-Aug-25
1307 2011-Aug-26
1408 2011-Aug-29
1510 2011-Aug-30
1613 2011-Aug-31
1718 2011-Sep-01
1823 2011-Sep-02
1929 2011-Sep-06
2035 2011-Sep-07
2143 2011-Sep-08
2251 2011-Sep-09
2359 2011-Sep-13
2470 2011-Sep-14
2581 2011-Sep-15
2692 2011-Sep-16
2785 2011-Sep-19
2869 2011-Sep-20
2965 2011-Sep-21
3062 2011-Sep-22
3160 2011-Sep-23
3258 2011-Sep-26
3356 2011-Sep-27
3455 2011-Sep-28
3555 2011-Sep-29
3655 2011-Sep-30
3755 2011-Oct-03
3856 2011-Oct-04
3957 2011-Oct-05
4059 2011-Oct-06
4164 2011-Oct-07
4269 2011-Oct-10
4374 2011-Oct-11
4479 2011-Oct-12
4584 2011-Oct-13
4689 2011-Oct-14
str(test.sub)
'data.frame': 35 obs. of 2 variables:
$ V1:Class 'Date' num [1:35] NA NA NA NA NA NA NA NA NA NA ...
$ V4: num 0.475 0.452 0.423 0.418 0.403 ...
head(test.sub) V1 V4
1212 <NA> 0.474697
1313 <NA> 0.451907
1414 <NA> 0.423184
1516 <NA> 0.417709
1620 <NA> 0.402966
1725 <NA> 0.414264
Now that this is working, I'd like to add a 3rd variable to plot a 3d chart - any suggestions how I can do that. thx!
So I think there are a few things going on here that are worth talking through:
first, some example data:
test <- data.frame(End = Sys.Date()+1:5,
Start = Sys.Date()+0:4,
tck = rep("GOOG",5),
EndP= 1:5,
StartP= 0:4)
test.sub = subset(test, tck=="GOOG",select = c(End, EndP))
First, note that test and test.sub are both data frames, so calls like test.sub[1] don't really "mean" anything to R.** It's more R-ish to write test.sub[,1] by virtue of consistency with other R structures. If you compare the results of str(test.sub[1]) and str(test.sub[,1]) you'll see that R treats them slightly differently.
You said you typed:
as.ts(test.sub)
plot(test.sub)
I'd guess you have extensive experience with some sort of OO-language; and while R does have some OO flavor to it, it doesn't apply here. Rather than transforming test.sub to something of class ts, this just does the transformation and throws it away, then moves on to plot the data frame you started with. It's an easy fix though:
test.sub.ts <- as.ts(test.sub)
plot(test.sub.ts)
But, this probably isn't what you were looking for either. Rather, R creates a time series that has two variables called "End" (which is the date now coerced to an integer) and "EndP". Funny business like this is part of the reason time series packages like zoo and xts have caught on so I'll detail them instead a little further down.
(Unfortunately, to the best of my understanding, R doesn't keep date stamps with its default ts class, choosing instead to keep start and end dates as well as a frequency. For more general time series work, this is rarely flexible enough)
You could perhaps get what you wanted by typing
plot(test.sub[,1], test.sub[,2])
instead of
plot(test.sub[1], test.sub[2])
since the former runs into trouble given that you are passing two sub-data frames instead of two vectors (even though it looks like you would be).*
Anyways, with xts (and similarly for zoo):
library(xts) # You may need to install this
xtemp <- xts(test.sub[,2], test.sub[,1]) # Create the xts object
plot(xtemp)
# Dispatches a xts plot method which does all sorts of nice time series things
Hope some of this helps and sorry for the inline code that's not identified as such: still getting used to stack overflow.
Michael
**In reality, they access the lists that are used to structure a data frame internally, but that's more a code nuance than something worth relying on.
***The nitty-gritty is that when you pass plot(test.sub[1], test.sub[2]) to R, it dispatches the method plot.data.frame which takes a single data frame and tries to interpret the second data frame as an additional plot parameter which gets misinterpreted somewhere way down the line, giving your error.
The reason that you get the Error about different x and y lengths is immediately apparent if you do a traceback immediately upon raising the error:
> plot(test.sub[1],test.sub[2])
Error in xy.coords(x, y, xlabel, ylabel, log) :
'x' and 'y' lengths differ
> traceback()
6: stop("'x' and 'y' lengths differ")
5: xy.coords(x, y, xlabel, ylabel, log)
4: plot.default(x1, ...)
3: plot(x1, ...)
2: plot.data.frame(test.sub[1], test.sub[2])
1: plot(test.sub[1], test.sub[2])
The problems in your call are manifold. First, as mentioned by #mweylandt test.sub[1] is a data frame with the single component, not a vector comprised of the contents of the first component of test.sub.
From the traceback, we see that the plot.data.frame method was called. R is quite happy to plot a data frame as long as it has at least two columns. R took you at your word and passed test.sub[1] (as a data.frame) on to plot() - test.sub[2] never gets a look in. test.sub[1] is eventually passed on to xy.coords() which correctly informs you that you have lots of rows for x but 0 rows for y because test.sub[1] only contains a single component.
It would have worked if you'd done plot(test.sub[,1], test.sub[,2], type = "l") or used the formula interface to name the variables plot(V4 ~ V1, data = test.sub, type = "l") as I show in my other Answer.
Surely it is easier to use the formula interface:
> test <- data.frame(End = Sys.Date()+1:5,
+ Start = Sys.Date()+0:4,
+ tck = rep("GOOG",5),
+ EndP= 1:5,
+ StartP= 0:4)
>
> test.sub = subset(test, tck=="GOOG",select = c(End, EndP))
> head(test.sub)
End EndP
1 2011-10-19 1
2 2011-10-20 2
3 2011-10-21 3
4 2011-10-22 4
5 2011-10-23 5
> plot(EndP ~ End, data = test.sub, type = "l")
I work extensively with time series type data and rarely, if ever, have any need for the "ts" class of objects. Packages zoo and xts are very useful, but if all you want to do is plot the data, i) get the date/time information correctly formatted/set-up as a "Date" or "POSIXt" class object, and then ii) just plot it using standard graphics and type = "l" (or type = "b" or type = "o" if you want to see the observation times).