I have a table in R like:
start duration
02/01/2012 20:00:00 5
05/01/2012 07:00:00 6
etc... etc...
I got to this by importing a table from Microsoft Excel that looked like this:
date time duration
2012/02/01 20:00:00 5
etc...
I then merged the date and time columns by running the following code:
d.f <- within(d.f, { start=format(as.POSIXct(paste(date, time)), "%m/%d/%Y %H:%M:%S") })
I want to create a third column called 'end', which will be calculated as the number of hours after the start time. I am pretty sure that my time is a POSIXct vector. I have seen how to manipulate one datetime object, but how can I do that for the entire column?
The expected result should look like:
start duration end
02/01/2012 20:00:00 5 02/02/2012 01:00:00
05/01/2012 07:00:00 6 05/01/2012 13:00:00
etc... etc... etc...
Using lubridate
> library(lubridate)
> df$start <- mdy_hms(df$start)
> df$end <- df$start + hours(df$duration)
> df
# start duration end
#1 2012-02-01 20:00:00 5 2012-02-02 01:00:00
#2 2012-05-01 07:00:00 6 2012-05-01 13:00:00
data
df <- structure(list(start = c("02/01/2012 20:00:00", "05/01/2012 07:00:00"
), duration = 5:6), .Names = c("start", "duration"), class = "data.frame", row.names = c(NA,
-2L))
You can simply add dur*3600 to start column of the data frame. E.g. with one date:
start = as.POSIXct("02/01/2012 20:00:00",format="%m/%d/%Y %H:%M:%S")
start
[1] "2012-02-01 20:00:00 CST"
start + 5*3600
[1] "2012-02-02 01:00:00 CST"
Related
I have a column of dates in an R data frame, that look like this,
Date
2020-08-05
2020-08-05
2020-08-05
2020-08-07
2020-08-08
2020-08-08
So the dates are formatted as 'yyyy-mm-dd'.
I am writing this data frame to a CSV that needs to be formatted in a very specific manner. I need to convert these dates to the format 'mm/dd/yyyy hh:mm:ss', so this is what I want the columns to look like:
Date
8/5/2020 12:00:00 AM
8/5/2020 12:00:00 AM
8/5/2020 12:00:00 AM
8/7/2020 12:00:00 AM
8/8/2020 12:00:00 AM
8/8/2020 12:00:00 AM
The dates do not have a timestamp attached to begin with, so all dates will need a midnight timestamp in the format shown above.
I spent quite some time trying to coerce this format yesterday and was unable. I am easily able to change 2020-08-05 to 8/5/2020 using as.Date(), but the issue arises when I attempt to add the midnight time stamp.
How can I add a midnight timestamp to these reformatted dates?
Thanks so much for any help!
You can use format:
df <- data.frame(Date = as.Date(c("2020-08-05", "2020-08-07")))
format(df$Date, "%d-%m-%Y 12:00:00 AM")
[1] "05-08-2020 12:00:00 AM" "07-08-2020 12:00:00 AM"
dat <- data.frame(
Date = as.Date("2020-08-05") + c(0, 0, 0, 2, 3, 3)
)
dat[["Date"]] <- format(dat[["Date"]], "%m/%d/%Y %I:%M:%S %p")
dat[["Date"]] <- sub("([ap]m)$", "\\U\\1", dat[["Date"]], perl = T)
dat
## Date
## 1 08/05/2020 12:00:00 AM
## 2 08/05/2020 12:00:00 AM
## 3 08/05/2020 12:00:00 AM
## 4 08/07/2020 12:00:00 AM
## 5 08/08/2020 12:00:00 AM
## 6 08/08/2020 12:00:00 AM
Try this:
format(as.POSIXct("2022-11-08", tz = "Australia/Sydney"), "%Y-%m-%d %H:%M:%S")
I was trying to see if it is possible to set the start and end parameters of the ts() function in the forecast R package. The reason for this is to then use window() to subset a train and test set by date.
The time frame is from 2015-01-01 00:00:00 to 12/31/2017 23:00
index esti
2015-01-01 00:00:00 1
2015-01-01 01:00:00 2
2015-01-01 02:00:00 3
2015-01-01 03:00:00 2
2015-01-01 04:00:00 5
2015-01-01 05:00:00 2
...
2017-12-31 18:00:00 0
2017-12-31 19:00:00 1
2017-12-31 20:00:00 0
2017-12-31 21:00:00 2
2017-12-31 22:00:00 0
2017-12-31 23:00:00 4
I used the following syntax to create the time series object:
tmp <- ts(dat, start = c(2015,1), frequency=24)
The returned object is this:
Time Series:
Start = c(2015, 1)
End = c(2015, 6)
Frequency = 24
It looks as if the ts object isn't correct here...
As far as I understand, the ts object does not work well with hourly input. It is recommended that you work with xts or zoo package instead. See this SO post.
Try the following:
## Creating an entire hourly dataframe similar to the example dat
x <-
lubridate::parse_date_time(
c("2015-01-01 00:00:00", "2017-12-31 23:00:00"),
orders = "ymdHMS"
)
y <- seq(x[1], x[2], by = "hour")
dat <- data.frame(
index = y, esti = sample(seq(0, 10), size = length(y),
replace = TRUE)
)
## xts package
library(xts)
tmp <- xts(dat, order.by = dat$index)
## Example window-ing
window(tmp, end = y[100])
Let me know if this does not work out.
How do you set 0:00 as end of day instead of 23:00 in an hourly data? I have this struggle while using period.apply or to.period as both return days ending at 23:00. Here is an example :
x1 = xts(seq(as.POSIXct("2018-02-01 00:00:00"), as.POSIXct("2018-02-05 23:00:00"), by="hour"), x = rnorm(120))
The following functions show periods ends at 23:00
to.period(x1, OHLC = FALSE, drop.date = FALSE, period = "days")
x1[endpoints(x1, 'days')]
So when I am aggregating the hourly data to daily, does someone have an idea how to set the end of day at 0:00?
As already pointed out by another answer here, to.period on days computes on the data with timestamps between 00:00:00 and 23:59:59.9999999 on the day in question. so 23:00:00 is seen as the last timestamp in your data, and 00:00:00 corresponds to a value in the next day "bin".
What you can do is shift all the timestamps back 1 hour, use to.period get the daily data points from the hour points, and then using align.time to get the timestamps aligned correctly.
(More generally, to.period is useful for generating OHLCV type data, and so if you're say generating say hourly bars from ticks, it makes sense to look at all the ticks between 23:00:00 and 23:59:59.99999 in the bar creation. then 00:00:00 to 00:59:59.9999.... would form the next hourly bar and so on.)
Here is an example:
> tail(x1["2018-02-01"])
# [,1]
# 2018-02-01 18:00:00 -1.2760349
# 2018-02-01 19:00:00 -0.1496041
# 2018-02-01 20:00:00 -0.5989614
# 2018-02-01 21:00:00 -0.9691905
# 2018-02-01 22:00:00 -0.2519618
# 2018-02-01 23:00:00 -1.6081656
> head(x1["2018-02-02"])
# [,1]
# 2018-02-02 00:00:00 -0.3373271
# 2018-02-02 01:00:00 0.8312698
# 2018-02-02 02:00:00 0.9321747
# 2018-02-02 03:00:00 0.6719425
# 2018-02-02 04:00:00 -0.5597391
# 2018-02-02 05:00:00 -0.9810128
> head(x1["2018-02-03"])
# [,1]
# 2018-02-03 00:00:00 2.3746424
# 2018-02-03 01:00:00 0.8536594
# 2018-02-03 02:00:00 -0.2467268
# 2018-02-03 03:00:00 -0.1316978
# 2018-02-03 04:00:00 0.3079848
# 2018-02-03 05:00:00 0.2445634
x2 <- x1
.index(x2) <- .index(x1) - 3600
> tail(x2["2018-02-01"])
# [,1]
# 2018-02-01 18:00:00 -0.1496041
# 2018-02-01 19:00:00 -0.5989614
# 2018-02-01 20:00:00 -0.9691905
# 2018-02-01 21:00:00 -0.2519618
# 2018-02-01 22:00:00 -1.6081656
# 2018-02-01 23:00:00 -0.3373271
x.d2 <- to.period(x2, OHLC = FALSE, drop.date = FALSE, period = "days")
> x.d2
# [,1]
# 2018-01-31 23:00:00 0.12516594
# 2018-02-01 23:00:00 -0.33732710
# 2018-02-02 23:00:00 2.37464235
# 2018-02-03 23:00:00 0.51797747
# 2018-02-04 23:00:00 0.08955208
# 2018-02-05 22:00:00 0.33067734
x.d2 <- align.time(x.d2, n = 86400)
> x.d2
# [,1]
# 2018-02-01 0.12516594
# 2018-02-02 -0.33732710
# 2018-02-03 2.37464235
# 2018-02-04 0.51797747
# 2018-02-05 0.08955208
# 2018-02-06 0.33067734
Want to convince yourself? Try something like this:
x3 <- rbind(x1, xts(x = matrix(c(1,2), nrow = 2), order.by = as.POSIXct(c("2018-02-01 23:59:59.999", "2018-02-02 00:00:00"))))
x3["2018-02-01 23/2018-02-02 01"]
# [,1]
# 2018-02-01 23:00:00.000 -1.6081656
# 2018-02-01 23:59:59.999 1.0000000
# 2018-02-02 00:00:00.000 -0.3373271
# 2018-02-02 00:00:00.000 2.0000000
# 2018-02-02 01:00:00.000 0.8312698
x3.d <- to.period(x3, OHLC = FALSE, drop.date = FALSE, period = "days")
> x3.d <- align.time(x3.d, 86400)
> x3.d
[,1]
2018-02-02 1.00000000
2018-02-03 -0.09832625
2018-02-04 -0.65075506
2018-02-05 -0.09423664
2018-02-06 0.33067734
See that the value of 2 on 00:00:00 did not form the last observation in the day for 2018-02-02 (00:00:00), which went from 2018-02-01 00:00:00 to 2018-02-01 23:59:59.9999.
Of course, if you want the daily timestamp to be the start of the day, not the end of the day, which would be 2018-02-01 as start of bar for the first row, in x3.d above, you could shift back the day by one. You could do this relatively safely for most timezones, when your data doesn't involve weekend dates:
index(x3.d) = index(x3.d) - 86400
I say relatively safetly, because there are corner cases when there are time shifts in a time zone. e.g. Be careful with day light savings. Simply subtracting -86400 can be a problem when going from Sunday to Saturday in time zones where day light saving occurs:
#e.g. bad: day light savings occurs on this weekend for US EST
z <- xts(x = 9, order.by = as.POSIXct("2018-03-12", tz = "America/New_York"))
> index(z) - 86400
[1] "2018-03-10 23:00:00 EST"
i.e. the timestamp is off by one hour, when you really want the midnight timestamp (00:00:00).
You could get around this problem using something much safer like this:
library(lubridate)
# right
> index(z) - days(1)
[1] "2018-03-11 EST"
I don't think this is possible because 00:00 is the start of the day. From the manual:
These endpoints are aligned in POSIXct time to the zero second of the day at the beginning, and the 59.9999th second of the 59th minute of the 23rd hour of the final day
I think the solution here is to use minutes instead of hours. Using your example:
x1 = xts(seq(as.POSIXct("2018-02-01 00:00:00"), as.POSIXct("2018-02-05 23:59:99"), by="min"), x = rnorm(7200))
to.period(x1, OHLC = FALSE, drop.date = FALSE, period = "day")
x1[endpoints(x1, 'day')]
I am new to R and I have a data frame with date time as variable. For every hour each day temperature is recorded, and date time is in format of YYYY-MM-DD 00:00:00.
Now I would like to convert the time into a factor ranging from 0 to 23 each day.
So For each day my new column should have factors 0 to 23. Could anyone help me with this? My 2015-01-01 00:00:00, should give me 0, while 2015-01-01 01:00:00, should give me 1 and so on. Also my 2015-01-02 00:00:00 should be 0 again.
You can convert your timestamp into a POSIXlt object. Once you have that, you can obtain the hour directly like this:
> timestamp <- as.POSIXlt("2015-01-01 00:00:00")
> timestamp
[1] "2015-01-01 MYT"
> timestamp$hour
[1] 0
Using a sample data, one way would be the following.
mydf <- data.frame(id = c(1,1,1,2,2,1,1),
event = c("start", "valid", "end", "start", "bad", "start", "bad"),
time = as.POSIXct(c("2015-05-16 20:46:53", "2015-05-16 20:46:56", "2015-05-16 21:46:59",
"2015-05-16 22:46:53", "2015-05-16 22:47:00", "2015-05-16 22:49:05",
"2015-05-16 23:49:09"), format = "%Y-%m-%d %H:%M:%S"),
stringsAsFactors = FALSE)
library(dplyr)
mutate(mydf, group = factor(format(time, "%H")))
# id event time group
#1 1 start 2015-05-16 20:46:53 20
#2 1 valid 2015-05-16 20:46:56 20
#3 1 end 2015-05-16 21:46:59 21
#4 2 start 2015-05-16 22:46:53 22
#5 2 bad 2015-05-16 22:47:00 22
#6 1 start 2015-05-16 22:49:05 22
#7 1 bad 2015-05-16 23:49:09 23
Tim's answer using POSIXlt is probably the best option, but here's a regex way just in case:
> times <- c("2015-01-01 00:00:00", "2015-01-01 01:00:00", "2015-01-02 00:00:00")
> regmatches(times, regexpr("(?<=-\\d{2} )\\d{2}", times, perl=TRUE))
[1] "00" "01" "00"
With the extracted hours you can make them factors or integers as necessary.
#Sairam, in addition to #jazzurro's use of 'dplyr' (which, like jazzurro, many R-insitas routinely use)...in the future, if you need/want a simple & powerful way to manipulate dates, you're encouraged to gain familiarity with another package: 'lubridate.'
lubridate makes working with dates a snap. Hope this helps and best regards on your project.
I have read in and formatted my data set like shown under.
library(xts)
#Read data from file
x <- read.csv("data.dat", header=F)
x[is.na(x)] <- c(0) #If empty fill in zero
#Construct data frames
rawdata.h <- data.frame(x[,2],x[,3],x[,4],x[,5],x[,6],x[,7],x[,8]) #Hourly data
rawdata.15min <- data.frame(x[,10]) #15 min data
#Convert time index to proper format
index.h <- as.POSIXct(strptime(x[,1], "%d.%m.%Y %H:%M"))
index.15min <- as.POSIXct(strptime(x[,9], "%d.%m.%Y %H:%M"))
#Set column names
names(rawdata.h) <- c("spot","RKup", "RKdown","RKcon","anm", "pp.stat","prod.h")
names(rawdata.15min) <- c("prod.15min")
#Convert data frames to time series objects
data.htemp <- xts(rawdata.h,order.by=index.h)
data.15mintemp <- xts(rawdata.15min,order.by=index.15min)
#Select desired subset period
data.h <- data.htemp["2013"]
data.15min <- data.15mintemp["2013"]
I want to be able to combine hourly data from data.h$prod.h with data, with 15 min resolution, from data.15min$prod.15min corresponding to the same hour.
An example would be to take the average of the hourly value at time 2013-12-01 00:00-01:00 with the last 15 minute value in that same hour, i.e. the 15 minute value from time 2013-12-01 00:45-01:00. I'm looking for a flexible way to do this with an arbitrary hour.
Any suggestions?
Edit: Just to clarify further: I want to do something like this:
N <- NROW(data.h$prod.h)
for (i in 1:N){
prod.average[i] <- mean(data.h$prod.h[i] + #INSERT CODE THAT FINDS LAST 15 MIN IN HOUR i )
}
I found a solution to my problem by converting the 15 minute data into hourly data using the very useful .index* function from the xts package like shown under.
prod.new <- data.15min$prod.15min[.indexmin(data.15min$prod.15min) %in% c(45:59)]
This creates a new time series with only the values occuring in the 45-59 minute interval each hour.
For those curious my data looked like this:
Original hourly series:
> data.h$prod.h[1:4]
2013-01-01 00:00:00 19.744
2013-01-01 01:00:00 27.866
2013-01-01 02:00:00 26.227
2013-01-01 03:00:00 16.013
Original 15 minute series:
> data.15min$prod.15min[1:4]
2013-09-30 00:00:00 16.4251
2013-09-30 00:15:00 18.4495
2013-09-30 00:30:00 7.2125
2013-09-30 00:45:00 12.1913
2013-09-30 01:00:00 12.4606
2013-09-30 01:15:00 12.7299
2013-09-30 01:30:00 12.9992
2013-09-30 01:45:00 26.7522
New series with only the last 15 minutes in each hour:
> prod.new[1:4]
2013-09-30 00:45:00 12.1913
2013-09-30 01:45:00 26.7522
2013-09-30 02:45:00 5.0332
2013-09-30 03:45:00 2.6974
Short answer
df %>%
group_by(t = cut(time, "30 min")) %>%
summarise(v = mean(value))
Long answer
Since, you want to compress the 15 minutes time series to a smaller resolution (30 minutes), you should use dplyr package or any other package that computes the "group by" concept.
For instance:
s = seq(as.POSIXct("2017-01-01"), as.POSIXct("2017-01-02"), "15 min")
df = data.frame(time = s, value=1:97)
df is a time series with 97 rows and two columns.
head(df)
time value
1 2017-01-01 00:00:00 1
2 2017-01-01 00:15:00 2
3 2017-01-01 00:30:00 3
4 2017-01-01 00:45:00 4
5 2017-01-01 01:00:00 5
6 2017-01-01 01:15:00 6
The cut.POSIXt, group_by and summarise functions do the work:
df %>%
group_by(t = cut(time, "30 min")) %>%
summarise(v = mean(value))
t v
1 2017-01-01 00:00:00 1.5
2 2017-01-01 00:30:00 3.5
3 2017-01-01 01:00:00 5.5
4 2017-01-01 01:30:00 7.5
5 2017-01-01 02:00:00 9.5
6 2017-01-01 02:30:00 11.5
A more robust way is to convert 15 minutes values into hourly values by taking average. Then do whatever operation you want to.
### 15 Minutes Data
min15 <- structure(list(V1 = structure(1:8, .Label = c("2013-01-01 00:00:00",
"2013-01-01 00:15:00", "2013-01-01 00:30:00", "2013-01-01 00:45:00",
"2013-01-01 01:00:00", "2013-01-01 01:15:00", "2013-01-01 01:30:00",
"2013-01-01 01:45:00"), class = "factor"), V2 = c(16.4251, 18.4495,
7.2125, 12.1913, 12.4606, 12.7299, 12.9992, 26.7522)), .Names = c("V1",
"V2"), class = "data.frame", row.names = c(NA, -8L))
min15
### Hourly Data
hourly <- structure(list(V1 = structure(1:4, .Label = c("2013-01-01 00:00:00",
"2013-01-01 01:00:00", "2013-01-01 02:00:00", "2013-01-01 03:00:00"
), class = "factor"), V2 = c(19.744, 27.866, 26.227, 16.013)), .Names = c("V1",
"V2"), class = "data.frame", row.names = c(NA, -4L))
hourly
### Convert 15min data into hourly data by taking average of 4 values
min15$V1 <- as.POSIXct(min15$V1,origin="1970-01-01 0:0:0")
min15 <- aggregate(. ~ cut(min15$V1,"60 min"),min15[setdiff(names(min15), "V1")],mean)
min15
names(min15) <- c("time","min15")
names(hourly) <- c("time","hourly")
### merge the corresponding values
combined <- merge(hourly,min15)
### average of hourly and 15min values
rowMeans(combined[,2:3])