I'm importing data from Excel and then trying to manipulate dates and times in R and it's giving me SUCH A HEADACHE. In the Excel file, one column contained a date and time, and another column contained a different time for that same day. The data in R looks basically like this example:
mydata <- data.frame(DateTime1 = as.POSIXct(c("2014-12-13 04:56:00",
"2014-12-13 09:30:00",
"2014-12-13 11:30:00",
"2014-12-13 13:30:00"),
origin = "1970-01-01", tz = "GMT"),
Time2 = c(0.209, 0.209, 0.715, 0.715))
I'd like to have a new column in POSIXct format with the date and the 2nd time, and I can't get that to work. I've tried:
mydata$DateTime2 <- as.POSIXct(as.numeric(as.Date(mydata$DateTime1)
+ mydata$Time2), origin = "1970-01-01",
tz = "GMT")
but that gives me dates and times close to 1/1/1970.
This is more convoluted, but one thing that has worked in other similiar situations that I've also tried is:
library(lubridate)
mydata$DateTime2 <- ymd_hms(format(as.POSIXct(as.Date(mydata$DateTime1) +
mydata$Time2,
origin = "1899-12-30", tz = "GMT")))
but that gives me dates and times that are off by 8 hours. That time difference makes me think that the problem is the time zone since I'm on Pacific Standard Time, but I set it to GMT both in the input data and when trying to convert! What gives? I'm hesitant to just add 8 hours to everything because of daylight savings time complications.
Really, both of the attempts I'm listing here seem to have problems with interconversion, i.e., if you start with a POSIXct object and convert it to numeric and then convert it back again to POSIXct, you should end up back where you started and you don't. Similarly, if you start with the time zone being GMT and then you add something that also is set to have the time zone as GMT, then you shouldn't have a problem with things somewhere mysteriously getting converted to the system time zone.
Advice?
I found an answer based on chris holbrook's answer here: How do you convert dates/times from one time zone to another in R?
This worked:
mydata$DateTime2 <- as.POSIXct(as.Date(mydata$DateTime1) +
mydata$Time2)
attributes(mydata$DateTime2)$tzone <- "GMT"
#MichaelChirico and I were correct that the time zone was the problem. I'm still not sure why, but the time zone for DateTime2 was apparently PST. It didn't list "PST" when I checked str(mydata$DateTime2), but based on the time difference, it must have been, in fact, PST until I set the attributes. Crazy. It did that even though DateTime1 was GMT.
I have a vector of decimal numbers, which represent 'decimal day', the fraction of a day. I want to convert it into HH:MM format using R.
For example, the number 0.8541667 would correspond to 20:30. How can I convert the numbers to HH:MM format?
Using chron:
chron::times(0.8541667)
#[1] 20:30:00
Try this:
R> format(as.POSIXct(Sys.Date() + 0.8541667), "%H:%M", tz="UTC")
[1] "20:30"
R>
We start with a date--which can be any date, so we use today--and add your desired fractional day.
We then convert the Date type into a Datetime object.
Finally, we format the hour and minute part of the Datetime object, ensuring that UTC is used for the timezone.
One option with data.table:
> library(data.table)
> structure(as.integer(0.4305556*60*60*24), class="ITime")
[1] "10:20:00"
We convert from day fraction to seconds since midnight; coerce to integer; and apply ITime class. (ITime works with integer-stored seconds since midnight.)
Other resources:
#GaborGrothendieck re chron package and link to his R News article with Thomas Petzoldt about converting from Excel in particular Converting a time decimal/fraction representing days to its actual time in R?
#JorisChau re RStudio's hms package how to convert excel internal coding for hours to hours in R?
Currently we are implementing some sort of direct marketing
We send an SMS (timestamp) on a certain day from a certain country within Europe and then receive a response from a customer in the local country if they are interested in the product
I have a sample data set for two months, of SMS's sent and Responses received (if at all). The end goal is , I want to calculate the time difference in hours between the sent time and the response time factoring in Time Zones and Day Light Savings
The file i received has the time stamps in character format. I wish to convert them to dates of their local timezones, convert them to a standard zone and calculate the hours difference
I have tried the methods below to just create the conversion but it comes back with NA.
Any help would be greatly appreciated
as.POSIXlt("12/02/2015 11:23", tz = "Europe/London")
strptime("12/02/2015 11:23", "%d/%m/%Y %h:%m")
as.Date("12/02/2015 11:23","%d/%m/%y %h:%m")
Note that you do not say if you are using 24-hour or 12-hour clock, but given that you do not have AM/PM, I am going to assume the former:
strptime("12/02/2015 11:23 AM", "%d/%m/%Y %I:%M %p", tz = "Europe/London") #12-hour
strptime("12/02/2015 11:23", "%d/%m/%Y %H:%M", tz = "Europe/London") # 24-hour
t1 <- strptime("2015-02-17 12:20:00", format = "%Y-%m-%d %H:%M:%S", tz = "America/Chicago")
t2 <- strptime("2015-03-13 15:00:00", format = "%Y-%m-%d %H:%M:%S", tz = "America/Chicago")
as.numeric(abs(difftime(t1, t2, units = "mins", tz = "America/Chicago")))
The above returns 34,660 while Excel and http://www.timeanddate.com/date/duration.html both return 34,720. R recognizes t1 as CST and t2 as CDT due to daylight savings on 3/8/15. I wanted to confirm that R is correct, while these other two sources are not.
Depends if you want to take into account local daylight savings, or if these times are two points in time irrelevant of local shifts. Excel and that site are just taking the exact difference between those two times, discarding daylight savings - e.g. R can get the same result by changing the timezones to UTC, which doesn't observe daylight saving:
difftime(as.POSIXct("2015-03-13 15:00:00",tz="UTC"),
as.POSIXct("2015-02-17 12:20:00",tz="UTC"), units="mins")
# Time difference of 34720 mins
If I'm sitting in Chicago with a stopwatch and counting the minutes difference between when a local clock in the pub says "2015-02-17 12:20:00" and then hits "2015-03-13 15:00:00", I'd count 60 minutes less as the clock got wound forward an hour for daylight savings while I was counting.
I have time series data from 1974-2013 with a column for datetimeUTC (YYYY-MM-DD hh:mm +0000), and a column for the timezones in Olson format (e.g., Canada/Pacific, Canada/Eastern). I can convert the whole UTCdatetime column to a common timezone like this:
dataset$datetimeEST <- strptime(
dataset$datetimeUTC, format="%Y-%m-%d %H:%M:%S%z", tz="Canada/Eastern"
)
How do I convert datetimeUTC to datetimeLOCAL, given the corresponding timezone in each row?
Let me back up a bit. I have data from across the country (6 timezones) formatted in ISO8601 representation for 1974-2013. The timestamps are in local standard time throughout the year (i.e. DST is disregarded even if civilian time in the region observes DST). I need to do datetime calculations which are probably safest to do in UTC time, so that's easy. But, I also need to pull data for specific civil time periods, taking into account DST, and do calculations and plots (e.g., all the data for rush hour at locations across all 6 timezones) for that subsetted data.
The datetimeCLOCKTIME that I calculated below appears to be doing what I want for plotting, but gives the wrong answer when doing datetime calculations because it stored the datetime in the timezone of my local machine without having actually converted the time. The solution offered by #thelatemail is what I'm looking for, but I haven't been able to get it to work in Windows on the test dataset for 2012 (see below). Also, I was using strptime which converts to POXITlt, and his solution is in POXITct. I'm new to R, so any help would be infinitely appreciated.
Test dataset:
dataset <- data.frame (timestampISO8601 = c("2012-04-25T22:00:00-08:00","2012-04-25T22:15:00-08:00","2012-04-25T22:30:00-08:00","2012-04-25T22:45:00-08:00","2012-04-25T23:00:00-08:00","2012-04-25T23:15:00-08:00","2012-04-25T23:30:00-08:00","2012-04-25T23:45:00-08:00","2012-04-26T00:00:00-08:00","2012-04-26T00:15:00-08:00","2012-04-26T00:30:00-08:00","2012-04-26T00:45:00-08:00","2012-04-26T01:00:00-08:00","2012-04-26T01:15:00-08:00","2012-04-26T01:30:00-08:00","2012-04-26T01:45:00-08:00","2012-04-26T02:00:00-08:00","2012-04-25T22:00:00-03:30","2012-04-25T22:15:00-03:30","2012-04-25T22:30:00-03:30","2012-04-25T22:45:00-03:30","2012-04-25T23:00:00-03:30","2012-04-25T23:15:00-03:30","2012-04-25T23:30:00-03:30","2012-04-25T23:45:00-03:30","2012-04-26T00:00:00-03:30","2012-04-26T00:15:00-03:30","2012-04-26T00:30:00-03:30","2012-04-26T00:45:00-03:30","2012-04-26T01:00:00-03:30","2012-04-26T01:15:00-03:30","2012-04-26T01:30:00-03:30","2012-04-26T01:45:00-03:30","2012-04-26T02:00:00-03:30"), olson = c("Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland"), value = c(0,0,1,2,5,11,17,19,20,19,17,11,5,2,1,0,0,-3,-3,-2,-1,2,8,14,16,17,16,14,8,2,-1,-2,-3,-3), stringsAsFactors=FALSE)
Remove the ":" from the UTC offset. (R is expecting the format nnnn for the UTC offset):
dataset$timestampR<- paste(substr(dataset$timestampISO8601,1,22),substr(dataset$timestampISO8601,24,25),sep="")
When converting to UTC time, R defaults to -ve for the UTC offset, making -ve offsets in the timestamps positive:
dataset$datetimeUTC <- strptime(dataset$timestampR, format="%Y-%m-%dT%H:%M:%S%z", tz="UTC")
When converting to MACHINE time like this, R reads the input time and converts it to the time in the timezone of the local machine - in my case, this is Canada/Eastern:
dataset$datetimeMACHINE <- strptime(dataset$timestampR, format="%Y-%m-%dT%H:%M:%S%z")
When converting to CLOCKTIME time like this, R reads the input time and assigns the time zone of the local machine (currently EDT on my machine) without doing any time conversions:
dataset$datetimeCLOCKTIME <- strptime(dataset$timestampR,format="%Y-%m-%dT%H:%M:%S")
See the structure of the dataset:
str(dataset)
Plotting behaviours are different
library(ggplot2)
qplot(data=dataset,x=datetimeUTC,y=value)
qplot(data=dataset,x=datetimeMACHINE,y=value)
qplot(data=dataset,x=datetimeCLOCKTIME,y=value)
Calculation results differ. Incorrect calculation result for datetimeCLOCKTIME:
range (dataset$datetimeUTC)
range (dataset$datetimeMACHINE)
range (dataset$datetimeCLOCKTIME)
dataset$datetimeUTC[34] - dataset$datetimeUTC[1]
dataset$datetimeMACHINE[34] - dataset$datetimeMACHINE[1]
dataset$datetimeCLOCKTIME[34] - dataset$datetimeCLOCKTIME[1]
You could format back and forth a bit to get a local time representation in a character format. E.g.:
dataset <- data.frame(
datetimeUTC=c("2014-01-01 00:00 +0000","2014-01-01 00:00 +0000"),
olson=c("Canada/Eastern", "Canada/Pacific"),
stringsAsFactors=FALSE
)
# datetimeUTC olson
#1 2014-01-01 00:00 +0000 Canada/Eastern
#2 2014-01-01 00:00 +0000 Canada/Pacific
dataset$localtime <- with(dataset,
mapply(function(dt,ol) format(
as.POSIXct(dt,"%Y-%m-%d %H:%M %z",tz=ol),
"%Y-%m-%d %H:%M %z"),
datetimeUTC, olson
)
)
# datetimeUTC olson localtime
#1 2014-01-01 00:00 +0000 Canada/Eastern 2013-12-31 19:00 -0500
#2 2014-01-01 00:00 +0000 Canada/Pacific 2013-12-31 16:00 -0800
If you have only two time zones to convert to and know the difference in time between UTC and those two. Using #thelatemail's dataset
transform(dataset,
localtime=as.POSIXct(datetimeUTC, "%Y-%m-%d %H:%M %z")-
c(5*3600,8*3600)[as.numeric(factor(olson))])
# datetimeUTC olson localtime
#1 2014-01-01 00:00 +0000 Canada/Eastern 2013-12-31 19:00:00
#2 2014-01-01 00:00 +0000 Canada/Pacific 2013-12-31 16:00:00