R - UTC to LOCAL time given Olson timezones - r
I have time series data from 1974-2013 with a column for datetimeUTC (YYYY-MM-DD hh:mm +0000), and a column for the timezones in Olson format (e.g., Canada/Pacific, Canada/Eastern). I can convert the whole UTCdatetime column to a common timezone like this:
dataset$datetimeEST <- strptime(
dataset$datetimeUTC, format="%Y-%m-%d %H:%M:%S%z", tz="Canada/Eastern"
)
How do I convert datetimeUTC to datetimeLOCAL, given the corresponding timezone in each row?
Let me back up a bit. I have data from across the country (6 timezones) formatted in ISO8601 representation for 1974-2013. The timestamps are in local standard time throughout the year (i.e. DST is disregarded even if civilian time in the region observes DST). I need to do datetime calculations which are probably safest to do in UTC time, so that's easy. But, I also need to pull data for specific civil time periods, taking into account DST, and do calculations and plots (e.g., all the data for rush hour at locations across all 6 timezones) for that subsetted data.
The datetimeCLOCKTIME that I calculated below appears to be doing what I want for plotting, but gives the wrong answer when doing datetime calculations because it stored the datetime in the timezone of my local machine without having actually converted the time. The solution offered by #thelatemail is what I'm looking for, but I haven't been able to get it to work in Windows on the test dataset for 2012 (see below). Also, I was using strptime which converts to POXITlt, and his solution is in POXITct. I'm new to R, so any help would be infinitely appreciated.
Test dataset:
dataset <- data.frame (timestampISO8601 = c("2012-04-25T22:00:00-08:00","2012-04-25T22:15:00-08:00","2012-04-25T22:30:00-08:00","2012-04-25T22:45:00-08:00","2012-04-25T23:00:00-08:00","2012-04-25T23:15:00-08:00","2012-04-25T23:30:00-08:00","2012-04-25T23:45:00-08:00","2012-04-26T00:00:00-08:00","2012-04-26T00:15:00-08:00","2012-04-26T00:30:00-08:00","2012-04-26T00:45:00-08:00","2012-04-26T01:00:00-08:00","2012-04-26T01:15:00-08:00","2012-04-26T01:30:00-08:00","2012-04-26T01:45:00-08:00","2012-04-26T02:00:00-08:00","2012-04-25T22:00:00-03:30","2012-04-25T22:15:00-03:30","2012-04-25T22:30:00-03:30","2012-04-25T22:45:00-03:30","2012-04-25T23:00:00-03:30","2012-04-25T23:15:00-03:30","2012-04-25T23:30:00-03:30","2012-04-25T23:45:00-03:30","2012-04-26T00:00:00-03:30","2012-04-26T00:15:00-03:30","2012-04-26T00:30:00-03:30","2012-04-26T00:45:00-03:30","2012-04-26T01:00:00-03:30","2012-04-26T01:15:00-03:30","2012-04-26T01:30:00-03:30","2012-04-26T01:45:00-03:30","2012-04-26T02:00:00-03:30"), olson = c("Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Pacific","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland","Canada/Newfoundland"), value = c(0,0,1,2,5,11,17,19,20,19,17,11,5,2,1,0,0,-3,-3,-2,-1,2,8,14,16,17,16,14,8,2,-1,-2,-3,-3), stringsAsFactors=FALSE)
Remove the ":" from the UTC offset. (R is expecting the format nnnn for the UTC offset):
dataset$timestampR<- paste(substr(dataset$timestampISO8601,1,22),substr(dataset$timestampISO8601,24,25),sep="")
When converting to UTC time, R defaults to -ve for the UTC offset, making -ve offsets in the timestamps positive:
dataset$datetimeUTC <- strptime(dataset$timestampR, format="%Y-%m-%dT%H:%M:%S%z", tz="UTC")
When converting to MACHINE time like this, R reads the input time and converts it to the time in the timezone of the local machine - in my case, this is Canada/Eastern:
dataset$datetimeMACHINE <- strptime(dataset$timestampR, format="%Y-%m-%dT%H:%M:%S%z")
When converting to CLOCKTIME time like this, R reads the input time and assigns the time zone of the local machine (currently EDT on my machine) without doing any time conversions:
dataset$datetimeCLOCKTIME <- strptime(dataset$timestampR,format="%Y-%m-%dT%H:%M:%S")
See the structure of the dataset:
str(dataset)
Plotting behaviours are different
library(ggplot2)
qplot(data=dataset,x=datetimeUTC,y=value)
qplot(data=dataset,x=datetimeMACHINE,y=value)
qplot(data=dataset,x=datetimeCLOCKTIME,y=value)
Calculation results differ. Incorrect calculation result for datetimeCLOCKTIME:
range (dataset$datetimeUTC)
range (dataset$datetimeMACHINE)
range (dataset$datetimeCLOCKTIME)
dataset$datetimeUTC[34] - dataset$datetimeUTC[1]
dataset$datetimeMACHINE[34] - dataset$datetimeMACHINE[1]
dataset$datetimeCLOCKTIME[34] - dataset$datetimeCLOCKTIME[1]
You could format back and forth a bit to get a local time representation in a character format. E.g.:
dataset <- data.frame(
datetimeUTC=c("2014-01-01 00:00 +0000","2014-01-01 00:00 +0000"),
olson=c("Canada/Eastern", "Canada/Pacific"),
stringsAsFactors=FALSE
)
# datetimeUTC olson
#1 2014-01-01 00:00 +0000 Canada/Eastern
#2 2014-01-01 00:00 +0000 Canada/Pacific
dataset$localtime <- with(dataset,
mapply(function(dt,ol) format(
as.POSIXct(dt,"%Y-%m-%d %H:%M %z",tz=ol),
"%Y-%m-%d %H:%M %z"),
datetimeUTC, olson
)
)
# datetimeUTC olson localtime
#1 2014-01-01 00:00 +0000 Canada/Eastern 2013-12-31 19:00 -0500
#2 2014-01-01 00:00 +0000 Canada/Pacific 2013-12-31 16:00 -0800
If you have only two time zones to convert to and know the difference in time between UTC and those two. Using #thelatemail's dataset
transform(dataset,
localtime=as.POSIXct(datetimeUTC, "%Y-%m-%d %H:%M %z")-
c(5*3600,8*3600)[as.numeric(factor(olson))])
# datetimeUTC olson localtime
#1 2014-01-01 00:00 +0000 Canada/Eastern 2013-12-31 19:00:00
#2 2014-01-01 00:00 +0000 Canada/Pacific 2013-12-31 16:00:00
Related
Different parsing behaviour for the first day of April in R as.POSIXct and as.POSIXlt, is R april fooling me?
I am unexperienced in working with data format in R, and I am struggling to understand the different behaviour with the first of April... is it an april fool?? :)
They have the same format, but it seems that the first day can't be parsed using as.POSIXct (when other dates show no issues) or it does not returns the time zone with as.POSIXlt?
(as.POSIXct("1/04/2012 02:58", format = "%d/%m/%Y %H:%M")) # this doesn't work
(as.POSIXct("2/04/2012 02:58", format = "%d/%m/%Y %H:%M")) # this works
(as.POSIXct("01/04/2012 02:58", format = "%d/%m/%Y %H:%M")) # this doesn't
(as.POSIXct("02/04/2012 02:58", format = "%d/%m/%Y %H:%M")) # this does...
(as.POSIXlt("1/04/2012 02:58", format = "%d/%m/%Y %H:%M")) # This works, but does not returns a time zone
(as.POSIXlt("2/04/2012 02:58", format = "%d/%m/%Y %H:%M")) # This works, and returns a time zone
(as.POSIXlt("01/04/2012 02:58", format = "%d/%m/%Y %H:%M")) # This works, and does not returns a time zone
(as.POSIXlt("02/04/2012 02:58", format = "%d/%m/%Y %H:%M")) # This works, and returns a time zone
Any direction as to why? Thanks!
This is almost certainly a daylight savings time issue. Not sure why POSIXct and POSIXlt behave differently though. From your profile, it looks like you're in Mexico. From here: most of Mexico, including capital Mexico City, will set the clocks 1 hour forward 3 weeks later, on Sunday, April 1, 2012. So the problem is that 2:58 AM on 1 April 2012 did not exist in the time zone that is currently active in your locale. Unless there is something specific having to do with the POSIXct/POSIXlt difference, this should probably be closed as a duplicate of e.g.: What is wrong with this date and time? R POSIXct returns NA with "03/12/2017 02:17:13" PosixCT conversion in R fails Weird as.POSIXct behavior depending on daylight savings time Strange strptime behavior in R as.POSIX error, can not convert a particular date Weird POSIX behaviour for two closely time strings with and without specifying the format And this r help question If you want to deal with this e.g. by setting all times to UTC (i.e. ignoring your local time zone settings), I believe there are lots of suggestions on Stack Overflow (now that you know to search for "daylight savings time" it should be easy to find them). obligatory xkcd
#Ben Bolker is correct that this is a daylight saving time issue. Specifically, this is what I call a nonexistent time issue. In Mexico City, on April 1st 2012, there was a DST gap of 1 hour where the clocks jumped from 01:59:59 AM straight to 03:00:00 AM, skipping the two o'clock hour entirely. So 02:58:00 AM is a nonexistent time on that day.
These problems can be really frustrating, so in the clock package I've made parsing issues like this an error by default, with many ways to get around them according to your needs.
For future visitors to this post, here is a reprex with the full output from as.POSIXc/lt() vs clock. The relevant clock function is date_time_parse().
library(clock)
x <- c("1/04/2012 02:58", "2/04/2012 02:58")
zone <- "America/Mexico_City"
format <- "%d/%m/%Y %H:%M"
# Nonexistent time - returns NA
as.POSIXct(x, tz = zone, format = format)
#> [1] NA "2012-04-02 02:58:00 CDT"
# Nonexistent time - can't determine zone
as.POSIXlt(x, tz = zone, format = format)
#> [1] "2012-04-01 02:58:00" "2012-04-02 02:58:00 CDT"
# Errors on nonexistent time so you don't have surprising results
date_time_parse(x, zone = zone, format = format)
#> Error: Nonexistent time due to daylight saving time at location 1.
#> ℹ Resolve nonexistent time issues by specifying the `nonexistent` argument.
# Next valid time
date_time_parse(x, zone = zone, format = format, nonexistent = "roll-forward")
#> [1] "2012-04-01 03:00:00 CDT" "2012-04-02 02:58:00 CDT"
# Previous valid time
date_time_parse(x, zone = zone, format = format, nonexistent = "roll-backward")
#> [1] "2012-04-01 01:59:59 CST" "2012-04-02 02:58:00 CDT"
# Shift forward by the size of the gap (1 hour)
date_time_parse(x, zone = zone, format = format, nonexistent = "shift-forward")
#> [1] "2012-04-01 03:58:00 CDT" "2012-04-02 02:58:00 CDT"
# NA on nonexistent times
date_time_parse(x, zone = zone, format = format, nonexistent = "NA")
#> [1] NA "2012-04-02 02:58:00 CDT"
R POSIXct and as.Date
I have a date and time value which I'm using POSIXct to store in a variable. and when I use as.Date the date is different - why ?
x<-as.POSIXct("2012-02-25 19:00:00")
as.Date(x)
[1] "2012-02-**26**"
Why is it incrementing day by 1?
By simply using x<-as.POSIXct("2012-02-25 19:00:00") you will be setting the time zone, tz, to your current systems time zone, EST.
However when you use as.Date without declaring a time zone it will default to GMT. GMT is 5 hours ahead of EST so 25/02/2012 19:00 is 00:00 26/02/2012.
R: Posix (Unix) Time Crazy Conversion
Unix time is 1435617000. as.Date(1435617000,origin="01-01-1970") [1] "3930586-11-23" Which is wrong. I'm trying to (a) get the correct date, which, per epoch converter is GMT: Mon, 29 Jun 2015 22:30:00 GMT. How do I get R to tell me the month, day, year, hour, minute & second? Thank you.
I think the reason why that happen is because as.Date converts arguments to class date objects. In this case you do not need a date but a class POSIXct object because your input, the x vector, contains other informations that as.Date is not able to manage. Another problem that even with the right function could appear, is that if when you do not specify the right time zone with the tz argument (except the case where your time zone is the same as the original time). The following code does the job. x <- 1435617000 as.POSIXct(x, origin = "1970-01-01", tz ="GMT") [1] "2015-06-29 22:30:00 GMT" Use as.Date Just in the case you wanted only the date but you have a complete Unix time like x, you have to just divide by 86400 (which is the number of seconds in a day!) to get only the right date. as.Date(x/86400L, origin = "1970-01-01") [1] "2015-06-29" Another important detail The origin argument has to be supplied with YYYY-MM-DD and not like you did DD-MM-YYYY I am not sure but I think that the former is the only accepted and correct way.
as.POSIXct assigns different timezones [duplicate]
I am trying to convert UTC time to local standard time. I have found many functions which convert to Local Daylight Time, but I was not successful in getting the standard time. Right now, I have the following code which converts to local daylight time at my specific timezone: pb.date <- as.POSIXct(date,tz="UTC") format(pb.date, tz="timeZone",usetz=TRUE) I would appreciate any help.
First, POSIXct date-times are always UCT internally. The print.POSIXt and format.POSIXt methods will appropriately make the TZ shift on output from their internal representations: pb.date <- as.POSIXct(Sys.Date()) Sys.Date() #[1] "2015-07-09" So that was midnight of the current date in Greenwich: format(pb.date, tz="America/Los_Angeles",usetz=TRUE) #[1] "2015-07-08 17:00:00 PDT" When it's midnight in Greenwich, it's 5PM Daylight Time in the previous day on the Left Coast of the US. You need to use the correct character values for your TZ (and your OS) both of which at the moment are unspecified. The US Pacific timezone is 8 hours behind GMT (in winter months) so you can use a timezone that is Standard/Daylight-agnostic: > format(pb.date,usetz=TRUE, tz="Etc/GMT+8") [1] "2015-07-08 16:00:00 GMT+8" (Notice the reversal of + with "behind" and - with "ahead".)
I know this question has an accepted answer, but in case anyone comes along and this can help. I needed a function to convert UTC times to MTN time (Server is in UTC, we operate in MTN).
Not sure why, but needed to force it to UTC/GMT first and the convert it to MTN. However it does work
mtn_ts = function(utcTime){
library(lubridate)
toTz = "us/mountain"
utcTime = force_tz(utcTime,tzone= 'GMT')
dt = as.POSIXct(format(utcTime,tz = toTz,origin ='GMT', usetz=TRUE))
dt = force_tz(dt,tzone= toTz)
return(dt)
}
mtn_ts(as.POSIXct("2021-09-27 14:48:51.000000000"))
Converting UTC time to local standard time in R
I am trying to convert UTC time to local standard time. I have found many functions which convert to Local Daylight Time, but I was not successful in getting the standard time. Right now, I have the following code which converts to local daylight time at my specific timezone: pb.date <- as.POSIXct(date,tz="UTC") format(pb.date, tz="timeZone",usetz=TRUE) I would appreciate any help.
First, POSIXct date-times are always UCT internally. The print.POSIXt and format.POSIXt methods will appropriately make the TZ shift on output from their internal representations: pb.date <- as.POSIXct(Sys.Date()) Sys.Date() #[1] "2015-07-09" So that was midnight of the current date in Greenwich: format(pb.date, tz="America/Los_Angeles",usetz=TRUE) #[1] "2015-07-08 17:00:00 PDT" When it's midnight in Greenwich, it's 5PM Daylight Time in the previous day on the Left Coast of the US. You need to use the correct character values for your TZ (and your OS) both of which at the moment are unspecified. The US Pacific timezone is 8 hours behind GMT (in winter months) so you can use a timezone that is Standard/Daylight-agnostic: > format(pb.date,usetz=TRUE, tz="Etc/GMT+8") [1] "2015-07-08 16:00:00 GMT+8" (Notice the reversal of + with "behind" and - with "ahead".)
I know this question has an accepted answer, but in case anyone comes along and this can help. I needed a function to convert UTC times to MTN time (Server is in UTC, we operate in MTN).
Not sure why, but needed to force it to UTC/GMT first and the convert it to MTN. However it does work
mtn_ts = function(utcTime){
library(lubridate)
toTz = "us/mountain"
utcTime = force_tz(utcTime,tzone= 'GMT')
dt = as.POSIXct(format(utcTime,tz = toTz,origin ='GMT', usetz=TRUE))
dt = force_tz(dt,tzone= toTz)
return(dt)
}
mtn_ts(as.POSIXct("2021-09-27 14:48:51.000000000"))