I've downloaded some climate data from a website and am trying to analyse it in R.
The time format for the data is hours since 1800-01-01 00:00. For example:
ss <- seq(447042,455802, length.out = 1461)
which shows data at 6 hour intervals.
How can I convert this to a an actual time in R. This example should give data for 1851:
1851-01-01 00:00
1851-01-01 06:00
and so on...
How can this be done?
Any advice would be appreciated.
I think you have a typo as well as incorrect calculations. Let's assume you want time in the future of 1880 rather than the past. So it might be 1951 you wanted? Then to convert hours to seconds which are the basis for the POSIXt classed objects, just multiply by 3600 = 60*60:
> tail( as.POSIXct(ss*3600,origin='1880-01-01 00:00') )
[1] "1931-12-30 04:00:00 PST" "1931-12-30 10:00:00 PST" "1931-12-30 16:00:00 PST"
[4] "1931-12-30 22:00:00 PST" "1931-12-31 04:00:00 PST" "1931-12-31 10:00:00 PST"
As you can see it's nowhere the year 1951 either, but maybe you had two digits off and you wanted 1931? Conversions that span the range of years before the onset of daylight savings time and cross century boundaries where leap years and leap seconds were used may not "line up" with your expectations.
Related
I am hoping somebody can help me with a logic question in R-studio. I have a rather large data set, with "Time" as one of my columns. This column has values from 00:00:00 to 23:59:00, and is in HH:MM:SS format.
Because of some trouble I have had with analysis of time in this format, I am trying to create a new column, called "ZT" where I convert this time column to ZT time. Lights turn on at 7am, so need the time 07:00:00 to correspond to ZT=0, with 07:01:00 to correspond to ZT=0.016... and so on and so forth.
Can anybody help me with this? It would be much appreciated!
Not sure if this is what you are going for or not but this seems to work at converting a character vector of times in the format HH:MM:SS to your ZT time in the format HH:MM:SS starting at 7am as 00:00:00.
I am unclear exactly what you mean when you state that 07:01:00 should correspond to ZT=0.016, but maybe this can be a start.
Fair warning this is a little slow (took about 1 minute on my machine) but maybe someone else can help vectorize it and speed it up:
#Make Some Fake Data
df<-data.frame(Time=format(seq(ISOdate(2020,1,1), ISOdate(2020,2,1), by = "min"), '%H:%M:00'), Variable1=runif(n=44641))
#We need the help of a an external package to handle time in HH:MM:SS format
library(lubridate)
time_store<- hms(df$Time) #Convert your times to HMS format
ZT_vec<-vector() #Create an empty vector that we will fill in
for (i in 1:length(time_store)){ #iterate over each observation
if (hour(time_store[i])<7){ #Make sure the conversions are going the right direction
ZT<-time_store[i]+hours(17)
ZT_vec<-c(ZT_vec,sprintf("%02d:%02d:%02d", hour(ZT), minute(ZT), second(ZT))) #format the times in HH:MM:SS
} else {
ZT<-time_store[i]-hours(7)
ZT_vec<-c(ZT_vec,sprintf("%02d:%02d:%02d", hour(ZT), minute(ZT), second(ZT)))
}
}
df<-cbind(df,ZT_vec) #Bind on our new column
head(df)
Time Variable1 ZT_vec
12:00:00 0.6560604 05:00:00
12:01:00 0.3485023 05:01:00
12:02:00 0.8396784 05:02:00
12:03:00 0.4773929 05:03:00
12:04:00 0.6969242 05:04:00
12:05:00 0.5371502 05:05:00
head(df[4020:4025,])
Time Variable1 ZT_vec
06:59:00 0.6758364 23:59:00
07:00:00 0.1255861 00:00:00
07:01:00 0.2789485 00:01:00
07:02:00 0.2175933 00:02:00
07:03:00 0.1855100 00:03:00
07:04:00 0.1632865 00:04:00
I have dates and times stored in two columns. The first has the date as "20180831." The time is stored as the number of seconds from midnight; 3am would be stored as 10,800.
I need a combined date time column and am having a hard time with something that should be simple.
I can get the dates in no problem but lubridate "hms" interprets the time field as a period, not a 'time' per se.
I tried converting the date to posix.ct format and then using that as the origin for the time field but posix.ct does not set the time for midnight, instead it sets it for either 1800 or 1900 hours depending on the date. I need it set to midnight for all rows, I don't want any daylight savings time adjustment.
Here's the code:
First I made a function because there are several date and time fields I have to do this for.
mkdate<-function(x){
a<-as.Date(as.character(x),format='%Y%m%d')
a<-as.POSIXct(a)
return(a)
}
df$date<-mkdate(df$date) #applies date making function to date field
df$datetime<-as.POSIXct(df$time,origin=df$date)
I'm sure this has to do with time zones. I'm in Central time zone and I have experimented with adding the "tz" specification into these commands in both the mkdate function and in the time code creating "datetime" column.
I've tried:
tz="America/Chicago"
tz="CST"
tz="UTC"
Help would be much appreciated!
Edited with example:
x<-c(20180831,20180710,20160511,20170105,20180101) #these are dates.
as.POSIXct(as.Date(as.character(x),format="%Y%m%d"))
Above code converts dates to seconds from the Jan 1 1970. I could convert this to numeric and add my 'seconds' value to this field BUT it is not correct. This is what I see instead as the output:
[1] "2018-08-30 19:00:00 CDT" "2018-07-09 19:00:00 CDT" "2016-05-10 19:00:00 CDT" "2017-01-04 18:00:00 CST" "2017-12-31 18:00:00 CST"
Look at the first date - it should be 8/31 but instead it is 8/30. Somewhere in there there is a timezone adjustment taking place. It's moving the clock back 5 or 6 hours because I am on central time. The first entry should be 2018-08-31 00:00:00. I would then convert it to numeric and add the seconds field on and convert back to POSIXct format. I've tried including tz specification all over the place with no luck.
Sys.getlocale("LC_TIME")
returns "English_United States.1252"
I believe the following does what you want.
My locale is the following, so the results are different from yours.
Sys.getlocale("LC_TIME")
#[1] "Portuguese_Portugal.1252"
The difference will be due to the daylight savings time, the summer hour.
As for your problem, all you have to do is to remeber that the objects of class "POSIXct are coded as the number of seconds since an origin, and that origin is usually the midnight of 1970-01-01. So you have to add your seconds since midnight to the seconds of as.Date.
x <- "20180831"
xd <- mkdate(x)
y <- 10800
as.POSIXct(as.integer(xd) + y, origin = "1970-01-01")
#[1] "2018-08-31 04:00:00 BST"
as.POSIXct(as.integer(xd) + y, origin = "1970-01-01", tz = "America/Chicago")
#[1] "2018-08-30 22:00:00 CDT"
There are very many ways to do this:
mktime = function(a, b)modifyList(strptime(a, '%Y%m%d'), list(sec = as.numeric(gsub(',', '', b))))
mktime("20180831",'10,800')
[1] "2018-08-31 03:00:00 PDT"
mktime('20180301','10800')
[1] "2018-03-01 03:00:00 PST"
mktime('20180321','10800')
[1] "2018-03-21 03:00:00 PDT"
Looking at the above code, it does not adjust for the daylight saving time. Irrespective of the date, the seconds still show that it Is 3 AM, including the dates when ST-->DT. This will also take into consideration, your LOCAL timezone.
I have the 2 dates below both are in Pacific Time but one of them is Daylight savings (PDT) and 1 is standard time (PST).
First question: How can I convert both to the PST?
Second question: How can I convert both to the EST-eastern standard time (tz=America/New_York), which is not the Eastern daylight time?
date= as.POSIXct(c("2016-04-01 15:30:00 PDT","2016-12-06 16:00:00 PST"))
date
## [1] "2016-04-01 15:30:00 PDT" "2016-12-06 16:00:00 PST"
quite simple really. with Lubidridate as documented here:
library(lubridate)
corrected_date <- force_tz(date,tz = "America/New_York")
I have a series of datasets from a water quality continuous monitoring probe with the Date and Time set in Eastern Standard Time (EST) so there is no correction of Daylight Savings Time (EDT). In R, the fields are recognized as factors when importing the data table from an MS access database however when converting using as.POSIXct() dates and times from 02:00 (24 clock) on 2016-03-13 become NAs. This is due to the transition from EST to EDT...therefore technically 2016-03-13 02:00 doesn't exist.
Some created data as an example
test<-data.frame(Date=rep(as.Date("2016-03-13"),120),Hour=rep(seq(0,23,1),5),Min=rep(seq(0,60,15),24))
Is there a way to convert the factor or character field to as POSIXct field while retaining the EST timezone designation? Alternatively is there a way to identify and convert the proper date and times into EST and EDT?
I have gone around and around and can't get anything to work. I have attempted to convert to GMT (or UTC) then convert back to EST (tz="America/New_York"). I realize that this is an ongoing issue and people who work with date and time data, especially in R would love to move away from EDT.
Any help is appreciated...I am at my wits end on this one.
The trouble with using POSIX tz = "America/New_York" is that daylight savings time is accounted for (UTC-4 or UTC-5), even if the underlying timestamps are stored in U.S. Eastern Standard Time (UTC-5).
You should be able to designate your tz as Etc/GMT+5. From there, it is easy to convert between EST, Eastern local time, and GMT. Note that in R, time zones west of UTC are denoted with a positive offset (see Time Zone Names documentation in ?timezone).
Here is some example data (daylight savings took effect at 2:00 AM Eastern Time on 3/16/16):
StartTime=as.numeric(as.POSIXct("2016-03-11 0:00:00",format="%Y-%m-%d %H:%M",origin="1970-01-01",tz="Etc/GMT+5"))
EndTime=as.numeric(as.POSIXct("2016-03-15 0:00:00",format="%Y-%m-%d %H:%M",origin="1970-01-01",tz="Etc/GMT+5"))
Interval=15*60 #15-min
data.EST=as.POSIXct(seq(from = StartTime,to = EndTime, by=Interval),origin="1970-01-01",tz="Etc/GMT+5") #generate date stamps
# convert Eastern Standard Time (in R: GMT+5) to local time (accounts for daylight savings):
data.EastCoast<- format(data.EST, tz="America/New_York")
# convert Eastern Standard Time (in R: GMT+5) to UTC/GMT:
data.UTC<- format(data.EST, tz="GMT")
compare.times<-data.frame(data.EST,data.EastCoast,data.UTC)
compare.times[(198:203),]
data.EST data.EastCoast data.UTC
198 2016-03-13 01:15:00 2016-03-13 01:15:00 2016-03-13 06:15:00
199 2016-03-13 01:30:00 2016-03-13 01:30:00 2016-03-13 06:30:00
200 2016-03-13 01:45:00 2016-03-13 01:45:00 2016-03-13 06:45:00
201 2016-03-13 02:00:00 2016-03-13 03:00:00 2016-03-13 07:00:00
202 2016-03-13 02:15:00 2016-03-13 03:15:00 2016-03-13 07:15:00
203 2016-03-13 02:30:00 2016-03-13 03:30:00 2016-03-13 07:30:00
Good luck!
During the conversion to POSIX you need to specify the time zone. See this example:
test<-data.frame(Date=rep(as.Date("2016-03-13"),96),Hour=rep(seq(0,23,1), each=4),Min=rep(seq(0,45,15)))
wrong<-as.POSIXct(paste(test$Date, test$Hour, test$Min), format="%Y-%m-%d %H %M")
ans<-as.POSIXct(paste(test$Date, test$Hour, test$Min), format="%Y-%m-%d %H %M", tz="EST")
compare<-cbind(test, wrong, ans)
In the vector "wrong", no timezone was specified thus the NA, but in the second case Eastern Standard was specified and the desired result is given.
I have run into a similar issue using water quality data that does not observe daylight savings time. The workaround I have found useful is to use America/Jamaica rather than America/New_York. Below are a list of GMT offsets and the tz to use.
-4 = America/Virgin
-5 = America/Jamaica
-6 = America/Regina
-8 = Pacific/Pitcairn
-9 = Pacific/Gambier
-10 = Pacific/Honolulu
I want to subtract 1 day from a POSIX date and end up at the same time around DST.
For example, when I add a day:
> as.POSIXct('2009-03-08 23:00:00.000') + 86400
[1] "2009-03-09 23:00:00 EDT"
But when I go past, it offsets:
> as.POSIXct('2009-03-08 23:00:00.000') - 86400
[1] "2009-03-07 22:00:00 EST"
What's the best way to deal with absolute time differences around DST? Usually I deal with this by converting the times into strings and dealing with them separately so that DST isn't applied.
Your code is doing exactly what you asked it to do, because you didn't ask it to add or subtract one day, you asked it to add or subtract 24 hours. 24 hours (86400 seconds) before 2009-03-08 23:00:00 EDT is 2009-03-07 22:00:00 EST. I'm not familiar with the R libraries, but I am familiar with the POSIX functions that they wrap. If you take a POSIXct, decrease its day property by 1, and then "re-cast" it to POSIXct via POSIXlt ( to ensure that e.g. February -1st becomes January 31st) then you should be able to subtract one day reliably.
Store datetimes as UTC - or in this case convert to UTC subtract a day and convert back to local time. (This ignores leap seconds)
Thanks hobbs! I need to do a little more work with it, but subtracting from the day slot works in POSIXlt:
> a <- as.POSIXct('2009-03-08 23:00:00.000')
> as.POSIXlt(a)
[1] "2009-03-08 23:00:00 EDT"
> a <- as.POSIXlt(a)
> a$mday <- a$mday -1
> a
[1] "2009-03-07 23:00:00 EDT"
If you just want to count the days, you can use trunc to just move the day:
> trunc(Sys.time(), "day") + 86400
[1] "2009-09-13 PDT"
> trunc(Sys.time(), "day") - 86400
[1] "2009-09-11 PDT"
The package lubridate provides two additional timespan classes that specify what is somewhat unclear in the base package. From tha manual:
Durations
Durations measure the exact amount of time that occurs between two instants. This can create unexpected results in relation to clock times if a leap second, leap year, or change in daylight savings time (DST) occurs in the interval.
Periods
Periods measure the change in clock time that occurs between two instants. Periods provide robust predictions of clock time in the presence of leap seconds, leap years, and changes in DST.