What's the best way to get the length of time represented by an interval in lubridate, in specified units? All I can figure out is something like the following messy thing:
> ival
[1] 2011-01-01 03:00:46 -- 2011-10-21 18:33:44
> difftime(attr(ival, "start") + as.numeric(ival), attr(ival, "start"), 'days')
Time difference of 293.6479 days
(I also added this as a feature request at https://github.com/hadley/lubridate/issues/105, under the assumption that there's no better way available - but maybe someone here knows of one.)
Update - apparently the difftime function doesn't handle this either. Here's an example.
> (d1 <- as.POSIXct("2011-03-12 12:00:00", 'America/Chicago'))
[1] "2011-03-12 12:00:00 CST"
> (d2 <- d1 + days(1)) # Gives desired result
[1] "2011-03-13 12:00:00 CDT"
> (i2 <- d2 - d1)
[1] 2011-03-12 12:00:00 -- 2011-03-13 12:00:00
> difftime(attr(i2, "start") + as.numeric(i2), attr(i2, "start"), 'days')
Time difference of 23 hours
As I mention below, I think one nice way to handle this would be to implement a /.interval function that doesn't first cast its input to a period.
The as.duration function is what lubridate provides. The interval class is represented internally as the number of seconds from the start, so if you wanted the number of hours you could simply divide as.numeric(ival) by 3600, or by (3600*24) for days.
If you want worked examples of functions applied to your object, you should provide the output of dput(ival). I did my testing on the objects created on the help(duration) page which is where ?interval sent me.
date <- as.POSIXct("2009-03-08 01:59:59") # DST boundary
date2 <- as.POSIXct("2000-02-29 12:00:00")
span <- date2 - date #creates interval
span
#[1] 2000-02-29 12:00:00 -- 2009-03-08 01:59:59
str(span)
#Classes 'interval', 'numeric' atomic [1:1] 2.85e+08
# ..- attr(*, "start")= POSIXct[1:1], format: "2000-02-29 12:00:00"
as.duration(span)
#[1] 284651999s (9.02y)
as.numeric(span)/(3600*24)
#[1] 3294.583
# A check against the messy method:
difftime(attr(span, "start") + as.numeric(span), attr(span, "start"), 'days')
# Time difference of 3294.583 days
This question is really old, but I'm adding an update because this question has been viewed many times and when I needed to do something like this today, I found this page. In lubridate you can now do the following:
d1 <- ymd_hms("2011-03-12 12:00:00", tz = 'America/Chicago')
d2 <- ymd_hms("2011-03-13 12:00:00", tz = 'America/Chicago')
(d1 %--% d2)/dminutes(1)
(d1 %--% d2)/dhours(1)
(d1 %--% d2)/ddays(1)
(d1 %--% d2)/dweeks(1)
Ken, Dividing by days(1) will give you what you want. Lubridate doesn't coerce periods to durations when you divide intervals by periods. (Although the algorithm for finding the exact number of whole periods in the interval does begin with an estimate that uses the interval divided by the analagous number of durations, which might be what you are noticing).
The end result is the number of whole periods that fit in the interval. The warning message alerts the user that it is an estimate because there will be some fraction of a period that is dropped from the answer. Its not sensible to do math with a fraction of a period since we can't modify a clock time with it unless we convert it to multiples of a shorter period - but there won't be a consistent way to make the conversion. For example, the day you mention would be equal to 23 hours, but other days would be equal to 24 hours. You are thinking the right way - periods are an attempt to respect the variations caused by DST, leap years, etc. but they only do this as whole units.
I can't reproduce the error in subtraction that you mention above. It seems to work for me.
three <- force_tz(ymd_hms("2011-03-12 12:00:00"), "")
# note: here in TX, "" *is* CST
(four <- three + days(1))
> [1] "2011-03-13 12:00:00 CDT"
four - days(1)
> [1] "2011-03-12 12:00:00 CST"
Be careful when divinding time in seconds to obtain days as then you are no longer working with abstract representations of time but in bare numbers, which can lead to the following:
> date_f <- now()
> date_i <- now() - days(23)
> as.duration(date_f - date_i)/ddays(1)
[1] 22.95833
> interval(date_i,date_f)/ddays(1)
[1] 22.95833
> int_length(interval(date_i,date_f))/as.numeric(ddays(1))
[1] 22.95833
Which leads to consider that days or months are events in a calendar, not time amounts that can be measured in seconds, miliseconds, etc.
The best way to calculate differences in days is avoiding the transformation into seconds and work with days as a unit:
> e <- now()
> s <- now() - days(23)
> as.numeric(as.Date(s))
[1] 18709
> as.numeric(as.Date(e) - as.Date(s))
[1] 23
However, if you are considering a day as a pure 86400 seconds time span, as ddays() does, the previous approach can lead to the following:
> e <- ymd_hms("2021-03-13 00:00:10", tz = 'UTC')
> s <- ymd_hms("2021-03-12 23:59:50", tz = 'UTC')
> as.duration(e - s)
[1] "20s"
> as.duration(e - s)/ddays(1)
[1] 0.0002314815
> as.numeric(as.Date(e) - as.Date(s))
[1] 1
Hence, it depends on what you are looking for: time difference or calendar difference.
Related
I have datetimes written in format "%d-%b-%Y %H:%M:%S.%OS",
so for example "25-Apr-2021 18:31:56.234",
that is to the precision of milliseconds.
And when parse that to time object I see values are not the same, sometimes it adds 1 microsecond or decreases it, or similiar things.
Why is this and what to do about this?
I want to have a time object which is exactly 56 seconds and 234 milliseconds! (and zeroes after that if it needs to add higher precision
For example some of the values it prints when I call print(as.numeric(), digits=20) command: "1615310444.7509999 1615310442.5550001",
or when I ask for difference between some 2 values, it gives: "Time difference of 0.003999949 secs" for example.
You can use the options(digits.secs) to show the milliseconds. Here is an example below. The digits.secs must be set at zero. Also note that you should change the format of the date
> dp <- options(digits.secs=0)
> dp
$digits.secs
[1] 0
> strptime("25-Apr-2021 18:31:56.234", format = "%d-%b-%Y %H:%M:%OS")
[1] "2021-04-25 18:31:56 +08"
> dp <- options(digits.secs=3)
> dp
$digits.secs
[1] 0
> strptime("25-Apr-2021 18:31:56.234", format = "%d-%b-%Y %H:%M:%OS")
[1] "2021-04-25 18:31:56.234 +08"
I have been encountering a strange performance issue with R.
I have a csv file that contains close to 600,00 lines and 11 columns. The last column contains dates. I am trying filter rows based on whether date in the last column is a weekend or weekday. As you can see from the output below, it takes 12 seconds for this relatively simple filtering.
> library(lubridate)
> data335 = read.csv("data335.csv")
> Sys.time()
[1] "2017-10-29 00:50:16 IST"
> delete_variable = data335[ifelse((wday(data335$ticket_date) %in% c("1","6")), T , F),][11]
> Sys.time()
[1] "2017-10-29 00:50:28 IST"
However, filtering on other column values hardly takes a second or two.
> Sys.time()
[1] "2017-10-29 00:58:58 IST"
> delete_variable = data335[(data335$route_no == "V-335EUP") ,][11]
> Sys.time()
[1] "2017-10-29 00:58:58 IST"
I'm sure, in the earlier filtering case, I am not doing it in the R way. Is there a way to bring this time taken to filter within 2 seconds?
On my machine, your original code ran in ~7 seconds. I noticed that data335$ticket_date was stored as a factor, so I read it in as a string and coerced it to date format. Time dropped to .1 second.
Also took out the if_else statement, because %in% already returns a logical vector. And used numeric instead of character for the c(1,7) (you had c("1", "6"), but if you are looking for weekends, I think you want 1 & 7). Those resulted in minor speed improvements.
library(lubridate)
data335 <- read.csv('Downloads/data335.csv', stringsAsFactors=FALSE)
data335$ticket_date <- as.Date(data335$ticket_date, format="%d-%m-%Y")
start <- Sys.time()
delete_variable = data335[wday(data335$ticket_date) %in% c(1,7),][11]
end <- Sys.time()
end-start
I'm this has been asked before, but I just can't find the exact answer.
If I have a number which represents milliseconds since midnight, say 34200577, how do I turn this into an R time?
Construct a 'baseline time' at midnight, add the given millisecond once converted to seconds and interpret as a time:
R> as.POSIXct(as.numeric(ISOdatetime(2013,8,22,0,0,0)) + 34200577/1e3,
+ origin="1970-01-01")
[1] "2013-08-22 09:30:00.576 CDT"
R>
In fact, the shorter
R> ISOdatetime(2013,8,22,0,0,0) + 34200577/1e3
[1] "2013-08-22 09:30:00.576 CDT"
R>
works as well as ISOdatetime() returns a proper time object which operates in fractional seconds so we just apply the given offset.
This appears to be correct as
R> 34200577/1e3 # seconds
[1] 34200.6
R> 34200577/1e3/60 # minutes
[1] 570.01
R> 34200577/1e3/60/60 # hours
[1] 9.50016
R>
POSIXct uses 1970 as the origin of its time scale(measured in seconds.)
> time= as.POSIXct(34200577/1000 , origin=Sys.Date() )
> time
[1] "2013-08-22 02:30:00 PDT"
Note the discrepancy in results between Dirk's and my method. The POSIX times are input as assumed to occur in UCT, so there appeared the addition 8 hours for my location in UCT-8.
> difftime( as.POSIXct(34200577/1000 , origin=Sys.Date() ) , Sys.Date() )
Time difference of 9.50016 hours
You could get the time since midnight with:
format( as.POSIXct(34200577/1000 , origin=Sys.Date(), tz="UCT" ),
format="%H:%M:%S")
[1] "09:30:00"
A little "gottcha" which I think is worth pointing out...
In R 3.1.2 on windows 64 bit I get the following results for Dirk's example
> ISOdatetime(2013,8,22,0,0,0) + 34200577/1e3
[1] "2013-08-22 09:30:00 BST"
Note the lack of fractional seconds. This is due to the option setting for "digits.secs"
> getOption("digits.secs")
NULL
Setting this option as follows gives the expected result:
> options(digits.secs=3)
> ISOdatetime(2013,8,22,0,0,0) + 34200577/1e3
[1] "2013-08-22 09:30:00.576 BST"
As you can probably guess, this is to do with the formatting of output, not the actual values we get from our date arithmetic. See ?strptime and ?options for the documentation on this.
I have a large data frame with date variables, which reflect first day of the month. Is there an easy way to create a new data frame date variable that represents the last day of the month?
Below is some sample data:
date.start.month=seq(as.Date("2012-01-01"),length=4,by="months")
df=data.frame(date.start.month)
df$date.start.month
"2012-01-01" "2012-02-01" "2012-03-01" "2012-04-01"
I would like to return a new variable with:
"2012-01-31" "2012-02-29" "2012-03-31" "2012-04-30"
I've tried the following but it was unsuccessful:
df$date.end.month=seq(df$date.start.month,length=1,by="+1 months")
To get the end of months you could just create a Date vector containing the 1st of all the subsequent months and subtract 1 day.
date.end.month <- seq(as.Date("2012-02-01"),length=4,by="months")-1
date.end.month
[1] "2012-01-31" "2012-02-29" "2012-03-31" "2012-04-30"
Here is another solution using the lubridate package:
date.start.month=seq(as.Date("2012-01-01"),length=4,by="months")
df=data.frame(date.start.month)
library(lubridate)
df$date.end.month <- ceiling_date(df$date.start.month, "month") - days(1)
df$date.end.month
[1] "2012-01-31" "2012-02-29" "2012-03-31" "2012-04-30"
This uses the same concept given by James above, in that it gets the first day of the next month and subtracts one day.
By the way, this will work even when the input date is not necessarily the first day of the month. So for example, today is the 27th of the month and it still returns the correct last day of the month:
ceiling_date(Sys.Date(), "month") - days(1)
[1] "2017-07-31"
Use timeLastDayInMonth from the timeDate package:
df$eom <- timeLastDayInMonth(df$somedate)
library(lubridate)
as.Date("2019-09-01") - days(1)
[1] "2019-08-31"
or
library(lubridate)
as.Date("2019-09-01") + months(1) - days(1)
[1] "2019-09-30"
A straightforward solution would be using the yearmonfunction with the argument frac=1 from the xts-package. frac is a number between 0 and 1 that indicates the fraction of the way through the period that the result represents.
as.Date(as.yearmon(seq.Date(as.Date('2017-02-01'),by='month',length.out = 6)),frac=1)
[1] "2017-02-28" "2017-03-31" "2017-04-30" "2017-05-31" "2017-06-30" "2017-07-31"
Or if you prefer “piping” using magrittr:
seq.Date(as.Date('2017-02-01'),by='month',length.out = 6) %>%
as.yearmon() %>% as.Date(,frac=1)
[1] "2017-02-28" "2017-03-31" "2017-04-30" "2017-05-31" "2017-06-30" "2017-07-31"
A function as below would do the work (assume dt is scalar) -
month_end <- function(dt) {
d <- seq(dt, dt+31, by="days")
max(d[format(d,"%m")==format(dt,"%m")])
}
If you have a vector of Dates, then do the following -
sapply(dates, month_end)
you can use timeperiodsR
date.start.month=seq(as.Date("2012-01-01"),length=4,by="months")
df=data.frame(date.start.month)
df$date.start.month
# install.packages("timeperiodsR")
pm <- previous_month(df$date.start.month[1]) # get previous month
start(pm) # first day of previous month
end(pm) # last day of previous month
seq(pm) # vector with all days of previous month
We can also use bsts::LastDayInMonth:
transform(df, date.end.month = bsts::LastDayInMonth(df$date.start.month))
# date.start.month date.end.month
# 1 2012-01-01 2012-01-31
# 2 2012-02-01 2012-02-29
# 3 2012-03-01 2012-03-31
# 4 2012-04-01 2012-04-30
tidyverse has added the clock package in addition to the lubridate package that has nice functionality for this:
library(clock)
date_build(2012, 1:12, 31, invalid = "previous")
# [1] "2012-01-31" "2012-02-29" "2012-03-31" "2012-04-30" "2012-05-31" "2012-06-30"
# [7] "2012-07-31" "2012-08-31" "2012-09-30" "2012-10-31" "2012-11-30" "2012-12-31"
The invalid argument specifies what to do with an invalid date (e.g. 2012-02-31). From the documentation:
"previous": The previous valid instant in time.
"previous-day": The previous valid day in time, keeping the time of
day.
"next": The next valid instant in time.
"next-day": The next valid day in time, keeping the time of day.
"overflow": Overflow by the number of days that the input is invalid
by. Time of day is dropped.
"overflow-day": Overflow by the number of days that the input is
invalid by. Time of day is kept.
"NA": Replace invalid dates with NA.
"error": Error on invalid dates.
How can I add one hour to all the elements of the index of a zoo series?
I've tried
newseries <- myzooseries
index(newseries) <- index(myzooseries)+times("1:00:00")
but I get the message
Incompatible methods ("Ops.dates", "Ops.times") for "+"
thanks
My index is a chron object with date and time but I've tried with simpler examples and I can't get it
This is easily solved by adding the time you want in a numerical fashion :
newseries <- myzooseries
index(newseries) <- index(myzooseries) + 1/24
chron objects are represented as decimal numbers, so you can use that to calculate. A day is 1, so an hour is 1/24, a minute 1/1440 and so on. You can see this easily if you use the function times. This gives you the times of the object tested, eg :
> A <- chron(c("01/01/97","01/02/97","01/03/97"))
> B <- A + 1/24
> B
[1] (01/01/97 01:00:00) (01/02/97 01:00:00) (01/03/97 01:00:00)
> times(A)
Time in days:
[1] 9862 9863 9864
> times(B)
Time in days:
[1] 9862.042 9863.042 9864.042
> times(B-A)
[1] 01:00:00 01:00:00 01:00:00
> times(A[3]-B[1])
Time in days:
[1] 1.958333
Convert to POSIXct, add 60*60 (1h in s) and then convert back.