R - Intervals of date & Number of rows by date - r

Question

I have a question about a faster way to compute something about date intervals
My input:
A data frame : one row by couple (people, period). On each row, I have the ID of a person, a date of start and a date of end.
A period of time : All the dates day by day during two years
What i try to do is to calculate the number of people I have date by date.
I have a code which is working, but not enough efficient with a large dataset (~ from 100 k to 1 M rows).
The current problem is since I have two years of date, my code does 730 times (365x2) the following steps :
Filter the dataset with the specific date included between the start date and the end date
Calculate the number of unique id in the filtered dataset
And these operations are very long or impossible with a large dataset
I am wondering if it exists a better and faster way to do these operations, like with aggregation or with another technique.
An example with a short input and output :
library(lubridate)
library(dplyr)
# Vector of date
vector_day <- seq(ymd('2017-01-01'), ymd('2018-12-30'), by= "days")
# Input Data
df <- data.frame(
id_people = c(1, 2, 3, 4, 1),
StartDate = c(as.Date("2018-11-01"), as.Date("2018-11-03"),as.Date("2018-12-01"),as.Date("2018-11-15") ,as.Date("2018-11-15")),
EndDate = c(as.Date("2018-11-10"), as.Date("2018-12-04"),as.Date("2018-12-10"),as.Date("2018-11-17"), as.Date("2018-11-23")),
Gender = c("F", "F", "M", "F", "F"))
# Function to compute the number of people given a spécific date
compute_nb_f_by_day <- function(date) {
cond1 <- df_f$StartDate <= date
cond2 <- df_f$EndDate > date
cond <- cond1 & cond2
res <- length(unique(df_f[cond,]$id_people))
return(res)
}
# An example of how the function works for on date
compute_nb_f_by_day(as.Date("2018-12-01"))
# Computation for all the dates
nb_f_by_day <- cbind(
data.frame(vector_day),
data.frame(nb_f_by_day <- sapply(vector_day, compute_nb_f_by_day)))
Thanks.

Answer 1

This solution benchmarked significantly faster than your code in the given example (your code: 0.132s; this code: 0.032s in my system). Give it a try to see if it'll significantly improve for the large dataset!
#-- Create the 'Interval'
df2 <- df %>%
mutate(DateInterval = StartDate %--% EndDate)
#-- Create a result df instead of using cbind (more efficient)
result_df <- data.frame(Day = vector_day, Nb = NA)
#-- Get intervals that contain the days in vector_day
result_df$Nb <- sapply(vector_day, function(day) {sum(day %within% df2$DateInterval)})