Consecutive/Unbroken occurrences in time series - r

Question

I'm working with a data table in R containing information quarterly information on the products being sold in grocery stores across the United States. In particular, there is a column for the date, a column for the store, and a column for the product. For example, here is a (very small) subset of the data:
Date StoreID ProductID
2000-03-31 10001 20001
2000-03-31 10001 20002
2000-03-31 10002 20001
2000-06-30 10001 20001
For each product in each store, I want to find out for how many consecutive quarters the product has been sold in that store up until that date. For example, if we restrict to just looking at staplers that sold in a specific store, we would have:
Date StoreID ProductID
2000-03-31 10001 20001
2000-06-30 10001 20001
2000-09-30 10001 20001
2000-12-31 10001 20001
2001-06-30 10001 20001
2001-09-30 10001 20001
2001-12-31 10001 20001
Assuming that's all the data for that combination of StoreID and ProductID, I want to assign a new variable as:
Date StoreID ProductID V
2000-03-31 10001 20001 1
2000-06-30 10001 20001 2
2000-09-30 10001 20001 3
2000-12-31 10001 20001 4
2001-06-30 10001 20001 1
2001-09-30 10001 20001 2
2001-12-31 10001 20001 3
2002-03-31 10001 20001 4
2002-06-30 10001 20001 5
2002-09-30 10001 20001 6
2002-12-31 10001 20001 7
2004-03-30 10001 20001 1
2004-06-31 10001 20001 2
Note that we roll over after Q4 2000 because the product was not sold during Q1 of 2001. Additionally, we roll over after Q4 2002 because the product was not sold during Q1 2003. The next time the product was sold was Q1 2004, which was assigned a 1.
The issue I am having, is that my actual dataset is quite large (on the order of 10 Million rows) so this needs to be done efficiently. The only techniques I've been able to come up with are dreadfully inefficient. Any advice would be greatly appreciated.

Answer 1

You can use custom function that calculate difference between quarters.
# Load data.table
library(data.table)
# Set data as a data.table object
setDT(data)
# Set key as it might be big data
setkey(data, StoreID, ProductID)
consecutiveQuarters <- function(date, timeGap = 14) {
# Calculate difference in dates
# And check if this difference is less than 14 weeks
shifts <- cumsum(c(FALSE, abs(difftime(date[-length(date)], date[-1], units = "weeks")) > timeGap))
# Generate vector from 1 to number of consecutive quarters
ave(shifts, shifts, FUN = seq_along)
}
# Calculate consecutive months my storeID and productID
data[, V := consecutiveQuarters(Date), .(StoreID, ProductID)]

Answer 2

Create a variable which is 1 if the product is sold in a quarter and 0 if not. Order the variable so it starts at the present and goes backward in time.
Compare the cumulative sum of such a variable to a sequence of identical length. When sales drop to zero the cumulative sum will no longer equal the sequence. Sum the number of times the cumulative sum equals the sequence and this will tell the number of consecutive quarters sales were positive.
data <- data.frame(
quarter = c(1, 2, 3, 4, 1, 2, 3, 4),
store = as.factor(c(1, 1, 1, 1, 1, 1, 1, 1)),
product = as.factor(c(1, 1, 1, 1, 2, 2, 2, 2)),
numsold = c(5, 6, 0, 1, 7, 3, 2, 14)
)
sortedData <- data[order(-data$quarter),]
storeValues <- c("1")
productValues <- c("1","2")
dataConsec <- data.frame(store = NULL, product = NULL, ConsecutiveSales = NULL)
for (storeValue in storeValues ){
for(productValue in productValues){
prodSoldinQuarter <-
as.numeric(sortedData[sortedData$store == storeValue &
sortedData$product == productValue,]$numsold > 0)
dataConsec <- rbind(dataConsec,
data.frame(
store = storeValue,
product = productValue,
ConsecutiveSales =
sum(as.numeric(cumsum(prodSoldinQuarter) ==
seq(1,length(prodSoldinQuarter))
))
))
}
}

Answer 3

As I understand from your question your really need your V column as quarter of year, not sum of product in each quarter. You can use something like that.
# to_quarters returns year's quarter of given date in character string
# base on reg exp
to_quarters <- function(date_string) {
month <- as.numeric(substr(date_string, 6, 7))
as.integer((month - 1) / 3) + 1
}
# with tidyverse library
library(tidyverse)
# your data as tibble format of data frame
data_set_tibble <- as.tibble(YOUR_DATA)
# here you create your table
data_set_tibble %>% mutate(V = to_quarters(Date) %>% as.integer())
# alterative with data.table library
library(data.table)
# your data as data.table format of data frame
data_set <- as.data.table(YOUR_DATA)
# here you create your table
data_set[,.(Date, StoreID, ProductID, V = to_quarters(Date))]
For tidyverse and data.table performance is a same, in my case for 5 000 000 rows works in 12 seconds