R: Finding the most recent duplicate in a large data frame - r

Question

I have a large (~18 million records) database of insurance policies, and I need to determine if each policy has been renewed or not. Imagine that a few records look like this: (today is October 5, 2022):
policy_number
prior_policy_number
zip_code
expiration_date
123456
90210
2023-10-01
123456
987654
90210
2022-10-01
987654
90210
2021-10-01
456654
10234
2019-05-01
The first line is a current policy, because 2023-10-01 is in the future.
The second line was renewed (by the first line).
The third line was renewed by the second line--we can tell because the second line's prior policy number matches the third line's policy number.
The fourth line was not renewed.
So a policy is renewed if either:
a) there is another policy with the same policy number and zip code but a later expiration date
b) there is another policy whose prior policy number matches this policy number, they have the same zip code, and the other policy has a later expiration date.
(Zip code is necessary because some insurers use policy numbers like "00000002" and this disambiguates duplicates.)
I wrote the following code, which works but takes forever to execute. Basically, I sort the data frame by descending expiration date, and then for each observation I create a miniature data frame that consists of just policies that have the same policy number or previous policy number and zip code, and then check the expiration data of the first (and therefore latest) one to see if it's later than the policy in question. I realize this is probably a horrible way to do this.
Does anyone have suggestions for how to make it more efficient?
non_renewals <- valid_zip_policies %>% arrange(desc(expiration_date))
check_renewed <- function (policy,zip,exp) {
#We create a subset of the main data frame containing only that policy number, (or policies with this policy as the prior policy number) and filter it for the matching zip code
cat(policy,zip,exp)
test_renewed <- valid_zip_policies %>% select(c("policy_number","prior_policy_number","zip_code","expiration_date")) %>% filter(policy_number == policy | prior_policy_number == policy) %>% filter(zip_code == zip)
#These are all the policies for the given policy number, sorted from latest to earliest expiration date. Is the expiration date of the most recent one later than the expiration date of this one? If so, it was renewed
if (test_renewed$expiration_date[1] > exp) { return (TRUE)} else {return (FALSE)}
}
for (i in 1:nrow(non_renewals)) {
non_renewals$renewed [i] <- check_renewed(non_renewals$policy_number[i],non_renewals$zip_code[i],non_renewals$expiration_date[i])
}

Answer 1

So I was able to answer my own question! The following code is literally about 100 times faster! Two things helped:
by far the greatest speed boost was from using data tables from package data.table rather than data frames. That package also has the fifelse command you see below.
using package parallel and its mclapply command gave an additional speed boost on my system.
It may also have helped that instead of passing three items from the original table to the function, I just pass the number and let the function retrieve the items as necessary.
non_renewals <- setDT(non_renewals)
check_renewed <- function (obs) {
#If expiration date of latest example is later, then it was renewed
if (non_renewals[policy_number==policy_number[obs] & zip_code==zip_code[obs],expiration_date][1] > non_renewals$expiration_date[obs]) {return("RENEWED")}
#If not, check the prior policies
final <- fifelse(non_renewals[prior_policy_number==policy_number[obs] & zip_code==zip_code[obs],expiration_date][1] > non_renewals$expiration_date[obs],"RENEWED","NONRENEWED",na="NONRENEWED")
return(final)
}
renewed <- character(10000)
system.time(renewed <- mclapply(1:10000,function (i) {check_renewed(i)}))