Forward fill data with conditions - r

Question

I have a dataframe, DF, looking like this:
date permno ret sue bm gpa
1 202001 10000 0.01 0.4 0.4 NA
2 202002 10000 0.04 NA NA 0.5
3 202003 10000 -0.01 NA NA NA
4 202004 10000 0.00 1.3 0.5 NA
5 202005 10000 0.02 NA NA 0.3
6 202006 10000 0.01 NA NA NA
7 202007 10000 0.03 NA NA NA
8 202008 10000 -0.02 NA NA 0.4
9 202001 11000 0.05 0.1 0.3 NA
10 202002 11000 0.02 NA NA NA
11 202003 11000 0.01 NA NA NA
12 202004 11000 0.00 NA NA 0.3
13 202005 11000 0.01 NA NA NA
14 202006 11000 -0.01 NA NA NA
15 202007 11000 0.04 0.5 0.4 NA
16 202008 11000 0.30 NA NA NA
I am using this code to forward fill the variables sue, bm and gpa:
DF1 <-
DF %>%
arrange(permno,date) %>%
group_by(permno) %>%
mutate_at(vars(c(sue,bm,gpa)), funs(na.locf(.,na.rm=FALSE)))
This results in the following
date permno ret sue bm gpa
1 202001 10000 0.01 0.4 0.4 NA
2 202002 10000 0.04 0.4 0.4 0.5
3 202003 10000 -0.01 0.4 0.4 0.5
4 202004 10000 0 1.3 0.5 0.5
5 202005 10000 0.02 1.3 0.5 0.3
6 202006 10000 0.01 1.3 0.5 0.3
7 202007 10000 0.03 1.3 0.5 0.3
8 202008 10000 -0.02 1.3 0.5 0.4
9 202001 11000 0.05 0.1 0.3 NA
10 202002 11000 0.02 0.1 0.3 NA
11 202003 11000 0.01 0.1 0.3 NA
12 202004 11000 0 0.1 0.3 0.3
13 202005 11000 0.01 0.1 0.3 0.3
14 202006 11000 -0.01 0.1 0.3 0.3
15 202007 11000 0.04 0.5 0.4 0.3
16 202008 11000 0.3 0.5 0.4 0.3
I want to put a limit on how many months the data is forward filled. I want to forward fill the three variables until the next available value, but maximum 3 months. Thus, the results should look like this:
date permno ret sue bm gpa
1 202001 10000 0.01 0.4 0.4 NA
2 202002 10000 0.04 0.4 0.4 0.5
3 202003 10000 -0.01 0.4 0.4 0.5
4 202004 10000 0.00 1.3 0.5 0.5
5 202005 10000 0.02 1.3 0.5 0.3
6 202006 10000 0.01 1.3 0.5 0.3
7 202007 10000 0.03 1.3 0.5 0.3
8 202008 10000 -0.02 NA NA 0.4
9 202001 11000 0.05 0.1 0.3 NA
10 202002 11000 0.02 0.1 0.3 NA
11 202003 11000 0.01 0.1 0.3 NA
12 202004 11000 0.00 0.1 0.3 0.3
13 202005 11000 0.01 NA NA 0.3
14 202006 11000 -0.01 NA NA 0.3
15 202007 11000 0.04 0.5 0.4 0.3
16 202008 11000 0.30 0.5 0.4 NA
Does anyone know how I can do this in R?

Answer 1

We can write our own na.locf() that allows to make the adjustments you desire:
Code
library(zoo)
library(dplyr)
na.locf2 <- function(object, period = 3, ...){
# consecutive NAs
tmp1 <- rle(is.na(object))
# NA count in the length of the vector
tmp2 <- unlist(sapply(tmp1[[1]] , function(x){
1:x
}))
# remove all NAs
tmp3 <- na.locf(object, ...)
# reassign those that are greater than the desired period
tmp3[tmp2 > period] <- NA
# return
tmp3
}
# Then
DF %>%
arrange(permno,date) %>%
group_by(permno) %>%
mutate_at(vars(c(sue,bm,gpa)), funs(na.locf2(.,na.rm=FALSE)))
# Yields
# A tibble: 16 x 6
# Groups: permno [2]
# date permno ret sue bm gpa
# <int> <int> <dbl> <dbl> <dbl> <dbl>
# 1 202001 10000 0.01 0.4 0.4 NA
# 2 202002 10000 0.04 0.4 0.4 0.5
# 3 202003 10000 -0.01 0.4 0.4 0.5
# 4 202004 10000 0 1.3 0.5 0.5
# 5 202005 10000 0.02 1.3 0.5 0.3
# 6 202006 10000 0.01 1.3 0.5 0.3
# 7 202007 10000 0.03 1.3 0.5 0.3
# 8 202008 10000 -0.02 NA NA 0.4
# 9 202001 11000 0.05 0.1 0.3 NA
# 10 202002 11000 0.02 0.1 0.3 NA
# 11 202003 11000 0.01 0.1 0.3 NA
# 12 202004 11000 0 0.1 0.3 0.3
# 13 202005 11000 0.01 NA NA 0.3
# 14 202006 11000 -0.01 NA NA 0.3
# 15 202007 11000 0.04 0.5 0.4 0.3
# 16 202008 11000 0.3 0.5 0.4 NA

Answer 2

This sounds like a rolling-window thing. Since you need to limit the carry-forward, though, one problem is that when you look at a particular cell, its predecessor has already been fixed (un-NA'd), so we need to look at the vector in reverse.
A helper function, where 2:4 is based on your preference for no more than three months. In the context of a reversed rollapply, z[1] in this case is likely an NA with z[2:4] being the three preceding months.
func <- function(z) if (is.na(z[1])) na.omit(z[2:4])[1] else z[1]
Since we're using partial=TRUE in the rolling, it is feasible that z will not be length-4; this is fine, though, since even z[100000] will return NA, which is explicitly handled by na.omit. (This condition can also be addressed with a test on length(z).)
Another note: na.omit(.) can return a length-0 vector, which is obviously bad in this context. However, adding [1] after it forces it to return an NA, which is what we need it to be if no non-NA elements exist within z[2:4].
From here, most of the work is done by zoo::rollapply:
DF %>%
arrange(permno, date) %>%
group_by(permno) %>%
mutate(across(sue:gpa, ~ rev(zoo::rollapply(rev(.), 4, align="left", FUN = func, partial = TRUE)))) %>%
ungroup()
# # A tibble: 16 x 6
# date permno ret sue bm gpa
# <int> <int> <dbl> <dbl> <dbl> <dbl>
# 1 202001 10000 0.01 0.4 0.4 NA
# 2 202002 10000 0.04 0.4 0.4 0.5
# 3 202003 10000 -0.01 0.4 0.4 0.5
# 4 202004 10000 0 1.3 0.5 0.5
# 5 202005 10000 0.02 1.3 0.5 0.3
# 6 202006 10000 0.01 1.3 0.5 0.3
# 7 202007 10000 0.03 1.3 0.5 0.3
# 8 202008 10000 -0.02 NA NA 0.4
# 9 202001 11000 0.05 0.1 0.3 NA
# 10 202002 11000 0.02 0.1 0.3 NA
# 11 202003 11000 0.01 0.1 0.3 NA
# 12 202004 11000 0 0.1 0.3 0.3
# 13 202005 11000 0.01 NA NA 0.3
# 14 202006 11000 -0.01 NA NA 0.3
# 15 202007 11000 0.04 0.5 0.4 0.3
# 16 202008 11000 0.3 0.5 0.4 NA