Extracting latest non-NA value in data frame based on grouping - r

Question

I have a data frame that looks like the following:
Year Day ID V1 V2 ....
2003 35 1102 3 6
2003 35 1103 5 NA
2003 35 1104 8 100
.....
2003 40 1102 NA 8
2003 40 1103 NA 10
2003 40 1104 9 NA
.....
.....
2018 49 1104 5 NA
.....
2018 50 1102 3 6
2018 50 1103 7 NA
2018 50 1104 NA 100
I would like to build a data frame that extracts, for each combination of Year and ID, the the latest (high value per the Day column) non-NA value in V1, V2... Based on the above data set, for Year = 2018 and ID = 1104, I would like to extract V1 = 5 (on Day = 49) and V2 = 100 (on Day = 50). If all values for that Year and ID combination are NA then I would like it to return NA.

Answer 1

We can create a function which gives us the latest non-NA value based on Day for each Vn column
get_last_non_NA_value <- function(x) {
x[which.max(cumsum(!is.na(x)))]
}
and then apply that function for each Year and ID
library(dplyr)
df %>%
group_by(Year, ID) %>%
summarise_at(vars(V1:V2), funs(get_last_non_NA_value(.[order(Day)])))
# Year ID V1 V2
# <int> <int> <int> <int>
#1 2003 1102 3 8
#2 2003 1103 5 10
#3 2003 1104 9 100
#4 2018 1102 3 6
#5 2018 1103 7 NA
#6 2018 1104 5 100
EDIT
If we also want to extract corresponding Day for each value, we can change the function to return both values as comma-separated string
get_last_non_NA_value <- function(x, y) {
ind <- which.max(cumsum(!is.na(x[order(y)])))
paste(x[ind], y[ind], sep = ",")
}
and then use cSplit to separate these comma separated values into different columns.
library(dplyr)
library(splitstackshape)
cols <- c("V1", "V2")
df %>%
group_by(Year, ID) %>%
summarise_at(cols, funs(get_last_non_NA_value(., Day))) %>%
cSplit(cols) %>%
rename_at(vars(contains("_1")), funs(sub("_1", "_last_value", .))) %>%
rename_at(vars(contains("_2")), funs(sub("_2", "_days", .)))
# Year ID V1_last_value V1_days V2_last_value V2_days
#1: 2003 1102 3 35 8 40
#2: 2003 1103 5 35 10 40
#3: 2003 1104 9 40 100 35
#4: 2018 1102 3 50 6 50
#5: 2018 1103 7 50 NA 50
#6: 2018 1104 5 49 100 50
Note that rename_at part renames the columns for better understanding of what value it holds, you can skip that part if you are not interested in renaming columns.
data
df <- structure(list(Year = c(2003L, 2003L, 2003L, 2003L, 2003L, 2003L,
2018L, 2018L, 2018L, 2018L), Day = c(35L, 35L, 35L, 40L, 40L,
40L, 49L, 50L, 50L, 50L), ID = c(1102L, 1103L, 1104L, 1102L,
1103L, 1104L, 1104L, 1102L, 1103L, 1104L), V1 = c(3L, 5L, 8L,
NA, NA, 9L, 5L, 3L, 7L, NA), V2 = c(6L, NA, 100L, 8L, 10L, NA,
NA, 6L, NA, 100L)), .Names = c("Year", "Day", "ID", "V1", "V2"
), class = "data.frame", row.names = c(NA, -10L))

Answer 2

You can use dplyr
Assuming you want max for V1 and V2
library(dplyr)
df %>%
group_by(Year, ID) %>%
summarise(Day = max(Day, na.rm = TRUE),
V1 = max(V1, na.rm = TRUE),
V2 = max(V2, na.rm = TRUE))
If for V1 and V2, you want first non-NA then
df %>%
group_by(Year, ID) %>%
summarise(Day = max(Day, na.rm = TRUE),
V1 = first(setdiff(V1, NA)),
V2 = first(setdiff(V1, NA)))