Tackling missing data in R - r

Question

I have encountered this problem in a project that I'm currently doing.
I have a sparse dataframe and I need to calculate the difference between the first and the last observation per row under some conditions:
Conditions:
If the row only contains NA's then the difference is 0.
If the row contains only 1 observation then the difference is 0.
If row elements (>= 2) are non-NA's then their difference is the difference between the first and the last (tail - head).
The dataframe that I have:
S1 S2 S3 S4 S5
1 NA NA NA NA NA
2 NA 3 NA 5 NA
3 1 NA NA NA 5
4 1 NA 2 NA 7
5 2 NA NA NA NA
6 NA NA 3 4 NA
7 NA NA 3 NA NA
The dataframe that I need:
S1 S2 S3 S4 S5 diff
1 NA NA NA NA NA 0
2 NA 3 NA 5 NA 2
3 1 NA NA NA 5 4
4 1 NA 2 NA 7 6
5 2 NA NA NA NA 0
6 NA NA 3 4 NA 1
7 NA NA 3 NA NA 0
What I've written up till now:
last_minus_first <- function(x, y = na.omit(x)) tail(y, 1) - y[1]
But it doesn't resolve for the fact if the row contains all NA's.
Any help would be much appreciated.

Answer 1

I would suggest using a defined function with apply(). Here the code:
#Data
df <- structure(list(S1 = c(NA, NA, 1L, 1L, 2L, NA, NA), S2 = c(NA,
3L, NA, NA, NA, NA, NA), S3 = c(NA, NA, NA, 2L, NA, 3L, 3L),
S4 = c(NA, 5L, NA, NA, NA, 4L, NA), S5 = c(NA, NA, 5L, 7L,
NA, NA, NA)), class = "data.frame", row.names = c("1", "2",
"3", "4", "5", "6", "7"))
Code:
#Function
myown <- function(x)
{
#Check NA
i <- sum(!is.na(x))
#Compute
if(i<=1)
{
y <- 0
} else
{
#Detect positions
j1 <- max(which(!is.na(x)))
j2 <- min(which(!is.na(x)))
#Diff
y <- x[j1]-x[j2]
}
return(y)
}
#Apply function by row
df$NewVar <- apply(df,1,myown)
Output:
S1 S2 S3 S4 S5 NewVar
1 NA NA NA NA NA 0
2 NA 3 NA 5 NA 2
3 1 NA NA NA 5 4
4 1 NA 2 NA 7 6
5 2 NA NA NA NA 0
6 NA NA 3 4 NA 1
7 NA NA 3 NA NA 0

Answer 2

Here's an easier (in my mind) way to handle this, using rowwise from the dplyr package to do calculations along rows.
df %>%
dplyr::rowwise() %>%
dplyr::mutate(max_pop = max(which(!is.na(dplyr::c_across(S1:S5)))),
min_pop = min(which(!is.na(dplyr::c_across(S1:S5)))),
diff = tidyr::replace_na(dplyr::c_across()[max_pop] - dplyr::c_across()[min_pop], 0))
I've broken that mutate call down into the various parts to show what we're doing, but essentially, it goes across all columns in a row to find the last populated column (max_pop), the first populated column (min_pop) and then uses those values to retrieve the values therein.
You have to specify columns for max_pop and min_pop above because creating new interim columns affects the column indexing. c_across() defaults to using all columns, though, so you can actually do this all in one mutate call without specifying any columns.
df %>%
rowwise() %>%
mutate(diff = replace_na(c_across()[max(which(!is.na(c_across())))] - c_across()[min(which(!is.na(c_across())))], 0))

Answer 3

A vectorized option in base R would be to extract the values based on row/column index and then subtract
df1$NewVar <- df1[cbind(seq_len(nrow(df1)), max.col(!is.na(df1), 'last'))] -
df1[cbind(seq_len(nrow(df1)), max.col(!is.na(df1), 'first'))]
df1$NewVar[is.na(df1$NewVar)] <- 0
df1
# S1 S2 S3 S4 S5 NewVar
#1 NA NA NA NA NA 0
#2 NA 3 NA 5 NA 2
#3 1 NA NA NA 5 4
#4 1 NA 2 NA 7 6
#5 2 NA NA NA NA 0
#6 NA NA 3 4 NA 1
#7 NA NA 3 NA NA 0
data
df1 <- structure(list(S1 = c(NA, NA, 1L, 1L, 2L, NA, NA), S2 = c(NA,
3L, NA, NA, NA, NA, NA), S3 = c(NA, NA, NA, 2L, NA, 3L, 3L),
S4 = c(NA, 5L, NA, NA, NA, 4L, NA), S5 = c(NA, NA, 5L, 7L,
NA, NA, NA)), class = "data.frame", row.names = c("1", "2",
"3", "4", "5", "6", "7"))