How to replace NAs by group mean? - r

Question

I have two data frames, one containing the raw data (and lots of NAs) and one containing the means of the raw data at different intervals.
I would like to replace the NAs with the means at those intervals, while retaining the non-NAs.
I have googled a lot on "impute missing data r", or "replace missing data r", but have yet to find any solution that seems to fit, they all either seem to replace data with 0:s, or use a way to complex method for this assignment, like using the MICE package.
Code example:
This is the head of the first DF, with the raw data. As you can see, they are all NA for the first day.
steps date interval
1 NA 2012-10-01 0
2 NA 2012-10-01 5
3 NA 2012-10-01 10
4 NA 2012-10-01 15
5 NA 2012-10-01 20
6 NA 2012-10-01 25
...
The second data frame head contains the mean steps per interval, like this:
steps interval
1 1.72 0
2 0.340 5
3 0.132 10
4 0.151 15
5 0.0755 20
6 2.09 25
...
Now, what I am looking for is to be able to fill the NAs with the mean steps for the relevant interval, so it looks like this:
steps date interval
1 1.72 2012-10-01 0
2 0.340 2012-10-01 5
3 0.132 2012-10-01 10
4 0.151 2012-10-01 15
5 0.0755 2012-10-01 20
6 2.09 2012-10-01 25
...
Any tips, or links for resources I have missed? As this is a course assignment and I mostly wanna learn, any help without doing the assignment for me would be much appreciated! =)
edit: Also, since this is my first question on Stack Overflow, any comments on how to improve my question-making is also appreciated!

Answer 1

There are many ways to do that in R. For example
# generate dataframe with some interval vaulues
df1 <- data.frame(interval= rep(seq(0, 25, 5), 5))
# add a steps column
df1$steps <- 1:nrow(df)
# copy the dataframe
df2 <- df1
# replace some steps values with missings in df1
df1$steps[c(1,2,5, 14)] <- NA
# sapply goes thru every unique interval...
sapply(df1$interval, function(interval_i){
# replace missing steps of interval_i in df1 with the steps mean of interval_i of df2
df1$steps[is.na(df1$steps) & df1$interval == interval_i] <<- mean(df2$steps[df2$interval == interval_i], na.rm= TRUE)
# you must use <<- not <- to assign it to df outside of sapply
})

Answer 2

I'll add a solution using dplyr.
It's good practice to give some reproducible data in your question rather that can be put straight into R, rather than just pasting in the head of your data. I've created some dummy data instead:
# create random sample data
library(dplyr)
set.seed(100)
df1 <- tibble(
steps = runif(1e3),
date = lubridate::today() + runif(1e3) * 24,
interval = as.numeric(sample(seq(0,25, by=5), 1e3, replace = T))
)
# add 100 NAs at random
df1$steps[sample(1:1e3, 100)] <- NA
df1
# steps date interval
# <dbl> <date> <dbl>
# 1 0.308 2019-07-18 15
# 2 NA 2019-07-19 10
# 3 NA 2019-07-31 0
# 4 0.0564 2019-08-02 20
# 5 0.469 2019-07-25 0
# 6 0.484 2019-07-21 25
# 7 NA 2019-07-17 5
# 8 0.370 2019-07-28 0
# 9 0.547 2019-07-31 5
# 10 0.170 2019-08-08 15
# # … with 990 more rows
Using dplyr the imputation task is then pretty simple with group_by
df1 %>%
group_by(interval) %>%
mutate(steps = if_else(is.na(steps), mean(steps, na.rm = T), steps))
# # A tibble: 1,000 x 3
# # Groups: interval [6]
# steps date interval
# <dbl> <date> <dbl>
# 1 0.308 2019-07-18 15
# 2 0.573 2019-07-19 10
# 3 0.523 2019-07-31 0
# 4 0.0564 2019-08-02 20
# 5 0.469 2019-07-25 0
# 6 0.484 2019-07-21 25
# 7 0.527 2019-07-17 5
# 8 0.370 2019-07-28 0
# 9 0.547 2019-07-31 5
# 10 0.170 2019-08-08 15
# # … with 990 more rows
We can confirm that the imputed means are correct for each group by calculating the mean for each group and comparing it to the imputed values:
df1 %>%
group_by(interval) %>%
summarise(mean_int = mean(steps, na.rm=T))
# # A tibble: 6 x 2
# interval mean_int
# <dbl> <dbl>
# 1 0 0.523
# 2 5 0.527
# 3 10 0.573
# 4 15 0.511
# 5 20 0.475
# 6 25 0.485