Subset and loop calculation over each column in dplyr - r

Question

I have a dataframe of values for 50 IDs repeated over 10 iterations. I would like to subset by ID and then perform calculations, and repeat that for each column from x1 to x5. I used a for-loop but it is very inefficient (my actual dataset has a lot more IDs).
Here are the calculations I would like to perform. I've had varying success with the conversion to dplyr:
First calculation, gives me the correct value for x1, but need to repeat for each column from x1 to x5.
V1.x1 <- preds.df %>%
split(.$ID) %>%
sapply(function(ID) {
(ID$x1 - mean(ID$x1))^2 # for X1 only
}) %>%
mean()
A different calculation that involves subtracting from a corresponding value in another df data.pop. My dplyr attempt is wrong even for just x1:
## This is what I want to achieve, which I implemented using for-loop:
# df for for-loop
Bsq.perID <- data.frame(matrix(NA,
nrow = nrow(data.pop), # 50 observations
ncol = 5) # 5 models
# For-loop:
for (ids in 1:nrow(data.pop)){
current.ID <- preds.df[preds.df$ID == ids, ] # get current ID over all 10 iterations
for (i in 1:5){
Bsq.perID[ids, i] <- (mean(current.ID[, i]) - data.pop[ids, "real.val"])^2
}
}
Bsq.values <- colMeans(Bsq.perID)
## My wrong dplyr attempt of the above:
B1.x1 <- preds.df %>%
split(.$ID) %>%
sapply(function(ID) {
(mean(ID$x1) - data.pop$real.val)^2
}) %>%
mean()
The structure of preds.df looks like this:
head(preds.df)
x1 x2 x3 x4 x5 iteration ID
1 20.005984 6.78242996 3.526411 21.463892 8.792720 1 1
2 2.890490 7.28232755 18.670470 6.717213 19.830930 1 2
3 4.868658 24.88117301 1.883913 3.897779 14.371414 1 3
4 6.495532 5.79591685 7.745554 20.153269 7.935672 1 4
5 19.297779 0.05068784 21.744816 14.957751 14.232126 1 5
6 7.090456 22.06322779 8.388263 10.672151 9.921884 1 6
tail(preds.df)
x1 x2 x3 x4 x5 iteration ID
495 16.306927 2.8873609 9.7764755 23.798867 10.246443 10 45
496 4.767296 23.2086303 8.8394391 7.806442 24.898483 10 46
497 19.966301 13.7151699 10.2483011 15.199162 9.658736 10 47
498 18.134534 22.1658901 5.6481757 18.501411 23.787457 10 48
499 7.877636 7.2356274 8.2862336 3.790823 11.610848 10 49
500 8.554774 0.9199501 0.9650191 17.155611 1.158619 10 50

Answer 1

I would approach it like this:
library(dplyr)
library(rio)
preds.df <- import("~/Downloads/preds.df.csv")
data.pop <- import("~/Downloads/data.pop.csv")
## added a row because data.pop is only 49 rows in the data you sent
data.pop <- bind_rows(data.pop, data.pop[1,])
You could use dplyr with mutate() to do this:
dat1 <- preds.df %>%
group_by(ID) %>%
mutate(across(x1:x5, function(x)(x-mean(x))^2))
Then, for the second part you could merge the data
data.pop <- data.pop %>%
mutate(ID = 1:n())
dat2 <- dat1 %>% left_join(data.pop)
Next, summarise on ID to calculate the mean of x1 to x5 within ID, then from each one, you can subtract real.val and square.
dat2 <- dat2 %>%
select(c(ID, x1:x5, real.val)) %>%
group_by(ID) %>%
mutate(across(x1:x5, function(x)(x-real.val)^2)) %>%
summarise_all(mean) %>%
select(-real.val)