R mapply function running slowly - r

Question

I'm working on calculating item statistics for a series of multiple choice exams. I've got a solution using mapply that technically works, but it takes a few hours to calculate one of the more complicated statistics. The first dataset that I have is one that includes a separate row for every question that every student answered for every assessment.
df <- data.frame(c(rep("s1", 5), rep("s2", 5), rep("s3", 5),rep("s4", 5)),"a1", c("i1", "i2", "i3", "i4", "i5"), c(1, 0), 1)
colnames(df) <- c("student", "assessment", "item", "score", "points.possible")
The first step that I do (and only do once) is to create a table of all unique items. In this case, that would be simple, as there's only one assessment and 5 items.
unique <- subset(df[,c("assessment", "item")], !duplicated(df[,c("assessment", "item")]))
I then need to calculate a statistic for each one of these items. However, the tricky part is that the calculations requires calculating the overall scores that students got on the entire assessment. Here's the function I wrote to do that.
fun1 <- function(a.id, i.id) {
# subset original dataframe for just one assessment
subsetdf <- df[df$assessment == a.id,]
# generate list of students that got the item right and wrong
correct <- subsetdf$student[subsetdf$item==i.id & subsetdf$score==1]
wrong <- subsetdf$student[subsetdf$item==i.id & subsetdf$score==0]
# scores by student
scores <- aggregate(score ~ student, data=subsetdf,sum)/aggregate(points.possible ~ student, data=subsetdf, sum)
# average scores for students that got item right/wrong
x.1 <- sum(subsetdf$score[subsetdf$student %in% correct])/sum(subsetdf$points.possible[subsetdf$student %in% correct])
x.0 <- sum(subsetdf$score[subsetdf$student %in% wrong])/sum(subsetdf$points.possible[subsetdf$student %in% wrong])
# percent of students that got item right
p <- length(correct)/(length(correct)+length(wrong))
# final stat calculation
r <- ((x.1-x.0)*sqrt(p*(1-p)))/sd(scores[,2])
print(r)
}
I then used mapply to loop this function over the entire original dataset while using the smaller dataset for the inputs.
unique$r <- mapply(fun1, unique$assessment, unique$item)
I was happy that I was able to get it to work, but when I do it with the larger datasets (~7 million rows for "df", ~2000 rows for "unique), it takes quite a while (several hours). Any tips on other ways to tackle this problem that are more efficient? I've learned that one problem is that my function creates a copy of the original large dataset every time it loops through, but I don't know how to do the problem without that.
I still consider myself a beginner for this kind of usage for R, so any advice would be much appreciated!

Answer 1

When you perform
scores <- aggregate(score ~ student, data=subsetdf,sum)/aggregate(points.possible ~ student, data=subsetdf, sum)
the result is not strictly numeric, the result is a data frame (for example, for a.id = 'a1', i.id = 'i1'):
> aggregate(score ~ student, data=subsetdf,sum)
student score
1 s1 3
2 s2 2
3 s3 3
4 s4 2
So when you divide the two, the result of 's1'/'s1' is not numeric and throws a warning.
There is no need to create correct and wrong. Treat the value of that column as an indicator to tell you whether the student was right or wrong.
Instead, do the following:
scores <- aggregate(subsetdf[,c('score', 'points.possible')], by = list(subsetdf$student), sum)
names(scores) <- c('student', 'score','points.possible')
scores$avg.score <- scores$score/scores$points.possible
I would do the same for x.0 and x.1. If you create a subset by i.id and then aggregate that subset of the dataframe, this should also save you a few steps.
The fact that you are checking for each student whether or not they are in correct or wrong twice (for score and points.possible) is also pretty expensive.