Memory leak when using MNP package in R - r

Question

I have a question concerning memory use in R when using the MNP package. My goal is to estimate a multinomial probit model and then using the model to predict choices on a large set of data. I have split the predictor data in a list of pieces.
The problem is that when I loop over the list to predict, the memory used by R grows constantly and uses swap space after reaching the maximum memory of my computer. The allocated memory is not released even when hitting those boundaries. This happens even though I do not create any additional objects and so I don't understand what is going on.
Below I pasted an example code that suffers from the described problem. When running the example, the memory grows constantly and remains used even after removing all variables and calling gc().
The real data I have is much larger than what is generated in the example, so I need to find a workaround.
My questions are:
Why does this script use so much memory?
How can force R to release the allocated memory after each step?
library(MNP)
nr <- 10000
draws <- 500
pieces <- 100
# Create artificial training data
trainingData <- data.frame(y = sample(c(1,2,3), nr, rep = T), x1 = sample(1:nr), x2 = sample(1:nr), x3 = sample(1:nr))
# Create artificial predictor data
predictorData <- list()
for(i in 1:pieces){
predictorData[[i]] <- data.frame(y = NA, x1 = sample(1:nr), x2 = sample(1:nr), x3 = sample(1:nr))
}
# Estimate multinomial probit
mnp.out <- mnp(y ~ x1 + x2, trainingData, n.draws = draws)
# Predict using predictor data
predicted <- list()
for(i in 1:length(predictorData)){
cat('|')
mnp.pred <- predict(mnp.out, predictorData[[i]], type = 'prob')$p
mnp.pred <- colnames(mnp.pred)[apply(mnp.pred, 1, which.max)]
predicted[[i]] <- mnp.pred
rm(mnp.pred)
gc()
}
# Unite output into one string
predicted <- factor(unlist(predicted))
Here are the output statistics after running the script:
> rm(list = ls())
> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 158950 8.5 407500 21.8 407500 21.8
Vcells 142001 1.1 33026373 252.0 61418067 468.6
Here are my specifications of R:
> sessionInfo()
R version 2.13.1 (2011-07-08)
Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)
locale:
[1] en_US.UTF-8/en_US.UTF-8/C/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] MNP_2.6-2 MASS_7.3-14

Answer 1

The results don't seem anomalous, as in I don't think this evidences a memory leak. I suspect that you're misreading the data from gc(): the right hand column is the maximum memory used during the tracking of memory by R. If you use gc(reset = TRUE), then the maximum shown will be the memory used in the LHS, i.e. the 8.5MB and 1.1MB listed under "used".
I suspect that MNP just consumes a lot of memory during the prediction phase, so there's not much that can be done, other than to break up the prediction data into even smaller chunks, with fewer rows.
If you have multiple cores, you might consider using the foreach package, along with doSMP or doMC, as this will give you the speedup of independent calculations and the benefit of clearing the RAM allocated after each iteration of the loop is complete (as it involves a fork of R that uses a separate memory space, I believe).