How train data manually per fold with k-fold CV in R? - r

Question

I have the following code segment which works for me and I get the model result:
library(base)
library(caret)
library(tidyverse)
dataset <- read_csv("https://gist.githubusercontent.com/dmpe/bfe07a29c7fc1e3a70d0522956d8e4a9/raw/7ea71f7432302bb78e58348fede926142ade6992/pima-indians-diabetes.csv", col_names=FALSE)
X = dataset[, 1:8]
Y = as.factor(ifelse(dataset$X9 == 1, 'diabetes', 'nondiabetes'))
set.seed(88)
nfolds <- 3
cvIndex <- createFolds(Y, nfolds, returnTrain = T)
fit.control <- trainControl(method="cv",
index=cvIndex,
number=nfolds,
classProbs=TRUE,
savePredictions=TRUE,
verboseIter=TRUE,
summaryFunction=twoClassSummary,
allowParallel=FALSE)
model <- caret::train(X, Y,
method = "svmLinear",
trControl = fit.control,
preProcess=c("center","scale"),
tuneLength=10)
Using this I can access the final model as model$finalModel, however, in this case instead of having one final model, I actually want to have 3 models as I have 3-fold. So, I want to get the trained model after first fold, then after second fold and lastly after the third fold, which corresponds to the actual final model. Any ideas how to achieve this in R? Please note that usage of caret is not strict, if you can do it with mlr that's also welcomed.

Answer 1

The train function in caret streamlines model evaluation and training
https://cran.r-project.org/web/packages/caret/vignettes/caret.html
"evaluate, using resampling, the effect of model tuning parameters on performance
choose the ``optimal’’ model across these parameters
estimate model performance from a training set"
So, the model that it gives is the optimal final model.
There is no reason to use the models trained on each fold. I'm not aware of how to do this in R

Answer 2

Here is an approach using mlr package:
library(mlr)
library(base)
library(tidyverse)
dataset <- read_csv("https://gist.githubusercontent.com/dmpe/bfe07a29c7fc1e3a70d0522956d8e4a9/raw/7ea71f7432302bb78e58348fede926142ade6992/pima-indians-diabetes.csv", col_names=FALSE)
X = dataset[, 1:8]
Y = as.factor(ifelse(dataset$X9 == 1, 'diabetes', 'nondiabetes'))
create a mlr task:
mlr_task <- makeClassifTask(data = data.frame(X, Y),
target = "Y",
positive = "diabetes")
define the resampling:
set.seed(7)
cv3 <- makeResampleInstance(makeResampleDesc("CV", iters = 3),
task = mlr_task)
define the type of hyper parameter search
ctrl <- makeTuneControlRandom(maxit = 10L)
define a learner
lrn <- makeLearner("classif.ksvm", predict.type = "prob")
optionally check learner parameters to see which ones to tune
mlr::getLearnerParamSet(lrn)
define search space (vanilladot is linear kernel in kernlab package which is called internally for "classif.ksvm"). More info on integrated learners in mlr: https://mlr.mlr-org.com/articles/tutorial/integrated_learners.html
ps <- makeParamSet(makeDiscreteParam("kernel", "vanilladot"),
makeNumericParam("C", lower = 2e-6, upper = 2e-6))
tune hyper parameters. I just set some random measures, the first one listed is used to evaluate the performance, the others are there just for show.
res <- tuneParams(lrn,
mlr_task,
cv3,
measures = list(auc, bac, f1),
par.set = ps,
control = ctrl)
set optimal hyper parameters to a learner
lrn <- setHyperPars(lrn, par.vals = res$x)
resample with models = TRUE
rsmpls <- resample(lrn,
mlr_task,
cv3,
measures = list(auc, bac, f1),
models = TRUE)
models are in
rsmpls$models[[1]]$learner.model
rsmpls$models[[2]]$learner.model
rsmpls$models[[3]]$learner.model
What this does is it first tunes the hyper parameters and then performs another set of cross validation with tuned parameters on the same folds.
an alternative and in my opinion a better approach is to pick hyper parameters in the inner folds of nested cross validation and evaluate on the outer folds keeping outer fold models to fiddle with.
lrn <- makeLearner("classif.ksvm", predict.type = "prob")
define an inner resampling strategy
cv3_inner <- makeResampleDesc("CV", iters = 3)
create a tune wrapper - define what happens in inner cross validation loop
lrn <- makeTuneWrapper(lrn,
resampling = cv3_inner,
measures = list(auc, bac, f1),
par.set = ps,
control = ctrl)
perform outer cross validation
rsmpls <- resample(lrn,
mlr_task,
cv3,
measures = list(auc, bac, f1),
models = TRUE)
This performs three fold CV in the outer loop, in each training instance another, three fold CV is performed to tune the hyper parameters and a model is fit on the whole training instance with optimal hyper parameters, these models are evaluated on the outer loop test instances. This is done to reduce evaluation bias. See also: https://mlr.mlr-org.com/articles/tutorial/nested_resampling.html

Answer 3

Not a caret nor machine learning expert, but why not just train the model on a random sample and store the result in a list?
data <- read_csv("https://gist.githubusercontent.com/dmpe/bfe07a29c7fc1e3a70d0522956d8e4a9/raw/7ea71f7432302bb78e58348fede926142ade6992/pima-indians-diabetes.csv", col_names=FALSE)
train_multiple_models <- function(data, kfolds) {
resultlist <- list()
for(i in 1:kfolds) {
sample <- sample.int(n = nrow(data), size = floor(.75*nrow(data)), replace = F)
train <- data[sample, ]
X = train[, 1:8]
Y = as.factor(ifelse(train$X9 == 1, 'diabetes', 'nondiabetes'))
model <- caret::train(X, Y,
method = "svmLinear",
preProcess=c("center","scale"),
tuneLength=10)
resultlist[[i]] <- model
}
return(resultlist)
}
result <- train_multiple_models(data, kfolds = 3)
> result[[1]]$finalModel
Support Vector Machine object of class "ksvm"
SV type: C-svc (classification)
parameter : cost C = 1
Linear (vanilla) kernel function.
Number of Support Vectors : 307
Objective Function Value : -302.065
Training error : 0.230903