Cross-validation on comparing linear models - r

Question

I have 2 linear models to compare: One is reduced model and the other one is full model. I have already done F-Test on comparing 2 linear models. But I do not know how to do this using 5-fold cross-validation comparing 2 models. The following code is what I have in R.
library(carData) #make sure this package is installed!
data("Anscombe") #NOTE: lowercase was the data from before, uppercase is this dataset (annoying)
spending = subset(Anscombe,! rownames(Anscombe) %in% c("HI","AK","DC"),c(1,2,4)) ##Data file.
mod_full = lm(education ~ urban + income, data=spending) ##Full model.
mod_reduced = lm(education ~ income, data = spending) ## Reduced model.
FTest = anova(mod_reduced, mod_full)
print(FTest) ##Print out the F-test result.

Answer 1

You can use the following code for comparing 2 models using 5-fold cross-validation
library(caret)
library(carData)
data("Anscombe")
spending = subset(Anscombe,! rownames(Anscombe) %in% c("HI","AK","DC"),c(1,2,4)) ##Data file.
#Setting for 5-fold cross-validation
trainControl <- trainControl(method="cv", number=5,
savePredictions=TRUE, classProbs=F)
#Full model
set.seed(7) #To have reproducible results
fit.full <- train(education ~ urban + income, data=spending, method="lm", metric="RMSE",
preProc=c("center", "scale"),
trControl=trainControl)
#Reduced model
set.seed(7)
fit.reduced <- train(education ~ income, data=spending, method="lm", metric="RMSE",
preProc=c("center", "scale"),
trControl=trainControl)
#For comparing 2 models
results <- resamples(list(Full=fit.full,Reduced=fit.reduced))
summary(results)
dotplot(results,scale="free")
# correlation between results
modelCor(results)
splom(results)
#Difference in model predictions
diffs <- diff(results)
#Summarize p-values for pair-wise comparisons
summary(diffs)