Where does mlr3 save the final model, after training a learner --- learner$train(data)? By "final model", I mean something like a list produced by the following code:
model <- xgboost::xgb.train(data = data_train,
max.depth = 8, nthread = 2, nrounds = 15,
verbose = 0)
Is there a way to extract this list/object?
task <- TaskRegr$new("data", data, "y")
learner <- lrn("regr.xgboost")
preprocess <- po("scale", param_vals = list(center = TRUE, scale = TRUE))
pp <- preprocess %>>% learner
gg<- GraphLearner$new(pp)
gg$train(task)
In xgboost the 'model' is stored as:
model <- xgboost::xgb.train(data = data_train,
max.depth = 8, nthread = 2, nrounds = 15,
verbose = 0)
In MLR3, when trained using:
task <- TaskRegr$new("data", data, "y")
learner <- lrn("regr.xgboost")
preprocess <- po("scale", param_vals = list(center = TRUE, scale = TRUE))
pp <- preprocess %>>% learner
gg<- GraphLearner$new(pp)
gg$train(task)
The equivalent to 'model' is stored as
gg$model$regr.xgboost$model
Related
there is a function to tune threshold for say a binary classification described here: https://mlr3pipelines.mlr-org.com/reference/mlr_pipeops_tunethreshold.html
Here's my failed attempt:
RF_lrn <- lrn("classif.rfsrc", id = "rf", predict_type = "prob")
RF_lrn$param_set$values = list(na.action = "na.impute", seed = -123)
single_pred_rf = po("subsample", frac = 1, id = "rf_ss") %>>%
po("learner", RF_lrn) %>>% po("tunethreshold")
That did not work in my mlr3 pipeline and I did not find any solution explained anywhere so here is my code:
xgb_lrn <-
lrn("classif.xgboost", id = "xgb", predict_type = "prob")
single_pred_xgb = po("subsample", frac = 1, id = "xgb_ss") %>>%
po("learner", xgb_lrn)
lrnrs <- list(
RF_lrn,
xgb_lrn)
lrnrs <- lapply(lrnrs, function(x) {
GraphLearner$new(po("learner_cv", x) %>>% po("tunethreshold",
param_vals = list(
measure = "classif.prauc"
)
))
})
library("GenSA")
lrnrs$RF_lrn <- auto_tuner(
learner = RF_lrn,
search_space = ps(
ntree = p_int(lower = 20, upper = 300),
mtry = p_int(lower = 2, upper = 5),
nodesize = p_int(lower = 1, upper = 7)
),
resampling = rsmp("bootstrap", repeats = 2, ratio = 0.8),
measure = msr("classif.prauc"),
term_evals = 100,
method = "random_search"
)
which somehow works but I want the decision threshold to be tuned as a parameter the same way I tune other hyperparameters using the random search in benchmarking/cross validation. Any solution? Thanks in advance
the solution is to use po("threshold") instead of po("tunethreshold") as suggested in the comments and this mlr gallery example
I am trying to optimize an SVM for a classification task, which has worked for many other models I've tried this process on. Yet, when I used an SVM in my model based optimization function it returns an error: "Error in checkStuff(fun, design, learner, control) : Provided learner does not support factor parameters."
Attached is the relevant code. In my training task, all independent variables are numeric, the only factor is my outcome of interest.
library(mlr)
library(mlrMBO)
library(dplyr)
library(PRROC)
library(ggplot2)
library(DiceKriging)
traindf <- read.csv("/Users/njr/Google Drive/HMS IR Research/NSQIP Research/Endovascular/randomtraining.csv")
testdf <- read.csv("/Users/njr/Google Drive/HMS IR Research/NSQIP Research/Endovascular/randomtesting.csv")
traindf$Amputation<-as.factor(traindf$Amputation)
testdf$Amputation <- as.factor(testdf$Amputation)
trn.task = makeClassifTask(data = traindf, target = "Amputation", positive = "2")
test.task = makeClassifTask(data = testdf, target = "Amputation", positive = "2")
set.seed(9)
svmlrn = makeLearner("classif.svm", predict.type = "prob")
svm_model <- mlr::train(svmlrn, task = trn.task)
res = makeResampleDesc("CV", iters = 10, stratify = TRUE)
par5 = makeParamSet(
makeDiscreteParam("kernel", values = c("radial", "polynomial", "linear")),
makeNumericParam("cost", -15, 15, trafo = function(x) 2^x),
makeNumericParam("gamma", -15, 15, trafo = function(x) 2^x, requires = quote(kernel == "radial")),
makeIntegerParam("degree", lower = 1, upper = 4, requires = quote(kernel == "polynomial"))
)
mbo.ctrl = makeMBOControl()
mbo.ctrl = setMBOControlInfill(mbo.ctrl, crit = crit.ei)
mbo.ctrl = setMBOControlTermination(mbo.ctrl, iters = 35, max.evals = 25)
design.mat = generateRandomDesign(n = 50, par.set = par5)
surrogate.lrn = makeLearner("regr.km", predict.type = "se")
ctrl = mlr::makeTuneControlMBO(learner = surrogate.lrn, mbo.control = mbo.ctrl, mbo.design = design.mat)
parallelStartMulticore(cpus = 8L)
res.mbo = tuneParams(makeLearner("classif.svm"), trn.task, resampling = res, par.set = par5, control = ctrl,
show.info = TRUE, measures = auc)
parallelStop()
this is the traceback error code:
6.
stop("Provided learner does not support factor parameters.")
5.
checkStuff(fun, design, learner, control)
4.
initOptProblem(fun = fun, design = design, learner = learner, control = control, show.info = show.info, more.args = more.args)
3.
mlrMBO::mbo(tff, design = control$mbo.design, learner = control$learner, control = mbo.control, show.info = FALSE)
2.
sel.func(learner, task, resampling, measures, par.set, control, opt.path, show.info, resample.fun)
1.
tuneParams(makeLearner("classif.svm"), trn.task, resampling = res, par.set = par5, control = ctrl, show.info = TRUE, measures = auc)
The problem is that your parameter set has a categorical parameter (kernel) and the surrogate model you're using (regr.km) doesn't support that. You could try for example a random forest as surrogate model instead.
I am trying to predict the times table training a neural network. However, I couldn't really get how preProcess argument works in train function in Caret.
In the docs, it says:
The preProcess class can be used for many operations on predictors, including centering and scaling.
When we set preProcess like below,
tt.cv <- train(product ~ .,
data = tt.train,
method = 'neuralnet',
tuneGrid = tune.grid,
trControl = train.control,
linear.output = TRUE,
algorithm = 'backprop',
preProcess = 'range',
learningrate = 0.01)
Does it mean that the train function preprocesses (normalizes) the training data passed, in this case tt.train?
After the training is done, when we are trying to predict, do we pass normalized inputs to the predict function or are inputs normalized in the function because we set the preProcess parameter?
# Do we do
predict(tt.cv, tt.test)
# or
predict(tt.cv, tt.normalized.test)
And from the quote above, it seems that when we use preProcess, outputs are not normalized this way in training, how do we go about normalizing outputs? Or do we just normalize the training data beforehand like below and then pass it to the train function?
preProc <- preProcess(tt, method = 'range')
tt.preProcessed <- predict(preProc, tt)
tt.preProcessed.train <- tt.preProcessed[indexes,]
tt.preProcessed.test <- tt.preProcessed[-indexes,]
The whole code:
library(caret)
library(neuralnet)
# Create the dataset
tt = data.frame(multiplier = rep(1:10, times = 10), multiplicand = rep(1:10, each = 10))
tt = cbind(tt, data.frame(product = tt$multiplier * tt$multiplicand))
# Splitting
indexes = createDataPartition(tt$product,
times = 1,
p = 0.7,
list = FALSE)
tt.train = tt[indexes,]
tt.test = tt[-indexes,]
# Pre-process
preProc <- preProcess(tt, method = c('center', 'scale'))
tt.preProcessed <- predict(preProc, tt)
tt.preProcessed.train <- tt.preProcessed[indexes,]
tt.preProcessed.test <- tt.preProcessed[-indexes,]
# Train
train.control <- trainControl(method = "repeatedcv",
number = 10,
repeats = 3,
savePredictions = TRUE)
tune.grid <- expand.grid(layer1 = 8,
layer2 = 0,
layer3 = 0)
tt.cv <- train(product ~ .,
data = tt.train,
method = 'neuralnet',
tuneGrid = tune.grid,
trControl = train.control,
algorithm = 'backprop',
learningrate = 0.01,
stepmax = 100000,
preProcess = c('center', 'scale'),
lifesign = 'minimal',
threshold = 0.01)
How could I fine tune this so I can get better prediction? I don't know how
to make it a better model. Any insight will be greatly appreciated. Thanks a
ton.
Basically I meant to predict best corrected visual acuity (BCVA 0,1 with
0=20/20 vision, 1=worse than 20/20).
Liyan
#preparing data
library(xgboost)
train <- read_sas("Rtrain2.sas7bdat",NULL)
test <- read_sas("Rtest2.sas7bdat",NULL)
labels <- train$bcva01
test_label <- test$bcva01
#outcome variable
drops <- c("bcva01")
x<-train[ , !(names(train) %in% drops)]
x_test<-test[ , !(names(test) %in% drops)]
new_tr <- model.matrix(~.+0,data = x)
new_ts <- model.matrix(~.+0,data = x_test)
#preparing matrix
dtrain <- xgb.DMatrix(data = new_tr,label = labels)
dtest <- xgb.DMatrix(data = new_ts,label=test_label)
#parameters
?list
params <- list(booster = "gbtree", objective = "binary:logistic", eta=0.03,
gamma=0, max_depth=6,
min_child_weight=1, subsample=1, colsample_bytree=1)
#Using the inbuilt xgb.cv function
xgbcv <- xgb.cv( params = params, data = dtrain, nrounds = 21, nfold = 5,
showsd = T, stratified = T, print.every.n = 10, early.stop.round = 21,
maximize = F)
min(xgbcv$test.error.mean) #inf
#first default - model training
xgb1 <- xgb.train (params = params, data = dtrain, nrounds = 21, watchlist =
list(val=dtest,train=dtrain),
print.every.n = 10, early.stop.round = 21, maximize = F ,
eval_metric = "error")
#model prediction
xgbpred <- predict (xgb1,dtest)
cvAUC::AUC(predictions = xgbpred, labels = test[,"bcva01"]) #0.69 2018-10-25
There are a few ways to auto calibrate your hyper parameters:
scikit-learn GridSearch here and here
Hyperopt which I use, here with a nice example here and a short example on how to do it with xgboost
Bayesian Optimization with xgboost example here
All are technique of finding some kind of "minimum" in a defined "space" where that defined "space" is the "search space" you will define for your hypter parameters and the "minimum" is the models error you'd like to reduce.
Subject is quite wide and you have a lot to read, or you can just follow some examples and implement it in your code.
I use the following code to tune parameters for my Xgboost implementation adapted from here:
searchGridSubCol <- expand.grid(subsample = c(0.5, 0.75, 1),
colsample_bytree = c(0.6, 0.8, 1))
ntrees <- 100
#Build a xgb.DMatrix object
#DMMatrixTrain <- xgb.DMatrix(data = yourMatrix, label = yourTarget)
rmseErrorsHyperparameters <- apply(searchGridSubCol, 1, function(parameterList){
#Extract Parameters to test
currentSubsampleRate <- parameterList[["subsample"]]
currentColsampleRate <- parameterList[["colsample_bytree"]]
xgboostModelCV <- xgb.cv(data = as.matrix(train), nrounds = ntrees, nfold = 5, showsd = TRUE, label = traintarget,
metrics = "rmse", verbose = TRUE, "eval_metric" = "rmse",
"objective" = "reg:linear", "max.depth" = 15, "eta" = 2/ntrees,
"subsample" = currentSubsampleRate, "colsample_bytree" = currentColsampleRate)
xvalidationScores <- as.data.frame(xgboostModelCV)
#Save rmse of the last iteration
rmse <- tail(xvalidationScores$test.rmse.mean, 1)
return(c(rmse, currentSubsampleRate, currentColsampleRate))
})
However I recieve the following error when storing the XGBoostModelCV:
Error in as.data.frame.default(xgboostModelCV) :
cannot coerce class ""xgb.cv.synchronous"" to a data.frame
Can someone explain to me what is causing this error and how may I fix it?
The above should be fixed by:
xvalidationScores <- xgboostModelCV
#Save rmse of the last iteration
rmse <- tail(xvalidationScores$evaluation_log$test_rmse_mean, 1)