I want to use an SVM implementation in R to do some regression. I tried using svm from e1071 already but I am limited by the kernel functions there. So I moved on to ksvm from kernlab. But I have a major disadvantage that a tuning function has not been provided in kernlab (like tune.svm in e1071). Can someone explain how do I tune the parameters for different kernels there?
PS. I want to particularly use the rbfdot kernel. So if at least someone can help me understand how to tune sigma, I'd be extremely grateful.
PPS. I'm completely aware that the "automatic" value for kpar can be used "to calculate a good sigma". But I need something more tangible and more along the lines of tune.svm.
Either you write your own wrapper (wouldn't be that hard to be honest) or you could try already proven implemented solutions, like mlr and caret.
mlr tutorial has an example about it.
ps = makeParamSet(
makeDiscreteParam("C", values = 2^(-2:2)),
makeDiscreteParam("sigma", values = 2^(-2:2))
)
ctrl = makeTuneControlGrid()
rdesc = makeResampleDesc("CV", iters = 3L)
res = tuneParams("classif.ksvm", task = iris.task, resampling = rdesc, par.set = ps, control = ctrl)
This will perform 3-fold cross-validation to select parameters from the grid and evaluate accuracy on the iris dataset. You can, of course, change resampling strategies (leave-one-out, monte-carlo CV, CV, repeated CV, bootstrap validation and holdout are all implemented), search strategy (grid search, random search, generalized simulated annealing and iterated F-race are all supported) and evaluation metrics.
Related
The caret library in R has a hyper-parameter 'selectionFunction' inside trainControl().
It's used to prevent over-fitting models using Breiman's one standard error rule, or tolerance, etc.
Does mlr have an equivalent? If so, which function is it within?
mlr only allows to choose optimal hyperparameters by optimizing certain measures/metrics.
However, essentially each "measure" in mlr is just a function that specifies how a certain performance is handled.
You can try to write your own custom measure as outlined in this vignette.
Other than that, it might be worth opening this as a feature request in the new mlr3 framework, specifically in mlr3measures, since mlr itself is deprecated.
Posting an answer to my own question, I found this..
Estimate relative overfitting.
Source: R/relativeOverfitting.R
Estimates the relative overfitting of a model as the ratio of the difference in test and train performance to the difference of test performance in the no-information case and train performance. In the no-information case the features carry no information with respect to the prediction. This is simulated by permuting features and predictions.
estimateRelativeOverfitting(
predish,
measures,
task,
learner = NULL,
pred.train = NULL,
iter = 1
)
Arguments
predish - (ResampleDesc ResamplePrediction Prediction) Resampling strategy or resampling prediction or test predictions.
measures - (Measure list of Measure) Performance measure(s) to evaluate. Default is the default measure for the task, see here getDefaultMeasure.
task - (Task) The task.
learner - (Learner character(1)) The learner. If you pass a string the learner will be created via makeLearner.
pred.train - (Prediction) Training predictions. Only needed if test predictions are passed.
iter - (integer) Iteration number. Default 1, usually you don't need to specify this. Only needed if test predictions are passed.
I"m trying to understand how caret is coming to the decision it's making on the best-tuned model. I have looked through the documentation and I have not found (which could easily be my fault) a place to adjust how this decision is made. I'm using something similar to :
train(
y~.,
data=X,
num.trees = 1000,
method = "ranger",
trControl = trainControl(
method = "repeatedcv",
number = 100,
repeats = 100, verboseIter = T
)
I'm trying to use Caret more often, and I'm sure there is a smart way it's making the decision.. I'm just trying to understand how and if I can adjust it.
There is a lot of documentation but the best place to look for your question is here.
Basically, for grid search, multiple combinations of tuning parameters are evaluated using resampling. Each combination gets an associated resampling estimate of performance (let's say it is accuracy).
train() knows that accuracy should be maximized so, by default, it picks the parameter combination with the largest value and uses these to fit one final model (using these values and the entire training set).
I was reading the glmnet documentation and I found this:
Note also that the results of cv.glmnet are random, since the folds
are selected at random. Users can reduce this randomness by running
cv.glmnet many times, and averaging the error curves.
The following code uses caret with a repeated cv.
library(caret)
ctrl <- trainControl(verboseIter = TRUE, classProbs = TRUE,
summaryFunction = twoClassSummary, method = "repeatedcv",
repeats = 10)
fit <- train(x, y, method = "glmnet", metric = "ROC", trControl = ctrl)
Is that the best way to run glmnet with cross validation through caret?, or is it better to run glmnet directly?
You need to define best way. Do you want to use
A regularized regression alone on a dataset for feature selection? (in which case, use glmnet--Max Kuhn has implied that you may be better off using models with in-built CV features as they would have been optimized for both predictor selection and minimizing error). See below.
"In many cases, using these models with built-in feature selection will be more efficient than algorithms where the search routine for
the right predictors is external to the model. Built-in feature
selection typically couples the predictor search algorithm with the
parameter estimation and are usually optimized with a single
objective function (e.g. error rates or likelihood)." (Kuhn, caret
package documentation: caret feature selection overview)
Or are you comparing different models, one of which is glmnet? In which case, caret may be a great choice.
Using the defaults of the train in caret package, I am trying to train a random forest model for the dataset xtr2 (dim(xtr2): 765 9408). The problem is that it unbelievably takes too long (more than one day for one training) to fit the function. As far as I know train in its default uses bootstrap sampling (25 times) and three random selection of mtry, so why it should take so long?
Please notice that I need to train the rf, three times in each run (because I need to make a mean of the results of different random forest models with the same data), and it takes about three days, and I need to run the code for 10 different samples, so it would take me 30 days to have the results.
My question is how I can make it faster?
Can changing the defaults of train make the operation time less? for example using CV for training?
Can parallel processing with caret package help? if yes, how it can be done?
Can tuneRF of random forest package make any changes to the time?
This is the code:
rffit=train(xtr2,ytr2,method="rf",ntree=500)
rf.mdl =randomForest(x=xtr2,y=as.factor(ytr2),ntree=500,
keep.forest=TRUE,importance=TRUE,oob.prox =FALSE ,
mtry = rffit$bestTune$mtry)
Thank you,
My thoughts on your questions:
Yes! But don't forget you also have control over the search grid caret uses for the tuning parameters; in this case, mtry. I'm not sure what the default search grid is for mtry, but try the following:
ctrl <- trainControl("cv", number = 5, verboseIter = TRUE)
set.seed(101) # for reproducibility
rffit <- train(xtr2, ytr2, method = "rf", trControl = ctrl, tuneLength = 5)
Yes! See the caret website: http://topepo.github.io/caret/parallel-processing.html
Yes and No! tuneRF simply uses the OOB error to find an optimal value of mtry (the only tuning parameter in randomForest). Using cross-validation tends to work better and produce a more honest estimate of model performance. tuneRF can take a long time but should be quicker than k-fold cross-validation.
Overall, the online manual for caret is quite good: http://topepo.github.io/caret/index.html.
Good luck!
You use train for determining mtry only. I would skip the train step, and stay with default mtry:
rf.mdl =randomForest(x=xtr2,y=as.factor(ytr2),ntree=500,
keep.forest=TRUE,importance=TRUE,oob.prox =FALSE)
I strongly doubt that 3 different runs is a good idea.
If you do 10 fold cross-validation (I am not sure it should be done anyways, as validation is ingrained into the random forest), 10 parts is too much, if you are short in time. 5 parts would be enough.
Finally, the time of randomForest is proportional to nTree. Set nTree=100, and your program will run 5 time faster.
I would also just add, that it the main issue is speed, there are several other random forest implementations in caret, and many of them are much faster than the original randomForest which is notoriously slow. I've found ranger to be a nice alternative that suited my very simple needs.
Here is a nice summary of the random forest packges in R. Many of these are in caret already.
Also for consideration, here's an interesting study of the performance of ranger vs rborist, where you can see how performance is affected by the tradeoff between sample size and features.
what is the difference between tune.svm() and best.svm().
When we tune the parameters of svm kernel, aren't we expected to always choose the best values for our model.
Pardon as i am new to R and machine learning.
I noticed that there was no linear kernel option in tuning svm. Is there a possibility to tune my svm using a linear kernel
From ETHZ: best.svm() is really just a wrapper for tune.svm(...)$best.model. The
help page for tune() will tell you more on the available options.
Be sure to also go through the examples on the help page for tune(). e1071::svm offers linear, radial (the default), sigmoid and polynomial kernels, see help(svm). For example, to use the linear kernel the function call has to include the argument kernel = 'linear':
data(iris)
obj <- tune.svm(Species~., data = iris,
cost = 2^(2:8),
kernel = "linear")
If you are new to R and would like to train and cross validate SVM models you could also check the caret package and its train function which offers multiple types of kernels. The whole 'topics' section on that site might be of interest, too.