How to change the threshold for binary classification - r

Question

I've trained a gbm model in R. Since I'm trying to predict a very rare case, I get a lot of false positives. I wanted to change the threshold for a positive ("Good") case from the default to 0.7. Here is my code so far.
modFit.glm.ml <- train(as.factor(ml.training$one_lease)~., data=ml.training, method = "glm")
confusionMatrix(ml.testing$one_lease, predict(modFit.glm.ml, ml.testing), positive = "Good")
This code works but it uses the default cutoff.
Someone mentioned this is possible with the predict function, but I don't know how to do it.

Answer 1

You haven't provided a reproducible example, so here's one using the iris dataset to predict if an iris is of type setosa:
dat <- iris
dat$positive <- as.factor(ifelse(dat$Species == "setosa", "s", "ns"))
library(caret)
mod <- train(positive~Sepal.Length, data=dat, method="glm")
To generate a confusion matrix using a cutoff for predicted probabilities other than 0.5, you can threshold the probabilities returned by the predict function using whatever cutoff you want:
confusionMatrix(table(predict(mod, type="prob")[,"s"] >= 0.25,
dat$positive == "s"))
# Confusion Matrix and Statistics
#
#
# FALSE TRUE
# FALSE 88 3
# TRUE 12 47
#
# Accuracy : 0.9
# 95% CI : (0.8404, 0.9429)
# No Information Rate : 0.6667
# P-Value [Acc > NIR] : 2.439e-11
#
# Kappa : 0.7847
# Mcnemar's Test P-Value : 0.03887
#
# Sensitivity : 0.8800
# Specificity : 0.9400
# Pos Pred Value : 0.9670
# Neg Pred Value : 0.7966
# Prevalence : 0.6667
# Detection Rate : 0.5867
# Detection Prevalence : 0.6067
# Balanced Accuracy : 0.9100
#
# 'Positive' Class : FALSE

Answer 2

You haven't specified which package you want to use, so here's another solution using mlr:
library(mlr)
dat = iris
training.set = seq(1, nrow(iris), by = 2)
test.set = seq(2, nrow(iris), by = 2)
dat$positive = as.factor(ifelse(dat$Species == "setosa", "s", "ns"))
task = makeClassifTask(data = dat, target = "positive")
lrn = makeLearner("classif.glmnet", predict.type = "prob")
mod = train(lrn, task, subset = training.set)
pred = predict(mod, task, subset = test.set)
print(getConfMatrix(pred))
pred = setThreshold(pred, c(s = 1))
print(getConfMatrix(pred))
mlr allows you to set the threshold explicitly with setThreshold -- the advantage is that you can use the resulting predictions with any function that measures performance without having to make sure that the threshold is set correctly.
The mlr tutorial has a whole section on classifier calibration, which can help you figure out the best value for this threshold.