Highly imbalanced data on C5.0 tree model - r

Question

I have a imbalanced dataset with only 87 target events "F" out of all 496,978 obs, since I would like to see a rule/tree, I chose to use the tree models, I have been following the codes in "Applied Predictive Modeling in R" book by Dr Max Kuhn, in chapter 16 this imbalance issue is well addressed.
Here is the sample data structure:
str(training[,predictors])
'data.frame': 496978 obs. of 36 variables:
$ Point_Of_Sale_Code : Factor w/ 5 levels "c0","c2","c90",..: 3 3 5 5 3 3 5 5 5 5 ...
$ Delinquent_Amount : num 0 0 0 0 0 0 0 0 0 0 ...
$ Delinquent_Days_Count : num 0 0 0 0 0 0 0 0 0 0 ...
$ Overlimit_amt : num 0 0 0 0 0 0 0 0 0 0 ...
I tried the downsampling with random forest, it works well, with good auc= 0.9997 on test data, and confusion matrix
Reference
Prediction N F
N 140526 0
F 1442 24
however, rf does not give me a specific rule, so I tried the code in the book exactly as:
library(rpart)
library(e1071)
initialRpart <- rpart(flag ~ ., data = training,
control = rpart.control(cp = 0.0001))
rpartGrid <- data.frame(.cp = initialRpart$cptable[, "CP"])
cmat <- list(loss = matrix(c(0, 1, 20, 0), ncol = 2))
set.seed(1401)
cartWMod1 <- train(x = training[,predictors],
y = training$flag,
method = "rpart",
trControl = ctrlNoProb,
tuneGrid = rpartGrid,
metric = "Kappa",
parms = cmat)
cartWMod1
I got the error msg below everytime, no matter what I tried, like convert all int data type to num type, not sure why I get this warning msg,
Warning message:
In ni[1:m] * nj[1:m] : ***NAs produced by integer overflow***
Aggregating results
Selecting tuning parameters
Error in train.default(x = training[, predictors], y = training$flag, :
***final tuning parameters could not be determined***
I also tried the code for c5.0 package:
library(C50)
c5Grid <- expand.grid(.model = c("tree", "rules"),
.trials = c(1, (1:10)*10),
.winnow = FALSE)
finalCost <- matrix(c(0, 150, 1, 0), ncol = 2)
rownames(finalCost) <- colnames(finalCost) <- levels(training$flag)
set.seed(1401)
C5CostFit1 <- train(training[,predictors],
training$flag,
method = "C5.0",
metric = "Kappa",
tuneGrid = c5Grid,
cost = finalCost,
control = C5.0Control(earlyStopping = FALSE),
trControl = ctrlNoProb)
C5CostCM1 <- confusionMatrix(predict(C5CostFit, training), training$flag)
I got this result, which classify all the target event F to be nonevent N, Is it possible that I can increase the cost penalty from 150 to larger to fix this issue? Thank you!
C5CostCM1
Confusion Matrix and Statistics
Reference
Prediction N F
N 141968 ***24***
F 0 0
Accuracy : 0.9998
95% CI : (0.9997, 0.9999)
No Information Rate : 0.9998
P-Value [Acc > NIR] : 0.554
Kappa : NA
Mcnemar's Test P-Value : 2.668e-06
Sensitivity : 1.0000
Specificity : 0.0000
Pos Pred Value : 0.9998
Neg Pred Value : NaN
Prevalence : 0.9998
Detection Rate : 0.9998
Detection Prevalence : 1.0000
Balanced Accuracy : 0.5000
'Positive' Class : N
I have been googling this issue for the past week, but didn't see a solution, the code from the book work well though, but gives me error for my data... Any suggestion will be appriciated!! Thank you so much!

Answer 1

I think that it's telling you that something in the output (i.e., the list) has NAs in it--the Kappa stat.
Using something like this:
results.matrix = confusionMatrix(data, reference)
results.df = as.data.frame(results.matrix[3])
summary(is.finite(results.df$overall))
Gives you this:
Mode FALSE TRUE NA's
logical 1 6 0
So I'm guessing that's what it's picking up.