R nnet multiniom (multinomial logistic regression models) - assign penalties to avoid misclassification - r

I am using multinom from nnet package to fit a logistic regression model to data consists of 3 classes, however the prevalence of the classes is not balanced. I would like to assign weight/penalties in order to tell the model to avoid misclassification for a certain class.
Here is my code and a slice of my data:
mnm <- multinom(formula = cut.rank ~ ., data = training.logist, trace = FALSE, maxit = 1000, weights=c(10,5,1))
> str(head(training.logist))
'data.frame': 6 obs. of 15 variables:
$ is_top_rated_listing : Factor w/ 2 levels "0","1": 1 1 1 2 2 2
$ seller_is_top_rated_seller : int 1 1 1 1 1 1
$ is_auto_pay : Factor w/ 2 levels "0","1": 2 2 2 2 2 2
$ is_returns_accepted : Factor w/ 2 levels "0","1": 2 2 2 2 2 2
$ seller_feedback_rating_star : Factor w/ 11 levels "Blue","Green",..: 7 7 7 9 9 9
$ keywords_title_assoc : num 1 1 1 1 1 1
$ normalized.price_shipping : num 0 0 0.00871 0.01853 0.01853 ...
$ normalized.seller_feedback_score : num 0.7117 0.8791 0.0966 0.095 0.095 ...
$ normalized.seller_positive_feedback_percent: num 0.7117 0.8791 0.0966 0.095 0.095 ...
$ item_condition : Factor w/ 2 levels "New","New other (see details)": 1 1 1 1 1 1
$ listing_type : Factor w/ 2 levels "FixedPrice","StoreInventory": 2 2 2 1 1 1
$ best_offer_enabled : Factor w/ 2 levels "0","1": 1 1 1 1 1 1
$ shipping_handling_time : int 10 10 10 1 1 1
$ shipping_locations : Factor w/ 7 levels "AU,Americas,Europe,Asia",..: 5 5 5 5 5 5
$ cut.rank : Factor w/ 3 levels "1","2","3": 1 1 1 1 1 1
>
Anyone have an idea how to assign misclassification penalties? specifically I would like assign a penalty ratio of 10:5:1 (correspond to class 1,2,3) meaning I really like to be accurate on class 1.
The distribution of my target variable cut.rank is ~ 0.04,0.08,0.88.
Because class 1 has a low prevalence the model sensitivity for that class is low.

Related

Using lapply in group_by() of two factors in R

I have this data frame (named as OEM_final). This is the structure:
str(OEM_final)
'data.frame': 2265 obs. of 17 variables:
$ dia_hora_OEM : POSIXct, format: "2019-12-31 06:40:13" "2019-12-31 06:43:00" "2019-12-31 07:11:30" "2019-12-31 07:18:30" ...
$ coche_OEM : Factor w/ 6 levels "356232050832996",..: 3 3 3 3 3 3 3 3 6 6 ...
$ DTC_OEM_dec64: chr "[{\"code\":\"B1182\",\"description\":\"Tire pressure monitor module\",\"faultInformations\":[{\"description\":\"| __truncated__ "[{\"code\":\"B1182\",\"description\":\"Tire pressure monitor module\",\"faultInformations\":[{\"description\":\"| __truncated__ "[{\"code\":\"B1182\",\"description\":\"Tire pressure monitor module\",\"faultInformations\":[{\"description\":\"| __truncated__ "[{\"code\":\"B1182\",\"description\":\"Tire pressure monitor module\",\"faultInformations\":[{\"description\":\"| __truncated__ ...
$ rowname : Factor w/ 2265 levels "1","10","100",..: 1 1112 1489 1600 1711 1822 1933 2044 2155 2 ...
$ B1182 : Factor w/ 2 levels "B1182","NULL": 1 1 1 1 1 1 1 1 2 2 ...
$ B124D : Factor w/ 2 levels "B124D","NULL": 1 1 1 1 1 1 1 1 2 2 ...
$ NA. : Factor w/ 6 levels "c(NA, NA, NA, NA, NA, NA, NA, NA)",..: 3 3 3 3 3 3 3 3 1 1 ...
$ P2000 : Factor w/ 2 levels "c(\"P2000\", \"P2000\", \"P2000\")",..: 2 2 2 2 2 2 2 2 2 2 ...
$ U3003 : Factor w/ 2 levels "NULL","U3003": 1 1 1 1 1 1 1 1 1 1 ...
$ B1D01 : Factor w/ 3 levels "B1D01","c(\"B1D01\", \"B1D01\")",..: 3 3 3 3 3 3 3 3 3 3 ...
$ U0155 : Factor w/ 2 levels "NULL","U0155": 1 1 1 1 1 1 1 1 1 1 ...
$ C1B00 : Factor w/ 2 levels "C1B00","NULL": 2 2 2 2 2 2 2 2 2 2 ...
$ P037D : Factor w/ 2 levels "NULL","P037D": 1 1 1 1 1 1 1 1 1 1 ...
$ P0616 : Factor w/ 2 levels "NULL","P0616": 1 1 1 1 1 1 1 1 1 1 ...
$ P0562 : Factor w/ 2 levels "NULL","P0562": 1 1 1 1 1 1 1 1 1 1 ...
$ U0073 : Factor w/ 2 levels "NULL","U0073": 1 1 1 1 1 1 1 1 1 1 ...
$ P0138 : Factor w/ 2 levels "c(\"P0138\", \"P0138\", \"P0138\")",..: 2 2 2 2 2 2 2 2 2 2 ...
I would like to calculate the earlier date (dia_hora_OEM) that appears when grouping by two factors. The two factors are:
One of this factor, which is common in all the possible combinations, is coche_OEM.
The other one is one from column 8 (P2000) to the last one (P0138), one at a time.
So, the group_by() would be:
group_by(coche_OEM, P2000)
group_by(coche_OEM, U3003)
group_by(coche_OEM, B1D01)
group_by(coche_OEM, U0155)
...
I tried different ways to accomplish this:
Using for loops:
for (DTC in c(U3003, P2000)) {
OEM_final %>%
group_by(DTC, coche_OEM) %>%
filter(dia_hora_OEM == min(dia_hora_OEM))
}
But I get an error saying:
Error in c(U3003, P2000) : object 'U3003' not found
Using lapply
In this case, I created a function:
groupCombDTC <- function(x) {
OEM_final %>%
group_by(coche_OEM, x) %>%
filter(dia_hora_OEM == min(dia_hora_OEM))
}
And then I ran lapply():
lapply(colnames(OEM_final)[8:17], groupCombDTC)
I get this error:
Error: Column `x` is unknown
Can anybody help me iterating in different combinations using group_by()?
That's a standard problem of standard evaluation with dplyr. dplyr is based on non standard evaluation so quoted arguments need to be unquoted.
Several solutions exist. This one works well
groupCombDTC <- function(x) {
OEM_final %>%
group_by(coche_OEM, !!rlang::sym(x)) %>%
filter(dia_hora_OEM == min(dia_hora_OEM))
}
It requires to use together !! and rlang::sym to unquote and evaluate your variable name.
Column names as arguments are easier to handle with data.table. If you want more elements regarding SE/NSE in dplyr and data.table, you can have a look at a blog post I wrote a few days ago

Complex Counting of dataframe with factors

I have this table.
'data.frame': 5303 obs. of 9 variables:
$ Metric.ID : num 7156 7220 7220 7220 7220 ...
$ Metric.Name : Factor w/ 99 levels "Avoid accessing data by using the position and length",..: 51 59 59
$ Technical.Criterion: Factor w/ 25 levels "Architecture - Multi-Layers and Data Access",..: 4 9 9 9 9 9 9 9 9 9 ...
$ RT.Snapshot.name : Factor w/ 1 level "2017_RT12": 1 1 1 1 1 1 1 1 1 1 ...
$ Violation.status : Factor w/ 2 levels "Added","Deleted": 2 1 2 2 2 1 1 1 1 1 ...
$ Critical.Y.N : num 0 0 0 0 0 0 0 0 0 0 ...
$ Grouping : Factor w/ 29 levels "281","Bes",..: 27 6 6 6 6 7 7 7 7 7 ...
$ Object.type : Factor w/ 11 levels "Cobol Program",..: 8 7 7 7 7 7 7 7 7 7 ...
$ Object.name : Factor w/ 3771 levels "[S:\\SOURCES\\",..: 3771 3770 3769 3768 3767 3
I want to have a statistic output like this:
For every Technical.Criterion a row with the sum of all rows of Critical.Y.N = 0 and 1
So I have to combine the rows of my database to a new matrix. Using Values of the factor sums ...
But I have no idea how to start...? Any hints?
Thanks
I believe you're asking for a cross-tabulation. Because you did not provide a reproducible sample, I've used mine:
xtabs(~ Sub.Category + Category, retail)
Produce this:
And if you want the value to be say, based on Sales, instead of the count, then you can modify the code to:
xtabs(Sales ~ Sub.Category + Category, retail)
And you will get the following output:
EDIT based on extra information in the OP's comment
If you want to have your tables also share a common title and want to change the name of that title, you can use a combination of names() and dimnames(). An xtab is a cross-tabulation table and if you call dimnames() on it it returns a list of length 2, first one corresponding to the row and second to the column.
dimnames(xtab(dat))
$Technical.Criterion
[1] "TechnicalCrit1" "TechnicalCrit2" "TechnicalCrit3"
$`Object.type`
[1] "Object.type1" "Object.type2" "Object.type3"
So given a data frame, b:
'data.frame': 3 obs. of 9 variables:
$ Metric.ID : int 101 102 103
$ Metric.Name : Factor w/ 3 levels "A","B","C": 1 2 3
$ Technical.Criterion: Factor w/ 3 levels "TechnicalCrit1",..: 1 2 3
$ RT.Snapshot.name : Factor w/ 3 levels "A","B","C": 1 2 3
$ Violation.status : Factor w/ 2 levels "Added","Deleted": 1 2 1
$ Critical.Y.N : num 1 0 1
$ Grouping : Factor w/ 3 levels "A","B","C": 1 2 3
$ Object.type : Factor w/ 3 levels "Object.type1",..: 1 2 3
$ Object.name : Factor w/ 3 levels "A","B","C": 1 2 3
We can use xtab and then change the "common" header right at the top of our table. Since I don't know how many levels are in b$Violation.status, I would use a generic for loop:
for(i in 1:length(unique(b$Violation.status))){
tab[[i]] <- xtabs(Critical.Y.N ~ Technical.Criterion + Object.type, b)
names(dimnames(tab[[i]]))[2] <- paste("Violation.status", i)
}
This produces:
Violation.status 1
Technical.Criterion Object.type1 Object.type2 Object.type3
TechnicalCrit1 1 0 0
TechnicalCrit2 0 0 0
TechnicalCrit3 0 0 1
Which I can now use in my shiny app.

R - geeglm Error: contrasts can be applied only to factors with 2 or more levels

I have applied GEE to the following dataset (str as below). Everything is fine.
> str(cd4.5m2)
'data.frame': 1300 obs. of 7 variables:
$ id : Factor w/ 260 levels "1","5","29","32",..: 1 1 1 1 1 2 2 2 2 2 ...
$ Treatment: Factor w/ 4 levels "Alternating",..: 2 2 2 2 2 1 1 1 1 1 ...
$ Age : num 36.4 36.4 36.4 36.4 36.4 ...
$ Gender : Factor w/ 2 levels "Female","Male": 2 2 2 2 2 2 2 2 2 2 ...
$ logcd4 : num 3.14 3.04 2.77 2.83 3.22 ...
$ Week : num 0 7.57 15.57 23.57 32.57 ...
$ Time : int 0 1 2 3 4 0 1 2 3 4 ...
I then transformed the outcome variable, reason being we want to monitor the change over time. So the str of the transformed data looks like below, which is almost exactly the same as the previous one (other than some name changes).
> str(cd4.5m1)
'data.frame': 1300 obs. of 6 variables:
$ id : Factor w/ 260 levels "1","5","29","32",..: 1 2 3 4 5 6 7 8 9 10 ...
$ Treatment : Factor w/ 4 levels "Alternating",..: 2 1 4 1 3 3 1 4 1 3 ...
$ Age : num 36.4 35.9 47.5 37.3 42.7 ...
$ Gender : Factor w/ 2 levels "Female","Male": 2 2 2 1 2 2 2 2 2 2 ...
$ Week : num 1 1 1 1 1 1 1 1 1 1 ...
$ cd4.change.norm: num 0.572 0.572 0.572 0.572 0.572 ...
I then run the GEE again and it gives me the error.
> gee1.default <- geeglm(cd4.change.norm ~ Treatment, data=cd4.5m1, id=id, family=gaussian, corstr="unstructured")
Error in `contrasts<-`(`*tmp*`, value = contr.funs[1 + isOF[nn]]) :
contrasts can be applied only to factors with 2 or more levels
I also tested all variables in the data, they all contain multiple values. So I'm completely lost here. I also saw a lot of posts on this Error, but none seem to be able to address my issue here. Help appreciated!
I changed the correlation structure to AR1, and it worked. I did test the correlation (decreased over time) and AR1 is the correct structure to use.
But normally unstructured should be the save option?
I just reordered my data and it works. I'd like to suggest you try reordering your data like cd4.5m1<-cd4.5m1[order(cd4.5m1$id),]. Credits:KDG

Error in Cross Validation in GLMNET package R for Binomial Target Variable

This is in reference to https://stats.stackexchange.com/questions/72251/an-example-lasso-regression-using-glmnet-for-binary-outcome I am trying to use the Cross Validation in GLMNET (i.e. cv.glmnet) for a binomial target variable. The glmnet works fine but the cv.glmnet throws an error here is the error log:
Error in storage.mode(y) = "double" : invalid to change the storage mode of a factor
In addition: Warning messages:
1: In Ops.factor(x, w) : ‘*’ not meaningful for factors
2: In Ops.factor(y, ybar) : ‘-’ not meaningful for factors
Data Types:
'data.frame': 490 obs. of 13 variables:
$ loan_id : Factor w/ 614 levels "LP001002","LP001003",..: 190 381 259 310 432 156 179 24 429 408 ...
$ gender : Factor w/ 2 levels "Female","Male": 2 2 2 2 2 2 2 2 2 1 ...
$ married : Factor w/ 2 levels "No","Yes": 2 2 2 2 1 2 2 2 2 1 ...
$ dependents : Factor w/ 4 levels "0","1","2","3+": 1 1 1 3 1 4 2 3 1 1 ...
$ education : Factor w/ 2 levels "Graduate","Not Graduate": 1 1 1 2 1 1 1 2 1 2 ...
$ self_employed : Factor w/ 2 levels "No","Yes": 1 1 1 1 1 1 1 1 1 1 ...
$ applicantincome : int 9328 3333 14683 7667 6500 39999 3750 3365 2920 2213 ...
$ coapplicantincome: num 0 2500 2100 0 0 ...
$ loanamount : int 188 128 304 185 105 600 116 112 87 66 ...
$ loan_amount_term : Factor w/ 10 levels "12","36","60",..: 6 9 9 9 9 6 9 9 9 9 ...
$ credit_history : Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
$ property_area : Factor w/ 3 levels "Rural","Semiurban",..: 1 2 1 1 1 2 2 1 1 1 ...
$ loan_status : Factor w/ 2 levels "0","1": 2 2 1 2 1 2 2 1 2 2 ...
Codes Used:
xfactors<-model.matrix(loan_status ~ gender+married+dependents+education+self_employed+loan_amount_term+credit_history+property_area,data=data_train)[,-1]
x<-as.matrix(data.frame(applicantincome,coapplicantincome,loanamount,xfactors))
glmmod<-glmnet(x,y=as.factor(loan_status),alpha=1,family='binomial')
plot(glmmod,xvar="lambda")
grid()
cv.glmmod <- cv.glmnet(x,y=loan_status,alpha=1) #This Is Where It Throws The Error
The credit for the answer goes to #user20650.
Suspect you need to add the familyto cv.glmnet as well. An example:
x <- model.matrix(am ~ 0 + . , data=mtcars)
cv.glmnet(x, y=factor(mtcars$am), alpha=1)
cv.glmnet(x, y=factor(mtcars$am), alpha=1, family="binomial")

Error when I try to predict class probabilities in R - caret

I've build a model using caret. When the training was completed I got the following warning:
Warning message:
In train.default(x, y, weights = w, ...) :
At least one of the class levels are not valid R variables names; This may cause errors if class probabilities are generated because the variables names will be converted to: X0, X1
The names of the variables are:
str(train)
'data.frame': 7395 obs. of 30 variables:
$ alchemy_category : Factor w/ 13 levels "arts_entertainment",..: 2 8 6 6 11 6 1 6 3 8 ...
$ alchemy_category_score : num 3737 2052 4801 3816 3179 ...
$ avglinksize : num 2.06 3.68 2.38 1.54 2.68 ...
$ commonlinkratio_1 : num 0.676 0.508 0.562 0.4 0.5 ...
$ commonlinkratio_2 : num 0.206 0.289 0.322 0.1 0.222 ...
$ commonlinkratio_3 : num 0.0471 0.2139 0.1202 0.0167 0.1235 ...
$ commonlinkratio_4 : num 0.0235 0.1444 0.0426 0 0.0432 ...
$ compression_ratio : num 0.444 0.469 0.525 0.481 0.446 ...
$ embed_ratio : num 0 0 0 0 0 0 0 0 0 0 ...
$ frameTagRatio : num 0.0908 0.0987 0.0724 0.0959 0.0249 ...
$ hasDomainLink : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
$ html_ratio : num 0.246 0.203 0.226 0.266 0.229 ...
$ image_ratio : num 0.00388 0.08865 0.12054 0.03534 0.05047 ...
$ is_news : Factor w/ 2 levels "0","1": 2 2 2 2 2 1 2 1 2 1 ...
$ lengthyLinkDomain : Factor w/ 2 levels "0","1": 2 2 2 1 2 1 1 1 1 2 ...
$ linkwordscore : num 24 40 55 24 14 12 21 5 17 14 ...
$ news_front_page : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
$ non_markup_alphanum_characters: num 5424 4973 2240 2737 12032 ...
$ numberOfLinks : num 170 187 258 120 162 55 93 132 194 326 ...
$ numwords_in_url : num 8 9 11 5 10 3 3 4 7 4 ...
$ parametrizedLinkRatio : num 0.1529 0.1818 0.1667 0.0417 0.0988 ...
$ spelling_errors_ratio : num 0.0791 0.1254 0.0576 0.1009 0.0826 ...
$ label : Factor w/ 2 levels "0","1": 1 2 2 2 1 1 2 1 2 2 ...
$ isVideo : Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 1 1 ...
$ isFashion : Factor w/ 2 levels "0","1": 1 1 1 1 2 1 2 1 2 1 ...
$ isFood : Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
$ hasComments : Factor w/ 2 levels "0","1": 1 2 2 2 2 1 2 2 1 2 ...
$ hasGoogleAnalytics : Factor w/ 2 levels "0","1": 1 1 1 1 2 1 2 2 2 1 ...
$ hasInlineCSS : Factor w/ 2 levels "0","1": 1 2 2 2 1 1 2 1 2 2 ...
$ noOfMetaTags : num 10 12 6 10 13 2 6 6 9 5 ...
My code is the following:
ctrl <- trainControl(method = "CV",
number=10,
classProbs = TRUE,
allowParallel = TRUE,
summaryFunction = twoClassSummary)
set.seed(476)
rfFit <- train(formula,
data=train,
method = "rf",
tuneGrid = expand.grid(.mtry = seq(4,20,by=2)),
ntrees=1000,
importance = TRUE,
metric = "ROC",
trControl = ctrl)
pred <- predict.train(rfFit, newdata = test, type = "prob")
I get the error: Error in [.data.frame(out, , obsLevels, drop = FALSE) :
undefined columns selected
The variable names on the test data set are:
str(test)
'data.frame': 3171 obs. of 29 variables:
$ alchemy_category : Factor w/ 13 levels "arts_entertainment",..: 8 4 12 4 10 12 12 8 1 2 ...
$ alchemy_category_score : num 5307 4825 1 6708 5416 ...
$ avglinksize : num 2.56 3.77 2.27 2.52 1.85 ...
$ commonlinkratio_1 : num 0.39 0.462 0.496 0.706 0.471 ...
$ commonlinkratio_2 : num 0.257 0.205 0.385 0.346 0.161 ...
$ commonlinkratio_3 : num 0.0441 0.0513 0.1709 0.123 0.0323 ...
$ commonlinkratio_4 : num 0.0221 0 0.1709 0.0906 0 ...
$ compression_ratio : num 0.49 0.782 1.25 0.449 0.454 ...
$ embed_ratio : num 0 0 0 0 0 0 0 0 0 0 ...
$ frameTagRatio : num 0.0671 0.0429 0.0588 0.0581 0.093 ...
$ hasDomainLink : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
$ html_ratio : num 0.23 0.366 0.162 0.147 0.244 ...
$ image_ratio : num 0.19944 0.08 10 0.00596 0.03571 ...
$ is_news : Factor w/ 2 levels "0","1": 2 1 1 2 2 1 1 2 1 1 ...
$ lengthyLinkDomain : Factor w/ 2 levels "0","1": 2 2 2 2 1 2 2 1 1 1 ...
$ linkwordscore : num 15 62 42 41 34 35 15 22 41 7 ...
$ news_front_page : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
$ non_markup_alphanum_characters: num 5643 382 2420 5559 2209 ...
$ numberOfLinks : num 136 39 117 309 155 266 55 145 110 1 ...
$ numwords_in_url : num 3 2 1 10 10 7 1 9 5 0 ...
$ parametrizedLinkRatio : num 0.2426 0.1282 0.5812 0.0388 0.0968 ...
$ spelling_errors_ratio : num 0.0806 0.1765 0.125 0.0631 0.0653 ...
$ isVideo : Factor w/ 2 levels "0","1": 1 2 1 2 2 2 1 1 2 2 ...
$ isFashion : Factor w/ 2 levels "0","1": 1 1 1 1 1 2 1 1 1 1 ...
$ isFood : Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
$ hasComments : Factor w/ 2 levels "0","1": 2 1 1 2 2 2 1 2 2 1 ...
$ hasGoogleAnalytics : Factor w/ 2 levels "0","1": 1 2 2 2 2 1 1 2 1 1 ...
$ hasInlineCSS : Factor w/ 2 levels "0","1": 2 2 2 1 1 2 2 2 1 1 ...
$ noOfMetaTags : num 3 6 5 9 16 22 6 9 7 0 ...
If I omit the type="prob" part, I get no error.
Any ideas?
Could it be the length of the variable "alchemy_category" which is appended with the respective factor levels e.g. "alchemy_categoryarts_entertainment" inside the model??
The answer is in bold at the top of your post =]
What are you modeling? Is it alchemy_category? The code only says formula and we can't see it.
When you ask for class probabilities, model predictions are a data frame with separate columns for each class/level. If alchemy_category doesn't have levels that are valid column names, data.frame converts then to valid names. That creates a problem because the code is looking for a specific name but the data frame as a different (but valid) name.
For example, if I had
> test <- factor(c("level1", "level 2"))
> levels(test)
[1] "level 2" "level1"
> make.names(levels(test))
[1] "level.2" "level1"
the code would be looking for "level 2" but there is only "level.2".
As stated above the class values must be factors and must be valid names. Another way to insure this is,
levels(all.dat$target) <- make.names(levels(factor(all.dat$target)))
I have read through the answers above while facing a similar problem. A formal solution is to do this on the train and test datasets. Make sure you include the response variable in the feature.names too.
feature.names=names(train)
for (f in feature.names) {
if (class(train[[f]])=="factor") {
levels <- unique(c(train[[f]]))
train[[f]] <- factor(train[[f]],
labels=make.names(levels))
}
}
This creates syntactically correct labels for all factors.
As #Sam Firke already pointed out in comments (but I overlooked it) levels TRUE/FALSE also don't work. So I converted them to yes/no.
As per the above example, usually refactoring the outcome variable will fix the problem. It's better to change in the original dataset before partitioning into training and test datasets
levels <- unique(data$outcome)
data$outcome <- factor(data$outcome, labels=make.names(levels))
As others pointed out earlier, this problem only occurs when classProbs=TRUE which causes the train function to generate additional statistics related to the outcome class

Resources