I understand from reading various answers 1,2,3, that the train function from caret will create dummy variables to deal with factors that have multiple levels.
Here is an example using mtcars (model is useless other than to show point):
library(caret)
library(rattle)
df <- mtcars
df$cyl <- factor(df$cyl)
df$mpg_bound <- ifelse(df$mpg > 20, "good", "bad")
tc <- trainControl(classProbs = TRUE, summaryFunction = twoClassSummary)
mod <- as.formula(mpg_bound ~ cyl)
set.seed(666)
m1 <- train(mod, data = df,
method = "rpart",
preProcess = c("center", "scale"),
trControl = tc)
fancyRpartPlot(m1$finalModel)
m1$finalModel
n= 32
node), split, n, loss, yval, (yprob)
* denotes terminal node
1) root 32 14 bad (0.5625000 0.4375000)
2) cyl8>=0.124004 14 0 bad (1.0000000 0.0000000) *
3) cyl8< 0.124004 18 4 good (0.2222222 0.7777778) *
I don't understand this part cyl8>=0.124004. I get that cyl8 is the dummy variable for the factor but what does it mean that cyl8>=0.124004?
I'd like to extend the existing answer, because I don't think the conclusion reached in the comments is true. As you say, when using the formula interface, caret's train function will transform factor variables into dummy variables that only take the values 0 or 1, e.g. cyl8 == 1 means 'the car has 8 cylinders'. Each dummy variable makes a statement about a characteristic that is either true or false for the observation.
Rpart will nevertheless output a numeric value as the split criterion, so that cyl8 >= 0.5, cyl8 >= 0.2 and cyl8 == 1 all mean the same thing "This car has exactly 8 cylinders". By default, rpart will choose the split value cyl8 >= 0.5 for binary dummies to indicate that the dummy is true. The interpretation of cyl8 >= 0.5 is then "Does the car have 8 cylinders?" (and not "Does the car have more than 8 cylinders?")
df <- mtcars
df$cyl <- factor(df$cyl)
df$mpg_bound <- ifelse(df$mpg > 20, "good", "bad")
library(caret)
tc <- trainControl(classProbs = TRUE, summaryFunction = twoClassSummary)
set.seed(166)
m1 <- train(mod, data = df,
method = "rpart",
#preProcess = c("center", "scale"),
trControl = tc,
metric = "ROC")
m1$finalModel
#1) root 32 14 bad (0.5625000 0.4375000)
#2) cyl8>=0.5 14 0 bad (1.0000000 0.0000000) *
#3) cyl8< 0.5 18 4 good (0.2222222 0.7777778) *
The confusing value in your example is caused because caret apparently applies the preProcessing to the extended dataset where the dummies are numeric variables. The interpretation stays the same, but the (arbitrary) split value is transformed.
# Transform to dummies
mm <- model.matrix(mpg_bound ~ .-1, data = df)
# Do pre-processing
pp <- preProcess(mm, method = c("center", "scale"))
mm.pp <- as.matrix(predict(pp, mm))
# Dummy-Split in the middle
(max(mm.pp[,"cyl8"]) + min(mm.pp[,"cyl8"]) ) / 2
I think this value represents the split point based on the dummy var scale (0-1). This code produces the same outcome:
df = mtcars
df$cyl <- factor(df$cyl)
df$mpg_bound <- ifelse(df$mpg > 20, "good", "bad")
tc <- trainControl(classProbs = TRUE, summaryFunction = twoClassSummary)
data = cbind(df,model.matrix(~cyl+mpg_bound,df)) # binds the dummy transf to the data
mod <- as.formula(mpg_bound ~ cyl8)
m1 <- train(mod, data = data,
method = "rpart",
preProcess = c("center", "scale"),
trControl = tc)
m1$finalModel
It might be easier running the rpart code directly (incl original scale), although this might not allow you to specify the features you used. e.g.
rpart(mpg_bound~cyl,data=df,method="class")
Related
I want to perform a logistic regression with the train() function from the caret package. My model looks something like that:
model <- train(Y ~.,
data = train_data,
family = "binomial",
method = "glmnet")
With the resulting model, I want to make predictions:
pred <- predict(model, newdata = test_data, s = "lambda.min", type = "prob")
Now, I want to evaluate how good the model predictions are in comparison with the actual test data. For this I know how to receive the ROC and AUC. However I am also interested in receiveing the BRIER SCORE. The formula for the Brier Score is almost identical to the MSE.
The problem I am facing, is that the type argument in predict only allows "prob" (or "class" which I am not interested in) which gives the probability of one prediction beeing a ONE (e.g. 0.64) , and the complementing probability of beeing a ZERO (e.g. 0.37). For the Brier Score however, I need One probability estimate for each prediction that contains the information of both (e.g. a value above 0.5 would indicate a 1 and a value below 0.5 would indicate a 0).
I have not found any solution for receiving the Brier Score in the caret package. I am aware that with the package cv.glmnet the predict function allows the argument "response" which would solve my problem. However, for personal preferences I would like to stay with the caretpackage.
Thanks for the help!
If we go by the wiki definition of brier score:
The most common formulation of the Brier score is
where f_t is the probability that was forecast, o_t the actual outcome of the (0 or 1) and N is the number of forecasting instances.
In R, if your label is a factor, then the logistic regression will always predict with respect to the 2nd level, meaning you just calculate the probability and 0/1 with respect to that. For example:
library(caret)
idx = sample(nrow(iris),100)
data = iris
data$Species = factor(ifelse(data$Species=="versicolor","v","o"))
levels(data$Species)
[1] "o" "v"
In this case, o is 0 and v is 1.
train_data = data[idx,]
test_data = data[-idx,]
model <- train(Species ~.,data = train_data,family = "binomial",method = "glmnet")
pred <- predict(model, newdata = test_data)
So we can see the probability of the class:
head(pred)
o v
1 0.8367885 0.16321154
2 0.7970508 0.20294924
3 0.6383656 0.36163437
4 0.9510763 0.04892370
5 0.9370721 0.06292789
To calculate the score:
f_t = pred[,2]
o_t = as.numeric(test_data$Species)-1
mean((f_t - o_t)^2)
[1] 0.32
I use the Brier score to tune my models in caret for binary classification. I ensure that the "positive" class is the second class, which is the default when you label your response "0:1". Then I created this master summary function, based on caret's own suite of summary functions, to return all the metrics I want to see:
BigSummary <- function (data, lev = NULL, model = NULL) {
pr_auc <- try(MLmetrics::PRAUC(data[, lev[2]],
ifelse(data$obs == lev[2], 1, 0)),
silent = TRUE)
brscore <- try(mean((data[, lev[2]] - ifelse(data$obs == lev[2], 1, 0)) ^ 2),
silent = TRUE)
rocObject <- try(pROC::roc(ifelse(data$obs == lev[2], 1, 0), data[, lev[2]],
direction = "<", quiet = TRUE), silent = TRUE)
if (inherits(pr_auc, "try-error")) pr_auc <- NA
if (inherits(brscore, "try-error")) brscore <- NA
rocAUC <- if (inherits(rocObject, "try-error")) {
NA
} else {
rocObject$auc
}
tmp <- unlist(e1071::classAgreement(table(data$obs,
data$pred)))[c("diag", "kappa")]
out <- c(Acc = tmp[[1]],
Kappa = tmp[[2]],
AUCROC = rocAUC,
AUCPR = pr_auc,
Brier = brscore,
Precision = caret:::precision.default(data = data$pred,
reference = data$obs,
relevant = lev[2]),
Recall = caret:::recall.default(data = data$pred,
reference = data$obs,
relevant = lev[2]),
F = caret:::F_meas.default(data = data$pred, reference = data$obs,
relevant = lev[2]))
out
}
Now I can simply pass summaryFunction = BigSummary in trainControl and then metric = "Brier", maximize = FALSE in the train call.
I have detected a peculiar effect that RMSE gets lower for the testing set than that of the training set with the sample function with the caret package.
My code does a common split of training and testing set:
set.seed(seed)
training.index <- createDataPartition(dataset[[target_label]], p = 0.8, list = FALSE)
training.set <- dataset[training.index, ]
testing.set <- dataset[-training.index, ]
This e.g. gives an RMSE for testing set 0.651 which is higher than training set RMSE 0.575 - as expected.
Following the recommendation of many sources, e.g. here, the data should be shuffled, so I do it before the above split:
# shuffle data - short version:
set.seed(17)
dataset <- data %>% nrow %>% sample %>% data[.,]
After this shuffle, the testing set RMSE gets lower 0.528 than the training set RMSE 0.575! This finding is consistent across a number of algorithms including lm, glm, knn, kknn, rf, gbm, svmLinear, svmRadial etc.
According to my knowledge, the default of sample() is replace = FALSE so there can't be any data leakage into the testing set. The same observation occurs in classification (for accuracy and kappa) although the createDataPartition performs stratification, so any data imbalance should be handled.
I don't use any extraordinary configuration, just ordinary cross-validation:
training.configuration <- trainControl(
method = "repeatedcv", number = 10
, repeats = CV.REPEATS
, savePredictions = "final",
# , returnResamp = "all"
)
What did I miss here?
--
Update 1: Hunch about data leakage into testing set
I checked the data distribution and found a potential hint for the described effect.
Training set distribution:
. Freq prop
1 1 124 13.581599
2 2 581 63.636364
3 3 194 21.248631
4 4 14 1.533406
Testing set distribution without shuffle:
. Freq prop
1 1 42 18.502203
2 2 134 59.030837
3 3 45 19.823789
4 4 6 2.643172
Testing set distribution with shuffle:
. Freq prop
1 1 37 16.299559
2 2 139 61.233480
3 3 45 19.823789
4 4 6 2.643172
If we look at the mode (most frequent value), its proportion in the testing set with shuffle 61.2% is closer to the training set proportion 63.6% than without shuffle 59.0%.
I don't know how to interpret this statistically by underlying theory - can anybody?
An intuition of mine is that the shuffling makes the stratification of the testing set distribution (implicitly performed by createDataPartition()) "more stratified" - by that I mean "closer to the training set distribution". This may cause the effect of data leakage into the opposite direction - into the testing set.
Update 2: Reproducible Code
library(caret)
library(tidyverse)
library(magrittr)
library(mlbench)
data(BostonHousing)
seed <- 171
# shuffled <- TRUE
shuffled <- FALSE
if (shuffled) {
dataset <- BostonHousing %>% nrow %>% sample %>% BostonHousing[., ]
} else {
dataset <- BostonHousing %>% as_tibble()
}
target_label <- "medv"
features_labels <- dataset %>% select_if(is.numeric) %>%
select(-target_label) %>% names %T>% print
# define ml algorithms to train
algorithm_list <- c(
"lm"
, "glmnet"
, "knn"
, "gbm"
, "rf"
)
# repeated cv
training_configuration <- trainControl(
method = "repeatedcv", number = 10
, repeats = 10
, savePredictions = "final",
# , returnResamp = "all"
)
# preprocess by standardization within each k-fold
preprocess_configuration = c("center", "scale")
# select variables
dataset %<>% select(target_label, features_labels) %>% na.omit
# dataset subsetting for tibble: [[
set.seed(seed)
training.index <- createDataPartition(dataset[[target_label]], p = 0.8, list = FALSE)
training.set <- dataset[training.index, ]
testing.set <- testing.set <- dataset[-training.index, ]
########################################
# 3.2: Select the target & features
########################################
target <- training.set[[target_label]]
features <- training.set %>% select(features_labels) %>% as.data.frame
########################################
# 3.3: Train the models
########################################
models.list <- list()
models.list <- algorithm_list %>%
map(function(algorithm_label) {
model <- train(
x = features,
y = target,
method = algorithm_label,
preProcess = preprocess_configuration,
trControl = training_configuration
)
return(model)
}
) %>%
setNames(algorithm_list)
UPDATE: Code to calculate testingset performance
observed <- testing.set[[target_label]]
models.list %>%
predict(testing.set) %>%
map_df(function(predicted) {
sqrt(mean((observed - predicted)^2))
}) %>%
t %>% as_tibble(rownames = "model") %>%
rename(RMSE.testing = V1) %>%
arrange(RMSE.testing) %>%
as.data.frame
Running this code both for shuffled = FALSE and shuffled = TRUE on the testing.set gives:
model RMSE.testing RMSE.testing.shuffled
1 gbm 3.436164 2.355525
2 glmnet 4.516441 3.785895
3 knn 3.175147 3.340218
4 lm 4.501077 3.843405
5 rf 3.366466 2.092024
The effect is reproducible!
The reason you get a different test RMSE is because you have a different test set. You are shuffling your data and then using the same training.index each time so there's no reason to believe the test set would be the same each time.
In your original comparison you need to compare the RMSE from the shuffled test data with the RMSE of the shuffled training data, not the original training data.
Edit: the shuffling is also unnecessary as createDataPartition has its own sampling scheme. You can just change the seed if you want a different test/training split
I completely agree with the answer of Jonny Phelps. Based on your code and caret functions code there is no reason to suspect any kind of data leakage when using createDataPartition on shuffled data. So the variation in performance must be due to different train/test splits.
In order to prove this I checked the performance of the shuffled and non-shuffled workflow using 10 different seeds with 4 algorithms:
I omitted lm and replaced gbm with xgboost. The reason is my own preference.
In my opinion the result suggests there is no performance pattern between shuffled and non shuffled data. Perhaps only KNN looks suspicions. But this is just one algorithm.
Code:
create initial seeds:
set.seed(1)
gr <- expand.grid(sample = sample(1L:1e5L, 10),
shuffled = c(FALSE, TRUE))
loop over initial seeds:
apply(gr, 1, function(x){
print(x)
shuffled <- x[2]
set.seed(x[1])
if (shuffled) {
dataset <- BostonHousing %>% nrow %>% sample %>% BostonHousing[., ]
} else {
dataset <- BostonHousing %>% as_tibble()
}
target_label <- "medv"
features_labels <- dataset %>% select_if(is.numeric) %>%
select(-target_label) %>% names %T>% print
algorithm_list <- c(
"glmnet",
"knn",
"rf",
"xgbTree"
)
training_configuration <- trainControl(
method = "repeatedcv",
number = 5, #5 folds and 3 reps is plenty
repeats = 3,
savePredictions = "final",
search = "random") #tune hyper parameters
preprocess_configuration = c("center", "scale")
dataset %<>% select(target_label, features_labels) %>% na.omit
set.seed(x[1])
training.index <- createDataPartition(dataset[[target_label]], p = 0.8, list = FALSE)
training.set <- dataset[training.index, ]
testing.set <- testing.set <- dataset[-training.index, ]
target <- training.set[[target_label]]
features <- training.set %>% select(features_labels) %>% as.data.frame
models.list <- list()
models.list <- algorithm_list %>%
map(function(algorithm_label) {
model <- train(
x = features,
y = target,
method = algorithm_label,
preProcess = preprocess_configuration,
trControl = training_configuration,
tuneLength = 100 #get decent hyper parameters
)
return(model)
}
) %>%
setNames(algorithm_list)
observed <- testing.set[[target_label]]
models.list %>%
predict(testing.set) %>%
map_df(function(predicted) {
sqrt(mean((observed - predicted)^2))
}) %>%
t %>% as_tibble(rownames = "model") %>%
rename(RMSE.testing = V1) %>%
arrange(RMSE.testing) %>%
as.data.frame
}) -> perf
do.call(rbind, perf) %>%
mutate(shuffled = rep(c(FALSE, TRUE), each = 40)) %>%
ggplot()+
geom_boxplot(aes(x = model, y = RMSE.testing, color = shuffled)) +
theme_bw()
I know variations of this question have been asked before but I haven't yet seen an answer on how to implement the multinomial Poisson transformation with multilevel models.
I decided to make a fake dataset and follow the method outlined here, also consulting the notes the poster mentions as well as the Baker paper on MP transformation.
In order to check if I'm doing the coding correctly, I decided to create a binary outcome variable as a first step; because glmer can handle binary response variables, this will let me check I'm correctly recasting the logit regression as multiple Poissons.
The context of this problem is running multilevel regressions with survey data where the outcome variable is response to a question and the possible predictors are demographic variables. As I mentioned above, I wanted to see if I could properly code the binary outcome variable as a Poisson regression before moving on to multi-level outcome variables.
library(dplyr)
library(lme4)
key <- expand.grid(sex = c('Male', 'Female'),
age = c('18-34', '35-64', '45-64'))
set.seed(256)
probs <- runif(nrow(key))
# Make a fake dataset with 1000 responses
n <- 1000
df <- data.frame(sex = sample(c('Male', 'Female'), n, replace = TRUE),
age = sample(c('18-34', '35-64', '45-64'), n, replace = TRUE),
obs = seq_len(n), stringsAsFactors = FALSE)
age <- model.matrix(~ age, data = df)[, -1]
sex <- model.matrix(~ sex, data = df)[, -1]
beta_age <- matrix(c(0, 1), nrow = 2, ncol = 1)
beta_sex <- matrix(1, nrow = 1, ncol = 1)
# Create class probabilities as a function of age and sex
probs <- plogis(
-0.5 +
age %*% beta_age +
sex %*% beta_sex +
rnorm(n)
)
id <- ifelse(probs > 0.5, 1, 0)
df$y1 <- id
df$y2 <- 1 - df$y1
# First run the regular hierarchical logit, just with a varying intercept for age
glm_out <- glmer(y1 ~ (1|age), family = 'binomial', data = df)
summary(glm_out)
#Next, two Poisson regressions
glm_1 <- glmer(y1 ~ (1|obs) + (1|age), data = df, family = 'poisson')
glm_2 <- glmer(y2 ~ (1|obs) + (1|age), data = df, family = 'poisson')
coef(glm_1)$age - coef(glm_2)$age
coef(glm_out)$age
The outputs for the last two lines are:
> coef(glm_1)$age - coef(glm_2)$age
(Intercept)
18-34 0.14718933
35-64 0.03718271
45-64 1.67755129
> coef(glm_out)$age
(Intercept)
18-34 0.13517758
35-64 0.02190587
45-64 1.70852847
These estimates seem close but they are not exactly the same. I'm thinking I've specified an equation wrong with the intercept.
Suppose i'm doing seveal runs of the same model, but only with different complexity parameters, on the same (seed fixed) cross-validation with the caret package, for exemple :
library(caret)
data(iris)
# controls are the same for every models
c = trainControl(method = "cv",number=10,verboseIter = TRUE)
d = iris # data is also the same
f = Species ~ . # formula is also the same
m = "rpart" # method is also the same
set.seed(1234)
model1 <- train(form = f, data = d, trControl = c, method = m,
tuneGrid = expand.grid(cp = c(0,0.5)))
set.seed(1234)
model2 <- train(form = f, data = d, trControl = c, method = m,
tuneGrid = expand.grid(cp = c(0.1,0.2)))
set.seed(1234)
model3 <- train(form = f, data = d, trControl = c, method = m,
tuneGrid = expand.grid(cp = c(0,0.5,0.1,0.2)))
Is there a way i could "build up" the model3 train object only from model1 and the model2 ?
Calculations are long, and i did'nt ran all my different tuning in the same caret call. But having every run in the same train object will be much easier for comparing them (via the plot function, the update function, the resamples function, etc...)
I'm particularly looking for a way do do the same thing plot.train do but for all of them together.
I perfectly understand your concern, because my computation sources are also very limited. However I would approach it as follows, instead of "building up" the model3 object.
Suppose what you wish to achieve is highest accuracy. Then you simply need to evaluate the following: which among the model1 and model2 do we see highest accuracy? Then we are only interested in choosing the best-result tuning parameter. For example, we see the following:
> model1$bestTune$cp
[1] 0
> model2$bestTune$cp
[1] 0.2
> model1$results$Accuracy ## Respectively for cp = 0.0 and cp = 0.5
[1] 0.9333 0.3333
> model2$results$Accuracy ## Respectively for cp = 0.1 and cp = 0.2
[1] 0.9267 0.9267
We would choose cp = 0.
Suppose you have broken things down to model1, model2, model3, ... and wish to explore all manually input parameter values using them.
k = 2 ## Here we only have model1 and model2 to compare
evaluate <- list()
for (i in 1:k) {
model = eval(parse(text = paste0("model", i)))
evaluate[["cp"]][[paste0("model", i)]] <-
model$finalModel$tuneValue$cp
evaluate[["accuracy"]][[paste0("model", i)]] <-
model$results$Accuracy[[which(model$results$cp == model$bestTune$cp)]]
}
Then in our evaluate list, we have the following:
> evaluate
$cp
model1 model2
0.0 0.2
$accuracy
model1 model2
0.9333 0.9267
Upon this, we can do
> which(evaluate$accuracy == max(evaluate$accuracy))
model1
1
> evaluate$cp[[which(evaluate$accuracy == max(evaluate$accuracy))]]
[1] 0
Now we can happily choose cp = 0 and we also know that the result from the optimal cp is stored in model1.
If you wish to still "build up" the model3, you can simply substitute some of the components (e.g. results in which AccuracySD, KappaSD, and such metrics would be stored) after having chosen what we evaluated as the best model---model1 in this case.
I am trying to use AUPRC as my custom metric for a gbm model fit because I have imbalanced classifier. However, when i try to incorporate the custom metric I am getting the following error mentioned in the code. Not sure what I am doing wrong.
Also the auprcSummary() works on its own when i run it inline. It is giving me an error when i try to incorporate it in train().
library(dplyr) # for data manipulation
library(caret) # for model-building
library(pROC) # for AUC calculations
library(PRROC) # for Precision-Recall curve calculations
auprcSummary <- function(data, lev = NULL, model = NULL){
index_class2 <- data$Class == "Class2"
index_class1 <- data$Class == "Class1"
the_curve <- pr.curve(data$Class[index_class2],
data$Class[index_class1],
curve = FALSE)
out <- the_curve$auc.integral
names(out) <- "AUPRC"
out
}
ctrl <- trainControl(method = "repeatedcv",
number = 10,
repeats = 5,
summaryFunction = auprcSummary,
classProbs = TRUE)
set.seed(5627)
orig_fit <- train(Class ~ .,
data = toanalyze.train,
method = "gbm",
verbose = FALSE,
metric = "AUPRC",
trControl = ctrl)
This is the error I am getting:
Error in order(scores.class0) : argument 1 is not a vector
Is it because pr.curve() takes only numeric vectors as inputs (scores/probabilities?)
caret has a built-in function called prSummary that computes that for you. You don't have to write your own.
I think this approach yields an appropriate custom summary function:
library(caret)
library(pROC)
library(PRROC)
library(mlbench) #for the data set
data(Ionosphere)
in pr.curve function the classification scores may be either provided separately for the data points of each of the classes, i.e., as scores.class0 for the data points from the positive/foreground class and as scores.class1 for the data points of the negative/background class; or the classification scores for all data points are provided as scores.class0 and the labels are provided as numerical values (1 for the positive class, 0 for the negative class) as weights.class0 (I copied this from the help of the function I apologize if it is unclear).
I opted to provide the later - probability for all in scores.class0 and class assignment in weights.class0.
caret states that if the classProbs argument of the trainControl object is set to TRUE, additional columns in data will be present that contains the class probabilities. So for the Ionosphere data columns good and bad should be present:
levels(Ionosphere$Class)
#output
[1] "bad" "good"
to convert to 0/1 labeling one can just do:
as.numeric(Ionosphere$Class) - 1
good will become 1
bad will become 0
now we have all the data for the custom function
auprcSummary <- function(data, lev = NULL, model = NULL){
prob_good <- data$good #take the probability of good class
the_curve <- pr.curve(scores.class0 = prob_good,
weights.class0 = as.numeric(data$obs)-1, #provide the class labels as 0/1
curve = FALSE)
out <- the_curve$auc.integral
names(out) <- "AUPRC"
out
}
Instead of using data$good which will work on this data set alone one can extract the class names and use that to get the desired column:
lvls <- levels(data$obs)
prob_good <- data[,lvls[2]]
It is important to note each time you update the summaryFunction you need to update the trainControl object.
ctrl <- trainControl(method = "repeatedcv",
number = 10,
repeats = 5,
summaryFunction = auprcSummary,
classProbs = TRUE)
orig_fit <- train(y = Ionosphere$Class, x = Ionosphere[,c(1,3:34)], #omit column 2 to avoid a bunch of warnings related to the data set
method = "gbm",
verbose = FALSE,
metric = "AUPRC",
trControl = ctrl)
orig_fit$results
#output
shrinkage interaction.depth n.minobsinnode n.trees AUPRC AUPRCSD
1 0.1 1 10 50 0.9722775 0.03524882
4 0.1 2 10 50 0.9758017 0.03143379
7 0.1 3 10 50 0.9739880 0.03316923
2 0.1 1 10 100 0.9786706 0.02502183
5 0.1 2 10 100 0.9817447 0.02276883
8 0.1 3 10 100 0.9772322 0.03301064
3 0.1 1 10 150 0.9809693 0.02078601
6 0.1 2 10 150 0.9824430 0.02284361
9 0.1 3 10 150 0.9818318 0.02287886
Seems reasonable