I am working on surrogate modeling and having a problem with printing rmse value of the kriging surrogate object. It is inside an optimizer class and I want to print out rmse like in the last code in the documentation here.
Code is here
surr is the surrogate class, I want to get rmse value of the output from this class
surrOpt is the parent class, performs surrogate-based optimization.
Related
Does the "jags" function of R2jags give estimated states output when estimating a state space model?
Or is it to be derived manually using estimated mean/medians of parameter values.
I will try to explain my problem as best as i can. I am trying to externally validate a prediction model, made by one of my colleagues. For the external validation, I have collected data from a set of new patients.
I want to test the accuracy of the prediction model on this new dataset. Online i have found a way to do so, using the coef.orig function to extract de coefficients of the original prediction model (see the picture i added). Here comes the problem, it is impossible for me to repeat the steps my colleague did to obtain the original prediction model. He used multiple imputation and bootstrapping for the development and internal validation, making it very complex to repeat his steps. What I do have, is the computed intercept and coefficients from the original model. Step 1 from the picture i added, could therefor be skipped.
My question is, how can I add these coefficients into the regression model, without the use of the 'coef()' function?
Steps to externally validate the prediction model:
The coefficients I need to use:
I thought that the offset function would possibly be of use, however this does not allow me to set the intercept and all the coefficients for the variables at the same time
Do I understand correctly that the value (model output) retrieving from evaluate_model() for the linear regression model is RMSE?
No, the column output, model_output, is not the root mean square error (RMSE). It is the prediction value for your model.
So it appears that the evaluate_model() function is from the statisticalModeling package.
According to the documentation for this function, it "Evaluate a model for specified inputs" and (emphasis added below)
Find the model outputs for specified inputs. This is equivalent to the generic predict() function, except it will choose sensible values by default. This simplifies getting a quick look at model values.
In other words, this function, evaluate_model(), takes inputs and shows you the "prediction" of your input data using the model.
For your instance, evaluate_model() will take each row of your data and use the column/variable information (age, sex, coverage) for each row, and fits the data to estimate what the dependent variable would be based on this model.
I am using caret to train a classification model on a dataset, from which I take the outputted class probabilities and feed them into another set of calculations. Specifically, I sum the class probabilities along one dimension of the original data and use that to calculate my eventual summary statistic describing the model quality of fit.
I currently have to train the model to maximize some other metric (I'm using Kappa right now), but what I really want to do is to write a summaryFunction to pass to trainControl that will encapsulate the entire calculation from start-to-finish.
The problem is that this summaryFunction would require the original data points, since I have to aggregate the class probabilities along a dimension of the original data in order to calculate the summary statistic. The summaryFunction prototype doesn't seem to provide the data itself in any way that I can see.
Is there a simple solution here? I suppose I could just make the original data frame global and use its values in my summaryFunction, provided that the class probabilities and predictions passed to the summaryFunction are in the same number and row order as the original data set?
Thank you very much!!
Sorry for this novice question: if I fit a lm() model or loess() model, and save the model somewhere in a file or in a database, for later use by a third party with predict() method, do I have to save the entire model object? Since returned model object contains the original raw data, this returned object can be huge.
If you include the argument model = FALSE (it's true by default) when fitting the model, the model frame that was used will be excluded from the resulting object. You can get an estimate of the memory that is being used to store the model object giving:
object.size(my_model)