stackoverflowers , I need some help from tensorflow experts. Actually I've buid a multi-layer perceptron, trained it, tested it and everything seemed ok. However, When I restored the model and tried to use it again, its accuracy does not correspond to the trained model and the predictions are pretty different from the real labels. The code I am using for the restoring - prediction is the following : (I'm using R)
pred <- multiLayerPerceptron(test_data)
init <- tf$global_variables_initializer()
with(tf$Session() %as% sess, {
sess$run(init)
model_saver$restore(sess, "log_files/model_MLP1")
test_pred_1 <- sess$run(pred, feed_dict= dict(x = test_data))
})
Is everything Ok with the code ? FYI I wanted by this part of the code to get the predictions of my model for test_data.
Your code does not show where model_saver is initialized, but it should be created after you create the computational graph. If not, it does not know which variables to restore/save. So create your model_saver after pred <- multiLayerPerceptron(test_data).
Note that, if you made the same mistake during training, your checkpoint will be empty and you will need to retrain your model first.
Related
'bst' is the name of an xgboost model that I built in R. It gives me predicted values for the test dataset using this code. So it is definitely an xgboost model.
pred.xgb <- predict(bst , xdtest) # get prediction in test sample
cor(ytestdata, pred.xgb)
Now, I would like to save the model so another can use the model with their data set which has the same predictor variables and the same variable to be predicted.
Consistent with page 4 of xgboost.pdf, the documentation for the xgboost package, I use the xgb.save command:
xgb.save(bst, 'xgb.model')
which produces the error:
Error in xgb.save(bst, "xgb.model") : model must be xgb.Booster.
Any insight would be appreciated. I searched the stack overflow and could not locate relevant advice.
Mike
It's hard to know exactly what's going on without a fully reproducible example. But just because your model can make predictions on the test data, it doesn't mean it's an xgboost model. It can be any type of model with a predict method.
You can try class(bst) to see the class of your bst object. It should return "xgb.Booster," though I suspect it won't here (hence your error).
On another note, if you want to pass your model to another person using R, you can just save the R object rather than exporting to binary, via:
save(bst, model.RData)
I used H2O version 3.26.0.5 to train a GBM model in a binary problem, to predict the probability of positive class. I saved the model file as MOJO and used this file to generate predictions in new data:
## first, restart R session ##
# load the model
library(h2o)
h2o.init(nthreads = -1)
model <- h2o.import_mojo("path_to_mojo_file")
# load the new data input
input <- read_csv("path_to_new_data")
input_h2o <- as.h2o(input)
# predictions
predictions <- predict(model, input_h2o)
When I run this in my computer I get different predictions than when I use the same MOJO file to predict in a production environment.
Does this should happen with the MOJO file? I believed that once the model was saved in MOJO format, you could make predictions in any environment and get the same results. Does anyone knows why this is happening?
When I run this in my computer I get different predictions than when I use the same MOJO file to predict in a production environment.
Is the production environment running the exact same R script?
In the end I found out there was an error in the script for the production environment. After it was fixed, the predictions became pretty close.
After training an XGBoost model in R, I am presented with a model object called xgb which is a list of 7.
When I save the model using xgb.save and then reload using xgb.load, I am presented with what seems to be a 'smaller' model object which is a list of 2.
Obviously I can't share the code as you would need the training data which is massive, so all I can really show is a picture of the variable editor.
Below is model object xgb which is the original model after training, vs. the model object test1 which is the same model but saved and reloaded:
Why does this happen and am I losing valuable information upon saving/loading my models?
Any help is appreciated.
Maybe late, but I was having the same problem and found a solution.
Saving the xgb-model as "rds" doesn't loose any information and the reloaded model xgb_ does generate the same forecast values as the original xgb when I tested it. Hope that helps!
saveRDS(xgb, "model.rds")
xgb_ <- readRDS("model.rds")
all.equal(xgb, xgb_)
You are loosing information due to rounding errors after save/load. See this issue. I believe it is currently a bug.
As to why the loaded model is a smaller list, see here. So again, you are loosing information such as the callbacks and parameters. But these are not essential for prediction and not portable to e.g. python.
Reasonably new to this so sorry if I'm being thick.
Is there a way to pass existing models to caretEnsemble?
I have several models, run on the same training data, that I would like to ensemble with caretEnsemble.
Each model takes several hours to run, so I save them, then reload them when needed rather than re-run.
model_xgb <- train(oi_in_4_24_months~., method="xgbTree", data=training, trControl=train_control)
saveRDS(model_xgb, "model_xgb.rds")
model_logit <- train(oi_in_4_24_months~., method="LogitBoost", data=training, trControl=train_control)
saveRDS(model_logit, "model_logit.rds")
model_xgb <- readRDS("model_xgb.rds")
model_logit <- readRDS("model_logit.rds")
I want to pass these saved models to caretEnsemble, but as far as I can make out I can only pass a list of model types, e.g. "LogitBoost", "xgbTree", and caretEnsemble will both run the initial models, then ensemble them.
Is there a way to pass existing models, trained on the same data, to caretEnsemble?
The package author has an example script (https://gist.github.com/zachmayer/5152157) that suggests the following:
all_models <- list(model_xgb, model_logit)
names(all_models) <- sapply(all_models, function(x) x$method)
greedy <- caretEnsemble(all_models, iter=1000L)
But that produces an error
"Error: is(list_of_models, "caretList") is not TRUE".
I think that use of caretList previously wasn't compulsory, but now is.
I don't think you still need the solution to this but answering for anyone else that has the same question.
You can add models to be used by caretEnsemble or caretStack by using as.caretList(list(rpart2 = model_list1, gbm = model_list2))
But remember to use the same indexes for cross-validation/bootstrapping. 'If the indexes were different (or some stuff were not stored as "not/wrongly" specified in trainControl), it will throw an error when trying to use caretEnsemble or caretStack. Which is the expected behavior, obviously.' This issue on github has very clear and simple instructions.
I hope I have come to the right forum. I'm an ecologist making species distribution models using the maxent (version 3.3.3, http://www.cs.princeton.edu/~schapire/maxent/) function in R, through the dismo package. I have used the argument "replicates = 5" which tells maxent to do a 5-fold cross-validation. When running maxent from the maxent.jar file directly (the maxent software), an html file with statistics will be made, including the prediction maps. In R, an html file is also made, but the prediction maps have to be extracted afterwards, using the function "predict" in the dismo package in r. When I do this, I get 5 maps, due to the 5-fold cross-validation setting. However, (and this is the problem) I want only one output map, one "summary" prediction map. I assume this is possible, although I don't know how maxent computes it. The maxent tutorial (see link above) says that:
"...you may want to avoid eating up disk space by turning off the “write output grids” option, which will suppress writing of output grids for the replicate runs, so that you only get the summary statistics grids (avg, stderr etc.)."
A list of arguments that can be put into R is found in this forum https://groups.google.com/forum/#!topic/maxent/yRBlvZ1_9rQ.
I have tried to use the argument "outputgrids=FALSE" both in the maxent function itself, and in the predict function, but it doesn't work. I still get 5 maps, even though I don't get any errors in R.
So my question is: How do I get one "summary" prediction map instead of the five prediction maps that results from the cross-validation?
I hope someone can help me with this, I am really stuck and haven't found any answers anywhere on the internet. Not even a discussion about this. Hope my question is clear. This is the R-script that I use:
model1<-maxent(x=predvars, p=presence_points, a=target_group_absence, path="//home//...//model1", args=c("replicates=5", "outputgrids=FALSE"))
model1map<-predict(model1, predvars, filename="//home//...//model1map.tif", outputgrids=FALSE)
Best regards,
Kristin
Sorry to be the bearer of bad news, but based on the source code, it looks like Dismo's predict function does not have the ability to generate a summary map.
Nitty-gritty details for those who care: When you call maxent with replicates set to something greater than 1, the maxent function returns a MaxEntReplicates object, rather than a normal MaxEnt object. When predict receives a MaxEntReplicates object, it just iterates through all of the models that it contains and calls predict on them individually.
So, what next? Fortunately, all is not lost! The reason that Dismo doesn't have this functionality is that for most kinds of model-building, there isn't actually a valid way to average parameters across your cross-validation models. I don't want to go so far as to say that that's definitely the case for MaxEnt specifically, but I suspect it is. As such, cross-validation is usually used more as a way of checking that your model building methodology works for your data than as a way of building your model directly (see this question for further discussion of that point). After verifying via cross-validation that models built using a given procedure seem to be accurate for the phenomenon you're modelling, it's customary to build a final model using all of your data. In theory this new model should only be better than models trained on a subset of your data.
So basically, assuming your cross-validated models look reasonable, you can run MaxEnt again with only one replicate. Your final result will be a model accuracy estimate based on the cross-validation and a map based on the second run with all of your data lumped together. Depending on what exactly your question is, there might be other useful summary statistics from the cross-validation that you want to use, but those are all things you've already seen in the html output.
I may have found this a couple of years later. But you could do something like this:
xm <- maxent(predictors, pres_train) # basically the maxent model
px <- predict(predictors, xm, ext=ext, progress= '' ) #prediction
px2 <- predict(predictors, xm2, ext=ext, progress= '' ) #prediction #02
models <- stack(px,px2) # create a stack of prediction from all the models
final_map <- mean(px,px2) # Take a mean of all the prediction
plot(final_map) #plot the averaged map
xm1,xm2,.. would be the maxent models for each partitions in cross-validation, and px, px2,.. would be the predicted maps.