I am attempting to run h2o.glm in R but am encountering some strange behaviour. The same line of code sometimes works and sometimes errors with the following result
h2o.glm(x = Predictors.Revised, y = "NN", model_id = "GLM_FREQ_INITIAL",
offset_column = "Offset.To.Apply", nfolds = 5, family = "poisson",
link = "log", lambda_search = TRUE, training_frame = TrainDS.h2o,
alpha = 1, standardize = TRUE)
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32 at
water.util.ArrayUtils.subtract(ArrayUtils.java:1334) at
hex.glm.GLM$GLMDriver.fitIRLSM(GLM.java:824) at
hex.glm.GLM$GLMDriver.fitModel(GLM.java:1080) at
hex.glm.GLM$GLMDriver.computeSubmodel(GLM.java:1169) at
hex.glm.GLM.cv_computeAndSetOptimalParameters(GLM.java:132) at
hex.ModelBuilder.cv_buildModels(ModelBuilder.java:595) at
hex.ModelBuilder.computeCrossValidation(ModelBuilder.java:431) at
hex.glm.GLM.computeCrossValidation(GLM.java:100) at
hex.ModelBuilder$1.compute2(ModelBuilder.java:309) at
water.H2O$H2OCountedCompleter.compute(H2O.java:1395) at
jsr166y.CountedCompleter.exec(CountedCompleter.java:468) at
jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) at
jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) at
jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) at
jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
R Version: 3.3.1
Platform: x86_64-pc-linux-gnu (64-bit)
h2o Version: 3.22.1.5
Any ideas why? I am stumped.
If you get an "out of bounds" error (e.g. java.lang.ArrayIndexOutOfBoundsException), you should check if there is anything you are iterating over, during multiple runs of your glm function. To help debug the issue I would remove all but the simplest arguments and then slowly add arguments in while running whatever set of steps led to the issue, until you can identify the parameter that is causing the out of bounds error.
Related
I am using R library of xgboost version 1.4.1.1 for a binary classification task which I have been regularly delivering for nearly 2 years.
I recently had an upgrade to my company laptop and needed to install R and the libraries I am using.
I am now trying to run this task and it gives me the following error:
Error in xgb.DMatrix(data, label = label, missing = missing) :
[14:28:40] amalgamation/../src/data/data.cc:945: Check failed: valid: Input data contains `inf` or `nan`
The data pipeline has not been changed, the data structure is exactly the same as before. I convert the data to data matrix, as below:
xgbmodel <- xgboost(data = data.matrix(mydata),
label = res,
eta = 0.2,
max_depth = 10,
gamma = 0.4,
lambda = 0.5,
nround = 40,
subsample = 0.7,
colsample_bytree = 0.75,
seed = 21,
eval_metric = "logloss",
objective = "binary:logistic"
)
#
And I get this error which I have never got before.
R version: 4.1.0
xgboost: 1.4.1.1
Any ideas on how to resolve this?
Edit: I uninstalled xgboost 1.4.1.1 and installed version 1.1.1.1. It's working. It seems to be a problem with the version and not with the data. I would like to use the latest version, that's why it would be good to know if anyone else has a similar issue and how to fix it.
I got the same error.
I guess the previous version of xgboost wad handling missing values which is no longer the case.
Either we drop the NA, or we impute them (as xgboost was doing before the update version) with for example KNN imputation, using mean or median.
Not sure if it's the answer you were looking for.
I'm running JAGS with R v3.6.1 and Rstudio v1.2 in Windows 10, using package R2jags. JAGS does not appear to find the stored values that I've created in R for MCMC settings such as n.iter, n.burn-in, etc. because my code:
out2 <- jags.parallel (data = data, inits = inits, parameters.to.save = parameters,
model.file = "C:\\Users\\jj1995\\Desktop\\VIP\\Thanakorn\\DLM\\district model.txt", n.chains = n.chain,
n.thin = n.thin, n.iter = n.iter, n.burnin = n.burn)
Produces the error
Error in checkForRemoteErrors(lapply(cl, recvResult)) :
3 nodes produced errors; first error: object 'n.burn' not found
If I replace a stored value's name with a number (n.burnin = 10000), it will return the same error, but for a different MCMC setting. I assume JAGS will also be unable to find the lists and dataframes I've stored in the global environment also. I thought my system's firewall was preventing JAGS from accessing the global environment, so I disabled my antivirus measures and I ran both R and Rstudio as an administrator. This did not solve the problem.
has this issue been resolved by now? I encounter the same error message.
Usecase: I am doing binary classification using h2o's deeplearning() function. Below, I provide randomly generated data the same size as my actual usecase. System specs:
# R version 3.3.2 (2016-10-31)
# Platform: x86_64-w64-mingw32/x64 (64-bit)
# Running under: Windows >= 8 x64 (build 9200)
# h2o version h2o_3.20.0.2
I am currently learning how to use h2o, so I have played with that function quite a bit. Everything runs smoothly until I specify parameters for cross validation.
The problem occurs when specifying the nfolds parameter for cross-validation. Interestingly, I can specify low values for nfolds and everything goes fine. For my use case, even nfolds > 3 produced an error message (see below). I provide an example below, here I was able to specify nfolds < 7 (not really consistent... sometimes just up to nfolds = 3). Above those values, the REST API give the above mentioned error: object not found for argument: key.
# R version 3.3.2 (2016-10-31)
# Platform: x86_64-w64-mingw32/x64 (64-bit)
# Running under: Windows >= 8 x64 (build 9200)
# h2o version h2o_3.20.0.2
#does not matter whether run on debian or windows, does not matter how many threads are used
#error occurs with options for cross validation, otherwise works fine
#no error occurs with specifying a low nfold number(in my actual use case, maximum of 3 folds possible without running into that error message)
require(h2o)
h2o.init(nthreads = -1)
x = matrix(rnorm(900*96, mean=10, sd=2), nrow=900, ncol=96)
y = sample(size=900, x=c(0,1), replace=T)
sampleData = cbind(x, y)
sampleData = data.frame(sampleData)
sampleData[,97] = as.factor(sampleData[,97])
m = h2o.deeplearning(x = 1:96, y = 97,
training_frame = as.h2o(sampleData), reproducible = T,
activation = "Tanh", hidden = c(64,16), epochs = 10000, verbose=T,
nfolds = 4, balance_classes = TRUE, #Cross-validation
standardize = TRUE, variable_importances = TRUE, seed=123,
stopping_rounds=2, stopping_metric="misclassification", stopping_tolerance=0.01, #early stopping
)
performance = h2o.performance(model = m)
print(performance)
######### gives error message
# ERROR: Unexpected HTTP Status code: 404 Not Found (url = http://localhost:xxxxx/3/Models/DeepLearning_model_R_1535036938222_489)
#
# water.exceptions.H2OKeyNotFoundArgumentException
# [1] "water.exceptions.H2OKeyNotFoundArgumentException: Object 'DeepLearning_model_R_1535036938222_489' not found for argument: key"
I cannot understand why it does work only for low values of nfolds. Any suggestions? What am I missing here? I've searched most remotely related threads on Google Groups and also here on stackoverflow, but without success. If this is to do with a changed API for h2o 3.x as suggested above (though that post was 18 months ago...) I would highly appreciate some documentary on how to correctly specify the syntax to do CV with h2o.deeplearning(). Thanks in advance!
This is a bug caused by setting the verbose parameter to True, the workaround is to leave the verbose parameter as the default which is FALSE. I've created a jira ticket to track the issue here
I am running the train function in caret on a cluster via doRedis. For the most part, it works, but every so often I get errors at the very end of this nature:
error calling combine function:
<simpleError: obj$state$numResults <= obj$state$numValues is not TRUE>
and
Error in names(resamples) <- gsub("^\\.", "", names(resamples)) :
attempt to set an attribute on NULL
when I run traceback() I get:
5: nominalTrainWorkflow(dat = trainData, info = trainInfo, method = method,
ppOpts = preProcess, ctrl = trControl, lev = classLevels,
...)
4: train.default(x, y, weights = w, ...)
3: train(x, y, weights = w, ...)
2: train.formula(couple ~ ., training.balanced, method = "nnet",
preProcess = "range", tuneGrid = nnetGrid, MaxNWts = 2200)
1: caret::train(couple ~ ., training.balanced, method = "nnet",
preProcess = "range", tuneGrid = nnetGrid, MaxNWts = 2200)
These errors are not easily reproducible (i.e. they happen sometimes, but not consistently) and only occur at the end of the run. The stdout on the cluster shows all tasks running and completed, so I am a bit flummoxed.
Has anyone encountered these errors? and if so understand the cause and even better a fix?
I imagine you've already solved this problem, but I ran into the same issue on my cluster consisting of linux and windows systems. I was running the server on ubuntu 14.04 and had noticed the warnings when starting the server service about having 'transparent huge pages' enabled in the linux kernel. I ignored that message and began running training exercises where most of the machines were maxed out with workers. I received the same error at the end of the run:
error calling combine function:
<simpleError: obj$state$numResults <= obj$state$numValues is not TRUE>
After a lot of head scratching and useless tinkering, I decided to address that warning by following the instructions here: http://ubuntuforums.org/showthread.php?t=2255151
Essentially, I installed hugeadm using:
sudo apt-get install hugeadm
Then disabled the transparent pages using:
hugeadm --thp-never
Note that this change will be undone on restart of the computer.
When I re-ran my training process it ran without any errors.
Hope that helps.
Cheers,
Eric
I am attempting to use the caret package in R for several nested cross-validation processes with user-defined performance metrics. I have had all kinds of problems, so I pulled back to see see if there were issues with a more out of the box use of caret and it seems I have run into one.
If I run the following:
install.packages("caret")
install.packages("gbm")
library(caret)
library(gbm)
data(GermanCredit)
GermanCredit$Class<-ifelse(GermanCredit$Class=='Bad',1,0)
gbmGrid <- expand.grid(.interaction.depth = 1,
.n.trees = 150,
.shrinkage = 0.1)
gbmMOD <- train(Class~., data=GermanCredit
,method = "gbm",
tuneGrid= gbmGrid,
distribution="bernoulli",
bag.fraction = 0.5,
train.fraction = 0.5,
n.minobsinnode = 10,
cv.folds = 1,
keep.data=TRUE,
verbose=TRUE
)
I get the error (or similar):
Error in { :
task 1 failed - "arguments imply differing number of rows: 619, 381"
with warnings:
1: In eval(expr, envir, enclos) :
model fit failed for Resample01: interaction.depth=1, n.trees=150, shrinkage=0.1
But, if I run just the gbm routine everything finishes fine.
gbm1 <- gbm(Class~., data=GermanCredit,
distribution="bernoulli",
n.trees=150, # number of trees
shrinkage=0.10,
interaction.depth=1,
bag.fraction = 0.5,
train.fraction = 0.5,
n.minobsinnode = 10,
cv.folds = 1,
keep.data=TRUE,
verbose=TRUE
)
There were two issues: passing cv.folds caused a problem. Also, you don't need to convert the outcome to a binary number; this causes train to think that it is a regression problem. The idea behind the train function is to smooth out the inconsistencies with the modeling functions, so we use factors for classification and numbers for regression.
Just for note - although this issue has been caused by the reason described in the answer, the error message (given below) may also occur with older version of caret and gbm. I encountered this error and after spending a lot of time trying to figure out what the issue was it turned out that I had to upgrade to the most recent version of caret (5.17-7) and gbm (2.1-0.1). These are the most recent version as of today on CRAN.
Error in { :
task 1 failed - "arguments imply differing number of rows: ...