R svyglm na.exclude predict and na padding - r
I'm to use predict with the svlgm function and I'm having trouble getting predict to pad out the resulting vector with NAs as I would expect (and indeed can achieve with a non-survey glm using na.exclude).
Running svyglm with na.exclude gives the following error
Warning message:
In model.matrix(glm.object) * resid(glm.object, "working") :
longer object length is not a multiple of shorter object length
Am I asking svyglm/predict to do something I shouldn't, or is this an error with the svyglm package? Is there a way of getting predict to produce a vector padded with NAs from a svyglm object?
Any advice/help greatly appreciated
library(survey)
y=c(0,0,0,0,0,1,1,1,1,1,0,0,0,0,0,1,1,1,1,1)
x1=c(1,0,1,1,1,NA,2,2,2,2,1,0,NA,1,0,0,0,2,2,2)
x2=c(10,21,33,55,40,30,26,84,NA,87,20,21,23,25,NA,60,76,84,71,87)
x3=runif(20)
foo=data.frame(y,x1,x2,x3)
m1=glm(y~x1+x2, family=binomial(logit),na.action=na.exclude)
predict(m1)
svy1 <-
svydesign(
ids=~0,
data = foo,
weights = ~x3
)
m2 <- svyglm(y ~ x1+x2, svy1,na.action=na.exclude)
predict(m2)
Related
How can I include both my categorical and numeric predictors in my elastic net model? r
As a note beforehand, I think I should mention that I am working with highly sensitive medical data that is protected by HIPAA. I cannot share real data with dput- it would be illegal to do so. That is why I made a fake dataset and explained my processes to help reproduce the error.
I have been trying to estimate an elastic net model in r using glmnet. However, I keep getting an error. I am not sure what is causing it. The error happens when I go to train the data. It sounds like it has something to do with the data type and matrix.
I have provided a sample dataset. Then I set the outcomes and certain predictors to be factors. After setting certain variables to be factors, I label them. Next, I create an object with the column names of the predictors I want to use. That object is pred.names.min. Then I partition the data into the training and test data frames. 65% in the training, 35% in the test. With the train control function, I specify a few things I want to have happen with the model- random paraments for lambda and alpha, as well as the leave one out method. I also specify that it is a classification model (categorical outcome). In the last step, I specify the training model. I write my code to tell it to use all of the predictor variables in the pred.names.min object for the trainingset data frame.
library(dplyr)
library(tidyverse)
library(glmnet),0,1,0
library(caret)
#creating sample dataset
df<-data.frame("BMIfactor"=c(1,2,3,2,3,1,2,1,3,2,1,3,1,1,3,2,3,2,1,2,1,3),
"age"=c(0,4,8,1,2,7,4,9,9,2,2,1,8,6,1,2,9,2,2,9,2,1),
"L_TartaricacidArea"=c(0,1,1,0,1,1,1,0,0,1,0,1,1,0,1,0,0,1,1,0,1,1),
"Hydroxymethyl_5_furancarboxylicacidArea_2"=
c(1,1,0,1,0,0,1,0,1,1,0,1,1,0,1,1,0,1,0,1,0,1),
"Anhydro_1.5_D_glucitolArea"=
c(8,5,8,6,2,9,2,8,9,4,2,0,4,8,1,2,7,4,9,9,2,2),
"LevoglucosanArea"=
c(6,2,9,2,8,6,1,8,2,1,2,8,5,8,6,2,9,2,8,9,4,2),
"HexadecanolArea_1"=
c(4,9,2,1,2,9,2,1,6,1,2,6,2,9,2,8,6,1,8,2,1,2),
"EthanolamineArea"=
c(6,4,9,2,1,2,4,6,1,8,2,4,9,2,1,2,9,2,1,6,1,2),
"OxoglutaricacidArea_2"=
c(4,7,8,2,5,2,7,6,9,2,4,6,4,9,2,1,2,4,6,1,8,2),
"AminopentanedioicacidArea_3"=
c(2,5,5,5,2,9,7,5,9,4,4,4,7,8,2,5,2,7,6,9,2,4),
"XylitolArea"=
c(6,8,3,5,1,9,9,6,6,3,7,2,5,5,5,2,9,7,5,9,4,4),
"DL_XyloseArea"=
c(6,9,5,7,2,7,0,1,6,6,3,6,8,3,5,1,9,9,6,6,3,7),
"ErythritolArea"=
c(6,7,4,7,9,2,5,5,8,9,1,6,9,5,7,2,7,0,1,6,6,3),
"hpresponse1"=
c(1,0,1,1,0,1,1,0,0,1,0,0,1,0,1,1,1,0,1,0,0,1),
"hpresponse2"=
c(1,0,1,0,0,1,1,1,0,1,0,1,0,1,1,0,1,0,1,0,0,1))
#setting variables as factors
df$hpresponse1<-as.factor(df$hpresponse1)
df$hpresponse2<-as.factor(df$hpresponse2)
df$BMIfactor<-as.factor(df$BMIfactor)
df$L_TartaricacidArea<- as.factor(df$L_TartaricacidArea)
df$Hydroxymethyl_5_furancarboxylicacidArea_2<-
as.factor(df$Hydroxymethyl_5_furancarboxylicacidArea_2)
#labeling factor levels
df$hpresponse1 <- factor(df$hpresponse1, labels = c("group1.2", "group3.4"))
df$hpresponse2 <- factor(df$hpresponse2, labels = c("group1.2.3", "group4"))
df$L_TartaricacidArea <- factor(df$L_TartaricacidArea, labels =c ("No",
"Yes"))
df$Hydroxymethyl_5_furancarboxylicacidArea_2 <-
factor(df$Hydroxymethyl_5_furancarboxylicacidArea_2, labels =c ("No",
"Yes"))
df$BMIfactor <- factor(df$BMIfactor, labels = c("<40", ">=40and<50",
">=50"))
#creating list of predictor names
pred.start.min <- which(colnames(df) == "BMIfactor"); pred.start.min
pred.stop.min <- which(colnames(df) == "ErythritolArea"); pred.stop.min
pred.names.min <- colnames(df)[pred.start.min:pred.stop.min]
#partition data into training and test (65%/35%)
set.seed(2)
n=floor(nrow(df)*0.65)
train_ind=sample(seq_len(nrow(df)), size = n)
trainingset=df[train_ind,]
testingset=df[-train_ind,]
#specifying that I want to use the leave one out cross-
#validation method and
use "random" as search for elasticnet
tcontrol <- trainControl(method = "LOOCV",
search="random",
classProbs = TRUE)
#training model
elastic_model1 <- train(as.matrix(trainingset[,
pred.names.min]),
trainingset$hpresponse1,
data = trainingset,
method = "glmnet",
trControl = tcontrol)
After I run the last chunk of code, I end up with this error:
Error in { :
task 1 failed - "error in evaluating the argument 'x' in selecting a
method for function 'as.matrix': object of invalid type "character" in
'matrix_as_dense()'"
In addition: There were 50 or more warnings (use warnings() to see the first
50)
I tried removing the "as.matrix" arguemtent:
elastic_model1 <- train((trainingset[, pred.names.min]),
trainingset$hpresponse1,
data = trainingset,
method = "glmnet",
trControl = tcontrol)
It still produces a similar error.
Error in { :
task 1 failed - "error in evaluating the argument 'x' in selecting a method
for function 'as.matrix': object of invalid type "character" in
'matrix_as_dense()'"
In addition: There were 50 or more warnings (use warnings() to see the first
50)
When I tried to make none of the predictors factors (but keep outcome as factor), this is the error I get:
Error: At least one of the class levels is not a valid R variable name; This
will cause errors when class probabilities are generated because the
variables names will be converted to X0, X1 . Please use factor levels that
can be used as valid R variable names (see ?make.names for help).
How can I fix this? How can I use my predictors (both the numeric and categorical ones) without producing an error?
glmnet does not handle factors well. The recommendation currently is to dummy code and re-code to numeric where possible: Using LASSO in R with categorical variables
error r: invalid subscript type "closure" in a simple regression
unfortunately i am a beginner in r. I d like to run a simple linear regression model in r with the comand lm, but every time i try the following error occurs: Error in xj[i] : invalid subscript type 'closure' The regression model ist just as follows: REG1 <- lm(flowpercent~ret+tna+fundage+number_shr_cl,data = reg, na.omit) #-flowpercent is a calculated variable: reg$flowpercent <- reg$flow_dollar/lag(reg$tna, n=1) #-fundage is also calculated: reg$fundage <- as.numeric(difftime(ref_date,reg$InceptionDate, units = "days")/365.25) ret, tna, number_shr_cl are variables from a database hopefully some can help me to solve my problem. Many thanks in advance.
Your third argument is na.omit. You probably saw someone writing something like na.action = na.omit. However, if you look up the help for lm by typing ?lm, you will see: Usage: lm(formula, data, subset, weights, na.action, ... # etc which tells you that the third argument to lm is subset. So, you are passing the object called na.omit to the subset argument, which lm tries to use to subset your data. Unfortunately, na.omit is an R function (aka a "closure"). Not surprisingly, R does not know how to use this function to subset your data. Hence the error.
using ksvm of kernlab package for predicting has an error
I use ksvm function to train the data, but in predicting I have an error,here is the code:
svmmodel4 <- ksvm(svm_train[,1]~., data=svm_train,kernel = "rbfdot",C=2.4,
kpar=list(sigma=.12),cross=5)
Warning message:
In .local(x, ...) : Variable(s) `' constant. Cannot scale data.
pred <- predict(svmmodel4, svm_test[,-1])
Error in eval(expr, envir, enclos) : object 'res_var' not found.
If I add the response variable, it works:
pred <- predict(svmmodel4, svm_test)
But if you add the response variable,how can it be "predict"? what is wrong with my code? Thanks for your help!
The complete code:
library(kernlab)
svmData <- read.csv("svmData.csv",header=T,stringsAsFactors = F)
svmData$res_var <- as.factor(svmData$res_var)
svm_train <- svmData1[1:2110,]
svm_test <- svmData1[2111:2814,]
svmmodel4 <- ksvm(svm_train[,1]~.,data = svm_train,kernel = "rbfdot",C=2.4,
kpar=list(sigma=.12),cross=5)
pred1 <- predict(svmmodel4,svm_test[,-1])
You can not remove your response column from your test dataset. You simply divide your data horizontally, meaning the response column must be in your training and testing datasets, or even validation dataset if you have one. your function pred <- predict(svmmodel4, svm_test) is working just fine, the predict function will take your data, knowing your factored column, and test the rest against the model. Your training and testing datasets must have the same number of columns, but the number of rows could be different.
Extracting predictions from a GAM model with splines and lagged predictors
I have some data and am trying to teach myself about utilize lagged predictors within regression models. I'm currently trying to generate predictions from a generalized additive model that uses splines to smooth the data and contains lags. Let's say I have the following data and have split the data into training and test samples. head(mtcars) Train <- sample(1:nrow(mtcars), ceiling(nrow(mtcars)*3/4), replace=FALSE) Great, let's train the gam model on the training set. f_gam <- gam(hp ~ s(qsec, bs="cr") + s(lag(disp, 1), bs="cr"), data=mtcars[Train,]) summary(f_gam) When I go to predict on the holdout sample, I get an error message. f_gam.pred <- predict(f_gam, mtcars[-Train,]); f_gam.pred Error in ExtractData(object, data, NULL) : 'names' attribute [1] must be the same length as the vector [0] Calls: predict ... predict.gam -> PredictMat -> Predict.matrix3 -> ExtractData Can anyone help diagnose the issue and help with a solution. I get that lag(__,1) leaves a data point as NA and that is likely the reason for the lengths being different. However, I don't have a solution to the problem.
I'm going to assume you're using gam() from the mgcv library. It appears that gam() doesn't like functions that are not defined in "base" in the s() terms. You can get around this by adding a column which include the transformed variable and then modeling using that variable. For example tmtcars <- transform(mtcars, ldisp=lag(disp,1)) Train <- sample(1:nrow(mtcars), ceiling(nrow(mtcars)*3/4), replace=FALSE) f_gam <- gam(hp ~ s(qsec, bs="cr") + s(ldisp, bs="cr"), data= tmtcars[Train,]) summary(f_gam) predict(f_gam, tmtcars[-Train,]) works without error. The problem appears to be coming from the mgcv:::get.var function. It tires to decode the terms with something like eval(parse(text = txt), data, enclos = NULL) and because they explicitly set the enclosure to NULL, variable and function names outside of base cannot be resolved. So because mean() is in the base package, this works eval(parse(text="mean(x)"), data.frame(x=1:4), enclos=NULL) # [1] 2.5 but because var() is defined in stats, this does not eval(parse(text="var(x)"), data.frame(x=1:4), enclos=NULL) # Error in eval(expr, envir, enclos) : could not find function "var" and lag(), like var() is defined in the stats package.
Predict function from Caret package give an Error
I am doing just a regular logistic regression using the caret package in R. I have a binomial response variable coded 1 or 0 that is called a SALES_FLAG and 140 numeric response variables that I used dummyVars function in R to transform to dummy variables. data <- dummyVars(~., data = data_2, fullRank=TRUE,sep="_",levelsOnly = FALSE ) dummies<-(predict(data, data_2)) model_data<- as.data.frame(dummies) This gives me a data frame to work with. All of the variables are numeric. Next I split into training and testing: trainIndex <- createDataPartition(model_data$SALE_FLAG, p = .80,list = FALSE) train <- model_data[ trainIndex,] test <- model_data[-trainIndex,] Time to train my model using the train function: model <- train(SALE_FLAG~. data=train,method = "glm") Everything runs nice and I get a model. But when I run the predict function it does not give me what I need: predict(model, newdata =test,type="prob") and I get an ERROR: Error in dimnames(out)[[2]] <- modelFit$obsLevels : length of 'dimnames' [2] not equal to array extent On the other hand when I replace "prob" with "raw" for type inside of the predict function I get prediction but I need probabilities so I can code them into binary variable given my threshold. Not sure why this happens. I did the same thing without using the caret package and it worked how it should: model2 <- glm(SALE_FLAG ~ ., family = binomial(logit), data = train) predict(model2, newdata =test, type="response") I spend some time looking at this but not sure what is going on and it seems very weird to me. I have tried many variations of the train function meaning I didn't use the formula and used X and Y. I used method = 'bayesglm' as well to check and id gave me the same error. I hope someone can help me out. I don't need to use it since the train function to get what I need but caret package is a good package with lots of tools and I would like to be able to figure this out.
Show us str(train) and str(test). I suspect the outcome variable is numeric, which makes train think that you are doing regression. That should also be apparent from printing model. Make it a factor if you want to do classification. Max