I would like to use the gamlss package for fitting a model benefiting from more available distributions in that package. However, I am struggling to correctly specify my random effects or at least I think there is a mistake because if I compare the output of a lmer model with Gaussian distribution and the gamlss model with Gaussian distribution output differs. If comparing a lm model without the random effects and a gamlss model with Gaussian distribution and without random effects output is similar.
I unfortunately cannot share my data to reproduce it.
Here my code:
df <- subset.data.frame(GFW_food_agg, GFW_food_agg$fourC_area_perc < 200, select = c("ISO3", "Year", "Forest_loss_annual_perc_boxcox", "fourC_area_perc", "Pop_Dens_km2", "Pop_Growth_perc", "GDP_Capita_current_USD", "GDP_Capita_growth_perc",
"GDP_AgrForFis_percGDP", "Gini_2008_2018", "Arable_land_perc", "Forest_loss_annual_perc_previous_year", "Forest_extent_2000_perc"))
fourC <- lmer(Forest_loss_annual_perc_boxcox ~ fourC_area_perc + Pop_Dens_km2 + Pop_Growth_perc + GDP_Capita_current_USD +
GDP_Capita_growth_perc + GDP_AgrForFis_percGDP + Gini_2008_2018 + Arable_land_perc + Forest_extent_2000_perc + (1|ISO3) + (1|Year),
data = df)
summary(fourC)
resid_panel(fourC)
df <- subset.data.frame(GFW_food_agg, GFW_food_agg$fourC_area_perc < 200, select = c("ISO3", "Year", "Forest_loss_annual_perc_boxcox", "fourC_area_perc", "Pop_Dens_km2", "Pop_Growth_perc", "GDP_Capita_current_USD", "GDP_Capita_growth_perc",
"GDP_AgrForFis_percGDP", "Gini_2008_2018", "Arable_land_perc", "Forest_loss_annual_perc_previous_year", "Forest_extent_2000_perc"))
df <- na.omit(df)
df$ISO3 <- as.factor(df$ISO3)
df$Year <- as.factor(df$Year)
fourC <- gamlss(Forest_loss_annual_perc_boxcox ~ fourC_area_perc + Pop_Dens_km2 + Pop_Growth_perc + GDP_Capita_current_USD +
GDP_Capita_growth_perc + GDP_AgrForFis_percGDP + Gini_2008_2018 + Arable_land_perc + Forest_extent_2000_perc + random(ISO3) + random(Year),
data = df, family = NO, control = gamlss.control(n.cyc = 200))
summary(fourC)
plot(fourC)
How do the random effects need to be specified in gamlss to be similar to the random effects in lmer?
If I specify the random effects instead using
re(random = ~1|ISO3) + re(random = ~1|Year)
I get the following error:
Error in model.frame.default(formula = Forest_loss_annual_perc_boxcox ~ :
variable lengths differ (found for 're(random = ~1 | ISO3)')
I found the +re(random=~1|x) specification to work fairly well with my GAMLSS. Have you double check that the NA's are being removed from your dataset? Sometimes na.omit does not work properly.
Have a look at this thread that has the same error than yours, but in a GAM. You can try that code to remove your NA's
Error in model.frame.default: variable lengths differ
Related
I would like to ask for help with my project. My goal is to get ROC curve from existing logistic regression.
First of all, here is what I'm analyzing.
glm.fit <- glm(Severity_Binary ~ Side + State + Timezone + Temperature.F. + Wind_Chill.F. + Humidity... + Pressure.in. + Visibility.mi. + Wind_Direction + Wind_Speed.mph. + Precipitation.in. + Amenity + Bump + Crossing + Give_Way + Junction + No_Exit + Railway + Station + Stop + Traffic_Calming + Traffic_Signal + Sunrise_Sunset , data = train_data, family = binomial)
glm.probs <- predict(glm.fit,type = "response")
glm.probs = predict(glm.fit, newdata = test_data, type = "response")
glm.pred = ifelse(glm.probs > 0.5, "1", "0")
This part works fine, I am able to show a table of prediction and mean result. But here comes the problem for me, I'm using pROC library, but I am open to use anything else which you can help me with. I'm using test_data with approximately 975 rows, but variable proc has only 3 sensitivities/specificities values.
library(pROC)
proc <- roc(test_data$Severity_Binary,glm.probs)
test_data$sens <- proc$sensitivities[1:975]
test_data$spec <- proc$specificities[1:975]
ggplot(test_data, aes(x=spec, y=sens)) + geom_line()
Here´s what I have as a result:
With Warning message:
Removed 972 row(s) containing missing values (geom_path).
As I found out, proc has only 3 values as I said.
You can't (and shouldn't) assign the sensitivity and specificity to the data. They are summary data and exist in a different dimension than your data.
Specifically, these two lines are wrong and make no sense at all:
test_data$sens <- proc$sensitivities[1:975]
test_data$spec <- proc$specificities[1:975]
Instead you must either save them to a new data.frame, or use some of the existing functions like ggroc:
ggroc(proc)
If you consider what the ROC curve does, there is no reason to expect it to have the same dimensions as your dataframe. It provides summary statistics of your model performance (sensitivity, specificity) evaluated on your dataset for different thresholds in your prediction.
Usually you would expect some more nuance on the curve (more than the 3 datapoints at thresholds -Inf, 0.5, Inf). You can look at the distribution of your glm.probs - this ROC curve indicates that all predictions are either 0 or 1, with very little inbetween (hence only one threshold at 0.5 on your curve). [This could also mean that you unintentially used your binary glm.pred for calculating the ROC curve, and not glm.probs as shown in the question (?)]
This seems to be more an issue with your model than with your code - here an example from a random different dataset, using the same steps you took (glm(..., family = binomial, predict(, type = "response"). This produces a ROC curve with 333 steps for ~1300 datapoints.
PS: (Ingore the fact that this is evaluated on training data, the point is the code looks alright up to the point of generating the ROC curve)
m1 <- glm(survived ~ passengerClass + sex + age, data = dftitanic, family = binomial)
myroc <- roc(dftitanic$survived,predict(m1, dftitanic, type = "response"))
plot(myroc)
I would like to fit a nonlinear model just with the fixed structure specification using nlme R package.
model <- nlme(y ~ Asym/(1+exp((xmid-x)/scal)),
data = data,
fixed = list(Asym + xmid + scal ~ treatment))
#random = Asym ~ 1|subject)
However I am getting the following error:
Error in parse(text = paste("~", paste(nVal, collapse = "/"))) :
<text>:2:0: unexpected end of input
1: ~
^
Is there a way to circunvent this issue? Any advice is more than welcome.
I believe you want the gnls() function (also from the nlme package) with the params= argument rather than fixed=. Try this:
model <- gnls(y ~ Asym/(1+exp((xmid-x)/scal)),
data = data,
params = list(Asym + xmid + scal ~ treatment),
start= ...)
FWIW, if you're really fitting a logistic (and this isn't just a simplified example of what you want to do), fitting might be faster/more robust with the SSlogis() self-starting function in place of your explicit formula ...
I am using Afrobarometer survey data using 2 rounds of data for 10 countries. My DV is a binary 0-1 variable. I need to use logistic regression, fixed-effects, clustered standard errors (at country), and weighted survey data. A variable for the weights already exists in the dataframe.
I've been looking at help files for the following packages: clogit, glm, pglm, glm2, zelig, bife , etc. Typical errors include: can't add weights, can't do fixed effects, cant do either or etc.
#Glm
t3c1.fixed <- glm(formula = ethnic ~ elec_prox +
elec_comp + round + country, data=afb,
weights = afb$survey_weight,
index c("country", "round"),
family=binomial(link='logit'))
#clogit
t3c1.fixed2 <- clogit(formula = ethnic ~ elec_prox +
elec_comp + round + country, data=afb,
weights = afb$survey_weight,
method=c("within"))
#bife attempt
library(bife)
t3c1.fixed3 <- bife(ethnic ~ elec_prox + elec_comp + round +
country, model = logit,data=afb,
weights = afb$survey_weight,
bias_corr = "ana")
I either get error messages or the code doesn't include one of the conditions I need to include, so I can't use them. In Stata it appears this process is very simple, but in R it seems rather tedious. Any help would be appreciated!
I would check out the survey package which provides everything for which you are asking. The first step is to create the survey object, specify the survey weights and then you are off to the races.
library(survey)
my_survey <- svydesign(ids= ~1, strata = ~country, wts = ~wts, data = your_data)
# Then you can use the survey glm to do what you want via
svy_fit <- svy_glm(ethnic ~ elec_prox +
elec_comp + round + country, data = my_survey, family = binomial())
Or at least I would go down this path given you are using survey data.
Trying to fit a two-level imputation model with HMI (hierch. multiple imputation)...
The model I'm using is this (I want random intercept ONLY):
glmer(pica_yn ~ 1 + visit_c+visit_c2 + geo_child + hhloc + diar_c + hemo_c + (1|pid))
I keep getting this error:
Error in buildZ(rmodel.terms[r], data = data, nginverse =
names(ginverse)): object id not found
It seems as though HMI prefers the specified formula also has a random slope.
Has anyone fit a multilevel imputation model for a BINARY response?
Here is an example you can run that will get the same error:
data("sleepstudy", package="lme4")
sleepstudy[sample(1:nrow(sleepstudy), size = 20), "Reaction"] <- NA
sleep_formula<-Reaction ~ Days + (1|Subject)
hmi_imp <- hmi(data = sleepstudy, model_formula = sleep_formula, M = 5, maxit = 1)
I'm new to R and want to implement lasso on my data in order to feature selection according to the coefficient estimated by this algorithm. My data base is big and There are 40 predictors(continuous and categorical).when I apply lasso regression using glmnet package, all the coefficients that are estimated for each predictor in this algorithm are zero except the intercept, why this happen? Is the model over fitted? How can I fix it?The code I used for this section is:
#Transforming categorical variables:
xfactors <- model.matrix(Bill_TotalCharge ~addNA(P_AgeGroup) +
addNA(ADT_ConditionOnDischarge) + addNA(Provider_Profession) +
addNA(ADT_HospitalName) + addNA(ADT_Province) + addNA(ADT_City) +
addNA(DiagnosisValueGroup) + addNA(DiagnosisGroupLevel1) +
addNA(DiagnosisGroupLevel2) + addNA(Bill_Insurer) + addNA(Bill_InsurerType1)
+ addNA(Bill_InsurerType2) + addNA(Bill_InsurerBox) +
addNA(ADT_AdmissionType) + addNA(Bill_RecordType) + addNA(P_MaritalStatus) +
addNA(Gender) + addNA(MonthNumberOfYear) + addNA(CalenderYear) ,
na.action=na.exclude)[,-1]
#Creating matrix of combination of contniuous and categorical varriables
x <- as.matrix(data.frame(Bill_TotalBasicInsurance, Bill_TotalPatient
,Bill_TotalCost1,Bill_TotalCost2, Bill_TotalCost3 , Bill_TotalCost4 ,
Bill_TotalCost5 , Bill_TotalCost6 , Bill_TotalCost7 , Bill_TotalCost8
,Bill_TotalCost9 ,Bill_TotalCost10 ,Bill_TotalCost11 ,Bill_TotalCost12 ,
P_Age, xfactors))
#Running lasso
glmmod <- glmnet(x, y=Bill_TotalCharge, family="gaussian",alpha=1)
Then I want to use cv.glmnet function to determine the min_lambda with cross validation and unbelievably it returns a 6_digits number as a min lambda(lambda and subsequently alpha should be between zero and one).What is the problem and how can I fix it?The code I used for this reason is:
cv.glmmod <- cv.glmnet(x, y=Bill_TotalCharge, alpha=1)
best.lambda <- cv.glmmod$lambda.min
I appreciate any help greatly in advance.