The dataset which I am using here is unbalanced, but I balanced it manually like this by removing the multiple observations for same ID (this is a characteristic of my data as a single household later split to different ones). T is 2 here.
dataset %>% group_by(ID) %>% summarise(N =n()) %>% filter(N> 2 | N < 2)
Then I removed these rogue observations.So now the panel is balanced.I converted them to pdata afterwards
dataset <-plm.data(dataset, 30462)
And when I run is.pbalanced, it shows TRUE. But the problem is when I run the regression
plm(DEP~ VAR1 + VAR2, data= dataset, model= "within")
The summary shows this
Unbalanced Panel: n=20236, T=1-2, N=34920
I don't understand what I am missing here. Any suggestions will be greatly appreciated.
Related
In the multi linear regression lm(FE_FCE2 ~ Trial + .x, data = DF_FCE3) there is one fixed variable (trial) and many x variables. I am analysing each x variable against FE_FCE2 with trial as fixed effect. I than use the boom package for the many regressions and plot the results in one table. I have obtained the results for the regression results. However cannot add the data from ANOVA Table into the Broom packages with map function.
Is it possible? And Yes How?
I have used the following formula to obtain Data from Results from Regression:
DF_FCE3 %>%
select(-FE_FCE2, -Trial) %>% # exclude outcome, leave only predictors
map( ~lm(FE_FCE2 ~ Trial + .x, data = DF_FCE3)) %>%
map(summary) %>%
map_df(glance) %>%
round(3) -> rsme
However I would like to obtain the P-Value (**4.26e-08 *****) from the ANOVA Table of Trial.
To
see if Trial had a significant influence on the x variable.
**$x1
Analysis of Variance Table
**Response: FE_FCE2
Df Sum Sq Mean Sq F value Pr(>F)
Trial 3 0.84601 0.282002 15.0653 **4.26e-08 *****
.x 1 0.00716 0.007161 0.3826 0.5377
Residuals 95 1.77827 0.018719**
---**
Is it possible to use the broom package with map function to obtain a table which contains all the many p values of the anova regressions?
Like this (using mpg)?
This returns a dataframe with the original columns and one row containing the p-value except for the outcome and target (hwy and cyl in thisexample, FE_FCE2 and Trial in your case).
mpg %>%
select(-hwy, -cyl) %>% # exclude outcome, leave only predictors
map( ~lm(hwy ~ cyl + .x, data = mpg)) %>%
map(anova) %>%
map(broom::tidy) %>%
map_df(~.$p.value[1])
I am trying a problem what i found in redit and was experimenting how to do that using mtcars data set
This was the problem:
He is having list that looks like this: https://gyazo.com/0637f2226d8f53db4c90716bd3fb698c with 150 different "selskapsid".
He want to do a linear regression with "Return12" as the dependent variable and "SROE", "MktCap", and "y" and independent variables for each "Selskapsid". (Basically a row by row regression each row or for each id even the id got repeated i want separate model.)
I have read the comments in that didn't find any great solution so i was trying using dplyr and packages what I am bit comfort but the issue I was getting is cyl values are in factors so when I am trying to build the model cyl value is not repeating.
Does anyone know a simple loop to achieve this? I want to do training and testing in the same loop I wasn't getting training results also properly.
Using this libraries I was doing this:
library(tidyverse)
library(broom)
mtcars %>%
nest(-cyl) %>%
mutate(fit <-map(data, ~ lm(mpg ~ hp + wt + disp, data = .)),
results = map(fit, augment)) %>%
unnest(results)
I run the following code to generate regression models
library (dplyr)
fitted_models <- df %>%
group_by(sic, fyear) %>%
do (model = lm (TACCdTA ~ Inverse_TA + DeL_RevRec + PPEdTA , data = .))
Then to get the coefficients for each sic and fyear, I run the following code
library(broom)
fitted_models %>% tidy(model)
I got coefficients for each sic for each fyear.
Now my question is - under each sic for each fyear, there are many observations (e.g., 1000 observations) - how can I calculate the fitted value for each observation under each sic and fyear by using the coefficients that the model(s) generated above.
Another small question - for my first code in which I run the model, how can I ensure that the model(s) is(are) run only for the cases in which each sic and fyear combination (sic-fyear) has at least 10 observations.
I have a big dataset that I want to partition based on the values of a particular variable (in my case lifetime), and then run logistic regression on each partition. Following the answer of #tchakravarty in Fitting several regression models with dplyr I wrote the following code:
lifetimemodels = data %>% group_by(lifetime) %>% sample_frac(0.7)%>%
do(lifeModel = glm(churn ~., x= TRUE, family=binomial(link='logit'), data = .))
My question now is how I can use the resulting logistic models on computing the AUC on the rest of the data (the 0.3 fraction that was not chosen) which should again be grouped by lifetime?
Thanks a lot in advance!
You could adapt your dplyr approach to use the tidyr and purrr framework. You look at grouping/nesting, and the mutate and map functions to create list frames to store pieces of your workflow.
The test/training split you are looking for is part of modelr a package built to assist modelling within the purrr framework. Specifically the cross_vmc and cross_vkfold functions.
A toy example using mtcars (just to illustrate the framework).
library(dplyr)
library(tidyr)
library(purrr)
library(modelr)
analysis <- mtcars %>%
nest(-cyl) %>%
unnest(map(data, ~crossv_mc(.x, 1, test = 0.3))) %>%
mutate(model = map(train, ~lm(mpg ~ wt, data = .x))) %>%
mutate(pred = map2(model, train, predict)) %>%
mutate(error = map2_dbl(model, test, rmse))
This:
takes mtcars
nest into a list frame called data by cyl
Separate each data into a training set by mapping crossv_mc to each element, then using unnest to make the test and train list columns.
Map the lm model to each train, store that in model
Map the predict function to model and train and store in pred
Map the rmse function to model and test sets and store in error.
There are probably users out there more familiar than me with the workflow, so please correct/elaborate.
I have 100 replicates of coxph model fitted in loop. I am trying to extract out log-rank score test result with p-values for each replicate in a data frame or list. I am using the following. But, it gives me only log rank score, not p-value. Any help will be very appreciated.
I can share dataset, but am not sure how to attach here.
thanks,
Krina
Repl_List <- unique(dat3$Repl)
doLogRank = function(sel_name) {
dum <- dat3[dat3$Repl == sel_name,]
reg <- with(dum, coxph(Surv(TIME_day, STATUS) ~ Treatment, ties = "breslow"))
LogRank <- with(reg, reg$score)
}
LogRank <- t(as.data.frame(lapply(Repl_List, doLogRank)))
Here is a mock example that I took from the help page of the coxph function. I just replicated the dataset 100 times to create your scenario. I highly recommend to start using the tidyverse packages to do such work. broom is a great addition along with dplyr and tidyr.
library(survival)
library(tidyverse)
library(broom)
test <- data.frame(time=c(4,3,1,1,2,2,3),
status=c(1,1,1,0,1,1,0),
x=c(0,2,1,1,1,0,0),
sex=c(0,0,0,0,1,1,1))
Below I am replicating the dataset 100 times using the replicate function.
r <- replicate(test,n = 100,simplify = FALSE) %>% bind_rows %>%
mutate(rep = rep(seq(1,100,1),each=7))
I setup the cox model as a small function that I can them pass on to each replicate of the dataframe.
cxph_mod <- function(df) {
coxph(Surv(time, status) ~ x + strata(sex), df)
}
Below, is the step by step process of fitting the model and extracting the values.
tidyr::nest the dataframe
purrr::map the model into each nest
nest is function in library(tidyr)
map is a function similar to lapply in library(purrr)
nested <- r %>%
group_by(rep) %>%
nest %>%
mutate(model = data %>% map(cxph_mod))
look into the first rep to see the coxph output. You will see the model object stored in the cells of the dataframe allowing easier access.
nested %>% filter(rep==1)
With each model object, now use broom to get the parameter estimates and the prediction from the model into the nested dataset
nested <- nested %>%
mutate(
ests = model %>% map(broom::tidy)
)
tidyr::unnest to view your predictions for fitting each resampled dataset
ests <- unnest(nested,ests,.drop=TRUE) %>% dplyr::select(rep,estimate:conf.high)
In this case since I am repeating the same dataset 100 times, the pvalue will be the same, but in your case you will have 100 different datasets and hence 100 different p.values.
ggplot(data=ests,aes(y=p.value,x=rep))+geom_point()
Vijay