I have asked about this already on stats.exchange (original question), now I re-posted the same content here - hoping to get help from a wider population.
I would like to know the way to exclude all the unwanted pairs from the output generated from two-way ANOVA, so when there shows a significant result from summary(aov()), the post-hoc test won't give me any comparisons I don't want. Details as follows:
I have datTable contain proportion data under two factor site (four levels: A, B, C, D) and treatment(two levels: control and treated). Specifically, I want to do a pair-wise test among all the site under each same treatment (e.g. control-A VS. control-B, control-A VS.control-C, treated-A VS.treated-C, etc.), while excludes comparisons between different sites and different treatments(e.g., pairs such as control-A VS. treated-B, control-B VS. treated-C).
The data looks like this:
> datTable
site treatment proportion
A control 0.5000000
A control 0.4444444
A treated 0.1000000
A treated 0.4000000
B control 0.4444444
B control 0.4782609
B treated 0.0500000
B treated 0.3000000
C control 0.3214286
C control 0.4705882
C treated 0.1200000
C treated 0.4000000
D control 0.3928571
D control 0.4782609
D treated 0.4000000
D treated 0.4100000
I did a two-way ANOVA (also not sure whether to use within subject site/treatment or between subject site*treatment...), and summarised the results.
m1 <- aov(proportion~site*treatment,data=datTable) # Or should I use 'site/treatment'?
Then my summary(m1) gave me the following:
> summary(m1)
Df Sum Sq Mean Sq F value Pr(>F)
site 3 0.02548 0.00849 0.513 0.6845
treatment 1 0.11395 0.11395 6.886 0.0305 *
site:treatment 3 0.03686 0.01229 0.742 0.5561
Residuals 8 0.13239 0.01655
Next step is to use TukeyHSD post-hoc test to see actually which pair caused the * significance in site factor.
> TukeyHSD(m1)
Tukey multiple comparisons of means
95% family-wise confidence level
Fit: aov(formula = proportion ~ site * treatment, data = datTable)
$site
diff lwr upr p adj
B-A -0.042934783 -0.3342280 0.2483585 0.9631797
C-A -0.033106909 -0.3244002 0.2581863 0.9823452
D-A 0.059168392 -0.2321249 0.3504616 0.9124774
C-B 0.009827873 -0.2814654 0.3011211 0.9995090
D-B 0.102103175 -0.1891901 0.3933964 0.6869754
D-C 0.092275301 -0.1990179 0.3835685 0.7461309
$treatment
diff lwr upr p adj
treated-control -0.1687856 -0.3171079 -0.02046328 0.0304535
$`site:treatment`
diff lwr upr p adj
B:control-A:control -0.010869565 -0.5199109 0.4981718 1.0000000
C:control-A:control -0.076213819 -0.5852551 0.4328275 0.9979611
D:control-A:control -0.036663216 -0.5457045 0.4723781 0.9999828
A:treated-A:control -0.222222222 -0.7312635 0.2868191 0.6749021
B:treated-A:control -0.297222222 -0.8062635 0.2118191 0.3863364 # Not wanted
C:treated-A:control -0.212222222 -0.7212635 0.2968191 0.7154690 # Not wanted
D:treated-A:control -0.067222222 -0.5762635 0.4418191 0.9990671 # Not wanted
C:control-B:control -0.065344254 -0.5743856 0.4436971 0.9992203
D:control-B:control -0.025793651 -0.5348350 0.4832477 0.9999985
A:treated-B:control -0.211352657 -0.7203940 0.2976887 0.7189552 # Not wanted
B:treated-B:control -0.286352657 -0.7953940 0.2226887 0.4233804 # Not wanted
C:treated-B:control -0.201352657 -0.7103940 0.3076887 0.7583437 # Not wanted
D:treated-B:control -0.056352657 -0.5653940 0.4526887 0.9996991
D:control-C:control 0.039550603 -0.4694907 0.5485919 0.9999713
A:treated-C:control -0.146008403 -0.6550497 0.3630329 0.9304819 # Not wanted
B:treated-C:control -0.221008403 -0.7300497 0.2880329 0.6798628 # Not wanted
C:treated-C:control -0.136008403 -0.6450497 0.3730329 0.9499131
D:treated-C:control 0.008991597 -0.5000497 0.5180329 1.0000000 # Not wanted
A:treated-D:control -0.185559006 -0.6946003 0.3234823 0.8168230 # Not wanted
B:treated-D:control -0.260559006 -0.7696003 0.2484823 0.5194129 # Not wanted
C:treated-D:control -0.175559006 -0.6846003 0.3334823 0.8505865 # Not wanted
D:treated-D:control -0.030559006 -0.5396003 0.4784823 0.9999950
B:treated-A:treated -0.075000000 -0.5840413 0.4340413 0.9981528
C:treated-A:treated 0.010000000 -0.4990413 0.5190413 1.0000000
D:treated-A:treated 0.155000000 -0.3540413 0.6640413 0.9096378
C:treated-B:treated 0.085000000 -0.4240413 0.5940413 0.9960560
D:treated-B:treated 0.230000000 -0.2790413 0.7390413 0.6429921
D:treated-C:treated 0.145000000 -0.3640413 0.6540413 0.9326207
However, there are some pairs I don't want to be included in the two-way ANOVA which I preformed, specified as # not wanted.
Is there any way that I can tweak the aov or TukeyHSD function to exclude those possibilities ('not wanted' ones) I listed above? I could easily select the significant entires that I am interested (with *) from the long list produced from TukeyHSD. But I don't want my result from anova to be biased by those! (It happens in the real data that the significance actually caused by those unwanted pairs!)
NB: You might have noticed that the site:treatment post-hoc tests doesn't show any significance, this is because I only selected a small sample from the original data.
If you mean to exclude those comparisons completely from the calculations, Tukey's test works by doing pairwise comparisons for all combinations of conditions. It doesn't make sense to "exclude" any pairs.
If you mean you want to exclude the unwanted comparisons from showing in your final results then yes, it is possible. The result of TukeyHSD is simply a list and site:treatment is simply a matrix which you can manipulate as you like.
lst <- TukeyHSD(m1)
lst[['site:treatment']] <- lst[['site:treatment']][-c(5,6,7,10,11,12,15,16,18,19,20,21),]
Related
I have a dataset with several variables that looks like this:
Competitor Disturbance Group MT CVt
1 M P A 17.416667 63.39274
2 M P A 11.055556 91.32450
3 M C N 13.928571 78.11438
4 B C N 13.500000 61.20542
5 B T E 12.700000 48.11819
6 B T E 27.250000 63.44356
I've made a GLMM (mMT1) with 3 predictors (Competitor, Disturbance and Group), one response (MT) and one random factor (Species, not shown in example dataset).
After fitting and checking the model, I calculated ls means with the package emmeans:
ls_MT <- emmeans(mMT1, pairwise~Disturbance*Competitor*Group, type="response")
And performed a post-hoc test:
post_MT <- emmeans(mMT1, transform="response", component="cond",list(~Disturbance|Competitor|Group,~Competitor|Group|Disturbance,~Group|Disturbance|Competitor))
pairs(post_MT)
Finally I produced a bar chart with ggplot2, based on the ls means and se
ggplot(ls_MT, aes(x=Disturbance, fill=Competitor, y=response))+
geom_bar(stat="identity",position=position_dodge())+
facet_grid(cols=vars(Group))+labs(y = "log10(MT)")+
scale_color_manual(values=c("#2ca02c","#d62728"))+
geom_errorbar(aes(ymin=ls_MT$lower, ymax=ls_MT$upper), width=.2,
position=position_dodge(.9))+
theme_light()
Which produces a plot like this:
At this point I'm struggling with 2 things:
Is there a way to have the bar plots side by side without the faceting by group? It looks ok but I'd prefer them to be one single plot rather than 3 side by side
How can I annotate on the graph the comparisons from the post-hoc test? I've seen that several functions allow to perform separate tests (e.g. stat_compare_means) and plot those results, but I can't find a solution for Tukey's post-hoc test. Another solution could be to add
geom_signif(map_signif_level = c(" * * * "=0.001, " * * "=0.01, "*"=0.05), comparisons = list(c("P","C"),c("P","T"),c("C","T"))
but that pools together the B and M (Competitor) values of each Disturbance (P, T, C). In this case I'd actually like to compare the B and M within each of P, T and C. Or even better, how can I specify between which bars I'd like to show the results of the post-hoc? For example I'd like to compare A-T-B with E-T-B, etc.
This is an example modified from https://rcompanion.org/rcompanion/b_05.html. I have a 4 by 2 matrix, with cell values less than 1. I would like to compare pairewise differences between supplement. I would like to show that Selenium is different when compared with Vitamin E and Selenium+E as shown. The current code shows all 1's for the p-values.
I would like to modify the current code to perform a test that accounts for the differences when the percentages are less than 1.
Input =("
Supplement No.cancer Cancer
'Selenium' 0.894 0.776
'Vitamin E' 0.973 0.916
'Selenium+E' 0.988 0.936
'Placebo' 0.967 0.847
")
Matriz = as.matrix(read.table(textConnection(Input),
header=TRUE,
row.names=1))
Matriz
pairwiseNominalIndependence(Matriz,
fisher = FALSE,
gtest = FALSE,
chisq = TRUE,
method = "fdr")
You can't use a chi-square test of association unless you have a table of (whole) counts. But some effect size statistics that are often applied to tables of counts might be applicable. For example, you could look at the odds ratio across pairwise rows.
I run a statistical test in R (running in RStudio). I save the result using a variable name. I want to extract one part of the result. How do I do this?
Here is an example with the R code at the end.
I set up an experiment with four treatments, and gather data. I next run ANOVA and perform a Tukey HSD test. The result is stored in a variable called "posthoc."
I look and note that posthoc is a list of 1. In RStudio I see a little blue arrow to the left of the name, and clicking on that gives more information. I am not sure how to interpret it in a way that I can use to answer my own question.
I can print(posthoc) and I get the following.
# Tukey multiple comparisons of means
# 95% family-wise confidence level
#
#Fit: aov(formula = Expt1$Treat1 ~ Expt1$Trt)
#
#$`Expt1$Trt`
# diff lwr upr p adj
#B-A 6.523841 2.664755 10.38292569 0.0001372
#C-A 18.584160 14.725075 22.44324507 0.0000000
#D-A 2.643719 -1.215367 6.50280370 0.2854076
#C-B 12.060319 8.201234 15.91940456 0.0000000
#D-B -3.880122 -7.739207 -0.02103681 0.0482260
#D-C -15.940441 -19.799527 -12.08135619 0.0000000
I can also type class(posthoc) and I get this:
[1] "TukeyHSD" "multicomp"
In this case, what I need are all the p-values in a new variable. The general problem is that R gives me output and I need to be able to figure out how to extract specific elements of that output. I might be using aov, lm, nlme, or something else.
Mean1=3.2
Sd1=3.2
Mean2=9.4
Sd2=2.4
Mean3=21.4
Sd3=6.4
Mean4=3.9
Sd4=10.7
Size1=30
Treat1=rnorm(Size1,mean=Mean1, sd=Sd1)
Trt="A"
Treat1M <- data.frame(Treat1, Trt)
Treat1=rnorm(Size1,mean=Mean2, sd=Sd2)
Trt="B"
Treat2M <- data.frame(Treat1, Trt)
Treat1=rnorm(Size1,mean=Mean3, sd=Sd3)
Trt="C"
Treat3M <- data.frame(Treat1, Trt)
Treat1=rnorm(Size1,mean=Mean4, sd=Sd4)
Trt="D"
Treat4M <- data.frame(Treat1, Trt)
Expt1=rbind(Treat1M, Treat2M, Treat3M, Treat4M)
Expt1R<-aov(Expt1$Treat1 ~ Expt1$Trt)
posthoc <-TukeyHSD(x=Expt1R, 'Expt1$Trt', conf.level=.95)
simply use subsetting of your posthoc variable.
posthoc$`Expt1$Trt`[,4]
or you can try the broom package.
library(broom)
res <- tidy(posthoc)
res
term comparison estimate conf.low conf.high adj.p.value
1 Expt1$Trt B-A 5.904138 1.3639293 10.444346 5.223255e-03
2 Expt1$Trt C-A 16.886340 12.3461316 21.426548 3.919087e-14
3 Expt1$Trt D-A 4.283597 -0.2566111 8.823805 7.189220e-02
4 Expt1$Trt C-B 10.982202 6.4419940 15.522410 3.226398e-08
5 Expt1$Trt D-B -1.620540 -6.1607487 2.919668 7.886097e-01
6 Expt1$Trt D-C -12.602743 -17.1429509 -8.062534 3.235351e-10
The output of the tidy function is a data.frame. Thus, you can access the p-values using res$adj.p.value.
class(res)
[1] "data.frame"
Jimbou has already provided an excellent solution. I would also go for broom especially when doing plots with ggplot2. I would just like to expound on the comment by John Coleman.
You can inspect the object by using str. In case of posthoc,
str(posthoc)
gives
List of 1
$ Expt1$Trt: num [1:6, 1:4] 6.46 18.19 -0.76 11.74 -7.22 ...
Typing
posthoc$`Expt1$Trt`
gives
diff lwr upr p adj
B-A 6.4562 2.130 10.782 9.530e-04
C-A 18.1922 13.866 22.519 2.454e-14
D-A -0.7598 -5.086 3.566 9.680e-01
C-B 11.7360 7.410 16.062 7.427e-10
D-B -7.2160 -11.542 -2.890 1.725e-04
D-C -18.9521 -23.278 -14.626 1.588e-14
So you can access the fourth column by typing
posthoc$`Expt1$Trt`[,4]
or
posthoc$`Expt1$Trt`[,'p adj']
For some objects like Expt1R, the output of str() can sometimes be overwhelming. Using names() to look at the different objects inside it is also helpful.
names(Expt1R)
[1] "coefficients" "residuals" "effects" "rank" "fitted.values" "assign"
[7] "qr" "df.residual" "contrasts" "xlevels" "call" "terms"
[13] "model"
So
Expt1R$df.residual
will give you the degrees of freedom of the residual.
I am trying to analyse the significant differences between different car company performance values across different countries. I am using ANOVA to do this.
Running ANOVA on my real dataset (30 countries, 1000 car companies and 90000 measurement scores) gave every car a zero p-value.
Confused by this, I created a reproducible example (below) with 30 groups, 3 car companies, 90000 random scores. Purposely, I kept a score of 1 for the Benz company where you shouldn't see any difference between countries. After running anova, I see a pvalue of 0.46 instead of 1.
Does any one know why is this ?
Reproducible example
set.seed(100000)
qqq <- 90000
df = data.frame(id = c(1:90000), country = c(rep("usa",3000), rep("usb",3000), rep("usc",3000), rep("usd",3000), rep("use",3000), rep("usf",3000), rep("usg",3000), rep("ush",3000), rep("usi",3000), rep("usj",3000), rep("usk",3000), rep("usl",3000), rep("usm",3000), rep("usn",3000), rep("uso",3000), rep("usp",3000), rep("usq",3000), rep("usr",3000), rep("uss",3000), rep("ust",3000), rep("usu",3000), rep("usv",3000), rep("usw",3000), rep("usx",3000), rep("usy",3000), rep("usz",3000), rep("usaa",3000), rep("usab",3000), rep("usac",3000), rep("usad",3000)), tesla=runif(90000), bmw=runif(90000), benz=rep(1, each=qqq))
str(df)
out<-data.frame()
for(j in 3:ncol(df)){
amod2 <- aov(df[,j]~df$country)
out[(j-2),1]<-colnames(df)[j]
out[(j-2),2]<-summary(amod2, test = adjusted("bonferroni"))[[1]][[1,"Pr(>F)"]]
}
colnames(out)<-c("cars","pvalue")
write.table(out,"df.output")
df.output
"cars" "pvalue"
"1" "tesla" 0.245931589754359
"2" "bmw" 0.382730335188437
"3" "benz" 0.465083026215268
With respect to the "benz" p-value in your reproducible example: an ANOVA analysis requires positive variance (i.e., non-constant data). If you violate this assumption, the model is degenerate. Technically, the p-value is based on an F-statistic whose value is a normalized ratio of the variance attributable to the "country" effect (for "benz" in your example, zero) divided by the total variance (for "benz" in your example, zero), so your F-statistic has "value" 0/0 or NaN.
Because of the approach R takes to calculating the F-statistic (using a QR matrix decomposition to improve numerical stability in "nearly" degenerate cases), it calculates an F-statistic equal to 1 (w/ 29 and 89970 degrees of freedom). This gives a p-value of:
> pf(1, 29, 89970, lower=FALSE)
[1] 0.465083
>
but it is, of course, largely meaningless.
With respect to your original problem, with large datasets relatively small effects will yield very small p-values. For example, if you add the following after your df definition above to introduce a difference in country usa:
df = within(df, {
o = country=="usa"
tesla[o] = tesla[o] + .1
bmw[o] = bmw[o] + .1
benz[o] = benz[o] + .1
rm(o)
})
you will find that out looks like this:
> out
cars pvalue
1 tesla 9.922166e-74
2 bmw 5.143542e-74
3 benz 0.000000e+00
>
Is this what you're seeing, or are you seeing all of them exactly zero?
I want to run MLR on my data using lm function in R. However, I am using data splitting cross validation method to access the reliability of the model. I intend using "sample" function to randomly split the data into the calibration and validation datasets by 80:20 ratio. This I want to repeat in say 100 times. Without setting a seed I believe the model from the different samplings will differ. I came across the function in previous post here and it solves the first part;
lst <- lapply(1:100, function(repetition) {
mod <- lm(...)
# Replace this with the code you need to train your model
return(mod)
})
save(lst, file="myfile.RData")
The concern now is how do I validate each of these 100 models and obtain reliability test statistics like RSME, ME, Rsquare for each of the models and hopefully obtain the confidence interval.
If I can get an output in the form of dataframe containing the predicted values for all the 100 models then I should proceed from there.
Any help please?
Thanks
To quickly recap your question: it seems that you want to fit an MLR model to a large training set and then use this model to make predictions on the remaining validation set. You want to repeat this process 100 times and afterwards you want to be able to analyze the characteristics and predictions of the individual models.
To accomplisch this you could just store temporary modelinformation in a datastructure during the modelgeneration and prediction process. You can then re-obtain and process all the information afterwards. You did not provide your own dataset in the description, so I will use one of R's built in datasets in order to demonstrate how this might work:
> library(car)
> Prestige <- Prestige[,c("prestige","education","income","women")]
> Prestige[,c("income")] <- log2(Prestige[,c("income")])
> head(Prestige,n=5)
prestige education income women
gov.administrators 68.8 13.11 -0.09620212 11.16
general.managers 69.1 12.26 -0.04955335 4.02
accountants 63.4 12.77 -0.11643822 15.70
purchasing.officers 56.8 11.42 -0.11972061 9.11
chemists 73.5 14.62 -0.12368966 11.68
We start by initializing some variables first. Let's say you want to create 100 models and use 80% of your data for training purposes:
nrIterations=100
totalSize <- nrow(Prestige)
trainingSize <- floor(0.80*totalSize)
We also want to create the datastructure that will be used to hold the intermediate modelinformation. R is quite a generic high level language in this regard, so we will just create a list of lists. This means that every listentry can by itself again hold another list of information. This gives us the flexibility to add whatever we need:
trainTestTuple <- list(mode="list",length=nrIterations)
We are now ready to create our models and predictions. During every loopiteration a different random trainingsubset is created while using the remaining data for testing purposes. Next, we fit our model to the trainingdata and we then use this obtained model to make predictions on the testdata. Note that we explicitly use the independent variables in order to predict the dependent variable:
for(i in 1:nrIterations)
{
trainIndices <- sample(seq_len(totalSize),size = trainingSize)
trainSet <- Prestige[trainIndices,]
testSet <- Prestige[-trainIndices,]
trainingFit <- lm(prestige ~ education + income + women, data=trainSet)
# Perform predictions on the testdata
testingForecast <- predict(trainingFit,newdata=data.frame(education=testSet$education,income=testSet$income,women=testSet$women),interval="confidence",level=0.95)
# Do whatever else you want to do (compare with actual values, calculate other stuff/metrics ...)
# ...
# add your training and testData to a tuple and add it to a list
tuple <- list(trainingFit,testingForecast) # Add whatever else you need ..
trainTestTuple[[i]] <- tuple # Add this list to the "list of lists"
}
Now, the relevant part: At the end of the iteration we put both the fitted model and the out of sample prediction results in a list. This list contains all the intermediate information that we want to save for the current iteration. We finish by putting this list in our list of lists.
Now that we are done with the modeling, we still have access to all the information we need and we can process and analyze it any way we want. We will take a look at the modeling and prediction results of model 50. First, we extract both the model and the prediction results from the list of lists:
> tuple_50 <- trainTestTuple[[50]]
> trainingFit_50 <- tuple_50[[1]]
> testingForecast_50 <- tuple_50[[2]]
We take a look at the model summary:
> summary(trainingFit_50)
Call:
lm(formula = prestige ~ education + log2(income) + women, data = trainSet)
Residuals:
Min 1Q Median 3Q Max
-15.9552 -4.6461 0.5016 4.3196 18.4882
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -287.96143 70.39697 -4.091 0.000105 ***
education 4.23426 0.43418 9.752 4.3e-15 ***
log2(income) 155.16246 38.94176 3.984 0.000152 ***
women 0.02506 0.03942 0.636 0.526875
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 7.308 on 77 degrees of freedom
Multiple R-squared: 0.8072, Adjusted R-squared: 0.7997
F-statistic: 107.5 on 3 and 77 DF, p-value: < 2.2e-16
We then explicitly obtain the model R-squared and RMSE:
> summary(trainingFit_50)$r.squared
[1] 0.8072008
> summary(trainingFit_50)$sigma
[1] 7.308057
We take a look at the out of sample forecasts:
> testingForecast_50
fit lwr upr
1 67.38159 63.848326 70.91485
2 74.10724 70.075823 78.13865
3 64.15322 61.284077 67.02236
4 79.61595 75.513602 83.71830
5 63.88237 60.078095 67.68664
6 71.76869 68.388457 75.14893
7 60.99983 57.052282 64.94738
8 82.84507 78.145035 87.54510
9 72.25896 68.874070 75.64384
10 49.19994 45.033546 53.36633
11 48.00888 46.134464 49.88329
12 20.14195 8.196699 32.08720
13 33.76505 27.439318 40.09079
14 24.31853 18.058742 30.57832
15 40.79585 38.329835 43.26187
16 40.35038 37.970858 42.72990
17 38.38186 35.818814 40.94491
18 40.09030 37.739428 42.44117
19 35.81084 33.139461 38.48223
20 43.43717 40.799715 46.07463
21 29.73700 26.317428 33.15657
And finally, we obtain some more detailed results about the 2nd forecasted value and the corresponding confidence intervals:
> testingPredicted_2ndprediction <- testingForecast_50[2,1]
> testingLowerConfidence_2ndprediction <- testingForecast_50[2,2]
> testingUpperConfidence_2ndprediction <- testingForecast_50[2,3]
EDIT
After rereading, it occured to me that you are obviously not splitting up the the same exact dataset each time. You are using completely different partitions of data during each iteration and they should be split up in a 80/20 fashion. However, the same solution can still be applied with minor modifications.
Also: For cross validation purposes you should probably take a look at cv.lm()
Description from the R help:
This function gives internal and cross-validation measures of predictive accuracy for multiple linear regression. (For binary logistic regression, use the CVbinary function.) The data are randomly assigned to a number of ‘folds’. Each fold is removed, in turn, while the remaining data is used to re-fit the regression model and to predict at the deleted observations.
EDIT: Reply to comment.
You can just take the means of the relevant performance metrics that you saved. For example, you can use an sapply on the trainTestTuple in order to extract the relevant elements from each sublist. sapply will return these elements as a vector from which you can calculate the mean. This should work:
mean_ME <- mean(sapply(trainTestTuple,"[[",2))
mean_MAD <- mean(sapply(trainTestTuple,"[[",3))
mean_MSE <- mean(sapply(trainTestTuple,"[[",4))
mean_RMSE <- mean(sapply(trainTestTuple,"[[",5))
mean_adjRsq <- mean(sapply(trainTestTuple,"[[",6))
Another small edit: The calculation of your MAD looks rather strange. It might be a good thing to double check if this is exactly what you want.