I would like to perform a bootstrapped paired t-test in R. I have tried this for multiple datasets that returned p<.05 when using a parametric paired t-test however when I run the bootstrap I get p-values between 0.4 and 0.5. Am I running this incorrectly?
differences<-groupA-groupB
t.test(differences) #To get the t-statistic e.g. 1.96
Repnumber <- 10000
tstat.values <- numeric(Repnumber)
for (i in 1:Repnumber) {
group1 = sample(differences, size=length(differences), replace=T)
tstat.values[i] = t.test(group1)$statistic
}
#### To get the bootstrap p-value compare the # of tstat.values
greater (or lesser) than or equal to the original t-statistic divided
by # of reps:
sum(tstat.values<=-1.96)/Repnumber
Thank you!
It looks like you're comparing apples and oranges. For the single t-test of differences you're getting a t-statistic, which, if greater than a critical value indicates whether the difference between group1 and group2 is significantly different from zero. Your bootstrapping code does the same thing, but for 10,000 bootstrapped samples of differences, giving you an estimate of the variation in the t-statistic over different random samples from the population of differences. If you take the mean of these bootstrapped t-statistics (mean(tstat.values)) you'll see it's about the same as the single t-statistic from the full sample of differences.
sum(tstat.values<=-1.96)/Repnumber gives you the percentage of bootstrapped t-statistics less than -1.96. This is an estimate of the percentage of the time that you would get a t-statistic less than -1.96 in repeated random samples from your population. I think this is essentially an estimate of the power of your test to detect a difference of a given size between group1 and group2 for a given sample size and significance level, though I'm not sure how robust such a power analysis is.
In terms of properly bootstrapping the t-test, I think what you actually need to do is some kind of permutation test that checks whether your actual data is an outlier when compared with repeatedly shuffling the labels on your data and doing a t-test on each shuffled dataset. You might want to ask a question on CrossValidated, in order to get advice on how to do this properly for your data. These CrossValidated answers might help: here, here, and here.
Related
I am conducting a propensity score match analysis on the outcome of two different new cancer treatments where the outcome is binary (cancer-free or not cancer free). Following successful matching I get my paired 2x2 contingency table for my outcome between my matched pairs which looks like below;
**Treatment 1**
Not-Cancer Free Cancer Free
**Treatment 2** Not-Cancer Free 50 39
Cancer Free 53 60
I'd like to compare the outcomes to figure out if one treatment is better than the other by comparing odds ratios of being cancer free. I've been advised to conduct a McNemar's test due to the matched nature of the data which I do and get a p-value of 0.17 (non-significant). However, I've also been advised that instead of simply using the odds ratio normally used for such 2x2 tables (B/C --> 39/53 = 0.78 OR) that I should calculate the odds ratio and 95% confidence intervals using the methods shown in Agresti Alan, Min Yongyi. Effects and non‐effects of paired identical observations in comparing proportions with binary matched‐pairs data. Statistics in medicine. 2004 Jan 15;23(1):65-75. as it accounts for the matched nature of the data.
Unfortunately after reading this paper numerous times (especially it's odds ratio section) I can't figure out what the equations given for the odds ratio and 95% CI calculations are that they are referring to but know that they must be in there somewhere as other papers have cited this paper when referring to their odds ratios but don't share their methodology making it difficult to traceback.
If anyone has read this paper or has experience with odds ratios for matched binary data, can you please let me know how I can go about to get matched pair odds ratios. Thank you incredibly much in advance!
You can use McNemar exact test for the paired data. A point they are making in the paper and what the exact test uses are the off-diagonal elements (b,c) in the calculations. You can use exact2x2 package (https://cran.r-project.org/web/packages/exact2x2/exact2x2.pdf) to get the test results with 95%CI:
library(exact2x2)
# Set up your data as matrix
x<-matrix(c(50,53,39,60),2,2)
mcnemar.exact(x)
Gives:
Exact McNemar test (with central confidence intervals)
data: x
b = 39, c = 53, p-value = 0.175
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
0.4738071 1.1339142
sample estimates:
odds ratio
0.7358491
(What I believe is) a very simple question. I have just performed a Pearson's correlation test in R, and I'd like to know the exact p-value. However, the p-value is so small R (or tdist in Excel, or any other online calculate-it software) tells me the p-value is <2.2e-16 or 0. I suspect it has something to do with the large number of observations I have (n = 11001).
Here's the output I get from running a pairwise correlation
cor.test(mets$s_M48153,mets$s_M48152)
Pearson's product-moment correlation
data: mets$s_M48153 and mets$s_M48152
t = 88.401, df = 10999, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
0.6334378 0.6552908
sample estimates:
cor
0.6444959
"cor.test(mets$s_M48153,mets$s_M48152)$p.value" also gives me a p-value of 0.
Because of this, I'd like to manually calculate the exact p-value using the t-statistic and degrees of freedom, but I can't find the formula anywhere. Does anyone know the formula, or can tell me how to extract exact p-values from R (if possible)?
Thanks everyone for your suggestions and advice, and sorry for not replying sooner. I've been juggling a few things around until recently. However, I did ask a statistician within my department about this, and he agreed with what r2evans said. If the p-value is smaller than 10^-16, there's little point in reporting an 'exact' value, since the point is that there is strong evidence that the result differs from the null hypothesis.
One case when p-values might be important is when you want to rank by order of significance, but you could get around this by using z-scores to rank instead.
To address the original question, I defer to this guide, which I found long after posting this question: https://stats.stackexchange.com/questions/315311/how-to-find-p-value-using-estimate-and-standard-error.
Running the below command in R:
cor.test(loandata$Age,loandata$Losses.in.Thousands)
loandata is the name of the dataset
Age is the independent Variable
Losses.in.Thousands is the dependent variable
Below is the result in R:
Pearson's product-moment correlation
data: loandata$Age and loandata$Losses.in.Thousands
t = -61.09, df = 15288, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
-0.4556139 -0.4301315
sample estimates:
cor
-0.4429622
How to decide whether Age is correlated with Losses.in.Thousand ?
How do we decide by looking at the p-value with alpha = 0.05?
As stated in the other answer, the correlation coefficient produced by cor.test() in the OP is -0.4429. The Pearson correlation coefficient is a measure of the linear association between two variables. It varies between -1.0 (perfect negative linear association) and 1.0 (perfect positive linear association), the magnitude is absolute value of the coefficient, or its distance from 0 (no association).
The t-test indicates whether the correlation is significantly different from zero, given its magnitude relative to its standard error. In this case, the probability value for the t-test, p < 2.2e-16, indicates that we should reject the null hypothesis that the correlation is zero.
That said, the OP question:
How to decide whether Age is correlated with Losses.in.Thousands?
has two elements: statistical significance and substantive meaning.
From the perspective of statistical significance, the t-test indicates that the correlation is non-zero. Since the standard error of a correlation varies inversely with degrees of freedom, the very large number of degrees of freedom listed in the OP (15,288) means that a much smaller correlation would still result in a statistically significant t-test. This is why one must consider substantive significance in addition to statistical significance.
From a substantive significance perspective, interpretations vary. Hemphill 2003 cites Cohen's (1988) rule of thumb for correlation magnitudes in psychology studies:
0.10 - low
0.30 - medium
0.50 - high
Hemphill goes on to conduct a meta analysis of correlation coefficients in psychology studies that he summarized into the following table.
As we can see from the table, Hemphill's empirical guidelines are much less stringent than Cohen's prior recommendations.
Alternative: coefficient of determination
As an alternative, the coefficient of determination, r^2 can be used as a proportional reduction of error measure. In this case, r^2 = 0.1962, and we can interpret it as "If we know one's age, we can reduce our error in predicting losses in thousands by approximately 20%."
Reference: Burt Gerstman's Statistics Primer, San Jose State University.
Conclusion: Interpretation varies by domain
Given the problem domain, if the literature accepts a correlation magnitude of 0.45 as "large," then treat it as large, as is the case in many of the social sciences. In other domains, however, a much higher magnitude is required for a correlation to be considered "large."
Sometimes, even a "small" correlation is substantively meaningful as Hemphill 2003 notes in his conclusion.
For example, even though the correlation between aspirin taking and preventing a heart attack is only r=0.03 in magnitude, (see Rosenthal 1991, p. 136) -- small by most statistical standards -- this value may be socially important and nonetheless influence social policy.
To know if the variables are correlated, the value to look at is cor = -0.4429
In your case, the values are negatively correlated, however the magnitude of correlation isn't very high.
A simple, less confusing way to check if two variables are correlated, you can do:
cor(loandata$Age,loandata$Losses.in.Thousands)
[1] -0.4429622
The null hypothesis of the Pearson test is that the two variables are not correlated: H0 = {rho = 0}
The p-value is the probability that the test's statistic (or its absolute value for a two tailed test) would be beyond the actual observed result (or its absolute value for a two tailed test). You can reject the hypothesis if the p-value is smaller than the confidence level. This is the case in your test, which means the variables are correlated.
I'm looking for a way to "block" a t-test that will allow me to use three measurements per subject, instead of just averaging them.
The problem is this: My way of measuring has (intrinsically) big errors, so I measured the same subject three times to account for that error (technical triplicates). I typically block my ANOVAs or consider subject and technical repetition as a Random Effects in a Linear Mixed Effect model, in order to take the nature of my measurements in consideration.
However, in this case I only have treatment (5 subjects, 3 measurements per subject = 15) and control (7*3 = 21) in my set, so a t-test would be more adequate, but I cannot find a way to "block" a t-test. A paired t-test is not applicable since the treatment is not applied to the same subjects (would be impossible) and a normal t-test does not take in consideration the error within repetitions. Is there a function in R to do this?
A t-test is just a special case of ANOVA. Take data that you would regularly do a t-test for and instead do ANOVA (2 groups, oneway) and you will notice that the p-value is exactly the same (if the t-test was 2 tailed) and the F statistic from the ANOVA is the square of the t statistic from the t-test (exactly equal if doing a pooled t-test, approximately if using the approximate t test).
So "blocking" a t-test is really just doing an ANOVA with blocking as well.
If you don't want to assume equal variances (the pooled t-test) then you can still do a mixed effects model instead of the ANOVA and just allow for unequal variances in the mixed effects model.
I'm trying to find if soil PH levels differ between 3 species of violet. I'm performing a one way ANOVA test in R and entered the text as follows:
anova<-aov(viola$PH~viola$species)
summary(anova)
This seems like it should be simple and I've done similar tests before, but when I run this, all I get is the degrees of freedom, sum sq, and mean sq. Any help would be appreciated.
Anova requires replicates to assess the variance and compute the statistic. Discard the mean values, and use the original replicated data as an input to anova. Then you can extract the p-values with summary(anova) or TukeyHSD(anova).