I am doing a meta-analysis in R. I have calculated effect sizes (Cohen's D) for all the studies (17) that qualified my filtering criterion. I want to use random effects model to do my analysis. I tried doing the analysis with metagen() and rma(), both returning completely different results (Copy-pasted below). I’m wondering if what I’m doing is correct? Is there another way to do the meta-analysis? Will really appreciate any help of guidance.
Result for meta-gen:
95%-CI z p-value
Fixed effect model 0.3687 [ 0.3385; 0.3990] 23.90 < 0.0001
Random effects model 0.0345 [-0.2560; 0.3249] 0.23 0.8160
Quantifying heterogeneity:
tau^2 = 0.3437; H = 8.16 [7.37; 9.03]; I^2 = 98.5% [98.2%; 98.8%]
Test of heterogeneity:
Q d.f. p-value
1065.79 16 < 0.0001
Details on meta-analytical method:
- Inverse variance method
- DerSimonian-Laird estimator for tau^2
Result for rma():
Random-Effects Model (k = 17; tau^2 estimator: REML)
tau^2 (estimated amount of total heterogeneity): 1.2737 (SE = 0.4612)
tau (square root of estimated tau^2 value): 1.1286
I^2 (total heterogeneity / total variability): 99.59%
H^2 (total variability / sampling variability): 244.16
Test for Heterogeneity:
Q(df = 16) = 1065.7913, p-val < .0001
Model Results:
estimate se zval pval ci.lb ci.ub
0.0332 0.2770 0.1200 0.9045 -0.5097 0.5762
Related
I tried to do a meta-analysis of a single proportion. Here is the R codes:
# Packages
library(metafor)
# Data
dat <- dat.debruin2009 #from metafor package
# Metafor package ----
dat <- escalc(measure = "PLO", xi = xi, ni = ni, data = dat)
## Calculate random effect
res <- rma(yi, vi, data = dat)
res
predict(res, transf = transf.ilogit)
Here is the raw result (logit) from res object:
Random-Effects Model (k = 13; tau^2 estimator: REML)
tau^2 (estimated amount of total heterogeneity): 0.4014 (SE = 0.1955)
tau (square root of estimated tau^2 value): 0.6336
I^2 (total heterogeneity / total variability): 90.89%
H^2 (total variability / sampling variability): 10.98
Test for Heterogeneity:
Q(df = 12) = 95.9587, p-val < .0001
Model Results:
estimate se zval pval ci.lb ci.ub
-0.1121 0.1926 -0.5821 0.5605 -0.4896 0.2654
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
And this is the result from predict():
pred ci.lb ci.ub pi.lb pi.ub
0.4720 0.3800 0.5660 0.1962 0.7660
So, my question is I get a non-significant result from raw result (p = 0.5605). But, a CI from predict() does not crosses zero (CI = 0.3800, 0.5660 ), which indicates a significant result. Do I misunderstand something or missing a step in the R code? or any explanation why the results are contradicting?
===================================================
Edit:
I tried using the meta package, I get a similar contradicting result as in metafor.
meta_pkg <- meta::metaprop(xi, ni, data = dat)
meta_pkg$.glmm.random
Here is the result (Similar result as predict() from above):
> meta_pkg
Number of studies combined: k = 13
Number of observations: o = 1516
Number of events: e = 669
proportion 95%-CI
Common effect model 0.4413 [0.4165; 0.4664]
Random effects model 0.4721 [0.3822; 0.5638]
Quantifying heterogeneity:
tau^2 = 0.3787; tau = 0.6154; I^2 = 87.5% [80.4%; 92.0%]; H = 2.83 [2.26; 3.54]
Test of heterogeneity:
Q d.f. p-value Test
95.96 12 < 0.0001 Wald-type
108.77 12 < 0.0001 Likelihood-Ratio
Details on meta-analytical method:
- Random intercept logistic regression model
- Maximum-likelihood estimator for tau^2
- Logit transformation
Similar raw result as in metafor:
> meta_pkg$.glmm.random
Random-Effects Model (k = 13; tau^2 estimator: ML)
tau^2 (estimated amount of total heterogeneity): 0.3787
tau (square root of estimated tau^2 value): 0.6154
I^2 (total heterogeneity / total variability): 90.3989%
H^2 (total variability / sampling variability): 10.4155
Tests for Heterogeneity:
Wld(df = 12) = 95.9587, p-val < .0001
LRT(df = 12) = 108.7653, p-val < .0001
Model Results:
estimate se zval pval ci.lb ci.ub
-0.1118 0.1880 -0.5946 0.5521 -0.4804 0.2567
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
The p-value is testing whether the average logit transformed proportion is significantly different from 0. That is not the same as testing whether the proportion is significantly different from 0. In fact, transf.ilogit(0) gives 0.5, so that is the corresponding value of a proportion that is being tested. And you will notice that 0.5 falls inside of the confidence interval after the back-transformation. So everything is fully consistent.
I'm currently working on a meta-analysis of proportions (number of mosquitoes transmitting a disease/number of mosquitoes tested), using metafor package (Viechtbauer, 2010). My aim is to compute a summarized effect size for each of 5 mosquitoes species. As far, my analysis strategy is:
Using PFT (double arcsine) transformation in order to normalize data (I have lots of 0 and 1 as values)
Running overall model to assess heterogeneity (test for residual heterogeneity significative)
As I have several measures coming from each of the articles included in the meta-analysis, I assessed the necessity of using a three-level model (LRT significative, for "measure" and "article" (measure nested in article)
Using subgroup analysis, with mosquitoes species as moderator
Assessing for residual heterogeneity
Testing some moderators to try to explain residual heterogeneity
When I performed subgroup analysis, I got a test for moderator significative for the variable "Specie". But I then wanted to know which part of variance is explained by this significative variable, and I obtained -0.9% (truncated at 0%) (I used "pseudo R-squared" method suggested by W. Viechtbauer here).
So, my question is: would it be possible/coherent to have a test of moderators significative and no variance explained by this moderator ? How to explain it ?
As I use REML estimation, I can't use LRT to test if the variable is significative (and I would rather not use ML back just to compute LRT).
Thanks in advance if someone can help me,
Best regards,
Alex
If useful, here is a abstract of the code I used:
ies.da <- escalc(xi = data_test[, "n"],
ni = data_test[, "n_tested"],
data = data,
measure = "PFT",
add = 0)
subganal.specie.mv <- rma.mv(yi, vi,
data = ies.da,
mods = ~factor(Specie),
method = "REML",
random = ~1|article/measure)
subganal.no.specie.mv <- rma.mv(yi, vi,
data = ies.da,
method = "REML",
random = ~1|article/measure)
pseudo.r.squared <- (sum(subganal.no.specie.mv$sigma2) - sum(subganal.specie.mv$sigma2)) / sum(subganal.no.specie.mv$sigma2)
As result, I get a test of moderator significative:
subganal.specie.mv
Multivariate Meta-Analysis Model (k = 165; method: REML)
Variance Components:
estim sqrt nlvls fixed factor
sigma^2.1 0.0184 0.1358 21 no article
sigma^2.2 0.0215 0.1466 165 no article/measure
Test for Residual Heterogeneity:
QE(df = 161) = 897.9693, p-val < .0001
Test of Moderators (coefficients 2:4):
QM(df = 3) = 12.3578, p-val = 0.0063
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt 0.6172 0.1051 5.8729 <.0001 0.4112 0.8232 ***
factor(Specie)1 -0.0123 0.1240 -0.0990 0.9211 -0.2554 0.2308
factor(Specie)2 -0.2110 0.1178 -1.7913 0.0733 -0.4419 0.0199 .
factor(Specie)3 -0.2299 0.1008 -2.2813 0.0225 -0.4274 -0.0324 *
But my "pseudo R squared" is null:
pseudo.r.squared
[1] -0.009012437
I'm conducting a meta-analysis of prevalence studies on viral infections. Prevalence can be seen as the proportion of individuals with a binary trait, thus the raw data needs to be transformed before we can start the analysis (preferably double-arcsine transformation).
Essentially, I have performed meta-regression with 3 moderators (hemisphere;dichotomous), (latitude; continuous) , (climate;dichotomous) on the double-arcsine transformed (proportion/prevalence) data. After obtaining the multi-variable meta-regression model I fail to perform a back-transformation on the regression coefficients and 95% CI to obtain the 'normal' proportion/prevalence numbers.
I have tried copying the code for the inverse of the double-arcsine transformation into the meta-regression code with the hopes of obtaining the multi-variable model with the untransformed data.
The following lines of codes were used to transform the data, conduct the meta-regression with 3 moderators, and print the output.
ies.da=escalc(xi=cases, ni=total, data=proportion, measure="PFT", add=0)
metareg.3mods=rma(yi, vi, data=ies.da, mods=~hemisphere+latitude+climate)
print(metareg.3mods)
The output of the lines of codes written above can be found below. I'm quite certain that the values in this output are based are based upon the transformed values and therefore still need a back-transformation (inverse double-arcsine?) in order to obtain the correct proportion values. How can I incorporate this back-transformation into the meta-regression so that RStudio provides me with a model that contains the correct proportion values?
OUTPUT:
Mixed-Effects Model (k = 22; tau^2 estimator: REML)
tau^2 (estimated amount of residual heterogeneity): 0.0067 (SE =
0.0037)
tau (square root of estimated tau^2 value): 0.0819
I^2 (residual heterogeneity / unaccounted variability): 67.46%
H^2 (unaccounted variability / sampling variability): 3.07
R^2 (amount of heterogeneity accounted for): 32.94%
Test for Residual Heterogeneity: QE(df = 18) = 66.9031, p-val < .0001
Test of Moderators (coefficient(s) 2:4): QM(df = 3) = 8.6072, p-val =
0.0350
Model Results:
Intercept: β=0.5313 SE= 0.1223 zval 4.3448 pval <0.0001 ci.lb
0.2916 ci.ub 0.7709
Sout hemisphere β=-0.4444 SE= 0.2116 zval -2.1001 pval 0.0357 ci.lb
-0.8592 ci.ub 0.-0.0297
Latitude β=-0.0055 SE= 0.0028 zval -1.9717 pval 0.0486 ci.lb
-0.0110 ci.ub -0.000
Tropical climate β=-0.1986 SE= 0.0701 zval -2.8340 pval 0.0046 ci.lb
-0.3360 ci.ub 0.-0.0613
I am working on a meta regression using the metafor package. My simple trial estimation is:
m1<-rma(yi=COEFF, sei=STDERR, mods = ~ MT_TIMESERIES + MT_BIVARIATE, method="REML", data=y)
Next I estimate the same model with rma.mv() and use a random term RID which is a factor identifying each single observation (no clusters of observations):
m2<-rma.mv(yi=COEFF, V=STDERR^2, random= ~ 1|RID, mods = ~ MT_TIMESERIES + MT_BIVARIATE, method="REML", data=y)
Estimations m1 and m2 should actually yield the same results (this idea is supported by a note from the package author on http://www.metafor-project.org/doku.php/tips:rma.uni_vs_rma.mv).
But in fact, they don't:
> summary(m1)
Mixed-Effects Model (k = 886; tau^2 estimator: REML)
logLik deviance AIC BIC AICc
-4847.7988 9695.5976 9703.5976 9722.7309 9703.6431
tau^2 (estimated amount of residual heterogeneity): 0.0000 (SE = 0.0000)
tau (square root of estimated tau^2 value): 0.0007
I^2 (residual heterogeneity / unaccounted variability): 1.21%
H^2 (unaccounted variability / sampling variability): 1.01
R^2 (amount of heterogeneity accounted for): 87.37%
Test for Residual Heterogeneity:
QE(df = 883) = 9083.3858, p-val < .0001
Test of Moderators (coefficient(s) 2,3):
QM(df = 2) = 104.7561, p-val < .0001
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt -0.0076 0.0009 -8.6343 <.0001 -0.0093 -0.0059 ***
MT_TIMESERIES 0.0004 0.0010 0.3669 0.7137 -0.0016 0.0023
MT_BIVARIATE 0.0062 0.0010 6.4595 <.0001 0.0043 0.0081 ***
> summary(m2)
Multivariate Meta-Analysis Model (k = 886; method: REML)
logLik Deviance AIC BIC AICc
-2948.3789 5896.7578 5904.7578 5923.8911 5904.8034
Variance Components:
estim sqrt nlvls fixed factor
sigma^2 3.2560 1.8044 886 no RID
Test for Residual Heterogeneity:
QE(df = 883) = 9083.3858, p-val < .0001
Test of Moderators (coefficient(s) 2,3):
QM(df = 2) = 13.7838, p-val = 0.0010
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt -0.5362 0.1262 -4.2475 <.0001 -0.7836 -0.2888 ***
MT_TIMESERIES -0.5021 0.1557 -3.2237 0.0013 -0.8073 -0.1968 **
MT_BIVARIATE -0.5016 0.1949 -2.5742 0.0100 -0.8835 -0.1197 *
Does anyone have an idea why that could be?
Many thanks in advance!
Best regards
Joachim
I run two meta-analysis and want to proof that the caluclated mean effect size (in fisher z) differs between both meta-analysis.
As i am quit new in R and not such a pro in statistics, could you provide the appropriate test and how to conduct it in R?
Here are my current results of the two meta analysis:
> results1GN
Random-Effects Model (k = 4; tau^2 estimator: REML)
tau^2 (estimated amount of total heterogeneity): 0.0921 (SE = 0.0752)
tau (square root of estimated tau^2 value): 0.3034
I^2 (total heterogeneity / total variability): 99.98%
H^2 (total variability / sampling variability): 5569.05
Test for Heterogeneity:
Q(df = 3) = 22183.0526, p-val < .0001
Model Results:
estimate se zval pval ci.lb ci.ub
0.3663 0.1517 2.4139 0.0158 0.0689 0.6637 *
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> results1NN
Random-Effects Model (k = 72; tau^2 estimator: REML)
tau^2 (estimated amount of total heterogeneity): 0.0521 (SE = 0.0096)
tau (square root of estimated tau^2 value): 0.2282
I^2 (total heterogeneity / total variability): 95.98%
H^2 (total variability / sampling variability): 24.85
Test for Heterogeneity:
Q(df = 71) = 1418.1237, p-val < .0001
Model Results:
estimate se zval pval ci.lb ci.ub
0.2594 0.0282 9.2016 <.0001 0.2042 0.3147 ***
I elaborate on the previous response, which was a bit rude.
On first step, you may consider the confidence interval of you result.
You may consult the Wikipedia page on the topic for a quick summary:
http://en.wikipedia.org/wiki/Confidence_interval
Anyway, in your example you have two effect sizes with overlapping confidence interval:
results1GN = 0.36 (95%CI: 0.069 to 0.66)
results1NN = 0.26 (95%CI: 0.20 to 0.31).
So, the two results are not statistically different.
The package 'metafor' also includes a function (anova) to compare "nested models", and I quote:
"For two (nested) models of class "rma.uni", the function provides a full versus reduced model comparison in terms of model fit statistics and a likelihood ratio test".
Note that "When specifying two models for comparison, the function provides a likelihood ratio test comparing the two models. The models must be based on the same set of data and should be nested for the likelihood ratio test to make sense".
You may found more details and examples on the use of the function here:
http://www.inside-r.org/packages/cran/metafor/docs/anova.rma.uni
Also, consult the manual of the package and the the site of the package maintainer.
Hope this may be helpful.