ANOVA LMER Eta squared - r

Question

I used the lmer package to run mixed models, when I use the anova function to retrieve the anova results, everything works. However, when I try and calculate the eta squared, I consistently get the error below. Any ideas?
Dyestuff is a dataset available with the lmerTest package. I
use package ‘lme4’ version 1.1-21. package ‘lmerTest’ version 3.1-0 and package ‘sjstats’ version 0.17.7
fm1 <- lmer(Yield ~ 1 + (1|Batch), Dyestuff)
am <- anova(fm1, test="F")
eta_sq(am, partial = FALSE, ci.lvl = NULL, n = 1000, method = c("dist", "quantile"))
Error: Result 2 is not a length 1 atomic vector
In addition: Warning message:
In tidy.anova(model) :
The following column names in ANOVA output were not recognized or transformed: NumDF, DenDF

Answer 1

tl;dr
it may be theoretically difficult to compute eta-squared for mixed models, see e.g. this CV question (it does suggest some ways of computing R^2 values for mixed models, which might satisfy your need for an effect size)
practically speaking, the proximal problem seems to be that internally the eta-squared computation in sjstats expects that the anova() method will return a table containing a row corresponding to the residual variance. ?anova.lmerModLmerTest returns a table with only rows corresponding to the fixed effect terms (not the residual variance).
in any case you might expect to have trouble computing an eta-squared for a model with no non-trivial fixed effects (i.e. a fixed-effect intercept only) ...
This might be more appropriate for the sjstats issues list but I'll use this space to share what I've figured out so far.
Fitting an intercept-only model gives a similar error even if it's just an lm() fit (which ought to work if anything does):
fm0 <- lm(Yield ~ 1 , Dyestuff)
am0 <- anova(fm0, test="F")
eta_sq(am0)
Error: Result 2 must be a single double, not a double vector of length 0
Run rlang::last_error() to see where the error occurred.
However: fitting a non-trivial (more fixed effects than just the intercept) lmer(Test) model also fails:
fm2 <- lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
am2 <- anova(fm2, test="F")
eta_sq(am2)
Error: Result 2 must be a single double, not a double vector of length 0
Run rlang::last_error() to see where the error occurred.
In addition: Warning message:
In tidy.anova(model) :
The following column names in ANOVA output were not recognized or transformed: NumDF, DenDF
(From what I can tell the warning message is actually harmless.)
The proximal cause of this problem seems to be that the internal sjstats:::aov_stat_summary() function returns a table with only a single row, for the SSQ/MSQ/etc. due to Days; it should also have a row for the residual SSQ/MSQ/etc.
sjstats:::aov_stat_summary(am3)
## term sumsq meansq NumDF DenDF statistic p.value
## 1 Days 30030.94 30030.94 1 16.99998 45.85296 3.263825e-06
The problem is that the number of terms is internally computed as (nrow(aov.sum)-1), which doesn't make sense here.
Compare this with what we get with a 1+Days model using lm():
fm3 <- lm(Reaction ~ Days , sleepstudy)
am3 <- anova(fm3, test="F")
sjstats:::aov_stat_summary(am3)
## term df sumsq meansq statistic p.value
## 1 Days 1 162702.7 162702.652 71.46442 9.894096e-15
## 2 Residuals 178 405251.6 2276.694 NA NA
Digging a little deeper, we can see that this is a direct consequence of the way the anova() results are reported for mixed models:
anova(fm2)
## Type III Analysis of Variance Table with Satterthwaite's method
## Sum Sq Mean Sq NumDF DenDF F value Pr(>F)
## Days 30031 30031 1 17 45.853 3.264e-06 ***
Note there is no "residuals" row. In contrast:
anova(fm3)
## Analysis of Variance Table
## Response: Reaction
## Df Sum Sq Mean Sq F value Pr(>F)
## Days 1 162703 162703 71.464 9.894e-15 ***
## Residuals 178 405252 2277

Answer 2

I think that if you use the function anova_stats from sjstats package it works.
fm2 <- lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
am2 <- anova_stats(fm2, test="F")