Making left-skewed distribution normal using log transformation? - r

Question

I have a dataset with a variable that has a left-skewed distribution (the tail is on the left).
variable <- c(rep(35, 2), rep(36, 4), rep(37, 16), rep(38, 44), rep(39, 72), rep(40, 30))
I just want to make this data have a more normal distribution so I can perform an anova, but using log10, or log2 makes it still way left-skewed. What transformation can I use to make this data more normal?
EDIT: My model is: mod <- lme(reponse ~ variable*variable2, random=~group, data=data), so Kruskal Wallace would work except for the random effect and one predictor term thing. I did a Shapiro Wilk test, and my data is definitely non-normal. If justifiable, I would like to transform my data to give the ANOVA a better chance of detecting a significant result. Either that, or a mixed effect test for non-normal data.
#Ben Bolker - Thank you for your reply; I appreciate it. I did read your answer, but I'm still reading up on exactly what some of your suggestions mean (I'm very new to statistics). My p-value is fairly close to significant and I don't want to p-hack, but I also want to give my data the best chance I can of being significant. If I can't justify transforming my data or using something besides ANOVA, then so be it.
I've provided a dataframe snapshot below. My response variable is "temp.max", the maximum temperature at which a plant dies. My predictor variables are "growth.chamber" (either a 29 or 21 degree growth chamber) and "environment" (either field or forest). My random variable is "groupID" (the group the plants were raised in, consisting of 5-10 individuals). This is a reciprocal transplant experiment, so I raised both forest and field plants in both 21 and 29 degree chambers. What I want to know is if "temp.max" differs between field and forest populations, whether "temp.max" differs between growth chambers, and whether there is any interaction between environment and growth chamber in regards to temp.max. I would very, very much appreciate any help. Thank you.
> dput(data)
structure(list(groupID = structure(c(12L, 12L, 12L, 12L, 12L,
12L, 12L, 12L, 12L, 12L, 14L, 14L, 14L, 14L, 14L, 14L, 14L, 14L,
14L, 14L, 15L, 15L, 15L, 15L, 15L, 15L, 15L, 15L, 15L, 16L, 16L,
16L, 16L, 16L, 19L, 19L, 19L, 19L, 19L, 19L, 19L, 19L, 19L, 19L,
18L, 18L, 18L, 18L, 18L, 18L, 18L, 18L, 18L, 18L, 17L, 17L, 17L,
17L, 17L, 17L, 17L, 20L, 20L, 20L, 20L, 20L, 20L, 20L, 20L, 20L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L,
1L, 1L, 1L, 1L, 3L, 3L, 3L, 3L, 3L, 6L, 6L, 6L, 6L, 6L, 6L, 6L,
6L, 6L, 6L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 7L, 7L, 7L, 7L, 7L, 7L,
7L, 7L, 7L, 7L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 13L, 13L, 13L,
13L, 13L, 13L, 13L, 13L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L,
9L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 11L, 11L,
11L, 11L, 11L, 8L, 8L, 8L, 8L, 8L), .Label = c("GRP_104", "GRP_111",
"GRP_132", "GRP_134", "GRP_137", "GRP_142", "GRP_145", "GRP_147",
"GRP_182", "GRP_192", "GRP_201", "GRP_28", "GRP_31", "GRP_40",
"GRP_68", "GRP_70", "GRP_78", "GRP_83", "GRP_92", "GRP_98"), class = "factor"),
individual = c(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L, 1L,
2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L, 1L, 2L, 3L, 4L, 5L,
6L, 7L, 8L, 9L, 1L, 2L, 3L, 4L, 5L, 1L, 2L, 3L, 4L, 5L, 6L,
7L, 8L, 9L, 10L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L,
1L, 2L, 3L, 4L, 5L, 6L, 7L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L,
9L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 16L, 17L, 1L, 2L, 3L,
4L, 5L, 6L, 7L, 8L, 9L, 10L, 1L, 2L, 3L, 4L, 5L, 1L, 2L,
3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L, 1L, 2L, 3L, 4L, 5L, 6L,
7L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 15L, 16L, 20L, 1L, 2L, 3L,
4L, 5L, 6L, 7L, 16L, 1L, 2L, 3L, 4L, 5L, 11L, 12L, 14L, 1L,
2L, 3L, 4L, 5L, 6L, 7L, 18L, 19L, 20L, 1L, 2L, 3L, 4L, 5L,
16L, 17L, 18L, 19L, 20L, 1L, 2L, 3L, 4L, 5L, 1L, 2L, 3L,
4L, 5L), temp.max = c(39L, 35L, 39L, 39L, 35L, 40L, 40L,
40L, 40L, 39L, 39L, 39L, 39L, 39L, 39L, 39L, 39L, 38L, 38L,
38L, 39L, 39L, 40L, 38L, 40L, 39L, 39L, 40L, 40L, 39L, 39L,
39L, 39L, 39L, 39L, 39L, 39L, 39L, 39L, 39L, 39L, 40L, 38L,
40L, 40L, 40L, 40L, 40L, 40L, 39L, 40L, 39L, 39L, 40L, 39L,
39L, 39L, 39L, 38L, 38L, 38L, 38L, 40L, 39L, 39L, 38L, 38L,
39L, 39L, 37L, 39L, 39L, 37L, 39L, 39L, 39L, 39L, 37L, 39L,
39L, 38L, 37L, 38L, 38L, 38L, 36L, 36L, 36L, 37L, 37L, 40L,
39L, 40L, 39L, 39L, 37L, 37L, 38L, 38L, 38L, 37L, 38L, 38L,
38L, 37L, 38L, 38L, 37L, 38L, 40L, 38L, 38L, 38L, 38L, 37L,
38L, 39L, 38L, 38L, 38L, 38L, 38L, 40L, 38L, 40L, 39L, 39L,
39L, 39L, 39L, 39L, 39L, 39L, 39L, 40L, 40L, 39L, 39L, 38L,
37L, 39L, 37L, 39L, 39L, 39L, 39L, 39L, 39L, 40L, 39L, 39L,
40L, 40L, 38L, 40L, 40L, 36L, 38L, 38L, 38L, 38L, 37L, 37L,
38L, 38L, 38L, 39L, 39L), environment = structure(c(1L, 1L,
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L,
1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L,
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
1L), .Label = c("field", "forest"), class = "factor"), growth.chamber = c(29L,
29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 21L, 21L, 21L,
21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L,
21L, 21L, 21L, 21L, 29L, 29L, 29L, 29L, 29L, 21L, 21L, 21L,
21L, 21L, 21L, 21L, 21L, 21L, 21L, 29L, 29L, 29L, 29L, 29L,
29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L,
21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 29L, 29L, 29L,
29L, 29L, 29L, 29L, 29L, 29L, 29L, 21L, 21L, 21L, 21L, 21L,
21L, 21L, 21L, 21L, 21L, 29L, 29L, 29L, 29L, 29L, 21L, 21L,
21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 29L, 29L, 29L, 29L,
29L, 29L, 29L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L,
21L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 21L, 21L, 21L,
21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L,
21L, 21L, 21L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L,
29L, 21L, 21L, 21L, 21L, 21L, 29L, 29L, 29L, 29L, 29L)), .Names = c("groupID",
"individual", "temp.max", "environment", "growth.chamber"), row.names = c(1L,
2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L, 21L, 22L, 23L, 24L, 25L,
26L, 27L, 28L, 29L, 30L, 41L, 42L, 43L, 44L, 45L, 46L, 47L, 48L,
49L, 58L, 59L, 60L, 61L, 62L, 68L, 69L, 70L, 71L, 72L, 73L, 74L,
75L, 76L, 77L, 88L, 89L, 90L, 91L, 92L, 93L, 94L, 95L, 96L, 97L,
108L, 109L, 110L, 111L, 112L, 113L, 114L, 122L, 123L, 124L, 125L,
126L, 127L, 128L, 129L, 130L, 139L, 140L, 141L, 142L, 143L, 144L,
145L, 146L, 147L, 148L, 158L, 159L, 160L, 161L, 162L, 163L, 164L,
165L, 166L, 167L, 178L, 179L, 180L, 181L, 182L, 188L, 189L, 190L,
191L, 192L, 193L, 194L, 195L, 196L, 197L, 208L, 209L, 210L, 211L,
212L, 213L, 214L, 222L, 223L, 224L, 225L, 226L, 227L, 228L, 229L,
230L, 231L, 242L, 243L, 244L, 245L, 246L, 247L, 248L, 249L, 258L,
259L, 260L, 261L, 262L, 263L, 264L, 265L, 272L, 273L, 274L, 275L,
276L, 277L, 278L, 279L, 280L, 281L, 292L, 293L, 294L, 295L, 296L,
297L, 298L, 299L, 300L, 301L, 312L, 313L, 314L, 315L, 316L, 322L,
323L, 324L, 325L, 326L), class = "data.frame")

Answer 1

tl;dr you probably don't actually need to worry about the skew here.
There are a few issues here, and since they're mostly statistical rather than programming-related, this question is probably more relevant for CrossValidated.
If I copied your data correctly, they're equivalent to this:
dd <- rep(35:40,c(2,4,16,44,72,30))
plot(table(dd))
Your data are discrete - that's why the density plot that #user113156 posts has distinct peaks.
Here are the issues:
the most important is that for most statistical purposes you're not actually interested in the Normality of the marginal distribution, which is what you're showing here. Rather, you want to know whether the distribution of the residuals from a model is Normal or not; for an ANOVA, this is equivalent to asking whether the distribution of values within each group is Normal (and the groups have similar within-group variances).
Normality is not very important; ANOVA is robust to moderate degrees of non-Normality (e.g. see here).
Log transformation modifies your data in the wrong direction (i.e. it will tend to increase the left skewness). In general fixing this kind of left-skewed data requires a transformation like raising to a power >1 (the opposite direction from log- or square root-transformation), but when the values are far from zero it doesn't usually help very much anyway.
Some statistical options if you are worried:
a non-parametric, rank-based test like the Kruskal-Wallis test (the rank-based analogue of 1-way ANOVA)
do an ANOVA, but use a permutation-based approach to test statistical significance.
use an ordinal model
use hierarchical bootstrapping (resample within replacement within and between clusters) to derive more robust confidence intervals on parameters

Answer 2

Your variable follows a discrete distribution. You have integer values ranging from 35 (n=2) to 40 (n=30). I think you need to carry out some ordinal analysis collapsing values from 35 to 37 that have fewer observations in one category. Otherwise you could perform a non-parametric analysis using kruskal.test() function.

Answer 3

I have bad news and good news.
the bad news is that I don't see statistically significant patterns in your data.
the good news is that, given the structure of your experimental design, you can analyze your data much more simply (you don't need mixed models)
load packages, adjust defaults
library(ggplot2); theme_set(theme_bw())
library(dplyr)
check structure of data
Tabulating the data confirms that this is a nested design; each group occurs within a single environment/growth chamber combination.
tt <- with(dd,table(groupID,
interaction(environment,growth.chamber)))
## exactly one non-zero entry per group
all(rowSums(tt>0)==1)
aggregate data
Convert growth.chamber to a categorical variable; collapse each group to its mean temp.max value (and record the number of observations per group)
dda <-(dd
%>% mutate(growth.chamber=factor(growth.chamber))
%>% group_by(groupID,environment,growth.chamber)
%>% summarise(n=n(),temp.max=mean(temp.max))
)
ggplot(dda,aes(growth.chamber,temp.max,
colour=environment))+
geom_boxplot(aes(fill=environment),alpha=0.2)+
geom_point(position=position_dodge(width=0.75),
aes(size=n),alpha=0.5)+
scale_size(range=c(3,7))
Analysis
Now that we've aggregated (without losing any information we care about), we can use a linear regression with weights specifying the number of samples per observation:
m1 <- lm(temp.max~growth.chamber*environment,weights=n,
data=dda)
Checking distribution etc. of residuals:
plot(m1)
This all looks fine; no indication of serious bias, heteroscedasticity, non-Normality, or outliers ...
summary(m1)
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 38.2558 0.2858 133.845 <2e-16 ***
## growth.chamber29 0.3339 0.4144 0.806 0.432
## environmentforest 0.2442 0.3935 0.620 0.544
## growth.chamber29:environmentforest 0.3240 0.5809 0.558 0.585
## Residual standard error: 1.874 on 16 degrees of freedom
## Multiple R-squared: 0.2364, Adjusted R-squared: 0.09318
## F-statistic: 1.651 on 3 and 16 DF, p-value: 0.2174
Or a coefficient plot (dotwhisker::dwplot(m1))
While the plot of the data doesn't look like it's just noise, the statistical analysis suggests that we can't really distinguish it from noise ...