Grouping scale_fill_gradient/continuous grouped bar chart - r

Question

I am trying to make a grouped bar chart in which the bars are colored based on one variable(binary/ e.g. Group 1 and group2), and then the transparency of the bars are based on another value(continuous/ e.g. p-value), but I want the transparency to be specific to each groups color, and I want the gradient to and legend to be continuous.
I have been able to get close using the color, group, and fill options in geom_bar. You will see that I can get the over all gradient to work and the outlines of the bars are colored correctly. But I would like the fill to be the colors of the outlines and retain the transparency. I also tried using scale_alpha, which maps the transparencies correctly, but doesn't produce a continuous legend.
Here is a small data set like the one I am working with
## data set
d <- data.frame(ID = rep(c(123, 456), 2),
description = rep(c("cancer", "infection"), 2),
variable = c("G2", "G2", "G1", "G1"),
value = c(1.535709, 1.582127, 4.093683, 4.658328),
pvals = c(9.806872e-12, 1.160182e-09, 3.179635e-05, 1.132216e-04))
Here is the ggplot code
ggplot(d, aes(x=reorder(description, -pvals), y=value)) +
geom_bar(stat="identity", aes(col=variable, group=variable, fill=pvals), position="dodge") +
ylim(0, max(d$value) + 0.6) + xlab("") +
coord_flip() +
scale_fill_brewer(palette = "Set1",
name="",
breaks=c("G1", "G2"),
labels=c("Group 1", "Group 2")) +
scale_fill_continuous(trans = 'log10') # I am using log10 transformation because I have many small p-values and this makes the shading look better
Here is attempt 2 where the fill works but the legend does not.
ggplot(d, aes(x=reorder(description, -pvals), y=value)) +
geom_bar(stat="identity", aes(fill=variable, alpha = pvals), position="dodge") +
ylim(0, max(d$value) + 0.6) + xlab("") +
coord_flip() +
scale_fill_brewer(palette = "Set1",
name="",
breaks=c("G1", "G2"),
labels=c("G1", "G2")) +
scale_alpha(trans = "log10")

Answer 1

I've come up with an ugly hack, but it works so here we are. The idea is to first plot your plot as you would per usual, take the layer data and use that as input in a new plot. In this new plot, we make two layers for G1 and G2 and use the ggnewscales package to map these layers to different aesthetics. There are a few caveats I'll warn about.
First, we'll make a plot and save it as a variable:
g <- ggplot(d, aes(x=reorder(description, -pvals), y=value)) +
geom_bar(stat="identity", aes(col=variable, group=variable, fill=pvals), position="dodge") +
ylim(0, max(d$value) + 0.6) + xlab("") +
coord_flip() +
scale_fill_brewer(palette = "Set1",
name="",
breaks=c("G1", "G2"),
labels=c("Group 1", "Group 2")) +
scale_fill_continuous(trans = 'log10')
Next, we'll take the coordinates of this layers data and match them back to the original data. Note that this highly dependent on having unique y-values in your original plot, but I suppose you could also figure this out in other ways.
ld <- layer_data(g)
ld <- ld[, c("xmin", "xmax", "ymin", "ymax")]
# Match back to original data
matches <- match(ld$ymax, d$value)
# Supplement with original data
ld$pvals <- log10(d$pvals[matches])
ld$descr <- d$description[matches]
ld$vars <- d$variable[matches]
Now we'll make a new plot with geom_rects as layers, separated by the vars. In between these layers, we the first fill scale for G1 and use the new_scale_fill() afterwards. Afterwards, we'll do the second geom_rect() and the second fill scale. Then we'll muddle around with the x-axis to have it resemble the original plot somewhat.
library(ggnewscale)
ggplot(mapping = aes(xmin = xmin, xmax = xmax, ymin = ymin, ymax = ymax)) +
geom_rect(data = ld[ld$vars == "G1", ], aes(fill = pvals)) +
scale_fill_gradient(low = "red", high = "transparent",
limits = c(min(ld$pvals), 0),
name = "Log10 P-values G1") +
new_scale_fill() +
geom_rect(data = ld[ld$vars == "G2", ], aes(fill = pvals)) +
scale_fill_gradient(low = "blue", high = "transparent",
limits = c(min(ld$pvals), 0),
name = "Log10 P-values G2") +
scale_x_continuous(breaks = seq_along(unique(d$description)),
labels = c("cancer", "infection")) +
coord_flip()
And that's the ugly hack. I might have the x-axis labels wrong, but I've found no elegant way to automatically reproduce the x-axis labels without the code getting too long.
Note: ggnewscales is known to throw errors in older versions of R, but if you use the github version they've fixed that error.

Answer 2

To make the script less verbose and the output is shown below if that is what you're after.
library(ggplot2)
base <- ggplot(d, aes(reorder(description, -pvals), value)) + geom_bar(stat = "identity", aes(col=variable, group=variable, fill=pvals), position = "dodge")
base_axes_flip <- base + ylim(0, max(d$value) + 0.6) + xlab("") + coord_flip()
bax_color <- base_axes_flip + scale_color_manual(values=c('#800020','#00FFFF'),
name="",
breaks=c("G1", "G2"),
labels=c("Group 1", "Group 2"))
# Note here the scale_color_manual
bax_color + scale_fill_continuous(trans = 'log10')
This produces the following output and hope it helps.