I'm trying to plot a distribution CDF using R and ggplot2. However, I am finding difficulties in plotting the CDF function after I transform the Y axis to obtain a straight line.
This kind of plot is frequently used in Gumbel paper plots, but here I'll use as example the normal distribution.
I generate the data, and plot the cumulative density function of the data along with the function. They fit well. However, when I apply an Y axis transformation, they don't fit anymore.
sim <- rnorm(100) #Simulate some data
sim <- sort(sim) #Sort it
cdf <- seq(0,1,length.out=length(sim)) #Compute data CDF
df <- data.frame(x=sim, y=cdf) #Build data.frame
library(scales)
library(ggplot2)
#Now plot!
gg <- ggplot(df, aes(x=x, y=y)) +
geom_point() +
stat_function(fun = pnorm, colour="red")
gg
And the output should be something on the lines of:
Good!
Now I try to transform the Y axis according to the distribution used.
#Apply transformation
gg + scale_y_continuous(trans=probability_trans("norm"))
And the result is:
The points are transformed correctly (they lie on a straight line), but the function is not!
However, everything seems to work fine if I do like this, calculating the CDF with ggplot:
ggplot(data.frame(x=sim), aes(x=x)) +
stat_ecdf(geom = "point") +
stat_function(fun="pnorm", colour="red") +
scale_y_continuous(trans=probability_trans("norm"))
The result is OK:
Why is this happening? Why doesn't calculating the CDF manually work with scale transformations?
This works:
gg <- ggplot(df, aes(x=x, y=y)) +
geom_point() +
stat_function(fun ="pnorm", colour="red", inherit.aes = FALSE) +
scale_y_continuous(trans=probability_trans("norm"))
gg
Possible explanation:
Documentation States:
inherit.aes If FALSE, overrides the default aesthetics, rather than combining with them. This is most useful for helper functions that define both data and aesthetics and shouldn't inherit behaviour from the default plot specification, e.g. borders.
My guess:
As scale_y_continuous changes the aesthetics of the main plot, we need to turn off the default inherit.aes=TRUE. It seems inherit.aes=TRUE in stat_function picks its aesthetics from the first layer of the plot, and so the scale transformation does not impact unless specifically chosen to.
Related
I'm trying to create a density curve in R using a set of random numbers between 1000, and shade the part that is less than or equal to a certain value. There are a lot of solutions out there involving geom_area or geom_ribbon, but they all require a yval, which I don't have (it's just a vector of 1000 numbers). Any ideas on how I could do this?
Two other related questions:
Is it possible to do the same thing for a cumulative density function (I'm currently using stat_ecdf to generate one), or shade it at all?
Is there any way to edit geom_vline so it will only go up to the height of the density curve, rather than the whole y axis?
Code: (the geom_area is a failed attempt to edit some code I found. If I set ymax manually, I just get a column taking up the whole plot, instead of just the area under the curve)
set.seed(100)
amount_spent <- rnorm(1000,500,150)
amount_spent1<- data.frame(amount_spent)
rand1 <- runif(1,0,1000)
amount_spent1$pdf <- dnorm(amount_spent1$amount_spent)
mean1 <- mean(amount_spent1$amount_spent)
#density/bell curve
ggplot(amount_spent1,aes(amount_spent)) +
geom_density( size=1.05, color="gray64", alpha=.5, fill="gray77") +
geom_vline(xintercept=mean1, alpha=.7, linetype="dashed", size=1.1, color="cadetblue4")+
geom_vline(xintercept=rand1, alpha=.7, linetype="dashed",size=1.1, color="red3")+
geom_area(mapping=aes(ifelse(amount_spent1$amount_spent > rand1,amount_spent1$amount_spent,0)), ymin=0, ymax=.03,fill="red",alpha=.3)+
ylab("")+
xlab("Amount spent on lobbying (in Millions USD)")+
scale_x_continuous(breaks=seq(0,1000,100))
There are a couple of questions that show this ... here and here, but they calculate the density prior to plotting.
This is another way, more complicated than required im sure, that allows ggplot to do some of the calculations for you.
# Your data
set.seed(100)
amount_spent1 <- data.frame(amount_spent=rnorm(1000, 500, 150))
mean1 <- mean(amount_spent1$amount_spent)
rand1 <- runif(1,0,1000)
Basic density plot
p <- ggplot(amount_spent1, aes(amount_spent)) +
geom_density(fill="grey") +
geom_vline(xintercept=mean1)
You can extract the x and y positions for the area to shade from the plot object using ggplot_build. Linear interpolation was used to get the y value at x=rand1
# subset region and plot
d <- ggplot_build(p)$data[[1]]
p <- p + geom_area(data = subset(d, x > rand1), aes(x=x, y=y), fill="red") +
geom_segment(x=rand1, xend=rand1,
y=0, yend=approx(x = d$x, y = d$y, xout = rand1)$y,
colour="blue", size=3)
this forum already helped me a lot for producing the code, which I expected to return a histogram of a specific variable overlayed with its empirical normal curve. I used ggplot2 and stat_function to write the code.
Unfortunately, the code produced a plot with the correct histogram but the normal curve is a straight line at zero (red line in plot produced by the following code).
For this minimal example I used the mtcars dataset - the same behavior of ggplot and stat_function is observed with my original data set.
This is the code is wrote and used:
library(ggplot2)
mtcars
hist_staff <- ggplot(mtcars, aes(x = mtcars$mpg)) +
geom_histogram(binwidth = 2, colour = "black", aes(fill = ..count..)) +
scale_fill_gradient("Count", low = "#DCDCDC", high = "#7C7C7C") +
stat_function(fun = dnorm, colour = "red")
print(hist_staff)
I also tried to specify dnorm:
stat_function(fun = dnorm(mtcars$mpg, mean = mean(mtcars$mpg), sd = sd(mtcars$mpg))
That did not work out either - an error message returned stating that the arguments are not numerical.
I hope you people can help me! Thanks a lot in advance!
Best, Jannik
Your curve and histograms are on different y scales and you didn't check the help page on stat_function, otherwise you'd've put the arguments in a list as it clearly shows in the example. You also aren't doing the aes right in your initial ggplot call. I sincerely suggest hitting up more tutorials and books (or at a minimum the help pages) vs learn ggplot piecemeal on SO.
Once you fix the stat_function arg problem and the ggplot``aes issue, you need to tackle the y axis scale difference. To do that, you'll need to switch the y for the histogram to use the density from the underlying stat_bin calculated data frame:
library(ggplot2)
gg <- ggplot(mtcars, aes(x=mpg))
gg <- gg + geom_histogram(binwidth=2, colour="black",
aes(y=..density.., fill=..count..))
gg <- gg + scale_fill_gradient("Count", low="#DCDCDC", high="#7C7C7C")
gg <- gg + stat_function(fun=dnorm,
color="red",
args=list(mean=mean(mtcars$mpg),
sd=sd(mtcars$mpg)))
gg
Suppose I have this plot:
ggplot(iris) + geom_point(aes(x=Sepal.Width, y=Sepal.Length, colour=Sepal.Length)) + scale_colour_gradient()
what is the correct way to discretize the color scale, like the plot shown below the accepted answer here (gradient breaks in a ggplot stat_bin2d plot)?
ggplot correctly recognizes discrete values and uses discrete scales for these, but my question is if you have continuous data and you want a discrete colour bar for it (with each square corresponding to a value, and squares colored in a gradient still), what is the best way to do it? Should the discretizing/binning happen outside of ggplot and get put in the dataframe as a separate discrete-valued column, or is there a way to do it within ggplot? an example of what I'm looking for is similar to the scale shown here:
except I'm plotting a scatter plot and not something like geom_tile/heatmap.
thanks.
The solution is slightly complicated, because you want a discrete scale. Otherwise you could probably simply use round.
library(ggplot2)
bincol <- function(x,low,medium,high) {
breaks <- function(x) pretty(range(x), n = nclass.Sturges(x), min.n = 1)
colfunc <- colorRampPalette(c(low, medium, high))
binned <- cut(x,breaks(x))
res <- colfunc(length(unique(binned)))[as.integer(binned)]
names(res) <- as.character(binned)
res
}
labels <- unique(names(bincol(iris$Sepal.Length,"blue","yellow","red")))
breaks <- unique(bincol(iris$Sepal.Length,"blue","yellow","red"))
breaks <- breaks[order(labels,decreasing = TRUE)]
labels <- labels[order(labels,decreasing = TRUE)]
ggplot(iris) +
geom_point(aes(x=Sepal.Width, y=Sepal.Length,
colour=bincol(Sepal.Length,"blue","yellow","red")), size=4) +
scale_color_identity("Sepal.Length", labels=labels,
breaks=breaks, guide="legend")
You could try the following, I have your example code modified appropriately below:
#I am not so great at R, so I'll just make a data frame this way
#I am convinced there are better ways. Oh well.
df<-data.frame()
for(x in 1:10){
for(y in 1:10){
newrow<-c(x,y,sample(1:1000,1))
df<-rbind(df,newrow)
}
}
colnames(df)<-c('X','Y','Val')
#This is the bit you want
p<- ggplot(df, aes(x=X,y=Y,fill=cut(Val, c(0,100,200,300,400,500,Inf))))
p<- p + geom_tile() + scale_fill_brewer(type="seq",palette = "YlGn")
p<- p + guides(fill=guide_legend(title="Legend!"))
#Tight borders
p<- p + scale_x_continuous(expand=c(0,0)) + scale_y_continuous(expand=c(0,0))
p
Note the strategic use of cut to discretize the data followed by the use of color brewer to make things pretty.
The result looks as follows.
The most commonly cited example of how to visualize a logistic fit using ggplot2 seems to be something very much like this:
data("kyphosis", package="rpart")
ggplot(data=kyphosis, aes(x=Age, y = as.numeric(Kyphosis) - 1)) +
geom_point() +
stat_smooth(method="glm", family="binomial")
This visualisation works great if you don't have too much overlapping data, and the first suggestion for crowded data seems to be to use injected jitter in the x and y coordinates of the points then adjust the alpha value of the points. When you get to the point where individual points aren't useful but distributions of points are, is it possible to use geom_density(), geom_histogram(), or something else to visualise the data but continue to split the categorical variable along the y-axis as it is done with geom_point()?
From what I have found, geom_density() and geom_histogram() can easily be split/grouped by the categorical variable and both levels can easily be reversed using scale_y_reverse() but I can't figure out if it is even possible to move only one of the categorical variable distributions to the top of the plot. Any help/suggestions would be appreciated.
The annotate() function in ggplot allows you to add geoms to a plot with properties that "are not mapped from the variables of a data frame, but are instead in as vectors," meaning that you can add layers that are unrelated to your data frame. In this case your two density curves are related to the data frame (since the variables are in it), but because you're trying to position them differently, using annotate() is useful.
Here's one way to go about it:
data("kyphosis", package="rpart")
model.only <- ggplot(data=kyphosis, aes(x=Age, y = as.numeric(Kyphosis) - 1)) +
stat_smooth(method="glm", family="binomial")
absents <- subset(kyphosis, Kyphosis=="absent")
presents <- subset(kyphosis, Kyphosis=="present")
dens.absents <- density(absents$Age)
dens.presents <- density(presents$Age)
scaling.factor <- 10 # Make the density plots taller
model.only + annotate("line", x=dens.absents$x, y=dens.absents$y*scaling.factor) +
annotate("line", x=dens.presents$x, y=dens.presents$y*scaling.factor + 1)
This adds two annotated layers with scaled density plots for each of the kyphosis groups. For the presents variable, y is scaled and increased by 1 to shift it up.
You can also fill the density plots instead of just using a line. Instead of annotate("line"...) you need to use annotate("polygon"...), like so:
model.only + annotate("polygon", x=dens.absents$x, y=dens.absents$y*scaling.factor, fill="red", colour="black", alpha=0.4) +
annotate("polygon", x=dens.presents$x, y=dens.presents$y*scaling.factor + 1, fill="green", colour="black", alpha=0.4)
Technically you could use annotate("density"...), but that won't work when you shift the present plot up by one. Instead of shifting, it fills the whole plot:
model.only + annotate("density", x=dens.absents$x, y=dens.absents$y*scaling.factor, fill="red") +
annotate("density", x=dens.presents$x, y=dens.presents$y*scaling.factor + 1, fill="green")
The only way around that problem is to use a polygon instead of a density geom.
One final variant: flipping the top density plot along y-axis = 1:
model.only + annotate("polygon", x=dens.absents$x, y=dens.absents$y*scaling.factor, fill="red", colour="black", alpha=0.4) +
annotate("polygon", x=dens.presents$x, y=(1 - dens.presents$y*scaling.factor), fill="green", colour="black", alpha=0.4)
I am not sure I get your point, but here an attempt:
dat <- rbind(kyphosis,kyphosis)
dat$grp <- factor(rep(c('smooth','dens'),each = nrow(kyphosis)),
levels = c('smooth','dens'))
ggplot(dat,aes(x=Age)) +
facet_grid(grp~.,scales = "free_y") +
#geom_point(data=subset(dat,grp=='smooth'),aes(y = as.numeric(Kyphosis) - 1)) +
stat_smooth(data=subset(dat,grp=='smooth'),aes(y = as.numeric(Kyphosis) - 1),
method="glm", family="binomial") +
geom_density(data=subset(dat,grp=='dens'))
I have a plot using ggplot, and I would like to add points and error bars to it. I am using geom_errorbar and geom_point, but I am getting an error: "Discrete value supplied to continuous scale" and I am not sure why. The data labels in the plot below should remain the same. I simply want to add new points to the existing graph. The new graph should look like the one below, except with two points/CI bars for each label on the Y axis.
The following example is from the lme4 package, and it produces a plot with confidence intervals using ggplot below (all can be replicated except the last two lines of borken code). My data is only different in that it includes about 15 intercepts instead of 6 below (which is why I am using scale_shape_manual).
The last two lines of code is my attempt at adding points/confidence intervals. I'm going to put a 50 bounty on this. Please let me know if I am being unclear. Thanks!
library("lme4")
data(package = "lme4")
# Dyestuff
# a balanced one-way classiï¬cation of Yield
# from samples produced from six Batches
summary(Dyestuff)
# Batch is an example of a random effect
# Fit 1-way random effects linear model
fit1 <- lmer(Yield ~ 1 + (1|Batch), Dyestuff)
summary(fit1)
coef(fit1) #intercept for each level in Batch
randoms<-ranef(fit1, postVar = TRUE)
qq <- attr(ranef(fit1, postVar = TRUE)[[1]], "postVar")
rand.interc<-randoms$Batch
#THESE ARE THE ADDITIONAL POINTS TO BE ADDED TO THE PLOT
Inter <- c(-25,-45,20,30,23,67)
SE2 <- c(20,20,20,20,20,20)
df<-data.frame(Intercepts=randoms$Batch[,1],
sd.interc=2*sqrt(qq[,,1:length(qq)]), Intercepts2=Inter, sd.iterc2=SE2,
lev.names=rownames(rand.interc))
df$lev.names<-factor(df$lev.names,levels=df$lev.names[order(df$Intercepts)])
library(ggplot2)
p <- ggplot(df,aes(lev.names,Intercepts,shape=lev.names))
#Added horizontal line at y=0
#Includes first set of points/confidence intervals. This works without error
p <- p + geom_hline(yintercept=0) +geom_errorbar(aes(ymin=Intercepts-sd.interc, ymax=Intercepts+sd.interc), width=0,color="black") + geom_point(aes(size=2))
#Removed legends and with scale_shape_manual point shapes set to 1 and 16
p <- p + guides(size=FALSE,shape=FALSE) + scale_shape_manual(values=c(16,16,16,16,16,16))
#Changed appearance of plot (black and white theme) and x and y axis labels
p <- p + theme_bw() + xlab("Levels") + ylab("")
#Final adjustments of plot
p <- p + theme(axis.text.x=element_text(size=rel(1.2)),
axis.title.x=element_text(size=rel(1.3)),
axis.text.y=element_text(size=rel(1.2)),
panel.grid.minor=element_blank(),
panel.grid.major.x=element_blank())
#To put levels on y axis you just need to use coord_flip()
p <- p+ coord_flip()
print(p)
#####
# code for adding more plots, NOT working yet
p <- p +geom_errorbar(aes(ymin=Intercepts2-sd.interc2, ymax=Intercepts2+sd.interc2),
width=0,color="gray40", lty=1, size=1)
p <- p + geom_point(aes(Intercepts2, lev.names),size=0,pch=7)
First, in your data frame df and geom_errorbar() there are two different variables sd.iterc2 and sd.interc2. Changed also in df to sd.interc2.
For the last line of geom_point() you get the error because your x and y values are in wrong order. As your are using coord_flip() then x and y values should be placed in the same order as in original plot before coord_flip(), that is, lev.names as x, and Intercepts2 as y. Changed also size= to 5 for better illustration.
+ geom_point(aes(lev.names,Intercepts2),size=5,pch=7)
Update - adding legend
To add legend for the points of intercept types, one option is to reshape your data to long format and add new column with intercept types. Other option with your existing data is, first, remove shape=lev.names from ggplot() call. Then in both geom_point() calls add shape="somename" inside aes(). Then with scale_shape_manual() set shape values you need.
ggplot(df,aes(lev.names,Intercepts))+
geom_hline(yintercept=0) +
geom_errorbar(aes(ymin=Intercepts-sd.interc, ymax=Intercepts+sd.interc), width=0,color="black")+
geom_point(aes(shape="Intercepts"),size=5)+
theme_bw() + xlab("Levels") + ylab("")+
theme(axis.text.x=element_text(size=rel(1.2)),
axis.title.x=element_text(size=rel(1.3)),
axis.text.y=element_text(size=rel(1.2)),
panel.grid.minor=element_blank(),
panel.grid.major.x=element_blank())+
coord_flip()+
geom_errorbar(aes(ymin=Intercepts2-sd.interc2, ymax=Intercepts2+sd.interc2),
width=0,color="gray40", lty=1, size=1) +
geom_point(aes(lev.names,Intercepts2,shape="Intercepts2"),size=5)+
scale_shape_manual(values=c(16,7))