Related
I would like to plot the line and the 95% confidence interval from a linear model where the response has been logit transformed back on the original scale of the data. So the result should be a curved line including the confidence intervals on the original scale, where it would be a straight line on the logit transformed scale. See code:
# Data
dat <- data.frame(c(45,75,14,45,45,55,65,15,3,85),
c(.37, .45, .24, .16, .46, .89, .16, .24, .23, .49))
colnames(dat) <- c("age", "bil.")
# Logit transformation
dat$bb_logit <- log(dat$bil./(1-dat$bil.))
# Model
modelbb <- lm(bb_logit ~ age + I(age^2), data=dat)
summary(modelbb)
# Backtranform
dat$bb_back <- exp(predict.lm(modelbb))/ (1 + exp(predict.lm(modelbb)))
# Plot
plot(dat$age, dat$bb_back)
abline(modelbb)
What do I try here is to plot the curved regression line and add the confidence interval. Within ggplot2 there is the geom_smooth function where the the linear model can be specified, but I could not find a way of plotting the predictions from the predict.lm(my model).
I would also like to know how to add a coloured polygon which will represent the confidence interval as in the image below. I know I have to use function polygon and coordinates but I do not know how.
You may use predict on an age range say 1:100, specify interval= option for the CIs. Plotting with type="l" will smooth a nice curve. Confidence intervals then can be added using lines.
p <- predict(modelbb, data.frame(age=1:100), interval="confidence")
# Backtransform
p.tr <- exp(p) / (1 + exp(p))
plot(1:100, p.tr[,1], type="l", ylim=range(p.tr), xlab="age", ylab="bil.")
sapply(2:3, function(i) lines(1:100, p.tr[,i], lty=2))
legend("topleft", legend=c("fit", "95%-CI"), lty=1:2)
Yields
Edit
To get shaded confidence bands use polygon. Since you want two confidence levels you probably need to make one prediction for each. The line will get covered by the polygons, so it's better to make an empty plot first using type="n" and draw the lines at the end. (Note that I'll also show you some hints for custom axis labeling.) The trick for the polygons is to express the values back and forth using rev.
p.95 <- predict(modelbb, data.frame(age=1:100), interval="confidence", level=.95)
p.99 <- predict(modelbb, data.frame(age=1:100), interval="confidence", level=.99)
# Backtransform
p.95.tr <- exp(p.95) / (1 + exp(p.95))
p.99.tr <- exp(p.99) / (1 + exp(p.99))
plot(1:100, p.99.tr[,1], type="n", ylim=range(p.99.tr), xlab="Age", ylab="",
main="", yaxt="n")
mtext("Tree biomass production", 3, .5)
mtext("a", 2, 2, at=1.17, xpd=TRUE, las=2, cex=3)
axis(2, (1:5)*.2, labels=FALSE)
mtext((1:5)*2, 2, 1, at=(1:5)*.2, las=2)
mtext(bquote(Production ~(kg~m^-2~year^-1)), 2, 2)
# CIs
polygon(c(1:100, 100:1), c(p.99.tr[,2], rev(p.99.tr[,3])), col=rgb(.5, 1, .2),
border=NA)
polygon(c(1:100, 100:1), c(p.95.tr[,2], rev(p.95.tr[,3])), col=rgb(0, .8, .5),
border=NA)
# fit
lines(1:100, p.99.tr[,1], ylim=range(p.99.tr), lwd=2)
#legend
legend("topleft", legend=c("fit", "99%-CI", "95%-CI"), lty=c(1, NA, NA), lwd=2,
pch=c(NA, 15, 15), bty="n",
col=c("#000000", rgb(.5, 1, .2), rgb(0, .8, .5)))
Yields
I have the following data:
I plotted the points of that data and then smoothed it on the plot using the following code :
scatter.smooth(x=1:length(Ticker$ROIC[!is.na(Ticker$ROIC)]),
y=Ticker$ROIC[!is.na(Ticker$ROIC)],col = "#AAAAAA",
ylab = "ROIC Values", xlab = "Quarters since Feb 29th 2012 till Dec 31st 2016")
Now I want to find the Point-wise slope of this smoothed curve. Also fit a trend line to the smoothed graph. How can I do that?
There are some interesting R packages that implement nonparametric derivative estimation. The short review of Newell and Einbeck can be helpful: http://maths.dur.ac.uk/~dma0je/Papers/newell_einbeck_iwsm07.pdf
Here we consider an example based on the pspline package (smoothing splines with penalties on order m derivatives):
The data generating process is a negative logistic models with an additive noise (hence y values are all negative like the ROIC variable of #ForeverLearner) :
set.seed(1234)
x <- sort(runif(200, min=-5, max=5))
y = -1/(1+exp(-x))-1+0.1*rnorm(200)
We start plotting the nonparametric estimation of the curve (the black line is the true curve and the red one the estimated curve):
library(pspline)
pspl <- smooth.Pspline(x, y, df=5, method=3)
f0 <- predict(pspl, x, nderiv=0)
Then, we estimate the first derivative of the curve:
f1 <- predict(pspl, x, nderiv=1)
curve(-exp(-x)/(1+exp(-x))^2,-5,5, lwd=2, ylim=c(-.3,0))
lines(x, f1, lwd=3, lty=2, col="red")
And here the second derivative:
f2 <- predict(pspl, x, nderiv=2)
curve((exp(-x))/(1+exp(-x))^2-2*exp(-2*x)/(1+exp(-x))^3, -5, 5,
lwd=2, ylim=c(-.15,.15), ylab=)
lines(x, f2, lwd=3, lty=2, col="red")
#DATA
set.seed(42)
x = rnorm(20)
y = rnorm(20)
#Plot the points
plot(x, y, type = "p")
#Obtain points for the smooth curve
temp = loess.smooth(x, y, evaluation = 50) #Use higher evaluation for more points
#Plot smooth curve
lines(temp$x, temp$y, lwd = 2)
#Obtain slope of the smooth curve
slopes = diff(temp$y)/diff(temp$x)
#Add a trend line
abline(lm(y~x))
Does anyone know why the t-distribution in the histogram overlay is just an horizontal line? The warnings() in fit.std result from the etimation of the dof, which can lead to an infinite likelihood - see Fernandez & Steel (1999).
library(zoo)
library(rugarch)
data(sp500ret)
g= zoo(sp500ret$SP500RET, as.Date(rownames(sp500ret)))
(fit.std = fitdistr(g,"t"))
mu.std = fit.std$estimate[["m"]]
lambda = fit.std$estimate[["s"]]
nu = fit.std$estimate[["df"]]
# plot
hist(g, density=20, breaks=20, prob=T)
curve(dt(x, nu, lambda), col="red", lwd=2, add=TRUE, yaxt="n")
From the help file for fitdistr:
For the "t" named distribution the density is taken to be the location-scale family with location m and scale s.
For a location-scale family if we have a location parameter m and a scale parameter s then we can get the density at 'x' using the standardized version (location = 0, scale = 1, call it f) by using:
f((x-m)/s)/s
So for you we have mu.std is the location parameter and lambda is the scale so we would want to change your line to:
curve(dt((x-mu.std)/lambda, nu)/lambda, col="red", lwd=2, add=TRUE, yaxt="n")
I am using the following code to create a standard normal distribution in R:
x <- seq(-4, 4, length=200)
y <- dnorm(x, mean=0, sd=1)
plot(x, y, type="l", lwd=2)
I need the x-axis to be labeled at the mean and at points three standard deviations above and below the mean. How can I add these labels?
The easiest (but not general) way is to restrict the limits of the x axis. The +/- 1:3 sigma will be labeled as such, and the mean will be labeled as 0 - indicating 0 deviations from the mean.
plot(x,y, type = "l", lwd = 2, xlim = c(-3.5,3.5))
Another option is to use more specific labels:
plot(x,y, type = "l", lwd = 2, axes = FALSE, xlab = "", ylab = "")
axis(1, at = -3:3, labels = c("-3s", "-2s", "-1s", "mean", "1s", "2s", "3s"))
Using the code in this answer, you could skip creating x and just use curve() on the dnorm function:
curve(dnorm, -3.5, 3.5, lwd=2, axes = FALSE, xlab = "", ylab = "")
axis(1, at = -3:3, labels = c("-3s", "-2s", "-1s", "mean", "1s", "2s", "3s"))
But this doesn't use the given code anymore.
If you like hard way of doing something without using R built in function or you want to do this outside R, you can use the following formula.
x<-seq(-4,4,length=200)
s = 1
mu = 0
y <- (1/(s * sqrt(2*pi))) * exp(-((x-mu)^2)/(2*s^2))
plot(x,y, type="l", lwd=2, col = "blue", xlim = c(-3.5,3.5))
An extremely inefficient and unusual, but beautiful solution, which works based on the ideas of Monte Carlo simulation, is this:
simulate many draws (or samples) from a given distribution (say the normal).
plot the density of these draws using rnorm. The rnorm function takes as arguments (A,B,C) and returns a vector of A samples from a normal distribution centered at B, with standard deviation C.
Thus to take a sample of size 50,000 from a standard normal (i.e, a normal with mean 0 and standard deviation 1), and plot its density, we do the following:
x = rnorm(50000,0,1)
plot(density(x))
As the number of draws goes to infinity this will converge in distribution to the normal. To illustrate this, see the image below which shows from left to right and top to bottom 5000,50000,500000, and 5 million samples.
In general case, for example: Normal(2, 1)
f <- function(x) dnorm(x, 2, 1)
plot(f, -1, 5)
This is a very general, f can be defined freely, with any given parameters, for example:
f <- function(x) dbeta(x, 0.1, 0.1)
plot(f, 0, 1)
I particularly love Lattice for this goal. It easily implements graphical information such as specific areas under a curve, the one you usually require when dealing with probabilities problems such as find P(a < X < b) etc.
Please have a look:
library(lattice)
e4a <- seq(-4, 4, length = 10000) # Data to set up out normal
e4b <- dnorm(e4a, 0, 1)
xyplot(e4b ~ e4a, # Lattice xyplot
type = "l",
main = "Plot 2",
panel = function(x,y, ...){
panel.xyplot(x,y, ...)
panel.abline( v = c(0, 1, 1.5), lty = 2) #set z and lines
xx <- c(1, x[x>=1 & x<=1.5], 1.5) #Color area
yy <- c(0, y[x>=1 & x<=1.5], 0)
panel.polygon(xx,yy, ..., col='red')
})
In this example I make the area between z = 1 and z = 1.5 stand out. You can move easily this parameters according to your problem.
Axis labels are automatic.
This is how to write it in functions:
normalCriticalTest <- function(mu, s) {
x <- seq(-4, 4, length=200) # x extends from -4 to 4
y <- (1/(s * sqrt(2*pi))) * exp(-((x-mu)^2)/(2*s^2)) # y follows the formula
of the normal distribution: f(Y)
plot(x,y, type="l", lwd=2, xlim = c(-3.5,3.5))
abline(v = c(-1.96, 1.96), col="red") # draw the graph, with 2.5% surface to
either side of the mean
}
normalCriticalTest(0, 1) # draw a normal distribution with vertical lines.
Final result:
I want to plot several probability distributions side-ways (density on the x-axis, variable on y-axis). Each distribution will be associated with a different category, and I want them side-by-side so that I can compare between them. This is a bit like a box-plot but instead I want an theoretical probability distribution that I will specify giving parameters. So if they were all normal distributions, I would simply provide the mean and std deviation for each. Thanks.
do you mean something like this?
x <- seq(-10, 10, length=100)
normal.dist <- dnorm(x, 0, 2)
f.dist <- df(x, 3, 4)
t.dist <- dt(x, 3)
chi.dist <- dchisq(x,3)
par(mfrow=c(2,2))
plot(x, normal.dist, type='l', lty=1 )
plot(x, f.dist, type='l', lty=1, xlab="x value", col='blue')
plot(x, t.dist, type='l', lty=1, xlab="x value", col='red')
plot(x, chi.dist, type='l', lty=1, xlab="x value", col='green')
see also Roman Luštrik's very helpful link as well as the helfiles (e.g. ?dnorm).
Rotated axis
x <- seq(-10, 10, length=100)
normal.dist <- dnorm(x, 0, 1)
normal.dist2 <- dnorm(x, 0, 2)
normal.dist3 <- dnorm(x, 0, 3)
normal.dist4 <- dnorm(x, 0, 4)
par(mfrow=c(2,2))
plot(normal.dist, x, type='l', lty=1 )
plot(normal.dist2, x, type='l', lty=1, col='red' )
plot(normal.dist3, x, type='l', lty=1, col='green' )
plot(normal.dist4, x, type='l', lty=1, col='blue' )
You can set up a frame for plot display and specify how many plots you want to show in a frame using par(mfrow()), for example:
par(mfrow=c(2,2))
plot(first plot)
plot(second plot)
hist(third histogram)
boxplot(fourth boxplot)
See the following link for the full description:
http://www.statmethods.net/advgraphs/layout.html