I have the following data:
I plotted the points of that data and then smoothed it on the plot using the following code :
scatter.smooth(x=1:length(Ticker$ROIC[!is.na(Ticker$ROIC)]),
y=Ticker$ROIC[!is.na(Ticker$ROIC)],col = "#AAAAAA",
ylab = "ROIC Values", xlab = "Quarters since Feb 29th 2012 till Dec 31st 2016")
Now I want to find the Point-wise slope of this smoothed curve. Also fit a trend line to the smoothed graph. How can I do that?
There are some interesting R packages that implement nonparametric derivative estimation. The short review of Newell and Einbeck can be helpful: http://maths.dur.ac.uk/~dma0je/Papers/newell_einbeck_iwsm07.pdf
Here we consider an example based on the pspline package (smoothing splines with penalties on order m derivatives):
The data generating process is a negative logistic models with an additive noise (hence y values are all negative like the ROIC variable of #ForeverLearner) :
set.seed(1234)
x <- sort(runif(200, min=-5, max=5))
y = -1/(1+exp(-x))-1+0.1*rnorm(200)
We start plotting the nonparametric estimation of the curve (the black line is the true curve and the red one the estimated curve):
library(pspline)
pspl <- smooth.Pspline(x, y, df=5, method=3)
f0 <- predict(pspl, x, nderiv=0)
Then, we estimate the first derivative of the curve:
f1 <- predict(pspl, x, nderiv=1)
curve(-exp(-x)/(1+exp(-x))^2,-5,5, lwd=2, ylim=c(-.3,0))
lines(x, f1, lwd=3, lty=2, col="red")
And here the second derivative:
f2 <- predict(pspl, x, nderiv=2)
curve((exp(-x))/(1+exp(-x))^2-2*exp(-2*x)/(1+exp(-x))^3, -5, 5,
lwd=2, ylim=c(-.15,.15), ylab=)
lines(x, f2, lwd=3, lty=2, col="red")
#DATA
set.seed(42)
x = rnorm(20)
y = rnorm(20)
#Plot the points
plot(x, y, type = "p")
#Obtain points for the smooth curve
temp = loess.smooth(x, y, evaluation = 50) #Use higher evaluation for more points
#Plot smooth curve
lines(temp$x, temp$y, lwd = 2)
#Obtain slope of the smooth curve
slopes = diff(temp$y)/diff(temp$x)
#Add a trend line
abline(lm(y~x))
Related
I want to plot regression lines in R for technical analysis.
First, I regress the price on the date and I get the main regression line. However, also, I need lines that correspond to (Main regression line +- 2*standard deviation).
Do you know how I can implement this? I already checked the TTR package, but I couldn't find a built-in indicator for this purpose.
Thank you.
To obtain points on the regression line, you can use the function predict on the fitted model. For confidence intervals, use the options interval and level, e.g.:
lsq <- lm(y ~ x, data)
predict(lsq, data.frame(x=c(12,45), interval="confidence", level=0.95)
To expand on #cdalitz answer this is how you plot the regression line with the confidence interval:
# Generate data
set.seed(123)
n = 100
x = runif(n)
y = 2 * x + rnorm(n, sd = 0.5)
m = lm(y ~ x)
newx = seq(min(x), max(x), length.out = 100)
pred = predict(m, newdata = data.frame(x = newx), interval="confidence", level=0.95)
# Plot data
plot(x, y)
# Plot model
abline(m)
# Plot 95% confidence interval
lines(newx, pred[, 2], col = "red", lty = 2)
lines(newx, pred[, 3], col = "red", lty = 2)
This question also shows many ways to do the same thing.
The question is that is this the correct way to specify the knots in the smoothing spline in gam in mgcv?
The confusion part is that in the vignette, it says the k is the dimension of the basis used to represent the smooth term.
(Previously I thought that in the "cr" setting, the dimension of the basis is 3. After reading p. 149-150 (GAM, an introduction to R), it seems that the gam uses a set of k basis to write the cubic regression splines.)
However, in the post below, it shows that k is actually the number of knots. This is verified by the code below
# reference
# https://stackoverflow.com/questions/40056566/mgcv-how-to-set-number-and-or-locations-of-knots-for-splines
library(mgcv)
## toy data
set.seed(0); x <- sort(rnorm(400, 0, pi)) ## note, my x are not uniformly sampled
set.seed(1); e <- rnorm(400, 0, 0.4)
y0 <- sin(x) + 0.2 * x + cos(abs(x))
y <- y0 + e
## fitting natural cubic spline
cr_fit <- gam(y ~ s(x, bs = 'cr', k = 20))
cr_knots <- cr_fit$smooth[[1]]$xp ## extract knots locations
par(mfrow = c(1,2))
plot(x, y, col= "blue", main = "natural cubic spline");
lines(x, cr_fit$linear.predictors, col = 2, lwd = 2)
abline(v = cr_knots, lty = 2)
Then, to use the smoothing spline, should I assign the knots manually in the argument of gam? The attempted code is below:
## fitting natural cubic spline, smoothing spline
cr_fit <- gam(y ~ s(x, bs = 'cr', k = length(x)), knots=list(x))
cr_knots <- cr_fit$smooth[[1]]$xp ## extract knots locations
## summary plot
par(mfrow = c(1,2))
plot(x, y, col= "blue", main = "natural cubic spline");
lines(x, cr_fit$linear.predictors, col = 2, lwd = 2)
abline(v = cr_knots, lty = 2)
plot(x,cr_knots)
cr_fit$sp
Is this understanding correct?
If yes, then how can I implement the smoothing splines method with the gam in the mgcv?
I perform the linear interpolation and specifies where interpolation is to take place. I get the interpolated data, but I don't know how to get the confidence interval for these interpolated data (red) by linear interpolation.
Is there any other way (loess,splines ) to do interpolation meeting two requirements:
Specifying where interpolation is to take place
getting the interpolated data and confidence interval for interpolated data.
Here is the code that I used.
x <- rnorm(100)
y <- 0.4 * x+ rnorm(100, 0, 1)
ptsLin <- approx(x, y, method="linear", xout=seq(-2,2, 0.1))
plot(x, y, xlab=NA, ylab=NA, pch=19, main="Linear interpolation",
cex=1.5)
points(ptsLin, pch=16, col="red", lwd=1.5)
lines(ptsLin)
legend(x="bottomleft", c("Data", "linear"), pch=c(19, 16), col=c("black", "red"
), bg="white")
Does anyone know why the t-distribution in the histogram overlay is just an horizontal line? The warnings() in fit.std result from the etimation of the dof, which can lead to an infinite likelihood - see Fernandez & Steel (1999).
library(zoo)
library(rugarch)
data(sp500ret)
g= zoo(sp500ret$SP500RET, as.Date(rownames(sp500ret)))
(fit.std = fitdistr(g,"t"))
mu.std = fit.std$estimate[["m"]]
lambda = fit.std$estimate[["s"]]
nu = fit.std$estimate[["df"]]
# plot
hist(g, density=20, breaks=20, prob=T)
curve(dt(x, nu, lambda), col="red", lwd=2, add=TRUE, yaxt="n")
From the help file for fitdistr:
For the "t" named distribution the density is taken to be the location-scale family with location m and scale s.
For a location-scale family if we have a location parameter m and a scale parameter s then we can get the density at 'x' using the standardized version (location = 0, scale = 1, call it f) by using:
f((x-m)/s)/s
So for you we have mu.std is the location parameter and lambda is the scale so we would want to change your line to:
curve(dt((x-mu.std)/lambda, nu)/lambda, col="red", lwd=2, add=TRUE, yaxt="n")
How can I add a poisson regression line to a plot? I tried the following, but the abline function doesn't not work. This is because abline() uses the intercept and slope, whereas a poisson regression line uses a log-link.
x = rpois(12, 5)
plot(x, axes = F)
axis(1,at=1:length(month.name), labels = month.name)
axis(side = 2)
y = c(1:12)
poislm = glm(x~y, family=poisson)
abline(poislm)
How about from R-help
predProbs<-predict(poislm,data.frame(y=seq(min(y), max(y), length.out=100)), type="response")
lines(seq(min(y), max(y), length.out=100), predProbs, col=2, lwd=2)