I have an algorithm that uses an x,y plot of sorted y data to produce an ogive.
I then derive the area under the curve to derive %'s.
I'd like to do something similar using kernel density estimation. I like how the upper/lower bounds are smoothed out using kernel densities (i.e. the min and max will extend slightly beyond my hard coded input).
Either way... I was wondering if there is a way to treat an ogive as a type of cumulative distribution function and/or use kernel density estimation to derive a cumulative distribution function given y data?
I apologize if this is a confusing question. I know there is a way to derive a cumulative frequency graph (i.e. ogive). However, I can't determine how to derive a % given this cumulative frequency graph.
What I don't want is an ecdf. I know how to do that, and I am not quite trying to capture an ecdf. But, rather integration of an ogive given two intervals.
I'm not exactly sure what you have in mind, but here's a way to calculate the area under the curve for a kernel density estimate (or more generally for any case where you have the y values at equally spaced x-values (though you can, of course, generalize to variable x intervals as well)):
library(zoo)
# Kernel density estimate
# Set n to higher value to get a finer grid
set.seed(67839)
dens = density(c(rnorm(500,5,2),rnorm(200,20,3)), n=2^5)
# How to extract the x and y values of the density estimate
#dens$y
#dens$x
# x interval
dx = median(diff(dens$x))
# mean height for each pair of y values
h = rollmean(dens$y, 2)
# Area under curve
sum(h*dx) # 1.000943
# Cumulative area
# cumsum(h*dx)
# Plot density, showing points at which density is calculated
plot(dens)
abline(v=dens$x, col="#FF000060", lty="11")
# Plot cumulative area under curve, showing mid-point of each x-interval
plot(dens$x[-length(dens$x)] + 0.5*dx, cumsum(h*dx), type="l")
abline(v=dens$x[-length(dens$x)] + 0.5*dx, col="#FF000060", lty="11")
UPDATE to include ecdf function
To address your comments, look at the two plots below. The first is the empirical cumulative distribution function (ECDF) of the mixture of normal distributions that I used above. Note that the plot of this data looks the same below as it does above. The second is a plot of the ECDF of a plain vanilla normal distribution, mean=0, sd=1.
set.seed(67839)
x = c(rnorm(500,5,2),rnorm(200,20,3))
plot(ecdf(x), do.points=FALSE)
plot(ecdf(rnorm(1000)))
Related
I calculated the cumulative probability (in English, cdf) of my data, based on the probability of exceedance (edf). No problem at all.
However, does anyone know if there is any command to transform this data into probability density (pdf)?
I have already tested using the histogram function, but it does not work correctly.
x <- c (0.00000000, 0.03505324, 0.07005407, 0.10512053, 0.14021308,
0.17533767, 0.21051443, 0.24570116, 0.28090087, 0.31592221,
0.35092739, 0.38591441,0.42085712, 0.45599341, 0.49119521, 0.52646341,
0.56159558, 0.59673546, 0.63172464, 0.66674853, 0.70177413, 0.73712542,
0.77225123, 0.80750715, 0.84250460, 0.87720473, 0.91172191, 0.94588810,
0.98056348)
Is the function you are looking for density() ?
You can plot with
plot(density(x))
You can see x and y values with:
density(x)$x
density(x)$y
uppose that i have a poisson distribution with mean of 6 i would like to plot a probability mass function which includes an overlay of the approximating normal density.
This is what i have tried
plot( dpois( x=0:10, lambda=6 ))
this produces
which is wrong since it doesnt contain an overlay of approxiamating noral density
How do i go about this?
Something like what you seem to be asking for (I'm outlining the commands and the basic ideas, but checking the help on the functions and trying should fill in the remaining details):
taking a wider range of x-values (out to at least 13 or so) and use xlim to extend the plot slightly into the negatives (maybe to -1.5) and
plotting the pmf of the Poisson with solid dots (similar to your command but with pch=16 as an argument to plot) with a suitable color, then
call points with the same x and y arguments as above and have type=h and lty=3 to get vertical dotted lines (to give a clear impression of the relative heights, somewhat akin to the appearance of a Cleveland dot-chart); I'd use the same colour as the dots or a slightly lighter/greyer version of the dot-colour
use curve to draw the normal curve with the same mean and standard deviation as the Poisson with mean 6 (see details at the Wikipedia page for the Poisson which gives the mean and variance), but across the wider range we plotted; I'd use a slightly contrasting colour for that.
I'd draw a light x-axis in (e.g. using abline with the h argument)
Putting all those suggestions together:
(However, while it's what you're asking for it's not strictly a suitable way to compare discrete and continuous variables since density and pmf are not on the same scale, since density is not probability -- the "right" comparison between a Poisson and an approximating normal would be on the scale of the cdfs so you compare like with like -- they'd both be on the scale of probabilities then)
I have a multi-parameter function on which I infer the parameters using MCMC. This means that I have many samples of the parameters, and I can plot the functions:
# Simulate some parameters. Really, I get these from MCMC sampling.
first = rnorm(1000) # a
second = rnorm(1000) # b
# The function (geometric)
geometric = function(x, a, b) b*(1 - a^(x + 1)/a)
# Plot curves. Perhaps not the most efficient way, but it works.
curve(geometric(x, first[1], second[1]), ylim=c(-3, 3)) # first curve
for(i in 2:length(first)) {
curve(geometric(x, first[i], second[i]), add=T, col='#00000030') # add others
}
How do I make this into a density plot instead of plotting the individual curves? For example, it's hard to see just how much denser it is around y=0 than around other values.
The following would be nice:
The ability to draw observed values on top (points and lines).
Drawing a contour line in the density, e.g. the 95% Highest Posterior Density interval or the 2.5 and 97.5 quantiles.
I would like to create a Student's t distribution density plot with a mean of 0.02 instead of 0. is that possible to do?
the distribtion should have 2 degrees of freedom.
is this possible to do?
I tried the following:
X<-rnorm(100000,mean=0.02, sd=(1/sqrt(878)))
pop.mean<-mean(X)
t<-sapply(1:10000, function(x) (mean(sample(X,100))-pop.mean)/(1/sqrt(878)))
plot(density(t))
Is this approach correct?
If it is correct, how can I get the real densities, not just the approximation?
Your statement and example contradict each other somewhat.
Do you want a non-central t distribution which is based on a normal with mean 0.02? This is what your example suggests, but note that the non-central t is not just a shifted t, it is now skewed.
If you want the non-central t then you can plot it with a command like:
curve(dt(x,2,0.02), from=-5, to=6)
Or, do you want a shifted t distribution? A distribution that is symmetric around 0.02 with the shape of a t distribution?
You can plot the curve shifted by using a command like:
curve(dt(x-0.02,2), from=-5, to=6 )
The curve function has an add argument that you could use to plot both on the same plot if you want to compare them (not much difference in this case), changing the color on one of them would be suggested.
I have a question about the kde2d (Kernel density estimator). I am computing two different kde2d for two different sets of data in the same space of variables. When I compare both with a filled.contour2 or contours I see that the set with lower density of points in a scatter plot(Also has less points in the total with a factor of 10) has an higher density for the contours values. I was expecting that the set with higher point density will have higher density contours values, but like I said above is not the case. It has to be with the choice of bandwidth (h)? I am using equals h, and i tried to change but the result did not changed a lot. What is my error?
An example
a <- runif(1000, 5.0, 7.5)
b <- runif(1000, 2.0, 3.0)
c <- runif(100000,5.0, 7.5)
d <- runif(100000, 2.0, 3.0)
library(MASS)
abdens <- kde2d(a,b,n=100,h=0.5)
cddens <- kde2d(c,d,n=100,h=0.5)
mylevels <- seq(-2.4,30,0.9)
filled.contour2(abdens,xlab="a",ylab="b",xlim=c(5,7.5),ylim=c(2,3),
col=hsv(seq(0,1,length=length(mylevels))))
plot(a,b)
contour(abdens,nlevels=5,add=T,col="blue")
plot(c,d)
contour(cddens,nlevels=5,add=T,col="orange")
I'm not sure I agree that the densities should be different in the uniform case. I would have expected a set with a higher number of randomly drawn points from a Normal distribution to have more support for extreme regions and therefore to have lower (estimated) density in the center. That effect might be also be occasionally apparent with bibariate Uniform draws with 1,000 points versus 100,000. I hope my modifications to your code are acceptable. It's easier to see the contours if done after the plots.
(The theoretic density would be the same in both these cases since the density distribution is normalized to an integral of 1.0. We are only looking at estimates with some expected artifacts from "edge" effects. In the case of univariate densities adding the information about bounds can be done with the desity functions in package::logspline.)