I am trying to fit a certain truncated distribution to a data set. For example, for a lognormal distribution, I define the density function of the truncated distribution as:
dtlnorm <- function(x, meanlog, sdlog,low)
dlnorm(x,meanlog,sdlog)/(1-plnorm(low,meanlog,sdlog))* (x >= low)
where low is the truncation point.
My data is the following vector
Data <- c(1068295.00589834, 1406446.49289834, 1540330.78489834, 1152321.94489834,
3108649.66189834, 3718417.97089834, 2981945.18089834, 4552923.31989834,
5747260.98289834, 2105461.57989834, 1044515.95889834, 1133641.75289834,
3847920.72789834, 2536441.02989834, 3073854.15789834, 1591039.28389834,
2592446.73289834, 4989152.55189834, 2426457.45489834, 120265066.499898,
6888222046.1999, 1092811.87089834, 3440123.51689834, 74684298.1398983,
1475038.27689834, 1124226.39489834, 11739544.5798983, 1187688.74489834,
1023193.88789834, 18784663.9698983)
To fit the distribution, I write:
fitdist(Data,distr="tlnorm",method="mle",start = list(meanlog=0,sdlog=0),fix.arg = list(low=100))
But the following error appears:
Error in fitdist(Data, distr = "tlnorm", method = "mle", start = list(meanlog = 0, :
the function mle failed to estimate the parameters,
with the error code 100
I do not know what is happening. Can somebody help me? Thank you!
Related
I am trying to fit shifted exponential distribution to my data but fitdist function giving error of 100 and failing to estimate starting values. I also used plotdist function to find starting or initial values in order to fit the distribution and I have obtained the followings plots with parameters rate = 0.155 shift = 0.00001 after iteration process and even I used these values in fitdist as well.
I used mledist function to calculate starting values of distribution parameters as well but it is also not working. I used fitdist function as well it gives the following error:
Error in fitdist(x, "sexp", start = list(rate = 0.155, shift = 1e-05)) :
the function mle failed to estimate the parameters,
with the error code 100
The code is as below:
library(fitdistrplus)
library(readxl)
library(tidyverse)
library(here)
library(janitor)
# Load data-------------------------------
pvr <- read_excel(here("data", "pvr.xlsx"))
pvr <- pvr %>%
select(-starts_with("...")) %>%
clean_names(case = "snake")
x <- pvr$headway
rate <- 0.155
shift <- 0.00001
dsexp <- function(x, rate, shift)
dexp(x-shift, rate=rate)
psexp <- function(x, rate, shift)
pexp(x-shift, rate=rate)
qsexp <- function(x, rate, shift)
qexp(x-shift, rate=rate)
f12 <- fitdist(x, "sexp", start = list(rate=0.155, shift=0.00001), lower = c(0, -min(x)))
The data may be downloaded from the following link below:
https://ptagovsa-my.sharepoint.com/:x:/g/personal/kkhan_tga_gov_sa/EfzCE5h0jexCkVw0Ak2S2_MBWf3WUywMd1izw41r0EsLeQ?e=EiqWDc
Can anyone help me in this regard?
The fitdist function uses mle method as default. The code works just by changing the method from mle to mse or mge.
I would need to calculate the parameters of a Fréchet distribution.
I am using the packages fitdistrplus and evd of R. But I don't know
what values to initialize the parameters.
library(fitdistrplus)
library(evd)
#Datos
x<-c(19.1,20.2,14.3,19.0,18.8,18.5,20.0,18.6,11.4,15.6,17.4,16.2,15.7,14.3,14.9,14.0,20.2,17.4,18.6,17.0,16.0,12.2,10.8,12.4,10.2,19.8,23.4)
fit.frechet<-fitdist(x,"frechet")
fit.frechet<-fitdist(x,"frechet")
generating the following error
Error in computing default starting values.
Error in manageparam(start.arg = start, fix.arg = fix.arg, obs = data, :
Error in start.arg.default(obs, distname) :
Unknown starting values for distribution frechet. `
When starting the parameters:
fit.frechet2<-fitdist(x,"frechet", start = list(loc=0,scale=1, shape=1))
Output:
Warning messages:
1: In fitdist(x, "frechet", start = list(loc = 0, scale = 1, shape = 1)) :
The dfrechet function should return a vector of with NaN values when input has inconsistent parameters and not raise an error
2: In fitdist(x, "frechet", start = list(loc = 0, scale = 1, shape = 1)) :
The pfrechet function should return a vector of with NaN values when input has inconsistent parameters and not raise an error
3: In sqrt(diag(varcovar)) : NaNs produced
4: In sqrt(1/diag(V)) : NaNs produced
5: In cov2cor(varcovar) :
diag(.) had 0 or NA entries; non-finite result is doubtful
Fitting of the distribution ' frechet ' by maximum likelihood
Parameters:
estimate Std. Error
loc -12128345 40.10705
scale 12128360 40.10705
shape 3493998 NaN
How can I estimate the parameters of the frechet in R?
Well, you could try limit your values and start with some reasonable estimates
F.e.
fit.frechet<-fitdist(x, "frechet", method = "mle", lower = c(0, 0, 0), start = list(loc=1,scale=12, shape=4))
will produce couple of expected warnings, and
print(fit.frechet)
will print somewhat reasonable values
loc 2.146861e-07
scale 1.449643e+01
shape 4.533351e+00
with plot of fit vs empirical
plot(fit.frechet,demp=TRUE)
UPDATE
I would say that Frechet might not be a good fit for your data. I tried Weibull and it looks a lot better, check it yourself
fit.weibull<-fitdist(x, "weibull", method = "mle", lower = c(0, 0))
print(fit.weibull)
plot(fit.weibull, demp=TRUE)
Output is
shape 5.865337
scale 17.837188
One could note that scale parameter is kind of similar and could have been guessed just from histogram. Plot for Weibull fit, given the data it looks quite good
I think my data might be described by a sum of Poisson distributions and I found out about mixdist package for R. I managed to fit gamma and lnorm distributions, but I can't figure out how to use Poisson. I tried with the example data first:
library(mixdist)
data(poisdat)
data(poispar)
plot.mixdata(poisdat) #this works
fitp<-mix(poisdat, coef(poispar), "pois") #this doesn't
but I get an error Error in if (usecondit & ncol(mixdat) - 2 != k) stop("Conditional data are not consistent with mixpar.") : argument is of length zero
How to get a working example of mixdist and Poisson?
Are you looking for this:
mix(poisdat, poispar, "pois", constr = mixconstr(consigma = "POIS"))
Simply mix(poisdat, poispar, "pois") gives an Error:
Error in testconstr(mixdat, mixpar, dist, constr) :
Poisson needs consigma = POIS.
thas why this constr parameter.
I'm trying to find the MLE of distribution whose pdf is specified as 'mixture' in the code. I've provided the code below that gives an error of
"Error in optim(start, f, method = method, hessian = TRUE, ...) :
L-BFGS-B needs finite values of 'fn'"
"claims" is the dataset im using. I tried the same code with just the first two values of "claims" and encountered the same problem, so for a reproducible example the first two values are 1536.77007 and 1946.92409.
The limits on the parameters of the distribution is that 0<.p.<1 and a>0 and b>0, hence the lower and upper bounds in the MLE function. Any help is much appreciated.
#create mixture of two exponential distribution
mixture<-function(x,p,a,b){
d<-p*a*exp(-a*x)+(1-p)*b*exp(-b*x)
d
}
#find MLE of mixture distribution
LL <- function(p,a,b) {
X = mixture(claims,p,a,b)
#
-sum(log(X))
}
mle(LL, start = list(p=0.5,a=1/100,b=1/100),method = "L-BFGS-B", lower=c(0,0,0), upper=c(1,Inf,Inf))
edit: Not really sure why dput(), but anyway,
#first two values of claims put into dput() (the actual values are above)
dput(claims[1:2])
c(307522.103, 195633.5205)
I generate some random variables using rplcon() function in package poweRlaw
data <- rplcon(1000,10,2)
Now, I want to know which known distributions fit the data best. Lognorm? exp? gamma? power law? power law with exponential cutoff?
So I use function fitdist() in package fitdistrplus:
fit.lnormdl <- fitdist(data,"lnorm")
fit.gammadl <- fitdist(data, "gamma", lower = c(0, 0))
fit.expdl <- fitdist(data,"exp")
Due to the power law distribution and power law with exponential cutoff are not the base probability function according to CRAN Task View: Probability Distributions, so I write the d,p,q function of power law based on the example 4 of ?fitdist
dplcon <- function (x, xmin, alpha, log = FALSE)
{
if (log) {
pdf = log(alpha - 1) - log(xmin) - alpha * (log(x/xmin))
pdf[x < xmin] = -Inf
}
else {
pdf = (alpha - 1)/xmin * (x/xmin)^(-alpha)
pdf[x < xmin] = 0
}
pdf
}
pplcon <- function (q, xmin, alpha, lower.tail = TRUE)
{
cdf = 1 - (q/xmin)^(-alpha + 1)
if (!lower.tail)
cdf = 1 - cdf
cdf[q < round(xmin)] = 0
cdf
}
qplcon <- function(p,xmin,alpha) alpha*p^(1/(1-xmin))
Finally, I use codes below to get parameter xmin and alpha of power law:
fitpl <- fitdist(data,"plcon",start = list(xmin=1,alpha=1))
But it throws an error:
<simpleError in optim(par = vstart, fn = fnobj, fix.arg = fix.arg, obs = data, ddistnam = ddistname, hessian = TRUE, method = meth, lower = lower, upper = upper, ...): function cannot be evaluated at initial parameters>
Error in fitdist(data, "plcon", start = list(xmin = 1, alpha = 1)) :
the function mle failed to estimate the parameters,
with the error code 100
I try to search in google and stackoverflow, and so many similar error questions appear, but after reading and trying, no solutions work in my issues, what should I do to complete it correctly to get the parameters?
Thank you for everyone who does me a favor!
This was an interesting one that I am not entirely happy with the discovery but I will tell you what I have found and see if it helps.
On calling the fitdist function, by default it wants to use mledist from the same package. This itself results in a call to stats::optim which is a general optimization function. In it's return value it gives a convergence error code, see ?optim for details. The 100 you see is not one of the ones returned by optim. So I pulled apart the code for mledist and fitdist to find where that error code comes from. Unfortunately it is defined in more than one case and is a general trap error code. If you break down all of the code, what fitdist is trying to do here is the following, subject to various checks etc beforehand.
fnobj <- function(par, fix.arg, obs, ddistnam) {
-sum(do.call(ddistnam, c(list(obs), as.list(par),
as.list(fix.arg), log = TRUE)))
}
vstart = list(xmin=5,alpha=5)
fnobj <- function(par, fix.arg obs, ddistnam) {
-sum(do.call(ddistnam, c(list(obs), as.list(par),
as.list(fix.arg), log = TRUE)))
}
ddistname=dplcon
fix.arg = NULL
meth = "Nelder-Mead"
lower = -Inf
upper = Inf
optim(par = vstart, fn = fnobj,
fix.arg = fix.arg, obs = data, ddistnam = ddistname,
hessian = TRUE, method = meth, lower = lower,
upper = upper)
If we run this code we find a more useful error "function cannot be evaluated at initial parameters". Which makes sense if we look at the function definition. Having xmin=0 or alpha=1 will yield a log-likelihood of -Inf. OK so think try different initial values, I tried a few random choices but all returned a new error, "non-finite finite-difference value 1".
Searching the optim source further for the source of these two errors they are not part of the R source itself, there is however a .External2 call so I can only assume the errors come from there. The non-finite error implies that one of the function evaluations somewhere gives a non numeric result. The function dplcon will do so when alpha <= 1 or xmin <= 0. fitdist lets you specify additional arguments that get passed to mledist or other (depending on what method you choose, mle is default) of which lower is one for controlling lower bounds on the parameters to be optimized. So I tried imposing these limits and trying again:
fitpl <- fitdist(data,"plcon",start = list(xmin=1,alpha=2), lower = c(xmin = 0, alpha = 1))
Annoyingly this still gives an error code 100. Tracking this down yields the error "L-BFGS-B needs finite values of 'fn'". The optimization method has changed from the default Nelder-Mead as you specifying the boundary and somewhere on the external C code call this error arises, presumably close to the limits of either xmin or alpha where the stability of the numerical calculation as we approach infinity is important.
I decided to do quantile matching rather than max likelihood to try to find out more
fitpl <- fitdist(data,"plcon",start = list(xmin=1,alpha=2),
method= "qme",probs = c(1/3,2/3))
fitpl
## Fitting of the distribution ' plcon ' by matching quantiles
## Parameters:
## estimate
## xmin 0.02135157
## alpha 46.65914353
which suggests that the optimum value of xmin is close to 0, it's limits. The reason I am not satisfied is that I can't get a maximum-likelihood fit of the distribution using fitdist however hopefully this explanation helps and the quantile matching gives an alternative.
Edit:
After learning a little more about power law distributions in general it makes sense that this does not work as you expect. The parameter power parameter has a likelihood function which can be maximised conditional on a given xmin. However no such expression exists for xmin since the likelihood function is increasing in xmin. Typically estimation of xmin comes from a Kolmogorov--Smirnov statistic, see this mathoverflow question and the d_jss_paper vignette of the poweRlaw package for more info and associated references.
There is functionality to estimate the parameters of the power law distribution in the poweRlaw package itself.
m = conpl$new(data)
xminhat = estimate_xmin(m)$xmin
m$setXmin(xminhat)
alphahat = estimate_pars(m)$pars
c(xmin = xminhat, alpha = alphahat)