EDITED: This is my first time posting and also using Julia so please forgive any mistakes I make in either and feel free to offer criticism where due.
1) I am getting an error when I make a call like
fit!(glmm(total_crashes ~ job_density + pop_density + (1|f), dat, Poisson()))
for any of the distributions. The error is:
ERROR: DimensionMismatch("length(wts) must be 0 or length(y)")
in MixedModels.LinearMixedModel{T<:AbstractFloat}(::DataFrames.Formula, ::Array{String,1}, ::Array{Any,1}, ::Array{Union{MixedModels.Identity{Float64},MixedModels.MaskedLowerTri{Float64},UniformScaling{Float64}},1}, ::Array{Float64,1}) at /Users/ilan/.julia/v0.5/MixedModels/src/pls.jl:51
in #lmm#30(::Array{Float64,1}, ::Function, ::DataFrames.Formula, ::DataFrames.DataFrame) at /Users/ilan/.julia/v0.5/MixedModels/src/pls.jl:136
in (::MixedModels.#kw##lmm)(::Array{Any,1}, ::MixedModels.#lmm, ::DataFrames.Formula, ::DataFrames.DataFrame) at ./<missing>:0
in #glmm#63(::Array{Any,1}, ::Array{Any,1}, ::Function, ::DataFrames.Formula, ::DataFrames.DataFrame, ::Distributions.Poisson{Float64}, ::GLM.LogLink) at /Users/ilan/.julia/v0.5/MixedModels/src/PIRLS.jl:47
in glmm(::DataFrames.Formula, ::DataFrames.DataFrame, ::Distributions.Poisson{Float64}) at /Users/ilan/.julia/v0.5/MixedModels/src/PIRLS.jl:64
However the documentation seems to say that weights defaults to 0. What am I missing here?
2) In any case, the distribution I'm really looking for is Negative Binomial - does MixedModels support that yet and if so, what is the right code for it?
Thanks!
EDIT: I'm ok on the first part, actually I'm really just curious if this package has Negative Binomial - I haven't been able to find anything to suggest it does.
Related
Right upfront: this is an issue I encountered when submitting an R package to CRAN. So I
dont have control of the stack size (as the issue occured on one of CRANs platforms)
I cant provide a reproducible example (as I dont know the exact configurations on CRAN)
Problem
When trying to submit the cSEM.DGP package to CRAN the automatic pretest (for Debian x86_64-pc-linux-gnu; not for Windows!) failed with the NOTE: C stack usage 7975520 is too close to the limit.
I know this is caused by a function with three arguments whose body is about 800 rows long. The function body consists of additions and multiplications of these arguments. It is the function varzeta6() which you find here (from row 647 onwards).
How can I adress this?
Things I cant do:
provide a reproducible example (at least I would not know how)
change the stack size
Things I am thinking of:
try to break the function into smaller pieces. But I dont know how to best do that.
somehow precompile? the function (to be honest, I am just guessing) so CRAN doesnt complain?
Let me know your ideas!
Details / Background
The reason why varzeta6() (and varzeta4() / varzeta5() and even more so varzeta7()) are so long and R-inefficient is that they are essentially copy-pasted from mathematica (after simplifying the mathematica code as good as possible and adapting it to be valid R code). Hence, the code is by no means R-optimized (which #MauritsEvers righly pointed out).
Why do we need mathematica? Because what we need is the general form for the model-implied construct correlation matrix of a recursive strucutral equation model with up to 8 constructs as a function of the parameters of the model equations. In addition there are constraints.
To get a feel for the problem, lets take a system of two equations that can be solved recursivly:
Y2 = beta1*Y1 + zeta1
Y3 = beta2*Y1 + beta3*Y2 + zeta2
What we are interested in is the covariances: E(Y1*Y2), E(Y1*Y3), and E(Y2*Y3) as a function of beta1, beta2, beta3 under the constraint that
E(Y1) = E(Y2) = E(Y3) = 0,
E(Y1^2) = E(Y2^2) = E(Y3^3) = 1
E(Yi*zeta_j) = 0 (with i = 1, 2, 3 and j = 1, 2)
For such a simple model, this is rather trivial:
E(Y1*Y2) = E(Y1*(beta1*Y1 + zeta1) = beta1*E(Y1^2) + E(Y1*zeta1) = beta1
E(Y1*Y3) = E(Y1*(beta2*Y1 + beta3*(beta1*Y1 + zeta1) + zeta2) = beta2 + beta3*beta1
E(Y2*Y3) = ...
But you see how quickly this gets messy when you add Y4, Y5, until Y8.
In general the model-implied construct correlation matrix can be written as (the expression actually looks more complicated because we also allow for up to 5 exgenous constructs as well. This is why varzeta1() already looks complicated. But ignore this for now.):
V(Y) = (I - B)^-1 V(zeta)(I - B)'^-1
where I is the identity matrix and B a lower triangular matrix of model parameters (the betas). V(zeta) is a diagonal matrix. The functions varzeta1(), varzeta2(), ..., varzeta7() compute the main diagonal elements. Since we constrain Var(Yi) to always be 1, the variances of the zetas follow. Take for example the equation Var(Y2) = beta1^2*Var(Y1) + Var(zeta1) --> Var(zeta1) = 1 - beta1^2. This looks simple here, but is becomes extremly complicated when we take the variance of, say, the 6th equation in such a chain of recursive equations because Var(zeta6) depends on all previous covariances betwenn Y1, ..., Y5 which are themselves dependend on their respective previous covariances.
Ok I dont know if that makes things any clearer. Here are the main point:
The code for varzeta1(), ..., varzeta7() is copy pasted from mathematica and hence not R-optimized.
Mathematica is required because, as far as I know, R cannot handle symbolic calculations.
I could R-optimze "by hand" (which is extremly tedious)
I think the structure of the varzetaX() must be taken as given. The question therefore is: can I somehow use this function anyway?
Once conceivable approach is to try to convince the CRAN maintainers that there's no easy way for you to fix the problem. This is a NOTE, not a WARNING; The CRAN repository policy says
In principle, packages must pass R CMD check without warnings or significant notes to be admitted to the main CRAN package area. If there are warnings or notes you cannot eliminate (for example because you believe them to be spurious) send an explanatory note as part of your covering email, or as a comment on the submission form
So, you could take a chance that your well-reasoned explanation (in the comments field on the submission form) will convince the CRAN maintainers. In the long run it would be best to find a way to simplify the computations, but it might not be necessary to do it before submission to CRAN.
This is a bit too long as a comment, but hopefully this will give you some ideas for optimising the code for the varzeta* functions; or at the very least, it might give you some food for thought.
There are a few things that confuse me:
All varzeta* functions have arguments beta, gamma and phi, which seem to be matrices. However, in varzeta1 you don't use beta, yet beta is the first function argument.
I struggle to link the details you give at the bottom of your post with the code for the varzeta* functions. You don't explain where the gamma and phi matrices come from, nor what they denote. Furthermore, seeing that beta are the model's parameter etimates, I don't understand why beta should be a matrix.
As I mentioned in my earlier comment, I would be very surprised if these expressions cannot be simplified. R can do a lot of matrix operations quite comfortably, there shouldn't really be a need to pre-calculate individual terms.
For example, you can use crossprod and tcrossprod to calculate cross products, and %*% implements matrix multiplication.
Secondly, a lot of mathematical operations in R are vectorised. I already mentioned that you can simplify
1 - gamma[1,1]^2 - gamma[1,2]^2 - gamma[1,3]^2 - gamma[1,4]^2 - gamma[1,5]^2
as
1 - sum(gamma[1, ]^2)
since the ^ operator is vectorised.
Perhaps more fundamentally, this seems somewhat of an XY problem to me where it might help to take a step back. Not knowing the full details of what you're trying to model (as I said, I can't link the details you give to the cSEM.DGP code), I would start by exploring how to solve the recursive SEM in R. I don't really see the need for Mathematica here. As I said earlier, matrix operations are very standard in R; analytically solving a set of recursive equations is also possible in R. Since you seem to come from the Mathematica realm, it might be good to discuss this with a local R coding expert.
If you must use those scary varzeta* functions (and I really doubt that), an option may be to rewrite them in C++ and then compile them with Rcpp to turn them into R functions. Perhaps that will avoid the C stack usage limit?
UPDATE: For anyone else who visits this page, it is worth having a look at this SO question and answer as I suspect the solution there is relevant to the problem I was having here.
This question duplicates one I have asked on the julia-users mailing list, but I haven't gotten a response there (admittedly it has only been 4 days), so thought I would ask here.
I am calling the NLopt API from Julia, although I think my question is independent of the Julia language.
I am trying to solve an optimisation problem using COBYLA but in many cases I am failing to trigger the stopping criteria. My problem is reasonably complicated, but I can reproduce the problem behaviour with a much simpler example.
Specifically, I try to minimize x1^2 + x2^2 + 1 using COBYLA, and I set both ftol_rel and ftol_abs to 0.5. My objective function includes a statement to print the current value to the console, so I can watch the convergence. The final five values printed to the console during convergence are:
1.161
1.074
1.004
1.017
1.038
My understanding is that any of these steps should have triggered the stopping criteria. All steps are less than 0.5, so that should trigger ftol_abs. Further, each value is approximately 1, and 0.5*1 = 0.5, so all steps should also have triggered ftol_rel. In fact, this behaviour is true of the last 8 steps in the convergence routine.
NLopt has been around for a while, so I'm guessing the problem is with my understanding of how ftol_abs and ftol_rel work, rather than being a bug.
Can anyone shed any light on why the stopping criteria are not being triggered much earlier?
If it is of any use, the following Julia code snippet can be used to reproduce everything I have just stated:
using NLopt
function objective_function(param::Vector{Float64}, grad::Vector{Float64})
obj_func_value = param[1]^2 + param[2]^2 + 1.0
println("Objective func value = " * string(obj_func_value))
println("Parameter value = " * string(param))
return(obj_func_value)
end
opt1 = Opt(:LN_COBYLA, 2)
lower_bounds!(opt1, [-10.0, -10.0])
upper_bounds!(opt1, [10.0, 10.0])
ftol_rel!(opt1, 0.5)
ftol_abs!(opt1, 0.5)
min_objective!(opt1, objective_function)
(fObjOpt, paramOpt, flag) = optimize(opt1, [9.0, 9.0])
Presumably, ftol_rel and ftol_abs are supposed to supply numerically guaranteed errors. The earlier values are close enough but the algorithm might not be able to guarantee it. For example, the gradient or Hessian at the evaluation point might supply such a numerical guarantee. So, it continues a bit further.
To be sure, it's best to look at the optimization algorithm source. If I manage this, I will add it to this answer.
Update: The COBYLA algorithm approximates the gradient (vector derivative) numerically using several evaluation points. As mentioned, this is used to model what the error might be. The errors can really be mathematically guaranteed only for functions restricted to some nice family (e.g. polynomials with some bound on degree).
Take home message: It's OK. Not a bug, but the best the algorithm can do. Let it have those extra iterations.
When I input any integral, say sin[x] or ln[x], all I get back is the input. And from what I understand that usually means that mathematica can't evaluate the integral...What am i missing here?
I'm new to stackoverflow so I can't post screenshots.
Read the documentation:
Sin[x] manual
Log[x] manual
Note also that by default Log function is of base e, therefore it is your Ln.
In addition, if you have further questions mathematica/wolfram has it's own subsite where you should ask related questions https://mathematica.stackexchange.com/
As of version 9 you can query wolfram alpha directly and it can even try to interpret human language (and fix syntax mistakes like this). To query wolfram alpha start line with =.
I'm self learning R and I keep seeing the mention of Integrals or calculating Integration in R when trawling through google examples. The three books I'm using (Discovering Statistics using R, R's Cookbook, Beginning R) have no mention of calculating these.
I found a couple of questions on here but they're not the same format/direction as this question. Also, this question didn't provide any information either.
So my query is, how can you calculate one:
I guess h(x) is a uniform in the unit interval?
Given that I have only found "text" and no actual examples of this implementation, I may just dismiss it and move on if you guys can't verify anything. Thanks in advance.
I'm not quite sure whether you're asking the very straightforward question here or not, but
f <- function(x) x*(x^5-1)
integrate(f,0,1)
## -0.3571429 with absolute error < 4e-15
seems to work fine. Checking analytically:
f_indef <- function(x) x^7/7-x^2/2
f_indef(1)-f_indef(0)
## [1] -0.3571429
or for the lazy:
library(Ryacas)
(intres <- yacas("Integrate (x) x*(x^5-1)",addSemi=FALSE))
## x^7/7-x^2/2;
I can't get Eval.yacas to give me anything but NULL, though ...
As said in the help(predict.nls), when se.fit=TRUE,the standard errors of the predictions should be calculated. However, my codes in the following do not display that, but only the predictions.
alloy <- data.frame(x=c(10,30,51,101,203,405,608,810,1013,2026,4052,6078,
8104,10130),
y=c(0.3561333,0.3453,0.3355,0.327453,0.3065299,0.2839316,
0.2675214,0.2552821,0.2455726,0.2264957,0.2049573,
0.1886496,0.1755897,0.1651624))
model <- nls(y ~ a * x^(-b), data=alloy, start=list(a=.5, b=.1))
predict(model, se=TRUE)
What's wrong with my codes? Thank you!
Nothing. The last line of the Description section of ?predict.nls says:
At present ‘se.fit’ and ‘interval’ are ignored.
Perhaps the deltaMethod function in the car package would help. (Results of library("sos"); findFn("{delta method} nls") ...)
update: I didn't get it work with car::deltaMethod on my first try. Here's an attempt to use the deltavar function from the emdbook package. The attach()/detach() stuff is a horrible hack, but it's what I could get to work quickly (a with-based solution didn't work because of non-standard evaluation). Improvements welcome.)
attach(alloy)
> deltavar(a*x^(-b),meanval=coef(model),Sigma=vcov(model))
[1] 1.445018e-04 6.956934e-05 4.897363e-05 3.287949e-05 2.527709e-05
[6] 2.379668e-05 2.487129e-05 2.626029e-05 2.762930e-05 3.300235e-05
[11] 3.933191e-05 4.317493e-05 4.588509e-05 4.795406e-05
> detach(alloy)
Caveats:
these are variances, not standard errors;
this is a delta-method approximation, which may be inaccurate in cases of strong nonlinearity or strongly non-Gaussian error (I believe Brian Ripley has stated in the past that he doesn't consider delta-method approximations reliable enough for general use, but he's a purist)
I haven't checked that this is a sensible answer