I am trying to execute radial basis interpolation in Julia using Surrogates package.
I have developed the following code:
using Surrogates
function t(train, test)
xy = zip(train[:,1],train[:,2]) |> collect
lb = vcat(minimum(train[:,1]), minimum(train([:,2]))
ub = vcat(maximum(train([:,1]),maximum(train[:,2]))
rbf = Surrogates.RadialBasis(xy, train[:,3], lb, ub)
testing = [rbf(i) for i in test]
return rbf, testing
end
train = [0.0 0.0 4;
0.0 0.0 2;
0.0 0.0 2;
0.0 0.0 2;
0.0 0.0 1;
0.0 0.0 3;
0.0 0.0 6;
0.0 0.0 7]
test = [(1.0, 1.0),
(1.0, 1.0),
(2.0, 2.0),
(2.0, 2.0),
(2.0, 2.0),
(2.0, 2.0),
(2.0, 2.0),
(2.0, 2.0)]
t(train, test)
However, upon executing this above code I am getting the following error:
LinearAlgebra.SingularException(7)
Stacktrace:
[1] checknonsingular at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\LinearAlgebra\src\factorization.jl:19 [inlined]
[2] checknonsingular at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\LinearAlgebra\src\factorization.jl:21 [inlined]
[3] bunchkaufman!(::LinearAlgebra.Symmetric{Float64,Array{Float64,2}}, ::Bool; check::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\LinearAlgebra\src\bunchkaufman.jl:99
[4] #bunchkaufman#142 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\LinearAlgebra\src\bunchkaufman.jl:186 [inlined]
[5] #_factorize#94 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\LinearAlgebra\src\symmetric.jl:638 [inlined]
[6] _factorize at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\LinearAlgebra\src\symmetric.jl:636 [inlined]
[7] factorize at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\LinearAlgebra\src\symmetric.jl:634 [inlined]
[8] \(::LinearAlgebra.Symmetric{Float64,Array{Float64,2}}, ::Array{Float64,1}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\LinearAlgebra\src\symmetric.jl:648
[9] _calc_coeffs(::Array{Tuple{Float64,Float64},1}, ::Array{Float64,1}, ::Array{Float64,1}, ::Array{Float64,1}, ::Function, ::Int64, ::Float64, ::Bool) at C:\Users\user\.julia\packages\Surrogates\wJbFN\src\Radials.jl:61
[10] RadialBasis(::Array{Tuple{Float64,Float64},1}, ::Array{Float64,1}, ::Array{Float64,1}, ::Array{Float64,1}; rad::Surrogates.RadialFunction{Int64,Surrogates.var"#1#2"}, scale_factor::Float64, sparse::Bool) at C:\Users\user\.julia\packages\Surrogates\wJbFN\src\Radials.jl:51
[11] RadialBasis(::Array{Tuple{Float64,Float64},1}, ::Array{Float64,1}, ::Array{Float64,1}, ::Array{Float64,1}) at C:\Users\user\.julia\packages\Surrogates\wJbFN\src\Radials.jl:49
Please suggest solution to resolve this issue!!
Thanks!
Your training data contains only a single sampling point with multiple values. Replace with more sound data and your code works:
julia> train = Float64[
1 2 3
4 5 6
7 8 9
]
3×3 Matrix{Float64}:
1.0 2.0 3.0
4.0 5.0 6.0
7.0 8.0 9.0
julia> t(train, test)
(RadialBasis{Surrogates.var"#1#2", Int64, Vector{Tuple{Float64, Float64}}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Float64, Bool}(Surrogates.var"#1#2"(), 0, [(1.0, 2.0), (4.0, 5.0), (7.0, 8.0)], [3.0, 6.0, 9.0], [1.0, 2.0], [7.0, 8.0], [0.3535533905932738, 0.0, -0.3535533905932738, 6.0], 1.0, false), [3.0939521879919494, 3.0939521879919494, 3.5922131362964587, 3.5922131362964587, 3.5922131362964587, 3.5922131362964587, 3.5922131362964587, 3.5922131362964587])
Related
Suppose I have this Matrix:
julia> mat = [
1 2 3 4
5 6 7 8
9 8 7 6
];
Then I want to put slices of this Matrix into a 3D Array with types of SMatrix{Int64}, like below:
julia> using StaticArrays
julia> arr = Array{SMatrix{Int64}, 3}(undef, 3, 2, 3);
julia> col_idx = [1, 2, 3];
julia> foreach(x->arr[:, :, x] = mat[:, x:x+1], col_idx)
ERROR: MethodError: Cannot `convert` an object of type
Int64 to an object of type
SMatrix{Int64}
Closest candidates are:
convert(::Type{T}, ::LinearAlgebra.Factorization) where T<:AbstractArray at C:\Users\JUL\.julia\juliaup\julia-1.8.3+0.x64\share\julia\stdlib\v1.8\LinearAlgebra\src\factorization.jl:58
convert(::Type{SA}, ::Tuple) where SA<:StaticArray at C:\Users\JUL\.julia\packages\StaticArrays\x7lS0\src\convert.jl:179
convert(::Type{SA}, ::SA) where SA<:StaticArray at C:\Users\JUL\.julia\packages\StaticArrays\x7lS0\src\convert.jl:178
...
Stacktrace:
[1] setindex!
# .\array.jl:968 [inlined]
[2] macro expansion
# .\multidimensional.jl:946 [inlined]
[3] macro expansion
# .\cartesian.jl:64 [inlined]
[4] macro expansion
# .\multidimensional.jl:941 [inlined]
[5] _unsafe_setindex!(::IndexLinear, ::Array{SMatrix{Int64}, 3}, ::Matrix{Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
# Base .\multidimensional.jl:953
[6] _setindex!
# .\multidimensional.jl:930 [inlined]
[7] setindex!(::Array{SMatrix{Int64}, 3}, ::Matrix{Int64}, ::Function, ::Function, ::Int64)
# Base .\abstractarray.jl:1344
[8] (::var"#5#6")(x::Int64)
# Main .\REPL[20]:1
[9] foreach(f::var"#5#6", itr::Vector{Int64})
# Base .\abstractarray.jl:2774
[10] top-level scope
# REPL[20]:1
How can I achieve it?
P.S.:
This is just a minimal and reproducible example. In the practical sense, I have a size of (10, 10, 2000) for arr and a big size for mat as well (10x2000, I guess)!
If I understood correctly, do you want an Array of SMatrices?
mat = [ 1 2 3 4
5 6 7 8
9 8 7 6 ];
using StaticArrays
col_idx = [1, 2, 3];
arr = [SMatrix{3,2}(mat[:, x:x+1]) for x in col_idx]
3-element Vector{SMatrix{3, 2, Int64, 6}}:
[1 2; 5 6; 9 8]
[2 3; 6 7; 8 7]
[3 4; 7 8; 7 6]
Then, what if I say:
julia> using StaticArrays
julia> mat = [
1 2 3 4
5 6 7 8
9 8 7 6
];
julia> arr = Array{Int64, 3}(undef, 3, 2, 3);
julia> foreach(x->arr[:, :, x] = mat[:, x:x+1], [1, 2, 3]);
julia> sarr = SArray{Tuple{3, 2, 3}}(arr)
3×2×3 SArray{Tuple{3, 2, 3}, Int64, 3, 18} with indices SOneTo(3)×SOneTo(2)×SOneTo(3):
[:, :, 1] =
1 2
5 6
9 8
[:, :, 2] =
2 3
6 7
8 7
[:, :, 3] =
3 4
7 8
7 6
julia> typeof(sarr[:, :, 1])
SMatrix{3, 2, Int64, 6} (alias for SArray{Tuple{3, 2}, Int64, 2, 6})
First, I created a regular 3D Array, then constructed a SArray based on it.
However, in the case of your practical situation, I tried the following:
julia> mat = rand(10, 2000);
julia> arr = Array{Float64, 3}(undef, 10, 2, 1999);
julia> foreach(x->arr[:, :, x] = mat[:, x:x+1], 1:1999);
julia> sarr = SArray{Tuple{10, 2, 1999}}(arr);
But it takes too much time to construct such a container. (I already canceled it, and I don't know the runtime of it.). Hence, in these cases, it's better to take #AboAmmar's advice.
Inspired by #Shayan and #AboAmmar, this answer explores using BlockArrays.jl package to construct the desired result. BlockArrays puts existing arrays into a 'meta-array'. The sub-arrays can be of SMatrix type.
In code:
using StaticArrays, BlockArrays
mat = rand(10,2000) # random demo matrix
# make all the slice SArrays
arr = [SArray{Tuple{10,2,1}, Float64, 3}(mat[:,i:i+1])
for i=1:1999]
arr = reshape(arr,1,1,1999)
# glue them into a BlockArray
bricked = mortar(arr)
After now:
julia> size(bricked)
(10, 2, 1999)
julia> bricked[:,:,25]
1×1-blocked 10×2 BlockMatrix{Float64}:
0.265972 0.258414
0.396142 0.863366
0.41708 0.648276
0.960283 0.773064
0.62513 0.268989
0.132796 0.0493077
0.844674 0.791772
0.59638 0.0769661
0.221536 0.388623
0.595742 0.50732
Hopefully this method gets the performance trade-off you wanted (or at least introduces some new ideas).
How is it possible to attain the value of #NLexpression when the variables are fixed? In the following code variables have fixed but value of K1 has not been reached.
using JuMP, Distributions,Juniper
#-----Model parameters--------------------------------------------------------
sig, C1, c0 = 2, 300, 10;
E, landa, T0, T1, T2, gam1, gam2, a1, a2, a3, ap = 0.05, 0.01, 0, 2, 2, 1, 1, 0.5, 0.1, 50, 25;
xhat=[2.807064523673271;23.0;1.3349699464500042];
f(x) = cdf(Normal(0, 1), x);
#---------------------------------------------------------------------------
ALT= Model(optimizer_with_attributes(Juniper.Optimizer, "nl_solver"=>optimizer_with_attributes(Ipopt.Optimizer, "print_level" => 0),
"mip_solver"=>optimizer_with_attributes(Gurobi.Optimizer, "logLevel" => 0),"registered_functions" =>[Juniper.register( :f, 1, f; autodiff = true)])
);
# variables-----------------------------------------------------------------
JuMP.register(ALT, :f, 1, f; autodiff = true);
#variable(ALT, hp == xhat[3]);
#variable(ALT, Lp ==xhat[1]);
#variable(ALT, np==xhat[2], Int);
#---------------------------------------------------------------------------
C1=rand(100:100:300);
sig=rand(0.5:0.5:2);
#---------------------------------------------------------------------------
k1=#NLexpression(ALT,hp/(1-f(Lp-sig*sqrt(np))+f(-Lp - sig*sqrt(np))));
JuMP.value(k1);
the error is this:
julia> JuMP.value(k1)
ERROR: type Nothing has no field status
Stacktrace:
[1] getproperty(::Nothing, ::Symbol) at .\Base.jl:33
[2] get at C:\Users\admin\.julia\packages\Juniper\dNHnx\src\MOI_wrapper\results.jl:4 [inlined]
[3] get(::MathOptInterface.Bridges.LazyBridgeOptimizer{Juniper.Optimizer}, ::MathOptInterface.TerminationStatus) at C:\Users\admin\.julia\packages\MathOptInterface\bygN7\src\Bridges\bridge_optimizer.jl:587
[4] get(::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.AbstractOptimizer,MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}, ::MathOptInterface.TerminationStatus) at C:\Users\admin\.julia\packages\MathOptInterface\bygN7\src\Utilities\cachingoptimizer.jl:553
[5] _moi_get_result(::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.AbstractOptimizer,MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}, ::MathOptInterface.VariablePrimal, ::Vararg{Any,N} where N) at C:\Users\admin\.julia\packages\JuMP\YXK4e\src\JuMP.jl:844
[6] get(::Model, ::MathOptInterface.VariablePrimal, ::VariableRef) at C:\Users\admin\.julia\packages\JuMP\YXK4e\src\JuMP.jl:877
[7] value(::VariableRef; result::Int64) at C:\Users\admin\.julia\packages\JuMP\YXK4e\src\variables.jl:767
[8] #103 at C:\Users\admin\.julia\packages\JuMP\YXK4e\src\nlp.jl:1159 [inlined]
[9] value(::NonlinearExpression, ::JuMP.var"#103#104"{Int64}) at C:\Users\admin\.julia\packages\JuMP\YXK4e\src\nlp.jl:1102
[10] #value#102 at C:\Users\admin\.julia\packages\JuMP\YXK4e\src\nlp.jl:1159 [inlined]
[11] value(::NonlinearExpression) at C:\Users\admin\.julia\packages\JuMP\YXK4e\src\nlp.jl:1159
[12] top-level scope at none:1
would you please help me how the error is solved?
Thanks.
Please update to Juniper v0.8.0. I fixed this issue a few days ago.
p.s., in future, please consider posting on the JuMP community forum: https://discourse.julialang.org/c/domain/opt/13. There are more readers of JuMP-specific questions.
I am trying to use maximum likelihood to estimate the normal linear model in Julia. I have used the following code to simulate the process with just an intercept and an anonymous function per the Optim documentation regarding values that do not change:
using Optim
nobs = 500
nvar = 1
β = ones(nvar)*3.0
x = [ones(nobs) randn(nobs,nvar-1)]
ε = randn(nobs)*0.5
y = x*β + ε
function LL_anon(X, Y, β, σ)
-(-length(X)*log(2π)/2 - length(X)*log(σ) - (sum((Y - X*β).^2) / (2σ^2)))
end
LL_anon(X,Y, pars) = LL_anon(X,Y, pars...)
res2 = optimize(vars -> LL_anon(x,y, vars...), [1.0,1.0]) # Start values: β=1.0, σ=1.0
This actually recovered the parameters and I received the following output:
* Algorithm: Nelder-Mead
* Starting Point: [1.0,1.0]
* Minimizer: [2.980587812647935,0.5108406803949835]
* Minimum: 3.736217e+02
* Iterations: 47
* Convergence: true
* √(Σ(yᵢ-ȳ)²)/n < 1.0e-08: true
* Reached Maximum Number of Iterations: false
* Objective Calls: 92
However, when I try and set nvar = 2, i.e. an intercept plus an additional covariate, I get the following error message:
MethodError: no method matching LL_anon(::Array{Float64,2}, ::Array{Float64,1}, ::Float64, ::Float64, ::Float64)
Closest candidates are:
LL_anon(::Any, ::Any, ::Any, ::Any) at In[297]:2
LL_anon(::Array{Float64,1}, ::Array{Float64,1}, ::Any, ::Any) at In[113]:2
LL_anon(::Any, ::Any, ::Any) at In[297]:4
...
Stacktrace:
[1] (::##245#246)(::Array{Float64,1}) at .\In[299]:1
[2] value!!(::NLSolversBase.NonDifferentiable{Float64,Array{Float64,1},Val{false}}, ::Array{Float64,1}) at C:\Users\dolacomb\.julia\v0.6\NLSolversBase\src\interface.jl:9
[3] initial_state(::Optim.NelderMead{Optim.AffineSimplexer,Optim.AdaptiveParameters}, ::Optim.Options{Float64,Void}, ::NLSolversBase.NonDifferentiable{Float64,Array{Float64,1},Val{false}}, ::Array{Float64,1}) at C:\Users\dolacomb\.julia\v0.6\Optim\src\multivariate/solvers/zeroth_order\nelder_mead.jl:136
[4] optimize(::NLSolversBase.NonDifferentiable{Float64,Array{Float64,1},Val{false}}, ::Array{Float64,1}, ::Optim.NelderMead{Optim.AffineSimplexer,Optim.AdaptiveParameters}, ::Optim.Options{Float64,Void}) at C:\Users\dolacomb\.julia\v0.6\Optim\src\multivariate/optimize\optimize.jl:25
[5] #optimize#151(::Array{Any,1}, ::Function, ::Tuple{##245#246}, ::Array{Float64,1}) at C:\Users\dolacomb\.julia\v0.6\Optim\src\multivariate/optimize\interface.jl:62
[6] #optimize#148(::Array{Any,1}, ::Function, ::Function, ::Array{Float64,1}) at C:\Users\dolacomb\.julia\v0.6\Optim\src\multivariate/optimize\interface.jl:52
[7] optimize(::Function, ::Array{Float64,1}) at C:\Users\dolacomb\.julia\v0.6\Optim\src\multivariate/optimize\interface.jl:52
I'm not sure why adding an additional variable is causing this issue but it seems like a type instability problem.
The second issue is that when I use my original working example and set the starting values to [2.0,2.0], I get the following error message:
log will only return a complex result if called with a complex argument. Try log(complex(x)).
Stacktrace:
[1] nan_dom_err at .\math.jl:300 [inlined]
[2] log at .\math.jl:419 [inlined]
[3] LL_anon(::Array{Float64,2}, ::Array{Float64,1}, ::Float64, ::Float64) at .\In[302]:2
[4] (::##251#252)(::Array{Float64,1}) at .\In[304]:1
[5] value(::NLSolversBase.NonDifferentiable{Float64,Array{Float64,1},Val{false}}, ::Array{Float64,1}) at C:\Users\dolacomb\.julia\v0.6\NLSolversBase\src\interface.jl:19
[6] update_state!(::NLSolversBase.NonDifferentiable{Float64,Array{Float64,1},Val{false}}, ::Optim.NelderMeadState{Array{Float64,1},Float64,Array{Float64,1}}, ::Optim.NelderMead{Optim.AffineSimplexer,Optim.AdaptiveParameters}) at C:\Users\dolacomb\.julia\v0.6\Optim\src\multivariate/solvers/zeroth_order\nelder_mead.jl:193
[7] optimize(::NLSolversBase.NonDifferentiable{Float64,Array{Float64,1},Val{false}}, ::Array{Float64,1}, ::Optim.NelderMead{Optim.AffineSimplexer,Optim.AdaptiveParameters}, ::Optim.Options{Float64,Void}, ::Optim.NelderMeadState{Array{Float64,1},Float64,Array{Float64,1}}) at C:\Users\dolacomb\.julia\v0.6\Optim\src\multivariate/optimize\optimize.jl:51
[8] optimize(::NLSolversBase.NonDifferentiable{Float64,Array{Float64,1},Val{false}}, ::Array{Float64,1}, ::Optim.NelderMead{Optim.AffineSimplexer,Optim.AdaptiveParameters}, ::Optim.Options{Float64,Void}) at C:\Users\dolacomb\.julia\v0.6\Optim\src\multivariate/optimize\optimize.jl:25
[9] #optimize#151(::Array{Any,1}, ::Function, ::Tuple{##251#252}, ::Array{Float64,1}) at C:\Users\dolacomb\.julia\v0.6\Optim\src\multivariate/optimize\interface.jl:62
Again, I’m not sure why this is happening and since start values are very important I’d like to know how to overcome this issue and they are not too far off from the true values.
Any help would be greatly appreciated!
Splatting causes the problem. E.g. it transforms [1, 2, 3] into three parameters while your function accepts only two.
Use the following call:
res2 = optimize(vars -> LL_anon(x,y, vars[1:end-1], vars[end]), [1.0,1.0,1.0])
and you can remove the following line from your code
LL_anon(X,Y, pars) = LL_anon(X,Y, pars...)
or replace it with:
LL_anon(X,Y, pars) = LL_anon(X,Y, pars[1:end-1], pars[end])
but it would not be called by optimization routine unless you change a call to:
res2 = optimize(vars -> LL_anon(x,y, vars), [1.0,1.0,1.0])
Finally - to get good performance of this code I would recommend to wrap it all in a function.
EDIT: now I see a second question. The reason is that σ can become negative in the optimization process and then log(σ) fails. The simplest thing to do in this case is to take log(abs(σ))) like this:
function LL_anon(X, Y, β, σ)
-(-length(X)*log(2π)/2 - length(X)*log(abs(σ)) - (sum((Y - X*β).^2) / (2σ^2)))
end
Of course then you have to take absolute value of σ as a solution as you might get a negative value from optimization routine.
A cleaner way would be to optimize over e.g. log(σ) not σ like this:
function LL_anon(X, Y, β, logσ)
-(-length(X)*log(2π)/2 - length(X)*logσ - (sum((Y - X*β).^2) / (2(exp(logσ))^2)))
end
but then you have to use exp(logσ) to recover σ after optimization finishes.
I have asked around regarding this and have another option. The main reason for looking at this problem is twofold. One, to learn how to use the optimization routines in Julia in a canonical situation and two, to expand this to spatial econometric models. With that in mind, I'm posting the other suggested code from the Julia message board so that others may see another solution.
using Optim
nobs = 500
nvar = 2
β = ones(nvar) * 3.0
x = [ones(nobs) randn(nobs, nvar - 1)]
ε = randn(nobs) * 0.5
y = x * β + ε
function LL_anon(X, Y, β, log_σ)
σ = exp(log_σ)
-(-length(X) * log(2π)/2 - length(X) * log(σ) - (sum((Y - X * β).^2) / (2σ^2)))
end
opt = optimize(vars -> LL_anon(x,y, vars[1:nvar], vars[nvar + 1]),
ones(nvar+1))
# Use forward autodiff to get first derivative, then optimize
fun1 = OnceDifferentiable(vars -> LL_anon(x, y, vars[1:nvar], vars[nvar + 1]),
ones(nvar+1))
opt1 = optimize(fun1, ones(nvar+1))
Results of Optimization Algorithm
Algorithm: L-BFGS
Starting Point: [1.0,1.0,1.0]
Minimizer: [2.994204150985705,2.9900626550063305, …]
Minimum: 3.538340e+02
Iterations: 12
Convergence: true
|x - x’| ≤ 1.0e-32: false
|x - x’| = 8.92e-12
|f(x) - f(x’)| ≤ 1.0e-32 |f(x)|: false
|f(x) - f(x’)| = 9.64e-16 |f(x)|
|g(x)| ≤ 1.0e-08: true
|g(x)| = 6.27e-09
Stopped by an increasing objective: true
Reached Maximum Number of Iterations: false
Objective Calls: 50
Gradient Calls: 50
opt1.minimizer
3-element Array{Float64,1}:
2.9942
2.99006
-1.0651 #Note: needs to be exponentiated
# Get Hessian, use Newton!
fun2 = TwiceDifferentiable(vars -> LL_anon(x, y, vars[1:nvar], vars[nvar + 1]),
ones(nvar+1))
opt2 = optimize(fun2, ones(nvar+1))
Results of Optimization Algorithm
Algorithm: Newton’s Method
Starting Point: [1.0,1.0,1.0]
Minimizer: [2.99420415098702,2.9900626550079026, …]
Minimum: 3.538340e+02
Iterations: 9
Convergence: true
|x - x’| ≤ 1.0e-32: false
|x - x’| = 1.36e-11
|f(x) - f(x’)| ≤ 1.0e-32 |f(x)|: false
|f(x) - f(x’)| = 1.61e-16 |f(x)|
|g(x)| ≤ 1.0e-08: true
|g(x)| = 6.27e-09
Stopped by an increasing objective: true
Reached Maximum Number of Iterations: false
Objective Calls: 45
Gradient Calls: 45
Hessian Calls: 9
fieldnames(fun2)
13-element Array{Symbol,1}:
:f
:df
:fdf
:h
:F
:DF
:H
:x_f
:x_df
:x_h
:f_calls
:df_calls
:h_calls
opt2.minimizer
3-element Array{Float64,1}:
2.98627
3.00654
-1.11313
numerical_hessian = (fun2.H) #.H is the numerical Hessian
3×3 Array{Float64,2}:
64.8715 -9.45045 0.000121521
-0.14568 66.4507 0.0
1.87326e-6 4.10675e-9 44.7214
From here, one can use the numerical Hessian to obtain the standard errors for the estimates and form t-statistics, etc. for their own functions.
Again, thank you for providing an answer and I hope people find this information useful.
I am interested in calculating quantity:
where x_i is a 1xD vector (one out of my N data of dimension D), μ is a DxK matrix and W is a list of K DxD matrices.
This should result in a 1XK vector. I try it for all N and K in the following way that works:
res = zeros(N,K);
for i in 1:N
for k in 1:K
res[i,k] = (x_matrix[i,:]-mus_matrix[:,k])'*
w_matrix[k]*(x_matrix[i,:]-mus_matrix[:,k])
If I try to vectorize it, using the following:
res = zeros(N,K);
for i in 1:N
res[i,:] = (x_matrix[i,:].-mus_matrix)'.*w_matrix.*(x_matrix[i,:].-mus_matrix)
I get the following error:
ERROR: DimensionMismatch("arrays could not be broadcast to a common size")
Stacktrace:
[1] _bcs1(::Base.OneTo{Int64}, ::Base.OneTo{Int64}) at ./broadcast.jl:70
[2] _bcs at ./broadcast.jl:63 [inlined]
[3] broadcast_shape(::Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}, ::Tuple{Base.OneTo{Int64}}, ::Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}, ::Vararg{Tuple{Base.OneTo{Int64},Base.OneTo{Int64}},N} where N) at ./broadcast.jl:57 (repeats 3 times)
[4] broadcast_indices(::Array{Float64,2}, ::Array{Any,1}, ::Array{Float64,1}, ::Vararg{Any,N} where N) at ./broadcast.jl:53
[5] broadcast_c(::Function, ::Type{Array}, ::Array{Float64,2}, ::Array{Any,1}, ::Vararg{Any,N} where N) at ./broadcast.jl:311
[6] broadcast(::Function, ::Array{Float64,2}, ::Array{Any,1}, ::Array{Float64,1}, ::Vararg{Any,N} where N) at ./broadcast.jl:434
Here is an example:
julia> N = 5
5
julia> D=2
2
julia> K = 4
4
julia> W=[]
0-element Array{Any,1}
julia> x = rand(N,D)
5×2 Array{Float64,2}:
0.576477 0.9575
0.184454 0.660436
0.470267 0.729649
0.648879 0.782561
0.626453 0.111332
julia> mu = rand(K,D)
4×2 Array{Float64,2}:
0.989281 0.00126782
0.659106 0.66136
0.50843 0.289442
0.327962 0.523229
julia> for i in 1:K
push!(W,rand(D,D))
end
And then run
julia> (x_matrix[i,:]-mus_matrix[:,k])'*
w_matrix[k]*(x_matrix[i,:]-mus_matrix[:,k])
34649.850360744866
But with the second code
julia> (x_matrix[i,:].-mus_matrix)'.*w_matrix.*(x_matrix[i,:].-mus_matrix)
ERROR: DimensionMismatch("arrays could not be broadcast to a common size")
Stacktrace:
[1] _bcs1(::Base.OneTo{Int64}, ::Base.OneTo{Int64}) at ./broadcast.jl:70
[2] _bcs at ./broadcast.jl:63 [inlined]
[3] broadcast_shape(::Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}, ::Tuple{Base.OneTo{Int64}}, ::Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}, ::Vararg{Tuple{Base.OneTo{Int64},Base.OneTo{Int64}},N} where N) at ./broadcast.jl:57 (repeats 3 times)
[4] broadcast_indices(::Array{Float64,2}, ::Array{Any,1}, ::Array{Float64,1}, ::Vararg{Any,N} where N) at ./broadcast.jl:53
[5] broadcast_c(::Function, ::Type{Array}, ::Array{Float64,2}, ::Array{Any,1}, ::Vararg{Any,N} where N) at ./broadcast.jl:311
[6] broadcast(::Function, ::Array{Float64,2}, ::Array{Any,1}, ::Array{Float64,1}, ::Vararg{Any,N} where N) at ./broadcast.jl:434
TL/DR: optimized variant below, but Einsum looks nicer, IMHO.
Looks like a case for using Einstein summation notation. In Julia, Einsum.jl can do this:
julia> N = 5
5
julia> D = 3
3
julia> K = 10
10
julia> x = rand(N, D)
5×3 Array{Float64,2}:
0.587436 0.210529 0.261725
0.527269 0.457477 0.482939
0.52726 0.411209 0.138872
0.89107 0.464789 0.758392
0.885267 0.931014 0.672959
julia> μ = rand(D, K)
3×10 Array{Float64,2}:
0.280792 0.265066 0.81437 0.503377 0.0717916 … 0.275872 0.609961 0.0820088 0.0042564
0.0177643 0.0959438 0.563948 0.332433 0.088527 0.691971 0.0296638 0.604488 0.956057
0.668128 0.444816 0.74203 0.518232 0.48689 0.465067 0.117469 0.729514 0.109973
julia> W = rand(K, D, D)
10×3×3 Array{Float64,3}:
[:, :, 1] =
0.320861 0.662103 0.219234
0.780944 0.769377 0.566203
0.466207 0.428527 0.330901
0.15534 0.035435 0.346737
0.810676 0.328116 0.469505
0.676575 0.668204 0.285334
0.455551 0.211295 0.85295
0.229995 0.741487 0.783361
0.0937583 0.401419 0.47032
0.956335 0.434213 0.967791
[:, :, 2] =
0.275903 0.130298 0.184485
0.941648 0.940107 0.439454
0.425292 0.252654 0.797115
0.0203406 0.594075 0.484809
0.164309 0.941597 0.455314
0.73628 0.109502 0.920664
0.906305 0.177235 0.540193
0.360038 0.0486971 0.20626
0.914357 0.699901 0.295872
0.284143 0.659117 0.291479
[:, :, 3] =
0.138311 0.921371 0.353719
0.345247 0.70865 0.246736
0.361364 0.636543 0.343837
0.752149 0.581561 0.346399
0.705888 0.24765 0.703952
0.992327 0.369668 0.109407
0.341624 0.223715 0.970667
0.762169 0.94248 0.917569
0.0367128 0.589345 0.121106
0.826602 0.692111 0.229499
julia> using Einsum
julia> #einsum r[n,k] := (x[n,i] - μ[i,k]) * W[k,i,j] * (x[n,j] - μ[j,k])
julia> r
5×10 Array{Float64,2}:
0.0176889 0.087092 0.522184 0.0417967 … -0.0430999 0.041266 -0.0596579 0.432076
0.0521066 0.364059 0.181515 0.00434307 -0.0248712 0.226976 -0.0686294 0.437169
-0.0472136 0.127803 0.458812 0.0119074 0.0391649 -0.0190299 -0.0585371 0.264379
0.468634 1.16498 -0.00263205 0.192809 0.273537 1.13787 -0.0653081 1.41321
0.749655 2.20266 0.0205068 0.420249 0.573358 1.42499 0.441232 1.67574
Which #macroexpands to essentially the following loops (plus preparation and bounds checking):
begin
local k
for k = 1:size(μ, 2)
begin
local n
for n = 1:size(x, 1)
begin
local s = zero(T)
begin
local j
for j = 1:size(W, 3)
begin
local i
for i = 1:size(x, 2)
s += (x[n, i] - μ[i, k]) * W[k, i, j] * (x[n, j] - μ[j, k])
end
end
end
end
r[n, k] = s
end
end
end
end
end
Now, to find something more performant, I compared a couple of variants using BenchmarkTools.jl. You can see the full code and results on my laptop here. It shows that the Einsum variant already is in fact better than the original:
# Original:
# memory estimate: 1017.73 MiB
# allocs estimate: 3429967
# median time: 361.982 ms (15.94% GC)
# Einsum:
# memory estimate: 2.64 MiB
# allocs estimate: 76
# median time: 127.536 ms (0.00% GC)
By far the most efficient and least allocating variant is the following, which requires x = x' and W = permutedims(W, [2, 3, 1]) (assuming you can change your representation easily):
function test_optimized!(res, x, μ, W)
z = zero(eltype(x))
for k = 1:size(μ, 1)
for n = 1:size(x, 1)
res[n, k] = z
for i = 1:size(W, 1)
for j = 1:size(W, 2)
#inbounds res[n, k] += (x[i, n] - μ[i, k]) * W[i, j, k] * (x[j, n] - μ[j, k])
end
end
end
end
end
function test_optimized(x, μ, W)
res = zeros(N, K)
test_optimized!(res, x, μ, W)
res
end
This brings us down to
# memory estimate: 2.63 MiB
# allocs estimate: 2
# median time: 521.215 μs (0.00% GC)
It uses a couple of "tricks" that can be found in the docs: filling a preallocated matrix in a separate method, accessing strides in column-major order, and using #inbounds (although that only improves things at the order of a microsecond).
There is also TensorOperations.jl, which I think does more intelligent things under the hood, but it fail on this:
julia> #tensor r[n,k] := (x[n,i] - μ[i,k]) * W[k,i,j] * (x[n,j] - μ[j,k])
ERROR: TensorOperations.IndexError{String}("invalid index specification: (:n, :i) to (:i, :k)")
Stacktrace:
[1] add_indices(::Tuple{Symbol,Symbol}, ::Tuple{Symbol,Symbol}) at /home/philipp/.julia/v0.6/TensorOperations/src/implementation/indices.jl:22
[2] + at /home/philipp/.julia/v0.6/TensorOperations/src/indexnotation/sum.jl:40 [inlined]
[3] -(::TensorOperations.IndexedObject{(:n, :i),:N,Array{Float64,2},Int64}, ::TensorOperations.IndexedObject{(:i, :k),:N,Array{Float64,2},Int64}) at /home/philipp/.julia/v0.6/TensorOperations/src/indexnotation/sum.jl:44
I guess that's deliberate and has to do with efficiency, see this issue.
I would like to produce an n x 3 matrix where n is the number of pixels (width * height).
x = linspace(-1, 1, width)
y = linspace(-1, 1, height)
r = 1.0
viewDirections = [[i j 1.0] for i in x for j in y]
However, when I run this I get a:
16-element Array{Array{Float64,2},1}
and not my desired a 16x3 Array{Float64,2}. I am obviously not using comprehensions properly to construct matrices. I tried using comprehensions to create an array of tuples, but I can't then convert those tuples into a matrix.
The problem here is array comprehension will give us a nested array instead of a Matrix. This is the right behavior of comprehension, it won't do extra guesswork for us, so we need to convert the nested array to matrix manually, which can be done using vcat with splating operator(...):
julia> vcat(viewDirections...)
6×3 Array{Float64,2}:
-1.0 -1.0 1.0
-1.0 1.0 1.0
0.0 -1.0 1.0
0.0 1.0 1.0
1.0 -1.0 1.0
1.0 1.0 1.0
It seems like you're constructing homogeneous coordinates from 2D Euclidean space. Using Base.Iterators.product is a more concise and robust way to create the iterator:
julia> w = linspace(-1,1,3)
-1.0:1.0:1.0
julia> h = linspace(-1,1,2)
-1.0:2.0:1.0
julia> r = 1.0
1.0
julia> viewDirections = [collect(i) for i in Iterators.product(w, h, r)]
3×2 Array{Array{Float64,1},2}:
[-1.0, -1.0, 1.0] [-1.0, 1.0, 1.0]
[0.0, -1.0, 1.0] [0.0, 1.0, 1.0]
[1.0, -1.0, 1.0] [1.0, 1.0, 1.0]
julia> hcat(viewDirections...).'
6×3 Array{Float64,2}:
-1.0 -1.0 1.0
0.0 -1.0 1.0
1.0 -1.0 1.0
-1.0 1.0 1.0
0.0 1.0 1.0
1.0 1.0 1.0
Note that, the order of coordinates is different from your original version, that's because Julia is column-major, Iterators.product will iterate the rightest dimension "outestly" i.e. [[i j r] for j in y for i in x ]. If the order is important in your use case, just pay attention to it.
Here are some benchmark results when width/height goes large:
julia> w = linspace(-1,1,300)
-1.0:0.006688963210702341:1.0
julia> h = linspace(-1,1,200)
-1.0:0.010050251256281407:1.0
julia> foo(w,h,r) = hcat([collect(i) for i in Iterators.product(w, h, r)]...).'
julia> bar(w,h,r) = vcat([[i j r] for i in w for j in h]...)
julia> #btime foo($w,$h,$r);
6.172 ms (60018 allocations: 10.99 MiB)
julia> #btime bar($w,$h,$r);
11.294 ms (360028 allocations: 17.02 MiB)