Find fixed point of multivariable function in Julia - julia

I need to find the fixed point of a multivariable function in Julia.
Consider the following minimal example:
function example(p::Array{Float64,1})
q = -p
return q
end
Ideally I'd use a package like Roots.jl and call find_zeros(p -> p - example(p)), but I can't find the analogous package for multivariable functions. I found one called IntervalRootFinding, but it oddly requires unicode characters and is sparsely documented, so I can't figure out how to use it.

There are many options. The choice of the best one depends on the nature of example function (you have to understand the nature of your example function and check against a documentation of a specific package if it would support it).
Eg. you can use fixedpoint from NLsolve.jl:
julia> using NLsolve
julia> function example!(q, p::Array{Float64,1})
q .= -p
end
example! (generic function with 1 method)
julia> fixedpoint(example!, ones(1))
Results of Nonlinear Solver Algorithm
* Algorithm: Anderson m=1 beta=1 aa_start=1 droptol=0
* Starting Point: [1.0]
* Zero: [0.0]
* Inf-norm of residuals: 0.000000
* Iterations: 3
* Convergence: true
* |x - x'| < 0.0e+00: true
* |f(x)| < 1.0e-08: true
* Function Calls (f): 3
* Jacobian Calls (df/dx): 0
julia> fixedpoint(example!, ones(3))
Results of Nonlinear Solver Algorithm
* Algorithm: Anderson m=3 beta=1 aa_start=1 droptol=0
* Starting Point: [1.0, 1.0, 1.0]
* Zero: [-2.220446049250313e-16, -2.220446049250313e-16, -2.220446049250313e-16]
* Inf-norm of residuals: 0.000000
* Iterations: 3
* Convergence: true
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: true
* Function Calls (f): 3
* Jacobian Calls (df/dx): 0
julia> fixedpoint(example!, ones(5))
Results of Nonlinear Solver Algorithm
* Algorithm: Anderson m=5 beta=1 aa_start=1 droptol=0
* Starting Point: [1.0, 1.0, 1.0, 1.0, 1.0]
* Zero: [0.0, 0.0, 0.0, 0.0, 0.0]
* Inf-norm of residuals: 0.000000
* Iterations: 3
* Convergence: true
* |x - x'| < 0.0e+00: true
* |f(x)| < 1.0e-08: true
* Function Calls (f): 3
* Jacobian Calls (df/dx): 0
If your function would require a global optimization tools to find a fixed point then you can e.g. use BlackBoxOptim.jl with norm(f(x) .-x) as an objective:
julia> using LinearAlgebra
julia> using BlackBoxOptim
julia> function example(p::Array{Float64,1})
q = -p
return q
end
example (generic function with 1 method)
julia> f(x) = norm(example(x) .- x)
f (generic function with 1 method)
julia> bboptimize(f; SearchRange = (-5.0, 5.0), NumDimensions = 1)
Starting optimization with optimizer DiffEvoOpt{FitPopulation{Float64},RadiusLimitedSelector,BlackBoxOptim.AdaptiveDiffEvoRandBin{3},RandomBound{ContinuousRectSearchSpace}}
0.00 secs, 0 evals, 0 steps
Optimization stopped after 10001 steps and 0.15 seconds
Termination reason: Max number of steps (10000) reached
Steps per second = 68972.31
Function evals per second = 69717.14
Improvements/step = 0.35090
Total function evaluations = 10109
Best candidate found: [-8.76093e-40]
Fitness: 0.000000000
julia> bboptimize(f; SearchRange = (-5.0, 5.0), NumDimensions = 3);
Starting optimization with optimizer DiffEvoOpt{FitPopulation{Float64},RadiusLimitedSelector,BlackBoxOptim.AdaptiveDiffEvoRandBin{3},RandomBound{ContinuousRectSearchSpace}}
0.00 secs, 0 evals, 0 steps
Optimization stopped after 10001 steps and 0.02 seconds
Termination reason: Max number of steps (10000) reached
Steps per second = 625061.23
Function evals per second = 631498.72
Improvements/step = 0.32330
Total function evaluations = 10104
Best candidate found: [-3.00106e-12, -5.33545e-12, 5.39072e-13]
Fitness: 0.000000000
julia> bboptimize(f; SearchRange = (-5.0, 5.0), NumDimensions = 5);
Starting optimization with optimizer DiffEvoOpt{FitPopulation{Float64},RadiusLimitedSelector,BlackBoxOptim.AdaptiveDiffEvoRandBin{3},RandomBound{ContinuousRectSearchSpace}}
0.00 secs, 0 evals, 0 steps
Optimization stopped after 10001 steps and 0.02 seconds
Termination reason: Max number of steps (10000) reached
Steps per second = 526366.94
Function evals per second = 530945.88
Improvements/step = 0.29900
Total function evaluations = 10088
Best candidate found: [-9.23635e-8, -2.6889e-8, -2.93044e-8, -1.62639e-7, 3.99672e-8]
Fitness: 0.000000391

I'm an author of IntervalRootFinding.jl. I'm happy to say that the documentation has been improved a lot recently, and no unicode characters are necessary. I suggest using the master branch.
Here's how to solve your example with the package. Note that this package should be able to find all of the roots within a box, and guarantee that it has found all of them. Yours only has 1:
julia> using IntervalArithmetic, IntervalRootFinding
julia> function example(p)
q = -p
return q
end
example (generic function with 2 methods)
julia> X = IntervalBox(-2..2, 2)
[-2, 2] × [-2, 2]
julia> roots(x -> example(x) - x, X, Krawczyk)
1-element Array{Root{IntervalBox{2,Float64}},1}:
Root([0, 0] × [0, 0], :unique)
For more information, I suggest looking at https://discourse.julialang.org/t/ann-intervalrootfinding-jl-for-finding-all-roots-of-a-multivariate-function/9515.

Related

Julia minimize simple scalar function

How do I minimize a simple scalar function in Julia using Newton's method? (or any other suitable optimization scheme)
using Optim
# Function to optimize
function g(x)
return x^2
end
x0 = 2.0 # Initial value
optimize(g, x0, Newton())
The above doesn't seem to work and returns
ERROR: MethodError: no method matching optimize(::typeof(g), ::Float64, ::Newton{LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}}})
Optim is designed for vector problems and not scalar ones like in your example. You can adjust the example to be a vector-problem with one variable though:
julia> using Optim
julia> function g(x) # <- g accepts x as a vector
return x[1]^2
end
julia> x0 = [2.0] # <- Make this a vector
1-element Vector{Float64}:
2.0
julia> optimize(g, x0, Newton())
* Status: success
* Candidate solution
Final objective value: 0.000000e+00
The optimize function expects an interval, not a starting point:
optimize(g, -10, 10)
returns
Results of Optimization Algorithm
* Algorithm: Brent's Method
* Search Interval: [-10.000000, 10.000000]
* Minimizer: 0.000000e+00
* Minimum: 0.000000e+00
* Iterations: 5
* Convergence: max(|x - x_upper|, |x - x_lower|) <= 2*(1.5e-08*|x|+2.2e-16): true
* Objective Function Calls: 6
Concerning available methods I have not read the doc, but you can directly have a look at the source code using the #edit macro:
#edit optimize(g, -10, 10)
Browsing the source you will see:
function optimize(f,
lower::Union{Integer, Real},
upper::Union{Integer, Real},
method::Union{Brent, GoldenSection};
kwargs...)
T = promote_type(typeof(lower/1), typeof(upper/1))
optimize(f,
T(lower),
T(upper),
method;
kwargs...)
end
Hence I think that you have only two methods for unidimensional minimization: Brent and GoldenSection.
By example you can try:
julia> optimize(g, -10, 10, GoldenSection())
Results of Optimization Algorithm
* Algorithm: Golden Section Search
* Search Interval: [-10.000000, 10.000000]
* Minimizer: 1.110871e-16
* Minimum: 1.234035e-32
* Iterations: 79
* Convergence: max(|x - x_upper|, |x - x_lower|) <= 2*(1.5e-08*|x|+2.2e-16): true
* Objective Function Calls: 80

Lagrange Multiplier Method using NLsolve.jl

I would like to minimize a distance function ||dz - z|| under the constraint that g(z) = 0.
I wanted to use Lagrange Multipliers to solve this problem. Then I used NLsolve.jl to solve the non-linear equation that I end up with.
using NLsolve
using ForwardDiff
function ProjLagrange(dz, g::Function)
λ_init = ones(size(g(dz...),1))
initial_x = vcat(dz, λ_init)
function gradL!(F, x)
len_dz = length(dz)
z = x[1:len_dz]
λ = x[len_dz+1:end]
F = Array{Float64}(undef, length(x))
my_distance(z) = norm(dz - z)
∇f = z -> ForwardDiff.gradient(my_distance, z)
F[1:len_dz] = ∇f(z) .- dot(λ, g(z...))
if length(λ) == 1
F[end] = g(z...)
else
F[len_dz+1:end] = g(z)
end
end
nlsolve(gradL!, initial_x)
end
g_test(x1, x2, x3) = x1^2 + x2 - x2 + 5
z = [1000,1,1]
ProjLagrange(z, g_test)
But I always end up with Zero: [NaN, NaN, NaN, NaN] and Convergence: false.
Just so you know I have already solved the equation by using Optim.jl and minimizing the following function: Proj(z) = b * sum(abs.(g(z))) + a * norm(dz - z).
But I would really like to know if this is possible with NLsolve. Any help is greatly appreciated!
Starting almost from scratch and wikipedia's Lagrange multiplier page because it was good for me, the code below seemed to work. I added an λ₀s argument to the ProjLagrange function so that it can accept a vector of initial multiplier λ values (I saw you initialized them at 1.0 but I thought this was more generic). (Note this has not been optimized for performance!)
using NLsolve, ForwardDiff, LinearAlgebra
function ProjLagrange(x₀, λ₀s, gs, n_it)
# distance function from x₀ and its gradients
f(x) = norm(x - x₀)
∇f(x) = ForwardDiff.gradient(f, x)
# gradients of the constraints
∇gs = [x -> ForwardDiff.gradient(g, x) for g in gs]
# Form the auxiliary function and its gradients
ℒ(x,λs) = f(x) - sum(λ * g(x) for (λ,g) in zip(λs,gs))
∂ℒ∂x(x,λs) = ∇f(x) - sum(λ * ∇g(x) for (λ,∇g) in zip(λs,∇gs))
∂ℒ∂λ(x,λs) = [g(x) for g in gs]
# as a function of a single argument
nx = length(x₀)
ℒ(v) = ℒ(v[1:nx], v[nx+1:end])
∇ℒ(v) = vcat(∂ℒ∂x(v[1:nx], v[nx+1:end]), ∂ℒ∂λ(v[1:nx], v[nx+1:end]))
# and solve
v₀ = vcat(x₀, λ₀s)
nlsolve(∇ℒ, v₀, iterations=n_it)
end
# test
gs_test = [x -> x[1]^2 + x[2] - x[3] + 5]
λ₀s_test = [1.0]
x₀_test = [1000.0, 1.0, 1.0]
n_it = 100
res = ProjLagrange(x₀_test, λ₀s_test, gs_test, n_it)
gives me
julia> res = ProjLagrange(x₀_test, λ₀s_test, gs_test, n_it)
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [1000.0, 1.0, 1.0, 1.0]
* Zero: [9.800027199717013, -49.52026655749088, 51.520266557490885, -0.050887973682118504]
* Inf-norm of residuals: 0.000000
* Iterations: 10
* Convergence: true
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: true
* Function Calls (f): 11
* Jacobian Calls (df/dx): 11
I altered your code as below (see my comments in there) and got the following output. It doesn't throw NaNs anymore, reduces the objective and converges. Does this differ from your Optim.jl results?
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [1000.0, 1.0, 1.0, 1.0]
* Zero: [9.80003, -49.5203, 51.5203, -0.050888]
* Inf-norm of residuals: 0.000000
* Iterations: 10
* Convergence: true
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: true
* Function Calls (f): 11
* Jacobian Calls (df/dx): 11
using NLsolve
using ForwardDiff
using LinearAlgebra: norm, dot
using Plots
function ProjLagrange(dz, g::Function, n_it)
λ_init = ones(size(g(dz),1))
initial_x = vcat(dz, λ_init)
# These definitions can go outside as well
len_dz = length(dz)
my_distance = z -> norm(dz - z)
∇f = z -> ForwardDiff.gradient(my_distance, z)
# In fact, this is probably the most vital difference w.r.t. your proposal.
# We need the gradient of the constraints.
∇g = z -> ForwardDiff.gradient(g, z)
function gradL!(F, x)
z = x[1:len_dz]
λ = x[len_dz+1:end]
# `F` is memory allocated by NLsolve to store the residual of the
# respective call of `gradL!` and hence doesn't need to be allocated
# anew every time (or at all).
F[1:len_dz] = ∇f(z) .- λ .* ∇g(z)
F[len_dz+1:end] .= g(z)
end
return nlsolve(gradL!, initial_x, iterations=n_it, store_trace=true)
end
# Presumable here is something wrong: x2 - x2 is not very likely, also made it
# callable directly with an array argument
g_test = x -> x[1]^2 + x[2] - x[3] + 5
z = [1000,1,1]
n_it = 10000
res = ProjLagrange(z, g_test, n_it)
# Ugly reformatting here
trace = hcat([[state.iteration; state.fnorm; state.stepnorm] for state in res.trace.states]...)
plot(trace[1,:], trace[2,:], label="f(x) inf-norm", xlabel="steps")
Evolution of inf-norm of f(x) over iteration steps
[Edit: Adapted solution to incorporate correct gradient computation for g()]

Solve linear Equation System

I have two (mathematical) functions:
y = x
y = -2x + 3
This is solved by y = 1 and x = 1. See picture:
How can I make Julia do this for me?
This is a set of linear equations so first rearrange them in the following way:
-x + y = 0
2x + y = 3
and you see that they are in the form of a linear equation system A*v=b where. A is a matrix:
julia> A = [-1 1; 2 1]
2×2 Array{Int64,2}:
-1 1
2 1
and b is a vector:
julia> b = [0, 3]
2-element Array{Int64,1}:
0
3
Now v contains your unknown variables x and y. You can now solve the system using the left division operator \:
julia> A\b
2-element Array{Float64,1}:
1.0
1.0
If you had a more general system of non-linear equations you should use NLsolve.jl package:
julia> using NLsolve
julia> function f!(F, v)
x = v[1]
y = v[2]
F[1] = -x + y
F[2] = 2*x + y - 3
end
f! (generic function with 1 method)
julia> res = nlsolve(f!, [0.0; 0.0])
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [0.0, 0.0]
* Zero: [1.0000000000003109, 0.9999999999999647]
* Inf-norm of residuals: 0.000000
* Iterations: 2
* Convergence: true
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: true
* Function Calls (f): 3
* Jacobian Calls (df/dx): 3
julia> res.zero
2-element Array{Float64,1}:
1.0000000000003109
0.9999999999999647
(note that in f! we define two outputs F[1] and F[2] to be equal to zero - you have to rearrange your equations in this way).
For more details how to use NLsolve.jl see https://github.com/JuliaNLSolvers/NLsolve.jl.
Mr #bogumił-kamiński gave an excellent answer. However, just a friendly reminder, the solution MAY NOT EXIST for some other system of linear equations. In that case, you'll get a SingularException. Consider checking if the solution exists or not. For example,
using LinearAlgebra;
"""
y = x => x - y = 0 => |1 -1| X = |0| => AX = B => X = A⁻¹B
y = -2x + 3 2x + y = 3 |2 1| |3|
"""
function solution()
A::Matrix{Int64} = Matrix{Int64}([1 -1; 2 1]);
br::Matrix{Int64} = Matrix{Int64}([0 3]);
bc = transpose(br);
# bc::Matrix{Int64} = Matrix{Int64}([0 ;3;;]); # I missed a semicolon, that's why I got an error
# println(bc);
if(det(A) == 0) # existence check
println("soln doesnot exist. returning...");
return
end
A⁻¹ = inv(A);
println("solution exists:")
X = A⁻¹ * bc;
println(X);
end
solution();

non-linear solver always produces zero residual

I am learning how to solve non-linear equations and Julia (1.3.1) in general and want to ask how we should use NLsolve.
As a first step, I try the following:
using NLsolve
uni = 1
z = 3
x = [3,2,4,5,6]
y = z .+ x
print(y)
function g!(F, x)
F[1:uni+4] = y .- z - x
end
nlsolve(g!, [0.5,0.9,1,2,3])
And, I confirm it works as below:
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [0.5, 0.9, 1.0, 2.0, 3.0]
* Zero: [2.999999999996542, 2.000000000003876, 4.000000000008193, 4.999999999990685, 5.999999999990221]
* Inf-norm of residuals: 0.000000
* Iterations: 2
* Convergence: true
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: true
* Function Calls (f): 3
* Jacobian Calls (df/dx): 3
Then, I try a more complicated model as following
using SpecialFunctions, NLsolve, Random
Random.seed!(1234)
S = 2
# Setting parameters
ε = 1.3
β = 0.4
γ = gamma((ε-1)/ε)
T = rand(S)
E = rand(S)
B = rand(S)
w = rand(S)
Q = rand(S)
d = [1 2 ;2 1 ]
# Construct a model
rvector = T.*Q.^(1-β).*B.^ε
svector = E.* w.^ε
Φ_all = (sum(sum(rvector * svector' .* d )))
π = rvector * svector' .* d ./ Φ_all
# These two are outcome the model
πR = (sum(π,dims=1))'
πM = sum(π,dims=2)
# E is now set as unknown and we want to estimate it given the outcome of the model
function f!(Res, Unknown)
rvector = T.*Q.^(1-β).*B.^ε
svector = Unknown[1:S].* w.^ε
Φ_all = (sum(sum(rvector * svector' .* d )))
π = rvector * svector' .* d ./ Φ_all
Res = ones(S+S)
Res[1:S] = πR - (sum(π,dims=1))'
Res[S+1:S+S] = πM - sum(π,dims=2)
end
nlsolve(f!, [0.5,0.6])
and this code produces a weird result as follows.
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [0.5, 0.6]
* Zero: [0.5, 0.6]
* Inf-norm of residuals: 0.000000
* Iterations: 0
* Convergence: true
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: true
* Function Calls (f): 1
* Jacobian Calls (df/dx): 1
So, essentially, the function always yields 0 as the return, and thus the initial input always becomes the solution. I cannot understand why this does not work and also why this behaves differently from the first example. Could I have your suggestion to fix it?
You need to update Res in place. Now you do
Res = ones(S+S)
which shadows the input Res. Just update Res directly.

How to use NLsolve to solve "only 1D equation"

The following is ok,
function f!(F,x)
F[1]=x[1]^3+8
F[2]=x[2]^2 -8
end
nlsolve(f!, [1.,2])
however, the following is bad:
function f!(F,x)
F[1]=x[1]^3+8
end
nlsolve(f!, [1.])
This is not an issue with 1D. The problem is with Jacobian of your function.
Here is an example of the same in 2D:
julia> function f!(F,x)
F[1]=x[1]^3+8
F[2]=x[2]^3-2
end
f! (generic function with 1 method)
julia> nlsolve(f!, [0.,0.])
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [0.0, 0.0]
* Zero: [NaN, NaN]
* Inf-norm of residuals: 8.000000
* Iterations: 1000
* Convergence: false
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: false
* Function Calls (f): 1001
* Jacobian Calls (df/dx): 2
Now let us go back to your function. If you start from [1.0] you are unlucky and nlsolve in the root finding process hits [0.0] exactly with its default parameters and you have a problem that then Jacobian is [0.0].
You can see it by running:
julia> function f!(F,x)
F[1]=x[1]^3+8
end
f! (generic function with 1 method)
julia> nlsolve(f!, [1.], show_trace=true, extended_trace=true, iterations=3);
Iter f(x) inf-norm Step 2-norm
------ -------------- --------------
0 9.000000e+00 NaN
* f(x): [9.0]
* g(x): [3.0]
* x: [1.0]
* delta: NaN
* rho: NaN
1 8.000000e+00 1.000000e+00
* f(x): [8.0]
* g(x): [0.0]
* x: [0.0]
* delta: 2.9999999999239875
* rho: 0.3777777777854353
2 NaN NaN
* f(x): [NaN]
* g(x): [0.0]
* x: [NaN]
* delta: 2.9999999999239875
* rho: NaN
3 NaN NaN
* f(x): [NaN]
* g(x): [0.0]
* x: [NaN]
* delta: 2.9999999999239875
* rho: NaN
You can fix it by changing a starting point or by changing factor:
julia> nlsolve(f!, [10.])
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [10.0]
* Zero: [-2.0]
* Inf-norm of residuals: 0.000000
* Iterations: 18
* Convergence: true
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: true
* Function Calls (f): 17
* Jacobian Calls (df/dx): 11
julia> nlsolve(f!, [1.], factor=0.5)
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [1.0]
* Zero: [-2.0]
* Inf-norm of residuals: 0.000000
* Iterations: 7
* Convergence: true
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: true
* Function Calls (f): 8
* Jacobian Calls (df/dx): 8
Also - as Chris suggests - Roots.jl has methods that are more robust because you have derivative free methods in it.
It's fine, but Roots.jl is probably better suited for this kind of equation. Its methods are directly for 1-dimensional rootfinding problems and can be more robust.

Resources