Restricting function signatures while using ForwardDiff in Julia - julia

I am trying to use ForwardDiff in a library where almost all functions are restricted to only take in Floats. I want to generalise these function signatures so that ForwardDiff can be used while still being restrictive enough so functions only take numeric values and not things like Dates. I have alot of functions with the same name but different types (ie functions that take in "time" as either a float or a Date with the same function name) and do not want to remove the type qualifiers throughout.
Minimal Working Example
using ForwardDiff
x = [1.0, 2.0, 3.0, 4.0 ,5.0]
typeof(x) # Array{Float64,1}
function G(x::Array{Real,1})
return sum(exp.(x))
end
function grad_F(x::Array)
return ForwardDiff.gradient(G, x)
end
G(x) # Method Error
grad_F(x) # Method error
function G(x::Array{Float64,1})
return sum(exp.(x))
end
G(x) # This works
grad_F(x) # This has a method error
function G(x)
return sum(exp.(x))
end
G(x) # This works
grad_F(x) # This works
# But now I cannot restrict the function G to only take numeric arrays and not for instance arrays of Dates.
Is there are a way to restict functions to only take numeric values (Ints and Floats) and whatever dual number structs that ForwardDiff uses but not allow Symbols, Dates, etc.

ForwardDiff.Dual is a subtype of the abstract type Real. The issue you have, however, is that Julia's type parameters are invariant, not covariant. The following, then, returns false.
# check if `Array{Float64, 1}` is a subtype of `Array{Real, 1}`
julia> Array{Float64, 1} <: Array{Real, 1}
false
That makes your function definition
function G(x::Array{Real,1})
return sum(exp.(x))
end
incorrect (not suitable for your use). That's why you get the following error.
julia> G(x)
ERROR: MethodError: no method matching G(::Array{Float64,1})
The correct definition should rather be
function G(x::Array{<:Real,1})
return sum(exp.(x))
end
or if you somehow need an easy access to the concrete element type of the array
function G(x::Array{T,1}) where {T<:Real}
return sum(exp.(x))
end
The same goes for your grad_F function.
You might find it useful to read the relevant section of the Julia documentation for types.
You might also want to type annotate your functions for AbstractArray{<:Real,1} type rather than Array{<:Real, 1} so that your functions can work other types of arrays, like StaticArrays, OffsetArrays etc., without a need for redefinitions.

This would accept any kind of array parameterized by any kind of number:
function foo(xs::AbstractArray{<:Number})
#show typeof(xs)
end
or:
function foo(xs::AbstractArray{T}) where T<:Number
#show typeof(xs)
end
In case you need to refer to the type parameter T inside the body function.
x1 = [1.0, 2.0, 3.0, 4.0 ,5.0]
x2 = [1, 2, 3,4, 5]
x3 = 1:5
x4 = 1.0:5.0
x5 = [1//2, 1//4, 1//8]
xss = [x1, x2, x3, x4, x5]
function foo(xs::AbstractArray{T}) where T<:Number
#show xs typeof(xs) T
println()
end
for xs in xss
foo(xs)
end
Outputs:
xs = [1.0, 2.0, 3.0, 4.0, 5.0]
typeof(xs) = Array{Float64,1}
T = Float64
xs = [1, 2, 3, 4, 5]
typeof(xs) = Array{Int64,1}
T = Int64
xs = 1:5
typeof(xs) = UnitRange{Int64}
T = Int64
xs = 1.0:1.0:5.0
typeof(xs) = StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}
T = Float64
xs = Rational{Int64}[1//2, 1//4, 1//8]
typeof(xs) = Array{Rational{Int64},1}
T = Rational{Int64}
You can run the example code here: https://repl.it/#SalchiPapa/Restricting-function-signatures-in-Julia

Related

Julia Flux withgradient operation

I am a newbie to Julia and Flux with some experience in Tensorflow Keras and python. I tried to use the Flux.withgradient command to write a user-defined training function with more flexibility. Here is the training part of my code:
loss, grad = Flux.withgradient(modelDQN.evalParameters) do
qEval = modelDQN.evalModel(evalInput)
Flux.mse(qEval, qTarget)
end
Flux.update!(modelDQN.optimizer, modelDQN.evalParameters, grad)
This code works just fine. But if I put the command qEval = modelDQN.evalModel(evalInput) outside the do end loop, as follows:
qEval = modelDQN.evalModel(evalInput)
loss, grad = Flux.withgradient(modelDQN.evalParameters) do
Flux.mse(qEval, qTarget)
end
Flux.update!(modelDQN.optimizer, modelDQN.evalParameters, grad)
The model parameters will not be updated. As far as I know, the do end loop works as an anonymous function that takes 0 arguments. Then why do we need the command qEval = modelDQN.evalModel(evalInput) inside the loop to get the model updated?
The short answer is that anything to be differentiated has to happen inside the (anonymous) function which you pass to gradient (or withgradient), because this is very much not a standard function call -- Zygote (Flux's auto-differentiation library) traces its execution to compute the derivative, and can't transform what it can't see.
Longer, this is Zygote's "implicit" mode, which relies on global references to arrays. The simplest use is something like this:
julia> using Zygote
julia> x = [2.0, 3.0];
julia> g = gradient(() -> sum(x .^ 2), Params([x]))
Grads(...)
julia> g[x] # lookup by objectid(x)
2-element Vector{Float64}:
4.0
6.0
If you move some of that calculation outside, then you make a new array y with a new objectid. Julia has no memory of where this came from, it is completely unrelated to x. They are ordinary arrays, not a special tracked type.
So if you refer to y in the gradient, Zygote cannot infer how this depends on x:
julia> y = x .^ 2 # calculate this outside of gradient
2-element Vector{Float64}:
4.0
9.0
julia> g2 = gradient(() -> sum(y), Params([x]))
Grads(...)
julia> g2[x] === nothing # represents zero
true
Zygote doesn't have to be used in this way. It also has an "explicit" mode which does not rely on global references. This is perhaps less confusing:
julia> gradient(x1 -> sum(x1 .^ 2), x) # x1 is a local variable
([4.0, 6.0],)
julia> gradient(x1 -> sum(y), x) # sum(y) is obviously indep. x1
(nothing,)
julia> gradient((x1, y1) -> sum(y1), x, y)
(nothing, Fill(1.0, 2))
Flux is in the process of changing to use this second form. On v0.13.9 or later, something like this ought to work:
opt_state = Flux.setup(modelDQN.optimizer, modelDQN) # do this once
loss, grads = Flux.withgradient(modelDQN.model) do m
qEval = m(evalInput) # local variable m
Flux.mse(qEval, qTarget)
end
Flux.update!(opt_state, modelDQN.model, grads[1])

Julia JuMP making sure nonlinear objective function has correct function signatures so that autodifferentiate works properly?

so I wrote a minimum example to show what I'm trying to do. Basically I want to solve a optimization problem with multiple variables. When I try to do this in JuMP I was having issues with my function obj not being able to take a forwardDiff object.
I looked here: and it seemed to do with the function signature :Restricting function signatures while using ForwardDiff in Julia . I did this in my obj function, and for insurance did it in my sub-function as well, but I still get the error
LoadError: MethodError: no method matching Float64(::ForwardDiff.Dual{ForwardDiff.Tag{JuMP.var"#110#112"{typeof(my_fun)},Float64},Float64,2})
Closest candidates are:
Float64(::Real, ::RoundingMode) where T<:AbstractFloat at rounding.jl:200
Float64(::T) where T<:Number at boot.jl:715
Float64(::Int8) at float.jl:60
This still does not work. I feel like I have the bulk of the code correct, just some weird of type thing going on that I have to clear up so autodifferentiate works...
Any suggestions?
using JuMP
using Ipopt
using LinearAlgebra
function obj(x::Array{<:Real,1})
println(x)
x1 = x[1]
x2 = x[2]
eye= Matrix{Float64}(I, 4, 4)
obj_val = tr(eye-kron(mat_fun(x1),mat_fun(x2)))
println(obj_val)
return obj_val
end
function mat_fun(var::T) where {T<:Real}
eye= Matrix{Float64}(I, 2, 2)
eye[2,2]=var
return eye
end
m = Model(Ipopt.Optimizer)
my_fun(x...) = obj(collect(x))
#variable(m, 0<=x[1:2]<=2.0*pi)
register(m, :my_fun, 2, my_fun; autodiff = true)
#NLobjective(m, Min, my_fun(x...))
optimize!(m)
# retrieve the objective value, corresponding x values and the status
println(JuMP.value.(x))
println(JuMP.objective_value(m))
println(JuMP.termination_status(m))
Use instead
function obj(x::Vector{T}) where {T}
println(x)
x1 = x[1]
x2 = x[2]
eye= Matrix{T}(I, 4, 4)
obj_val = tr(eye-kron(mat_fun(x1),mat_fun(x2)))
println(obj_val)
return obj_val
end
function mat_fun(var::T) where {T}
eye= Matrix{T}(I, 2, 2)
eye[2,2]=var
return eye
end
Essentially, anywhere you see Float64, replace it by the type in the incoming argument.
I found the problem:
in my mat_fun the type of the return had to be "Real" in order for it to propgate through. Before it was Float64, which was not consistent with the fact I guess all types have to be Real with the autodifferentiate. Even though a Float64 is clearly Real, it looks like the inheritence isn't perserved i.e you have to make sure everything that is returned and inputed are type Real.
using JuMP
using Ipopt
using LinearAlgebra
function obj(x::AbstractVector{T}) where {T<:Real}
println(x)
x1 = x[1]
x2 = x[2]
eye= Matrix{Float64}(I, 4, 4)
obj_val = tr(eye-kron(mat_fun(x1),mat_fun(x2)))
#println(obj_val)
return obj_val
end
function mat_fun(var::T) where {T<:Real}
eye= zeros(Real,(2,2))
eye[2,2]=var
return eye
end
m = Model(Ipopt.Optimizer)
my_fun(x...) = obj(collect(x))
#variable(m, 0<=x[1:2]<=2.0*pi)
register(m, :my_fun, 2, my_fun; autodiff = true)
#NLobjective(m, Min, my_fun(x...))
optimize!(m)
# retrieve the objective value, corresponding x values and the status
println(JuMP.value.(x))
println(JuMP.objective_value(m))
println(JuMP.termination_status(m))

Using ForwardDiff.jl for a function of many variables and parameters Julia

The github repo for ForwardDiff.jl has some examples. I am trying to extend the example to take in addition to a vector of variables, a parameter. I cannot get it to work.
This is the example (it is short so I will show it rather than linking)
using ForwardDiff
x = rand(5)
f(x::Vector) = sum(sin, x) .+ prod(tan, x) * sum(sqrt, x);
g = x -> ForwardDiff.gradient(f, x);
g(x) # this outputs the gradient.
I want to modify this since I use functions with multiple parameters as well as variables. As a simple modification I have tried adding a single parameter.
f(x::Vector, y) = (sum(sin, x) .+ prod(tan, x) * sum(sqrt, x)) * y;
I have tried the following to no avail:
fp = x -> ForwardDiff.gradient(f, x);
fp = x -> ForwardDiff.gradient(f, x, y);
y = 1
println("test grad: ", fp(x, y))
I get the following error message:
ERROR: LoadError: MethodError: no method matching (::var"#73#74")(::Array{Float64,1}, ::Int64)
A similar question was not answered in 2017. A comment led me to here and it seems the function can only accept one input?
The target function must be unary (i.e., only accept a single argument). ForwardDiff.jacobian is an exception to this rule.
Has this changed? It seems very limited to only be able to differentiate unary functions.
A possible workaround would be to concatenate the list of variables and parameters and then just slice the returned gradient to not include the gradients with respect to the parameters, but this seems silly.
I personally think it makes sense to have this unary-only syntax for ForwardDiff. In your case, you could just pack/unpack x and y into a single vector (nominally x2 below):
julia> using ForwardDiff
julia> x = rand(5)
5-element Array{Float64,1}:
0.4304735670747184
0.3939269364431113
0.7912705403776603
0.8942024934250143
0.5724373306715196
julia> f(x::Vector, y) = (sum(sin, x) .+ prod(tan, x) * sum(sqrt, x)) * y;
julia> y = 1
1
julia> f(x2::Vector) = f(x2[1:end-1], x2[end]) % unpacking in f call
f (generic function with 2 methods)
julia> fp = x -> ForwardDiff.gradient(f, x);
julia> println("test grad: ", fp([x; y])) % packing in fp call
test grad: [2.6105844240785796, 2.741442601659502, 1.9913192377198885, 1.9382805843854594, 2.26202717745402, 3.434350946190029]
But my preference would be to explicitly name the partial derivatives differently:
julia> ∂f∂x(x,y) = ForwardDiff.gradient(x -> f(x,y), x)
∂f∂x (generic function with 1 method)
julia> ∂f∂y(x,y) = ForwardDiff.derivative(y -> f(x,y), y)
∂f∂y (generic function with 1 method)
julia> ∂f∂x(x, y)
5-element Array{Float64,1}:
2.6105844240785796
2.741442601659502
1.9913192377198885
1.9382805843854594
2.26202717745402
julia> ∂f∂y(x, y)
3.434350946190029
Here's a quick attempt at a function which takes multiple arguments, the same signature as Zygote.gradient:
julia> using ForwardDiff, Zygote
julia> multigrad(f, xs...) = ntuple(length(xs)) do i
g(y) = f(ntuple(j -> j==i ? y : xs[j], length(xs))...)
xs[i] isa AbstractArray ? ForwardDiff.gradient(g, xs[i]) :
xs[i] isa Number ? ForwardDiff.derivative(g, xs[i]) : nothing
end;
julia> f1(x,y,z) = sum(x.^2)/y;
julia> multigrad(f1, [1,2,3], 4)
([0.5, 1.0, 1.5], -0.875)
julia> Zygote.gradient(f1, [1,2,3], 4)
([0.5, 1.0, 1.5], -0.875)
For a function with several scalar arguments, this evaluates each derivative separately, and perhaps it would be more efficient to use one evaluation with some Dual(x, (dx, dy, dz)). With large-enough array arguments, ForwardDiff.gradient will already perform multiple evaluations, each with some number of perturbations (the chunk size, which you can control).

Specializing method calls in order in meta-programming

I have issue after calling my macro:
#introspectable square(x) = x * x
Then when calling
square(3)
i should be able to get 9, cause the function call has been specialized to execute an attribute of the structure which is Julia code, however when I enter the macro, the code seems to be directly evaluated.
What i have tried:
struct IntrospectableFunction
name
parameters
native_function
end
(f::IntrospectableFunction)(x) = f.native_function(x)
macro introspectable(expr)
name = expr.args[1].args[1]
parameters = tuple(expr.args[1].args[2:end]...)
body = expr.args[2].args[2]
:( global $name = IntrospectableFunction( :( name ), $parameters, :( body ) ))
end
#introspectable square(x) = x * x
square(3)
The answer should be 9 , however i get "Object of type symbol are not callable ". However if i replace :( body ) with x -> x * x i get the desired result, my objective is generalizing the macro-call.
I usually find it easier to work with expressions in macros (it is not the shortest way to write things, but, from my experience, it is much easier to control what gets generated).
Therefore I would rewrite your code as:
macro introspectable(expr)
name = expr.args[1].args[1]
parameters = expr.args[1].args[2:end]
anon = Expr(Symbol("->"), Expr(:tuple, parameters...), expr.args[2].args[2])
constr = Expr(:call, :IntrospectableFunction, QuoteNode(name), Tuple(parameters), anon)
esc(Expr(:global, Expr(Symbol("="), name, constr)))
end
Now, as you said you wanted generality I would define your functor like this:
(f::IntrospectableFunction)(x...) = f.native_function(x...)
(in this way you allow multiple positional arguments to be passed).
Now let us test our definitions:
julia> #introspectable square(x) = x * x
IntrospectableFunction(:square, (:x,), getfield(Main, Symbol("##3#4"))())
julia> square(3)
9
julia> #macroexpand #introspectable square(x) = x * x
:(global square = IntrospectableFunction(:square, (:x,), ((x,)->x * x)))
julia> #introspectable toarray(x,y) = [x,y]
IntrospectableFunction(:toarray, (:x, :y), getfield(Main, Symbol("##5#6"))())
julia> toarray("a", 10)
2-element Array{Any,1}:
"a"
10
julia> #macroexpand #introspectable toarray(x,y) = [x,y]
:(global toarray = IntrospectableFunction(:toarray, (:x, :y), ((x, y)->[x, y])))
julia> function localscopetest()
#introspectable globalfun(x...) = x
end
localscopetest (generic function with 1 method)
julia> localscopetest()
IntrospectableFunction(:globalfun, (:(x...),), getfield(Main, Symbol("##9#10"))())
julia> globalfun(1,2,3,4,5)
(1, 2, 3, 4, 5)
julia> function f()
v = 100
#introspectable localbinding(x) = (v, x)
end
f (generic function with 1 method)
julia> f()
IntrospectableFunction(:localbinding, (:x,), getfield(Main, Symbol("##11#12")){Int64}(100))
julia> localbinding("x")
(100, "x")
(note that it is useful to use #macroexpand to make sure our macro works as expected)
EDIT - how to handle a minimal multiple dispatch
I am writing a non-macro example because it is related to the data structure:
Use e.g. such a definition:
struct IntrospectableFunction
name::Symbol
method_array::Vector{Pair{Type{<:Tuple}, Function}}
end
function (f::IntrospectableFunction)(x...)
for m in f.method_array
if typeof(x) <: first(m)
return last(m)(x...)
end
end
error("signature not found")
end
and now you can write:
julia> square = IntrospectableFunction(:square, [Tuple{Any}=>x->x*x,Tuple{Any,Any}=>(x,y)->x*y])
IntrospectableFunction(:square, Pair{DataType,Function}[Tuple{Any}=>##9#11(), Tuple{Any,Any}=>##10#12()])
julia> square(3)
9
julia> square(2,3)
6
Keep in mind that the approach I present is not perfect and universal - it just serves to give a very simple example how you could do it.

Generating type-stable `getfield` calls using generated functions

I would like to be able to create a dispatch for a user-defined type which will essentially do an inplace copy. However, I would like to do it in a type-stable manner, and thus I would like to avoid using getfield directly, and instead try to use a generated function. Is it possible for a type like
type UserType{T}
x::Vector{T}
y::Vector{T}
z::T
end
to generate some function
recursivecopy!(A::UserType,B::UserType)
# Do it for x
if typeof(A.x) <: AbstractArray
recursivecopy!(A.x,B.x)
else
A.x = B.x
end
# Now for y
if typeof(A.y) <: AbstractArray
recursivecopy!(A.y,B.y)
else
A.y = B.y
end
# Now for z
if typeof(A.z) <: AbstractArray
recursivecopy!(A.z,B.z)
else
A.z = B.z
end
end
The recursivecopy! in RecursiveArrayTools.jl makes this handle nested (Vector{Vector}) types well, but the only problem is that I do not know the fields the user will have in advance, just at compile-time when this function would be called. Sounds like a job for generated functions, but I'm not quite sure how to generate this.
You don't need to bend over backwards to avoid getfield and setfield. Julia can infer them just fine. The trouble comes when Julia can't figure out which field it's accessing… like in a for loop.
So the only special thing the generated function needs to do is effectively unroll the loop with constant values spliced into getfield:
julia> immutable A
x::Int
y::Float64
end
julia> #generated function f(x)
args = [:(getfield(x, $i)) for i=1:nfields(x)]
:(tuple($(args...)))
end
f (generic function with 1 method)
julia> f(A(1,2.4))
(1,2.4)
julia> #code_warntype f(A(1,2.4))
Variables:
#self#::#f
x::A
Body:
begin # line 2:
return (Main.tuple)((Main.getfield)(x::A,1)::Int64,(Main.getfield)(x::A,2)::Float64)::Tuple{Int64,Float64}
end::Tuple{Int64,Float64}
Just like you can splice in multiple arguments to a function call, you can also directly splice in multiple expressions to the function body.
julia> type B
x::Int
y::Float64
end
julia> #generated function f!{T}(dest::T, src::T)
assignments = [:(setfield!(dest, $i, getfield(src, $i))) for i=1:nfields(T)]
:($(assignments...); dest)
end
f! (generic function with 1 method)
julia> f!(B(0,0), B(1, 2.4))
B(1,2.4)
julia> #code_warntype f!(B(0,0), B(1, 2.4))
Variables:
#self#::#f!
dest::B
src::B
Body:
begin # line 2:
(Main.setfield!)(dest::B,1,(Main.getfield)(src::B,1)::Int64)::Int64
(Main.setfield!)(dest::B,2,(Main.getfield)(src::B,2)::Float64)::Float64
return dest::B
end::B
You can, of course, make the body of that comprehension as complicated as you'd like. That effectively becomes the inside of your for loop. Splatting the array into the body of the function does the unrolling for you.

Resources