I would like to form expressions like
julia> ex = :(a[$i,$j] + b[$i,$j])
because I am interested in writting a program that it is able to write a system of equations in which the user only gives the structure of them and then that julia writes explicitly the chunk of code involved. More generally, I would like to embed functions in a expression, something like
julia> ex = :(a[$myfunc(i,j),$j] + b[$i,$j])
Upon evaluating the first line of code, I get
ERROR: UndefVarError: i not defined
in eval(::Module, ::Any) at ./boot.jl:226
On the other hand, if I write
julia> ex = :(a[i,j] + b[i,j])
:(a[i,j] + b[i,j])
julia> for i in 1:2
for j in 1:2
println(eval(ex))
end
end
ERROR: UndefVarError: i not defined
in eval(::Module, ::Any) at ./boot.jl:226
[inlined code] from ./boot.jl:225
in anonymous at ./<no file>:4294967295
in eval(::Module, ::Any) at ./boot.jl:226
How is interpolation done in this case?
eval works at global scope, so it cannot reference any local variables. What you can do, however, is interpolate that expression into a larger chunk of code that's intended to be executed at the global scope:
julia> a = [1 2; 3 4]
b = [5 6; 7 8]
ex = :(a[i,j] + b[i,j]);
julia> #eval for i in 1:2
for j in 1:2
println($ex)
end
end
6
8
10
12
The #eval macro is a simple shorthand for quoting the passed syntax and passing it along to the eval() function. It'd probably even be better to define a custom function:
julia> #eval function f()
for i in 1:2
for j in 1:2
println($ex)
end
end
end
f (generic function with 1 method)
julia> f()
6
8
10
12
The usual caveats about eval apply: any passed code can be maliciously crafted to do nefarious or surprising things. Or just crash the program.
Related
I'm trying to build a function that will output an expression to be assigned to a new in-memory function. I might be misinterpreting the capability of metaprogramming but, I'm trying to build a function that generates a math series and assigns it to a function such as:
main.jl
function series(iter)
S = ""
for i in 1:iter
a = "x^$i + "
S = S*a
end
return chop(S, tail=3)
end
So, this will build the pattern and I'm temporarily working with it in the repl:
julia> a = Meta.parse(series(4))
:(x ^ 1 + x ^ 2 + x ^ 3 + x ^ 4)
julia> f =eval(Meta.parse(series(4)))
120
julia> f(x) =eval(Meta.parse(series(4)))
ERROR: cannot define function f; it already has a value
Obviously eval isn't what I'm looking for in this case but, is there another function I can use? Or, is this just not a viable way to accomplish the task in Julia?
The actual error you get has to do nothing with metaprogramming, but with the fact that you are reassigning f, which was assigned a value before:
julia> f = 10
10
julia> f(x) = x + 1
ERROR: cannot define function f; it already has a value
Stacktrace:
[1] top-level scope at none:0
[2] top-level scope at REPL[2]:1
It just doesn't like that. Call either of those variables differently.
Now to the conceptual problem. First, what you do here is not "proper" metaprogramming in Julia: why deal with strings and parsing at all? You can work directly on expressions:
julia> function series(N)
S = Expr(:call, :+)
for i in 1:N
push!(S.args, :(x ^ $i))
end
return S
end
series (generic function with 1 method)
julia> series(3)
:(x ^ 1 + x ^ 2 + x ^ 3)
This makes use of the fact that + belongs to the class of expressions that are automatically collected in repeated applications.
Second, you don't call eval at the appropriate place. I assume you meant to say "give me the function of x, with the body being what series(4) returns". Now, while the following works:
julia> f3(x) = eval(series(4))
f3 (generic function with 1 method)
julia> f3(2)
30
it is not ideal, as you newly compile the body every time the function is called. If you do something like that, it is preferred to expand the code once into the body at function definition:
julia> #eval f2(x) = $(series(4))
f2 (generic function with 1 method)
julia> f2(2)
30
You just need to be careful with hygiene here. All depends on the fact that you know that the generated body is formulated in terms of x, and the function argument matches that. In my opinion, the most Julian way of implementing your idea is through a macro:
julia> macro series(N::Int, x)
S = Expr(:call, :+)
for i in 1:N
push!(S.args, :($x ^ $i))
end
return S
end
#series (macro with 1 method)
julia> #macroexpand #series(4, 2)
:(2 ^ 1 + 2 ^ 2 + 2 ^ 3 + 2 ^ 4)
julia> #series(4, 2)
30
No free variables remaining in the output.
Finally, as has been noted in the comments, there's a function (and corresponding macro) evalpoly in Base which generalizes your use case. Note that this function does not use code generation -- it uses a well-designed generated function, which in combination with the optimizations results in code that is usually equal to the macro-generated code.
Another elegant option would be to use the multiple-dispatch mechanism of Julia and dispatch the generated code on type rather than value.
#generated function series2(p::Val{N}, x) where N
S = Expr(:call, :+)
for i in 1:N
push!(S.args, :(x ^ $i))
end
return S
end
Usage
julia> series2(Val(20), 150.5)
3.5778761722367333e43
julia> series2(Val{20}(), 150.5)
3.5778761722367333e43
This task can be accomplished with comprehensions. I need to RTFM...
https://docs.julialang.org/en/v1/manual/arrays/#Generator-Expressions
If I have a function signature like f(args...; kwargs...), how can I get a specific keyword out of kwargs? Naïvely typing kwargs.x does not work:
julia> f(args...; kwargs...) = kwargs.x
f (generic function with 1 method)
julia> f(x=1)
ERROR: type Pairs has no field x
Stacktrace:
[1] getproperty(::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:x,),Tuple{Int64}}}, ::Symbol) at ./Base.jl:20
[2] #f#7(::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:x,),Tuple{Int64}}}, ::typeof(f)) at ./REPL[2]:1
[3] (::var"#kw##f")(::NamedTuple{(:x,),Tuple{Int64}}, ::typeof(f)) at ./none:0
[4] top-level scope at REPL[3]:1
This question appeared on the JuliaLang Slack channel in the #helpdesk. For an automatic invite to the very helpful julia slack, simply fill out https://slackinvite.julialang.org
The reason this happens is that splatted keyword arguments are not stored in a named tuple by default. We can see how they're stored like so:
julia> g(;kwargs...) = kwargs
g (generic function with 1 method)
julia> g(a=1)
pairs(::NamedTuple) with 1 entry:
:a => 1
julia> g(a=1) |> typeof
Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:a,),Tuple{Int64}}}
So the splatted kwargs are instead stored as some sort of iterator object. However, we can easily convert that kwargs iterator to a NamedTuple like so: (;kwargs...) and then access it in the way we'd expect, so your example would translate into
julia> f(args...; kwargs...) = (;kwargs...).x
f (generic function with 1 method)
julia> f(x=1, y=2)
1
Of course, the more idiomatic way to do this would be to instead write the function as
julia> f(args...; x, kwargs...) = x
f (generic function with 1 method)
julia> f(x=1, y=2)
1
but this assumes you know the name you want to access (x) at the time when you write the function.
A brief sidenote: If we return to our example of g(;kwargs...) = kwargs, we can ask for the fieldnames of the iterator object the was returned like so:
julia> g(x=1, y=2) |> typeof |> fieldnames
(:data, :itr)
Hm, what is this data field?
julia> g(x=1, y=2).data
(x = 1, y = 2)
Aha! so we can actually get the kwargs as a named tuple using that, i.e. f(;kwargs...) = kwargs.data.x would work, but I wouldn't recommend this approach since it seems to rely on undocumented behaviour, so it may be a mere implementation detail that is not guaranteed to be stable across julia versions.
I'm writing a genetic program in order to test the fitness of randomly generated expressions. Shown here is the function to generate the expression as well a the main function. DIV and GT are defined elsewhere in the code:
function create_single_full_tree(depth, fs, ts)
"""
Creates a single AST with full depth
Inputs
depth Current depth of tree. Initially called from main() with max depth
fs Function Set - Array of allowed functions
ts Terminal Set - Array of allowed terminal values
Output
Full AST of typeof()==Expr
"""
# If we are at the bottom
if depth == 1
# End of tree, return function with two terminal nodes
return Expr(:call, fs[rand(1:length(fs))], ts[rand(1:length(ts))], ts[rand(1:length(ts))])
else
# Not end of expression, recurively go back through and create functions for each new node
return Expr(:call, fs[rand(1:length(fs))], create_single_full_tree(depth-1, fs, ts), create_single_full_tree(depth-1, fs, ts))
end
end
function main()
"""
Main function
"""
# Define functional and terminal sets
fs = [:+, :-, :DIV, :GT]
ts = [:x, :v, -1]
# Create the tree
ast = create_single_full_tree(4, fs, ts)
#println(typeof(ast))
#println(ast)
#println(dump(ast))
x = 1
v = 1
eval(ast) # Error out unless x and v are globals
end
main()
I am generating a random expression based on certain allowed functions and variables. As seen in the code, the expression can only have symbols x and v, as well as the value -1. I will need to test the expression with a variety of x and v values; here I am just using x=1 and v=1 to test the code.
The expression is being returned correctly, however, eval() can only be used with global variables, so it will error out when run unless I declare x and v to be global (ERROR: LoadError: UndefVarError: x not defined). I would like to avoid globals if possible. Is there a better way to generate and evaluate these generated expressions with locally defined variables?
Here is an example for generating an (anonymous) function. The result of eval can be called as a function and your variable can be passed as parameters:
myfun = eval(Expr(:->,:x, Expr(:block, Expr(:call,:*,3,:x) )))
myfun(14)
# returns 42
The dump function is very useful to inspect the expression that the parsers has created. For two input arguments you would use a tuple for example as args[1]:
julia> dump(parse("(x,y) -> 3x + y"))
Expr
head: Symbol ->
args: Array{Any}((2,))
1: Expr
head: Symbol tuple
args: Array{Any}((2,))
1: Symbol x
2: Symbol y
typ: Any
2: Expr
[...]
Does this help?
In the Metaprogramming part of the Julia documentation, there is a sentence under the eval() and effects section which says
Every module has its own eval() function that evaluates expressions in its global scope.
Similarly, the REPL help ?eval will give you, on Julia 0.6.2, the following help:
Evaluate an expression in the given module and return the result. Every Module (except those defined with baremodule) has its own 1-argument definition of eval, which evaluates expressions in that module.
I assume, you are working in the Main module in your example. That's why you need to have the globals defined there. For your problem, you can use macros and interpolate the values of x and y directly inside the macro.
A minimal working example would be:
macro eval_line(a, b, x)
isa(a, Real) || (warn("$a is not a real number."); return :(throw(DomainError())))
isa(b, Real) || (warn("$b is not a real number."); return :(throw(DomainError())))
return :($a * $x + $b) # interpolate the variables
end
Here, #eval_line macro does the following:
Main> #macroexpand #eval_line(5, 6, 2)
:(5 * 2 + 6)
As you can see, the values of macro's arguments are interpolated inside the macro and the expression is given to the user accordingly. When the user does not behave,
Main> #macroexpand #eval_line([1,2,3], 7, 8)
WARNING: [1, 2, 3] is not a real number.
:((Main.throw)((Main.DomainError)()))
a user-friendly warning message is provided to the user at parse-time, and a DomainError is thrown at run-time.
Of course, you can do these things within your functions, again by interpolating the variables --- you do not need to use macros. However, what you would like to achieve in the end is to combine eval with the output of a function that returns Expr. This is what the macro functionality is for. Finally, you would simply call your macros with an # sign preceding the macro name:
Main> #eval_line(5, 6, 2)
16
Main> #eval_line([1,2,3], 7, 8)
WARNING: [1, 2, 3] is not a real number.
ERROR: DomainError:
Stacktrace:
[1] eval(::Module, ::Any) at ./boot.jl:235
EDIT 1. You can take this one step further, and create functions accordingly:
macro define_lines(linedefs)
for (name, a, b) in eval(linedefs)
ex = quote
function $(Symbol(name))(x) # interpolate name
return $a * x + $b # interpolate a and b here
end
end
eval(ex) # evaluate the function definition expression in the module
end
end
Then, you can call this macro to create different line definitions in the form of functions to be called later on:
#define_lines([
("identity_line", 1, 0);
("null_line", 0, 0);
("unit_shift", 0, 1)
])
identity_line(5) # returns 5
null_line(5) # returns 0
unit_shift(5) # returns 1
EDIT 2. You can, I guess, achieve what you would like to achieve by using a macro similar to that below:
macro random_oper(depth, fs, ts)
operations = eval(fs)
oper = operations[rand(1:length(operations))]
terminals = eval(ts)
ts = terminals[rand(1:length(terminals), 2)]
ex = :($oper($ts...))
for d in 2:depth
oper = operations[rand(1:length(operations))]
t = terminals[rand(1:length(terminals))]
ex = :($oper($ex, $t))
end
return ex
end
which will give the following, for instance:
Main> #macroexpand #random_oper(1, [+, -, /], [1,2,3])
:((-)([3, 3]...))
Main> #macroexpand #random_oper(2, [+, -, /], [1,2,3])
:((+)((-)([2, 3]...), 3))
Thanks Arda for the thorough response! This helped, but part of me thinks there may be a better way to do this as it seems too roundabout. Since I am writing a genetic program, I will need to create 500 of these ASTs, all with random functions and terminals from a set of allowed functions and terminals (fs and ts in the code). I will also need to test each function with 20 different values of x and v.
In order to accomplish this with the information you have given, I have come up with the following macro:
macro create_function(defs)
for name in eval(defs)
ex = quote
function $(Symbol(name))(x,v)
fs = [:+, :-, :DIV, :GT]
ts = [x,v,-1]
return create_single_full_tree(4, fs, ts)
end
end
eval(ex)
end
end
I can then supply a list of 500 random function names in my main() function, such as ["func1, func2, func3,.....". Which I can eval with any x and v values in my main function. This has solved my issue, however, this seems to be a very roundabout way of doing this, and may make it difficult to evolve each AST with each iteration.
I would like to be able to create a dispatch for a user-defined type which will essentially do an inplace copy. However, I would like to do it in a type-stable manner, and thus I would like to avoid using getfield directly, and instead try to use a generated function. Is it possible for a type like
type UserType{T}
x::Vector{T}
y::Vector{T}
z::T
end
to generate some function
recursivecopy!(A::UserType,B::UserType)
# Do it for x
if typeof(A.x) <: AbstractArray
recursivecopy!(A.x,B.x)
else
A.x = B.x
end
# Now for y
if typeof(A.y) <: AbstractArray
recursivecopy!(A.y,B.y)
else
A.y = B.y
end
# Now for z
if typeof(A.z) <: AbstractArray
recursivecopy!(A.z,B.z)
else
A.z = B.z
end
end
The recursivecopy! in RecursiveArrayTools.jl makes this handle nested (Vector{Vector}) types well, but the only problem is that I do not know the fields the user will have in advance, just at compile-time when this function would be called. Sounds like a job for generated functions, but I'm not quite sure how to generate this.
You don't need to bend over backwards to avoid getfield and setfield. Julia can infer them just fine. The trouble comes when Julia can't figure out which field it's accessing… like in a for loop.
So the only special thing the generated function needs to do is effectively unroll the loop with constant values spliced into getfield:
julia> immutable A
x::Int
y::Float64
end
julia> #generated function f(x)
args = [:(getfield(x, $i)) for i=1:nfields(x)]
:(tuple($(args...)))
end
f (generic function with 1 method)
julia> f(A(1,2.4))
(1,2.4)
julia> #code_warntype f(A(1,2.4))
Variables:
#self#::#f
x::A
Body:
begin # line 2:
return (Main.tuple)((Main.getfield)(x::A,1)::Int64,(Main.getfield)(x::A,2)::Float64)::Tuple{Int64,Float64}
end::Tuple{Int64,Float64}
Just like you can splice in multiple arguments to a function call, you can also directly splice in multiple expressions to the function body.
julia> type B
x::Int
y::Float64
end
julia> #generated function f!{T}(dest::T, src::T)
assignments = [:(setfield!(dest, $i, getfield(src, $i))) for i=1:nfields(T)]
:($(assignments...); dest)
end
f! (generic function with 1 method)
julia> f!(B(0,0), B(1, 2.4))
B(1,2.4)
julia> #code_warntype f!(B(0,0), B(1, 2.4))
Variables:
#self#::#f!
dest::B
src::B
Body:
begin # line 2:
(Main.setfield!)(dest::B,1,(Main.getfield)(src::B,1)::Int64)::Int64
(Main.setfield!)(dest::B,2,(Main.getfield)(src::B,2)::Float64)::Float64
return dest::B
end::B
You can, of course, make the body of that comprehension as complicated as you'd like. That effectively becomes the inside of your for loop. Splatting the array into the body of the function does the unrolling for you.
I have a function, f. I want to add a method that takes any container of Strings. For example, I want to write a method that generates the following when needed:
f(xs::Array{String, 1}) = ...
f(xs::DataArray{String, 1}) = ...
f(xs::ITERABLE{String}) = ...
Is this possible to do in Julia's type system? Right now, I'm using a macro to write a specialized method when I need it.
#make_f(Array{String, 1})
#make_f(DataArray{String, 1})
This keeps things DRY, but it feels...wrong.
Can't you just use duck typing? I.e., just assume that you're feeding the function an object of the right type and throw an error if at some point e.g. you don't have a string in your iterable.
This should improve once you can really talk about iterables using traits; currently there is no iterable type. Scott's answer, for example, will not work with a tuple of strings, even though that is iterable.
E.g.
julia> f(x) = string(x...) # just concatenate the strings
f (generic function with 1 method)
julia> f(("a", "á"))
"aá"
julia> f(["a", "á"])
"aá"
julia> f(["a" "b"; "c" "d"]) # a matrix of strings!
"acbd"
At least in Julia 0.4, the following should work:
julia> abstract Iterable{T} <: AbstractVector{T}
julia> f{T<:Union{Vector{String},Iterable{String}}}(xs::T) = 1
f (generic function with 1 method)
julia> x = String["a", "é"]
2-element Array{AbstractString,1}:
"a"
"é"
julia> f(x)
1