First post here, thanks for reading!
Problem: I have a Vector{String} - call it A - where each element is a part of an equation, e.g. the first element of A is "x[1] - (0.8*x[1])". I would like to write a macro that takes as arguments i) a String - call it fn_name - with the name of a function, ii) the vector A, and returns a function named fn_name which looks like
function fn_name(f, x)
f[1] = x[1] - (0.8*x[1])
f[2] = (exp(x[4]) - 0.8*exp(x[3]))^(-1.1) - (0.99*(exp(x[4]) - 0.8*exp(x[4]))^(-1.1)*(1.0 - 0.025 + 0.30*exp(x[1])*exp(x[2])^(0.30 - 1.0)))
f[3] = exp(x[2]) - ((1.0 - 0.025)*exp(x[2]) + exp(x[1])*exp(x[2])^0.30 - exp(x[4]))
f[4] = x[3] - (x[4])
end
where each rhs is one element of
A = ["x[1] - (0.8*x[1])", "(exp(x[4]) - 0.8*exp(x[3]))^(-1.1) - (0.99*(exp(x[4]) - 0.8*exp(x[4]))^(-1.1)*(1.0 - 0.025 + 0.30*exp(x[1])*exp(x[2])^(0.30 - 1.0)))", "exp(x[2]) - ((1.0 - 0.025)*exp(x[2]) + exp(x[1])*exp(x[2])^0.30 - exp(x[4]))", "x[3] - (x[4])"]
What I tried: my best attempt at solving the problem is the following
macro make_fn(fn_name, A)
esc(quote
function $(Symbol(fn_name))(f, x)
for i = 1:length($(A))
f[$i] = Meta.parse($(A)[$i])
end
end
end)
end
which however doesn't work: when I run #make_fn("my_name", A) I get the error LoadError: UndefVarError: i not defined.
I find it quite hard to wrap my head around Julia metaprogramming, and while I'd be very happy to avoid using it, I think for this problem it's unavoidable.
Can you please help me understand where my mistake is?
Thanks
Macros in this case are not only avoidable, but even inapplicable, unless A is literal known at compile time.
I can provide a solution using eval and some closures:
julia> function make_fn2(A)
Af = [#eval(x -> $(Meta.parse(expr))) for expr in A]
function (f, x)
for i in eachindex(A, f)
f[i] = Af[i](x)
end
return f
end
end
make_fn2 (generic function with 1 method)
julia> fn_name = make_fn2(A)
#46 (generic function with 1 method)
julia> fn_name(zeros(4), [1,2,3,4])
4-element Array{Float64,1}:
0.19999999999999996
-0.06594092302655707
49.82984401122239
-1.0
with the restrictions that
eval will evaluate the expressions in a global scope of the module where this is defined (so it is potentially a different scope that a scope of the calling function), and
the newly created function will work only if you first return to global scope (i.e. it will not work if you try to run it within a function in which you have created it).
But I'd really recommend thinking about a better input format than strings.
Related
I'm very new to Julia but I've got a some background in Scheme/Rust/F#.
Today I wanted to make yesterday's AoC nicer without an explicit number of nested loops.
I arrived at this working solution, but I don't like the last if. In the languages mentioned above I would call a function (or use a computation expression) that gives me the first result that is not None. For Julia, I expected something to do that. It does, but unexpectedly in an eager fashion.
So When I tried return something(find(r, n, start + 1, which), find(r, n - 1, start + 1, extended)), that also evaluated the second argument when the first already had a result—and thus crashed.
Is there a macro/lazy version or something that I didn't find? How are you supposed to handle a case like that?
I also thought about (short-circuited) or'ing them together, but I guess Julia's strictness in that matter spoils that.
using DataStructures
function find(r::Array{Int}, n, start = 1, which = nil())::Union{Int,Nothing}
if start <= length(r)
extended = cons(start, which)
with_current = sum(i -> r[i], extended)
if with_current == 2020 && n == 1
return prod(i -> r[i], extended)
else
# Unfortunately no :(
#return something(find(r, n, start + 1, which), find(r, n - 1, start + 1, extended))
re = find(r, n, start + 1, which)
if isnothing(re)
return find(r, n - 1, start + 1, extended)
else
re
end
end
end
end
Let me comment more on it why it is not possible given the discussion in the comments.
In Julia function arguments are evaluated eagerly, so Julia evaluates both find(r, n, start + 1, which) and find(r, n - 1, start + 1, extended) before passing them to something function.
Now, with macros you have (I am not writing in a fully general case for simplicity and I hope I got the hygiene right :)):
julia> macro something(x, y)
quote
local vx = $(esc(x))
isnothing(vx) ? $(esc(y)) : vx
end
end
#something (macro with 1 method)
julia> #something 1 2
1
julia> #something nothing 2
2
julia> #something 1 sqrt(-1)
1
julia> #something nothing sqrt(-1)
ERROR: DomainError with -1.0:
sqrt will only return a complex result if called with a complex argument. Try sqrt(Complex(x)).
(in a full-blown version of the macro varargs and Some should be handled to replicate something exactly)
Piqued by Bogumił's answer I wanted to write my first Julia macro. It took some time and numerous attempts to figure out syntax, hygiene and escaping but I'm quite happy now.
I thought it might be worth sharing and provide opportunity for suggestions/improvements.
A lazy #something analog to Base.something
function _something_impl(thing)
:(something($(esc(thing))))
end
function _something_impl(thing, rest...)
quote
local evalued = $(esc(thing))
if isnothing(evalued)
$(_something_impl(rest...))
else
something(evalued)
end
end
end
macro something(things...)
_something_impl(things...)
end
Version without exceptions
As I found exceptions raised from a macro like this not quite suitable, I also made a version that falls back to nothing.
function _something_nothing_impl(thing)
quote
local evaluated = $(esc(thing))
if isa(evaluated, Some)
evaluated.value
else
evaluated
end
end
end
function _something_nothing_impl(thing, rest...)
quote
local evalued = $(esc(thing))
if isnothing(evalued)
$(_something_nothing_impl(rest...))
else
something(evalued)
end
end
end
macro something_nothing(things...)
_something_nothing_impl(things...)
end
Now I guess the recursive middle function could also generated by a macro. :)
I'm trying to build a function that will output an expression to be assigned to a new in-memory function. I might be misinterpreting the capability of metaprogramming but, I'm trying to build a function that generates a math series and assigns it to a function such as:
main.jl
function series(iter)
S = ""
for i in 1:iter
a = "x^$i + "
S = S*a
end
return chop(S, tail=3)
end
So, this will build the pattern and I'm temporarily working with it in the repl:
julia> a = Meta.parse(series(4))
:(x ^ 1 + x ^ 2 + x ^ 3 + x ^ 4)
julia> f =eval(Meta.parse(series(4)))
120
julia> f(x) =eval(Meta.parse(series(4)))
ERROR: cannot define function f; it already has a value
Obviously eval isn't what I'm looking for in this case but, is there another function I can use? Or, is this just not a viable way to accomplish the task in Julia?
The actual error you get has to do nothing with metaprogramming, but with the fact that you are reassigning f, which was assigned a value before:
julia> f = 10
10
julia> f(x) = x + 1
ERROR: cannot define function f; it already has a value
Stacktrace:
[1] top-level scope at none:0
[2] top-level scope at REPL[2]:1
It just doesn't like that. Call either of those variables differently.
Now to the conceptual problem. First, what you do here is not "proper" metaprogramming in Julia: why deal with strings and parsing at all? You can work directly on expressions:
julia> function series(N)
S = Expr(:call, :+)
for i in 1:N
push!(S.args, :(x ^ $i))
end
return S
end
series (generic function with 1 method)
julia> series(3)
:(x ^ 1 + x ^ 2 + x ^ 3)
This makes use of the fact that + belongs to the class of expressions that are automatically collected in repeated applications.
Second, you don't call eval at the appropriate place. I assume you meant to say "give me the function of x, with the body being what series(4) returns". Now, while the following works:
julia> f3(x) = eval(series(4))
f3 (generic function with 1 method)
julia> f3(2)
30
it is not ideal, as you newly compile the body every time the function is called. If you do something like that, it is preferred to expand the code once into the body at function definition:
julia> #eval f2(x) = $(series(4))
f2 (generic function with 1 method)
julia> f2(2)
30
You just need to be careful with hygiene here. All depends on the fact that you know that the generated body is formulated in terms of x, and the function argument matches that. In my opinion, the most Julian way of implementing your idea is through a macro:
julia> macro series(N::Int, x)
S = Expr(:call, :+)
for i in 1:N
push!(S.args, :($x ^ $i))
end
return S
end
#series (macro with 1 method)
julia> #macroexpand #series(4, 2)
:(2 ^ 1 + 2 ^ 2 + 2 ^ 3 + 2 ^ 4)
julia> #series(4, 2)
30
No free variables remaining in the output.
Finally, as has been noted in the comments, there's a function (and corresponding macro) evalpoly in Base which generalizes your use case. Note that this function does not use code generation -- it uses a well-designed generated function, which in combination with the optimizations results in code that is usually equal to the macro-generated code.
Another elegant option would be to use the multiple-dispatch mechanism of Julia and dispatch the generated code on type rather than value.
#generated function series2(p::Val{N}, x) where N
S = Expr(:call, :+)
for i in 1:N
push!(S.args, :(x ^ $i))
end
return S
end
Usage
julia> series2(Val(20), 150.5)
3.5778761722367333e43
julia> series2(Val{20}(), 150.5)
3.5778761722367333e43
This task can be accomplished with comprehensions. I need to RTFM...
https://docs.julialang.org/en/v1/manual/arrays/#Generator-Expressions
I'm trying to call BLAS in Julia using ccall like this
ccall((BLAS.#blasfunc(:zgemm_), BLAS.libblas),...other arguments)
For as far as I can tell, this is the same way the LinearAlgebra package calls BLAS (link to source)
I get the following error however:
ccall: could not find function :zgemm_64_ in library libopenblas64_
Anyone have any idea what could be the problem?
EDIT: found out that using :zgemm_64_ directly instead of BLAS.#blasfunc(:zgemm_) solved the error, but I'd still like to know why.
In case it becomes necessary, here is the full function where I make the BLAS call.
import LinearAlgebra: norm, lmul!, rmul!, BlasInt, BLAS
# Preallocated version of A = A*B
function rmul!(
A::AbstractMatrix{T},
B::AbstractMatrix{T},
workspace::AbstractVector{T}
) where {T<:Number}
m,n,lw = size(A,1), size(B,2), length(workspace)
if(size(A,2) !== size(B,1))
throw(DimensionMismatch("dimensions of A and B don't match"))
end
if(size(B,1) !== n)
throw(DimensionMismatch("A must be square"))
end
if(lw < m*n)
throw(DimensionMismatch("provided workspace is too small"))
end
# Multiplication via direct blas call
ccall((BLAS.#blasfunc(:zgemm_), BLAS.libblas), Cvoid,
(Ref{UInt8}, Ref{UInt8}, Ref{BlasInt}, Ref{BlasInt},
Ref{BlasInt}, Ref{T}, Ptr{T}, Ref{BlasInt},
Ptr{T}, Ref{BlasInt}, Ref{T}, Ptr{T},
Ref{BlasInt}),
'N', 'N', m, n,n, 1.0, A, max(1,stride(A,2)),B, max(1,stride(B,2)), 0.0, workspace, n)
# Copy temp to A
for j=1:n
for i=1:m
A[i,j] = workspace[j+*(i-1)*n]
end
end
end
function test_rmul(m::Integer, n::Integer)
BLAS.set_num_threads(1)
A = rand(ComplexF64, m,n)
Q = rand(ComplexF64, n,n)
workspace = similar(A, m*n)
A_original = copy(A)
Q_original = copy(Q)
rmul!(A,Q,workspace)
#show norm(A_original*Q_original - A)
#show norm(Q_original - Q)
end
test_rmul(100,50)
BLAS.#blasfunc(:zgemm_) returns Symbol(":zgemm_64_"), and not :zgemm_64_, which looks rather strange in the first place... it's hygienic in the technical sense, but admittedly confusing. The reason it works in the original implementation is because there, the symbol with the name is always spliced into #eval; compare:
julia> #eval begin
BLAS.#blasfunc(:zgemm_)
end
Symbol(":zgemm_64_")
julia> #eval begin
BLAS.#blasfunc($(:zgemm_))
end
:zgemm_64_
So, #blasfunc expects its argument to be a name (i.e., a symbol in the AST), not a symbol literal (a quoted symbol in the AST). You could equivalently write it like a variable name:
julia> #eval begin
BLAS.#blasfunc zgemm_
end
:zgemm_64_
(without zgemm_ being actually defined in this scope!)
I'm writing a genetic program in order to test the fitness of randomly generated expressions. Shown here is the function to generate the expression as well a the main function. DIV and GT are defined elsewhere in the code:
function create_single_full_tree(depth, fs, ts)
"""
Creates a single AST with full depth
Inputs
depth Current depth of tree. Initially called from main() with max depth
fs Function Set - Array of allowed functions
ts Terminal Set - Array of allowed terminal values
Output
Full AST of typeof()==Expr
"""
# If we are at the bottom
if depth == 1
# End of tree, return function with two terminal nodes
return Expr(:call, fs[rand(1:length(fs))], ts[rand(1:length(ts))], ts[rand(1:length(ts))])
else
# Not end of expression, recurively go back through and create functions for each new node
return Expr(:call, fs[rand(1:length(fs))], create_single_full_tree(depth-1, fs, ts), create_single_full_tree(depth-1, fs, ts))
end
end
function main()
"""
Main function
"""
# Define functional and terminal sets
fs = [:+, :-, :DIV, :GT]
ts = [:x, :v, -1]
# Create the tree
ast = create_single_full_tree(4, fs, ts)
#println(typeof(ast))
#println(ast)
#println(dump(ast))
x = 1
v = 1
eval(ast) # Error out unless x and v are globals
end
main()
I am generating a random expression based on certain allowed functions and variables. As seen in the code, the expression can only have symbols x and v, as well as the value -1. I will need to test the expression with a variety of x and v values; here I am just using x=1 and v=1 to test the code.
The expression is being returned correctly, however, eval() can only be used with global variables, so it will error out when run unless I declare x and v to be global (ERROR: LoadError: UndefVarError: x not defined). I would like to avoid globals if possible. Is there a better way to generate and evaluate these generated expressions with locally defined variables?
Here is an example for generating an (anonymous) function. The result of eval can be called as a function and your variable can be passed as parameters:
myfun = eval(Expr(:->,:x, Expr(:block, Expr(:call,:*,3,:x) )))
myfun(14)
# returns 42
The dump function is very useful to inspect the expression that the parsers has created. For two input arguments you would use a tuple for example as args[1]:
julia> dump(parse("(x,y) -> 3x + y"))
Expr
head: Symbol ->
args: Array{Any}((2,))
1: Expr
head: Symbol tuple
args: Array{Any}((2,))
1: Symbol x
2: Symbol y
typ: Any
2: Expr
[...]
Does this help?
In the Metaprogramming part of the Julia documentation, there is a sentence under the eval() and effects section which says
Every module has its own eval() function that evaluates expressions in its global scope.
Similarly, the REPL help ?eval will give you, on Julia 0.6.2, the following help:
Evaluate an expression in the given module and return the result. Every Module (except those defined with baremodule) has its own 1-argument definition of eval, which evaluates expressions in that module.
I assume, you are working in the Main module in your example. That's why you need to have the globals defined there. For your problem, you can use macros and interpolate the values of x and y directly inside the macro.
A minimal working example would be:
macro eval_line(a, b, x)
isa(a, Real) || (warn("$a is not a real number."); return :(throw(DomainError())))
isa(b, Real) || (warn("$b is not a real number."); return :(throw(DomainError())))
return :($a * $x + $b) # interpolate the variables
end
Here, #eval_line macro does the following:
Main> #macroexpand #eval_line(5, 6, 2)
:(5 * 2 + 6)
As you can see, the values of macro's arguments are interpolated inside the macro and the expression is given to the user accordingly. When the user does not behave,
Main> #macroexpand #eval_line([1,2,3], 7, 8)
WARNING: [1, 2, 3] is not a real number.
:((Main.throw)((Main.DomainError)()))
a user-friendly warning message is provided to the user at parse-time, and a DomainError is thrown at run-time.
Of course, you can do these things within your functions, again by interpolating the variables --- you do not need to use macros. However, what you would like to achieve in the end is to combine eval with the output of a function that returns Expr. This is what the macro functionality is for. Finally, you would simply call your macros with an # sign preceding the macro name:
Main> #eval_line(5, 6, 2)
16
Main> #eval_line([1,2,3], 7, 8)
WARNING: [1, 2, 3] is not a real number.
ERROR: DomainError:
Stacktrace:
[1] eval(::Module, ::Any) at ./boot.jl:235
EDIT 1. You can take this one step further, and create functions accordingly:
macro define_lines(linedefs)
for (name, a, b) in eval(linedefs)
ex = quote
function $(Symbol(name))(x) # interpolate name
return $a * x + $b # interpolate a and b here
end
end
eval(ex) # evaluate the function definition expression in the module
end
end
Then, you can call this macro to create different line definitions in the form of functions to be called later on:
#define_lines([
("identity_line", 1, 0);
("null_line", 0, 0);
("unit_shift", 0, 1)
])
identity_line(5) # returns 5
null_line(5) # returns 0
unit_shift(5) # returns 1
EDIT 2. You can, I guess, achieve what you would like to achieve by using a macro similar to that below:
macro random_oper(depth, fs, ts)
operations = eval(fs)
oper = operations[rand(1:length(operations))]
terminals = eval(ts)
ts = terminals[rand(1:length(terminals), 2)]
ex = :($oper($ts...))
for d in 2:depth
oper = operations[rand(1:length(operations))]
t = terminals[rand(1:length(terminals))]
ex = :($oper($ex, $t))
end
return ex
end
which will give the following, for instance:
Main> #macroexpand #random_oper(1, [+, -, /], [1,2,3])
:((-)([3, 3]...))
Main> #macroexpand #random_oper(2, [+, -, /], [1,2,3])
:((+)((-)([2, 3]...), 3))
Thanks Arda for the thorough response! This helped, but part of me thinks there may be a better way to do this as it seems too roundabout. Since I am writing a genetic program, I will need to create 500 of these ASTs, all with random functions and terminals from a set of allowed functions and terminals (fs and ts in the code). I will also need to test each function with 20 different values of x and v.
In order to accomplish this with the information you have given, I have come up with the following macro:
macro create_function(defs)
for name in eval(defs)
ex = quote
function $(Symbol(name))(x,v)
fs = [:+, :-, :DIV, :GT]
ts = [x,v,-1]
return create_single_full_tree(4, fs, ts)
end
end
eval(ex)
end
end
I can then supply a list of 500 random function names in my main() function, such as ["func1, func2, func3,.....". Which I can eval with any x and v values in my main function. This has solved my issue, however, this seems to be a very roundabout way of doing this, and may make it difficult to evolve each AST with each iteration.
Is there a way to check if a function has keywords arguments in Julia? I am looking for something like has_kwargs(fun::Function) that would return true if fun has a method with keyword arguments.
The high level idea is to build a function:
function master_fun(foo::Any, fun::Function, ar::Tuple, kw::Tuple)
if has_kwargs(fun)
fun(ar... ; kw...)
else
fun(ar...)
end
end
Basically, #Michael K. Borregaard's suggestion to use try-catch is correct and officially works.
Looking into the unofficial implementation details, I came up with the followng:
haskw(f,tup) = isdefined(typeof(f).name.mt,:kwsorter) &&
length(methods(typeof(f).name.mt.kwsorter,(Vector{Any},typeof(f),tup...)))>0
This function first looks if there is any keyword processing on any method of the generic function, and if so, looks at the specific tuple of types.
For example:
julia> f(x::Int) = 1
f (generic function with 1 method)
julia> f(x::String ; y="value") = 2
f (generic function with 2 methods)
julia> haskw(f,(Int,))
false
julia> haskw(f,(String,))
true
This should be tested for the specific application, as it probably doesn't work when non-leaf types are involved. As Michael commented, in the question's context the statement would be:
if haskw(fun, typeof.(ar))
...
I don't think you can guarantee that a given function has keyword arguments. Check
f(;x = 3) = println(x)
f(x) = println(2x)
f(3)
#6
f(x = 3)
#3
f(3, x = 3)
#ERROR: MethodError: no method matching f(::Int64; x=3)
#Closest candidates are:
# f(::Any) at REPL[2]:1 got unsupported keyword argument "x"
# f(; x) at REPL[1]:1
So, does the f function have keywords? You can only check for a given method. Note that, in your example above, you'd normally just do
function master_fun(foo, fun::Function, ar::Tuple, kw....)
fun(ar... ; kw...)
end
which should work, and if keywords are passed to a function that does not take them you'd just leave the error reporting to fun. If that is not acceptable you could try to wrap the fun(ar...; kw...) in a try-catch block.