I want to overwrite a function in Julia using its old definition. It seems the way to do this would be to clone the function and overwrite the original using the copy — something like the following. However, it appears deepcopy(f) just returns a reference to f, so this doesn't work.
f(x) = x
f_old = deepcopy(f)
f(x) = 1 + f_old(x)
How can I clone a function?
Background: I'm interesting in writing a macro #override that allows me to override functions pointwise (or maybe even piecewise).
fib(n::Int) = fib(n-1) + fib(n-2)
#override fib(0) = 1
#override fib(1) = 1
This particular example would be slow and could be made more efficient using #memoize. There may be good reasons not to do this, but there may also be situations in which one does not know a function fully when it is defined and overriding is necessary.
We can do this using IRTools.jl.
(Note, on newer versions of IRTools, you may need to ask for IRTools.Inner.code_ir instead of IRTools.code_ir.)
using IRTools
fib(n::Int) = fib(n-1) + fib(n-2)
const fib_ir = IRTools.code_ir(fib, Tuple{Int})
const fib_old = IRTools.func(fib_ir)
fib(n::Int) = n < 2 ? 1 : fib_old(fib, n)
julia> fib(10)
89
What we did there was captured the intermediate representation of the function fib, and then rebuilt it into a new function which we called fib_old. Then we were free to overwrite the definition of fib in terms of fib_old! Notice that since fib_old was defined as recursively calling fib, not fib_old, there's no stack overflow when we call fib(10).
The other thing to notice is that when we called fib_old, we wrote fib_old(fib, n) instead of fib_old(n). This is due to how IRTools.func works.
According to Mike Innes on Slack:
In Julia IR, all functions take a hidden extra argument that represents the function itself
The reason for this is that closures are structs with fields, which you need access to in the IR
Here's an implementation of your #override macro with a slightly different syntax:
function _get_type_sig(fdef)
d = splitdef(fdef)
types = []
for arg in d[:args]
if arg isa Symbol
push!(types, :Any)
elseif #capture(arg, x_::T_)
push!(types, T)
else
error("whoops!")
end
end
if isempty(d[:whereparams])
:(Tuple{$(types...)})
else
:((Tuple{$(types...)} where {$(d[:whereparams]...)}).body)
end
end
macro override(cond, fdef)
d = splitdef(fdef)
shadowf = gensym()
sig = _get_type_sig(fdef)
f = d[:name]
quote
const $shadowf = IRTools.func(IRTools.code_ir($(d[:name]), $sig))
function $f($(d[:args]...)) where {$(d[:whereparams]...)}
if $cond
$(d[:body])
else
$shadowf($f, $(d[:args]...))
end
end
end |> esc
end
Now one can type
fib(n::Int) = fib(n-1) + fib(n-2)
#override n < 2 fib(n::Int) = 1
julia> fib(10)
89
The best part is that this is nearly as fast (at runtime, not compile time!) as if we had written the conditions into the original function!
n = 15
fib2(n::Int) = n < 2 ? 1 : fib2(n-1) + fib2(n-2)
julia> #btime fib($(Ref(15))[])
4.239 μs (0 allocations: 0 bytes)
89
julia> #btime fib2($(Ref(15))[])
3.022 μs (0 allocations: 0 bytes)
89
I really don't see why you'd want to do this (there must a better way to get what you want!).
Nonetheless, although not exactly equivalent you can get what you want by using anonymous functions:
julia> f = x->x
#3 (generic function with 1 method)
julia> f_old = deepcopy(f)
#3 (generic function with 1 method)
julia> f = x->1+f_old(x)
#5 (generic function with 1 method)
julia> f(4)
5
Related
I'm playing around with the Collatz conjecture (see below), and I have this function:
function txpo(max_n)
for n ∈ 1:max_n
x = n
while x ≥ n && x != 1
x = iseven(x) ? x÷2 : 3x+1
end
end
end
On purpose I don't return anything or check anything, to make it as fast as possible. And boy, is it fast... - actually too fast, I think.
using BenchmarkTools
#btime txpo(100_000_000_000_000_000_000_000_000_000_000_000_000)
2.337 ns (0 allocations: 0 bytes)
Running through 10^38 while loops takes only 2ns? Julia is fast, but I would be surprised if it was that fast. It seems like the code is not evaluated. Or what am I missing here?
Collatz Conjecture
Take a whole number and build a series by using the following rule: if the number is even divide by 2; if the number is odd multiply by 3 and add 1. Conjecture: every series will end up in the loop 4-2-1-4.
Indeed, the whole function body is optimized out:
julia> function txpo(max_n)
for n ∈ 1:max_n
x = n
while x ≥ n && x != 1
x = iseven(x) ? x÷2 : 3x+1
end
end
end
txpo (generic function with 1 method)
julia> #code_llvm txpo(100000000000)
; # REPL[1]:1 within `txpo`
define void #julia_txpo_179(i64 signext %0) #0 {
top:
; # REPL[1]:5 within `txpo`
ret void
}
If you use 10^38, this only changes the function signature to accept 128-bit integers.
The LLVM IR code is basically the same as this Julia code:
julia_txpo_179(_0::Int64) = nothing
This makes perfect sense because why execute a function that doesn't return anything that depends on its computations and has no side-effects?
Finally, the native code that your CPU executes is literally retq, a.k.a. "return to caller":
julia> #code_native txpo(100000000000)
.section __TEXT,__text,regular,pure_instructions
; ┌ # REPL[1]:5 within `txpo`
retq
nopw %cs:(%rax,%rax)
; └
julia>
I'm trying to build a function that will output an expression to be assigned to a new in-memory function. I might be misinterpreting the capability of metaprogramming but, I'm trying to build a function that generates a math series and assigns it to a function such as:
main.jl
function series(iter)
S = ""
for i in 1:iter
a = "x^$i + "
S = S*a
end
return chop(S, tail=3)
end
So, this will build the pattern and I'm temporarily working with it in the repl:
julia> a = Meta.parse(series(4))
:(x ^ 1 + x ^ 2 + x ^ 3 + x ^ 4)
julia> f =eval(Meta.parse(series(4)))
120
julia> f(x) =eval(Meta.parse(series(4)))
ERROR: cannot define function f; it already has a value
Obviously eval isn't what I'm looking for in this case but, is there another function I can use? Or, is this just not a viable way to accomplish the task in Julia?
The actual error you get has to do nothing with metaprogramming, but with the fact that you are reassigning f, which was assigned a value before:
julia> f = 10
10
julia> f(x) = x + 1
ERROR: cannot define function f; it already has a value
Stacktrace:
[1] top-level scope at none:0
[2] top-level scope at REPL[2]:1
It just doesn't like that. Call either of those variables differently.
Now to the conceptual problem. First, what you do here is not "proper" metaprogramming in Julia: why deal with strings and parsing at all? You can work directly on expressions:
julia> function series(N)
S = Expr(:call, :+)
for i in 1:N
push!(S.args, :(x ^ $i))
end
return S
end
series (generic function with 1 method)
julia> series(3)
:(x ^ 1 + x ^ 2 + x ^ 3)
This makes use of the fact that + belongs to the class of expressions that are automatically collected in repeated applications.
Second, you don't call eval at the appropriate place. I assume you meant to say "give me the function of x, with the body being what series(4) returns". Now, while the following works:
julia> f3(x) = eval(series(4))
f3 (generic function with 1 method)
julia> f3(2)
30
it is not ideal, as you newly compile the body every time the function is called. If you do something like that, it is preferred to expand the code once into the body at function definition:
julia> #eval f2(x) = $(series(4))
f2 (generic function with 1 method)
julia> f2(2)
30
You just need to be careful with hygiene here. All depends on the fact that you know that the generated body is formulated in terms of x, and the function argument matches that. In my opinion, the most Julian way of implementing your idea is through a macro:
julia> macro series(N::Int, x)
S = Expr(:call, :+)
for i in 1:N
push!(S.args, :($x ^ $i))
end
return S
end
#series (macro with 1 method)
julia> #macroexpand #series(4, 2)
:(2 ^ 1 + 2 ^ 2 + 2 ^ 3 + 2 ^ 4)
julia> #series(4, 2)
30
No free variables remaining in the output.
Finally, as has been noted in the comments, there's a function (and corresponding macro) evalpoly in Base which generalizes your use case. Note that this function does not use code generation -- it uses a well-designed generated function, which in combination with the optimizations results in code that is usually equal to the macro-generated code.
Another elegant option would be to use the multiple-dispatch mechanism of Julia and dispatch the generated code on type rather than value.
#generated function series2(p::Val{N}, x) where N
S = Expr(:call, :+)
for i in 1:N
push!(S.args, :(x ^ $i))
end
return S
end
Usage
julia> series2(Val(20), 150.5)
3.5778761722367333e43
julia> series2(Val{20}(), 150.5)
3.5778761722367333e43
This task can be accomplished with comprehensions. I need to RTFM...
https://docs.julialang.org/en/v1/manual/arrays/#Generator-Expressions
Say I have a struct,
struct MyStruct
a
b
end
Is there some way to write a function like the following
function doSomething(x::MyStruct,fieldName::String)
y = x.fieldName
return f(y)
end
I could not find anything about this in the documentation / forums.
You can access fields with Symbols, so you can convert the string to a symbol and then use getproperty:
julia> struct MyStruct
a
b
end
julia> function doSomething(x::MyStruct, name::String)
s = Symbol(name)
return getproperty(x, s)
end
doSomething (generic function with 1 method)
julia> doSomething(MyStruct(1, 2), "a")
1
Note, however, that this will probably be very inefficient, since the compiler most likely can't see through this and thus your code might be type-unstable, see https://docs.julialang.org/en/v1/manual/performance-tips/.
If you plan to get the value only few times redrikekre's solution is OK. However, using metaprogramming you can write code without efficiency penalty, see the getDoSomething2() function below.
Consider those three functions:
function doSomethingNative(x)
return x.a
end
function doSomething(x, name::String)
return getproperty(x, Symbol(name))
end
function getDoSomething2(name::String)
field = Symbol(name)
code = quote
(obj) -> obj.$field
end
return eval(code)
end
Now the setup:
using BenchmarkTools
struct MyStruct
a
b
end
x = MyStruct(5,6)
Now the benchmarks:
julia> #btime doSomethingNative($x)
0.001 ns (0 allocations: 0 bytes)
5
julia> #btime doSomething($x,"a")
36.266 ns (0 allocations: 0 bytes)
5
julia> const doSomething2 = getDoSomething2("a");
julia> #btime doSomething2($x)
0.001 ns (0 allocations: 0 bytes)
5
If you run #code_native doSomethingNative(x) and #code_native doSomething2(x) you will see that the assembly output is identical.
Like in R:
a <- 2
or even better
a ← 2
which should translate to
a = 2
and if possible respect method overloading.
= is overloaded (not in the multiple dispatch sense) a lot in Julia.
It binds a new variable. As in a = 3. You won't be able to use ← instead of = in this context, because you can't overload binding in Julia.
It gets lowered to setindex!. As in, a[i] = b gets lowered to setindex!(a, b, i). Unfortunately, setindex! takes 3 variables while ← can only take 2 variables. So you can't overload = with 3 variables.
But, you can use only 2 variables and overload a[:] = b, for example. So, you can define ←(a,b) = (a[:] = b) or ←(a,b) = setindex!(a,b,:).
a .= b gets lowered to (Base.broadcast!)(Base.identity, a, b). You can overload this by defining ←(a,b) = (a .= b) or ←(a,b) = (Base.broadcast!)(Base.identity, a, b).
So, there are two potentially nice ways of using ←. Good luck ;)
Btw, if you really want to use ← to do binding (like in 1.), the only way to do it is using macros. But then, you will have to write a macro in front of every single assignment, which doesn't look very good.
Also, if you want to explore how operators get lowered in Julia, do f(a,b) = (a .= b), for example, and then #code_lowered f(x,y).
No. = is not an operator in Julia, and cannot be assigned to another symbol.
Disclaimer: You are fully responsible if you will try my (still beginner's) experiments bellow! :P
MacroHelper is module ( big thanks to #Alexander_Morley and #DanGetz for help ) I plan to play with in future and we could probably try it here :
julia> module MacroHelper
# modified from the julia source ./test/parse.jl
function parseall(str)
pos = start(str)
exs = []
while !done(str, pos)
ex, pos = parse(str, pos) # returns next starting point as well as expr
ex.head == :toplevel ? append!(exs, ex.args) : push!(exs, ex)
end
if length(exs) == 0
throw(ParseError("end of input"))
elseif length(exs) == 1
return exs[1]
else
return Expr(:block, exs...) # convert the array of expressions
# back to a single expression
end
end
end;
With module above you could define simple test "language":
julia> module TstLang
export #tst_cmd
import MacroHelper
macro tst_cmd(a)
b = replace("$a", "←", "=") # just simply replacing ←
# in real life you would probably like
# to escape comments, strings etc
return MacroHelper.parseall(b)
end
end;
And by using it you could probably get what you want:
julia> using TstLang
julia> tst```
a ← 3
println(a)
a +← a + 3 # maybe not wanted? :P
```
3
9
What about performance?
julia> function test1()
a = 3
a += a + 3
end;
julia> function test2()
tst```
a ← 3
a +← a + 3
```
end;
julia> test1(); #time test1();
0.000002 seconds (4 allocations: 160 bytes)
julia> test2(); #time test2();
0.000002 seconds (4 allocations: 160 bytes)
If you like to see syntax highlight (for example in atom editor) then you need to use it differently:
function test3()
#tst_cmd begin
a ← 3
a ← a + a + 3 # parser don't allow you to use "+←" here!
end
end;
We could hope that future Julia IDEs could syntax highlight cmd macros too. :)
What could be problem with "solution" above? I am not so experienced julian so many things. :P (in this moment something about "macro hygiene" and "global scope" comes to mind...)
But what you want is IMHO good for some domain specific languages and not to redefine basic of language! It is because readability very counts and if everybody will redefine everything then it will end in Tower of Babel...
Does Julia support the reflection just like java?
What I need is something like this:
str = ARGS[1] # str is a string
# invoke the function str()
The Good Way
The recommended way to do this is to convert the function name to a symbol and then look up that symbol in the appropriate namespace:
julia> fn = "time"
"time"
julia> Symbol(fn)
:time
julia> getfield(Main, Symbol(fn))
time (generic function with 2 methods)
julia> getfield(Main, Symbol(fn))()
1.448981716732318e9
You can change Main here to any module to only look at functions in that module. This lets you constrain the set of functions available to only those available in that module. You can use a "bare module" to create a namespace that has only the functions you populate it with, without importing all name from Base by default.
The Bad Way
A different approach that is not recommended but which many people seem to reach for first is to construct a string for code that calls the function and then parse that string and evaluate it. For example:
julia> eval(parse("$fn()")) # NOT RECOMMENDED
1.464877410113412e9
While this is temptingly simple, it's not recommended since it is slow, brittle and dangerous. Parsing and evaling code is inherently much more complicated and thus slower than doing a name lookup in a module – name lookup is essentially just a hash table lookup. In Julia, where code is just-in-time compiled rather than interpreted, eval is much slower and more expensive since it doesn't just involve parsing, but also generating LLVM code, running optimization passes, emitting machine code, and then finally calling a function. Parsing and evaling a string is also brittle since all intended meaning is discarded when code is turned into text. Suppose, for example, someone accidentally provides an empty function name – then the fact that this code is intended to call a function is completely lost by accidental similarity of syntaxes:
julia> fn = ""
""
julia> eval(parse("$fn()"))
()
Oops. That's not what we wanted at all. In this case the behavior is fairly harmless but it could easily be much worse:
julia> fn = "println(\"rm -rf /important/directory\"); time"
"println(\"rm -rf /important/directory\"); time"
julia> eval(parse("$fn()"))
rm -rf /important/directory
1.448981974309033e9
If the user's input is untrusted, this is a massive security hole. Even if you trust the user, it is still possible for them to accidentally provide input that will do something unexpected and bad. The name lookup approach avoids these issues:
julia> getfield(Main, Symbol(fn))()
ERROR: UndefVarError: println("rm -rf /important/directory"); time not defined
in eval(::Module, ::Any) at ./boot.jl:225
in macro expansion at ./REPL.jl:92 [inlined]
in (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:46
The intent of looking up a name and then calling it as a function is explicit, instead of implicit in the generated string syntax, so at worst one gets an error about a strange name being undefined.
Performance
If you're going to call a dynamically specified function in an inner loop or as part of some recursive computation, you will want to avoid doing a getfield lookup every time you call the function. In this case all you need to do is make a const binding to the dynamically specified function before defining the iterative/recursive procedure that calls it. For example:
fn = "deg2rad" # converts angles in degrees to radians
const f = getfield(Main, Symbol(fn))
function fast(n)
t = 0.0
for i = 1:n
t += f(i)
end
return t
end
julia> #time fast(10^6) # once for JIT compilation
0.010055 seconds (2.97 k allocations: 142.459 KB)
8.72665498661791e9
julia> #time fast(10^6) # now it's fast
0.003055 seconds (6 allocations: 192 bytes)
8.72665498661791e9
julia> #time fast(10^6) # see?
0.002952 seconds (6 allocations: 192 bytes)
8.72665498661791e9
The binding f must be constant for optimal performance, since otherwise the compiler can't know that you won't change f to point at another function at any time (or even something that's not a function), so it has to emit code that looks f up dynamically on every loop iteration – effectively the same thing as if you manually call getfield in the loop. Here, since f is const, the compiler knows f can't change so it can emit fast code that just calls the right function directly. But the compiler can sometimes do even better than that – in this case it actually inlines the implementation of the deg2rad function, which is just a multiplication by pi/180:
julia> #code_llvm fast(100000)
define double #julia_fast_51089(i64) #0 {
top:
%1 = icmp slt i64 %0, 1
br i1 %1, label %L2, label %if.preheader
if.preheader: ; preds = %top
br label %if
L2.loopexit: ; preds = %if
br label %L2
L2: ; preds = %L2.loopexit, %top
%t.0.lcssa = phi double [ 0.000000e+00, %top ], [ %5, %L2.loopexit ]
ret double %t.0.lcssa
if: ; preds = %if.preheader, %if
%t.04 = phi double [ %5, %if ], [ 0.000000e+00, %if.preheader ]
%"#temp#.03" = phi i64 [ %2, %if ], [ 1, %if.preheader ]
%2 = add i64 %"#temp#.03", 1
%3 = sitofp i64 %"#temp#.03" to double
%4 = fmul double %3, 0x3F91DF46A2529D39 ; deg2rad(x) = x*(pi/180)
%5 = fadd double %t.04, %4
%6 = icmp eq i64 %"#temp#.03", %0
br i1 %6, label %L2.loopexit, label %if
}
If you need to do this with many different dynamically specified functions, then you can even pass the function to be called in as an argument:
function fast(f,n)
t = 0.0
for i = 1:n
t += f(i)
end
return t
end
julia> #time fast(getfield(Main, Symbol(fn)), 10^6)
0.007483 seconds (1.70 k allocations: 76.670 KB)
8.72665498661791e9
julia> #time fast(getfield(Main, Symbol(fn)), 10^6)
0.002908 seconds (6 allocations: 192 bytes)
8.72665498661791e9
This generates the same fast code as single-argument fast above, but will generate a new version for every different function f that you call it with.