I have a problem where I would like to execute a number of statements in parallel that are independent of each other.
I have read plenty of examples that suggest this toy example should work:
function f1(x,y,z)
#sync #async begin
# these two statements can be computed independently
v1 = x+y;
v2 = y+z;
end
return v1*v2
end
However, it seems that the #sync part is not waiting for the results to be completed because I get the following error:
y1 = f1(1,2,3);
ERROR: LoadError: UndefVarError: v1 not defined
I have succeeded in making it work like this:
function f2(x,y,z)
v1 = #async x+y;
v2 = #async y+z;
return wait(v1)*wait(v2)
end
However, in many examples that I have seen, the wait statement seems to be unnecessary if one uses a #sync block.
I'm using Julia 0.6.2.
Any help would be highly appreciated.
The reason is that #async creates a closure around the expression you pass to it. You can see it by running:
julia> #macroexpand #async begin
v1 = x+y;
v2 = y+z;
end
:((Base.async_run_thunk)((()->begin # task.jl, line 335:
begin # REPL[22], line 2:
v1 = x + y # REPL[22], line 3:
v2 = y + z
end
end)))
This has the implication that v1 and v2 variables will not be visible outside the closure unless they are present in the enclosing function scope.
To fix this you can add local v1, v2 statement at the beginning of f1:
julia> function f1(x,y,z)
local v1, v2
#sync #async begin
v1 = x+y;
v2 = y+z;
end
return v1*v2
end
f1 (generic function with 1 method)
julia> f1(1,2,3)
15
and as you can see all worked as expected.
Additionally if your enclosing scope were not a function but a global scope you would have to use global keyword to get what you want:
julia> x,y,z = 1,2,3
(1, 2, 3)
julia> #sync #async begin
global v1 = x+y;
global v2 = y+z;
end
Task (done) #0x0000000018be30f0
julia> v1*v2
15
Related
What is wrong with my code? Do I have to declare x before using it?
function f(n::Int64, t::Int64)
A = ones(n,n)
for i=0:t
if i > 0
A[x,a] = rand()*A[x,a] + rand()
end
y = rand(1:n)
b = rand(1:n)
if i > 0
A[x,a] = rand()*A[x,a] + rand()*A[y,b]
end
x = y
a = min(b, rand(1:n))
end
return A
end
Here is the error thrown when trying to call f:
UndefVarError: x not defined
I think that the reason is more complex, as similar code in Python would work.
For example compare (Python):
>>> def f():
... for i in range(3):
... if i > 0:
... print(a)
... a = i
...
>>> f()
0
1
to (Julia):
julia> function f()
for i in 0:2
if i > 0
println(a)
end
a = i
end
end
f (generic function with 1 method)
julia> f()
ERROR: UndefVarError: a not defined
So what is the difference? As the Julia manual explains here you have:
for loops, while loops, and comprehensions have the following behavior: any new variables introduced in their body scopes are freshly allocated for each loop iteration, as if the loop body were surrounded by a let block
This means that in your code variables a and x as they are local to the for loop are freshly allocated in each iteration of the loop. Because of this the variable has to be assigned to before it is accessed inside the loop.
Therefore it is not needed to assign a value to x and a before the loop. It is enough to define them in scope outer to the loop (even without assigning of the value). For example like this:
julia> function f(n::Int64, t::Int64)
A = ones(n,n)
local x, a
for i=0:t
if i > 0
A[x,a] = rand()*A[x,a] + rand()
end
y = rand(1:n)
b = rand(1:n)
if i > 0
A[x,a] = rand()*A[x,a] + rand()*A[y,b]
end
x = y
a = min(b, rand(1:n))
end
return A
end
f (generic function with 1 method)
julia> f(1,1)
1Ă—1 Array{Float64,2}:
0.94526289614139
Now it works because x and a are not freshly allocated in each iteration of the loop.
In my original toy example it would look like:
julia> function f()
local a
for i in 0:2
if i > 0
println(a)
end
a = i
end
end
f (generic function with 2 methods)
julia> f()
0
1
and you see that you get exactly what you had in Python.
I have issue after calling my macro:
#introspectable square(x) = x * x
Then when calling
square(3)
i should be able to get 9, cause the function call has been specialized to execute an attribute of the structure which is Julia code, however when I enter the macro, the code seems to be directly evaluated.
What i have tried:
struct IntrospectableFunction
name
parameters
native_function
end
(f::IntrospectableFunction)(x) = f.native_function(x)
macro introspectable(expr)
name = expr.args[1].args[1]
parameters = tuple(expr.args[1].args[2:end]...)
body = expr.args[2].args[2]
:( global $name = IntrospectableFunction( :( name ), $parameters, :( body ) ))
end
#introspectable square(x) = x * x
square(3)
The answer should be 9 , however i get "Object of type symbol are not callable ". However if i replace :( body ) with x -> x * x i get the desired result, my objective is generalizing the macro-call.
I usually find it easier to work with expressions in macros (it is not the shortest way to write things, but, from my experience, it is much easier to control what gets generated).
Therefore I would rewrite your code as:
macro introspectable(expr)
name = expr.args[1].args[1]
parameters = expr.args[1].args[2:end]
anon = Expr(Symbol("->"), Expr(:tuple, parameters...), expr.args[2].args[2])
constr = Expr(:call, :IntrospectableFunction, QuoteNode(name), Tuple(parameters), anon)
esc(Expr(:global, Expr(Symbol("="), name, constr)))
end
Now, as you said you wanted generality I would define your functor like this:
(f::IntrospectableFunction)(x...) = f.native_function(x...)
(in this way you allow multiple positional arguments to be passed).
Now let us test our definitions:
julia> #introspectable square(x) = x * x
IntrospectableFunction(:square, (:x,), getfield(Main, Symbol("##3#4"))())
julia> square(3)
9
julia> #macroexpand #introspectable square(x) = x * x
:(global square = IntrospectableFunction(:square, (:x,), ((x,)->x * x)))
julia> #introspectable toarray(x,y) = [x,y]
IntrospectableFunction(:toarray, (:x, :y), getfield(Main, Symbol("##5#6"))())
julia> toarray("a", 10)
2-element Array{Any,1}:
"a"
10
julia> #macroexpand #introspectable toarray(x,y) = [x,y]
:(global toarray = IntrospectableFunction(:toarray, (:x, :y), ((x, y)->[x, y])))
julia> function localscopetest()
#introspectable globalfun(x...) = x
end
localscopetest (generic function with 1 method)
julia> localscopetest()
IntrospectableFunction(:globalfun, (:(x...),), getfield(Main, Symbol("##9#10"))())
julia> globalfun(1,2,3,4,5)
(1, 2, 3, 4, 5)
julia> function f()
v = 100
#introspectable localbinding(x) = (v, x)
end
f (generic function with 1 method)
julia> f()
IntrospectableFunction(:localbinding, (:x,), getfield(Main, Symbol("##11#12")){Int64}(100))
julia> localbinding("x")
(100, "x")
(note that it is useful to use #macroexpand to make sure our macro works as expected)
EDIT - how to handle a minimal multiple dispatch
I am writing a non-macro example because it is related to the data structure:
Use e.g. such a definition:
struct IntrospectableFunction
name::Symbol
method_array::Vector{Pair{Type{<:Tuple}, Function}}
end
function (f::IntrospectableFunction)(x...)
for m in f.method_array
if typeof(x) <: first(m)
return last(m)(x...)
end
end
error("signature not found")
end
and now you can write:
julia> square = IntrospectableFunction(:square, [Tuple{Any}=>x->x*x,Tuple{Any,Any}=>(x,y)->x*y])
IntrospectableFunction(:square, Pair{DataType,Function}[Tuple{Any}=>##9#11(), Tuple{Any,Any}=>##10#12()])
julia> square(3)
9
julia> square(2,3)
6
Keep in mind that the approach I present is not perfect and universal - it just serves to give a very simple example how you could do it.
I noticed that running a function within a #sync #parallel for loop is problematic if the argument of this function is defined within an if-else statement.
Example 1: It doesn't work
#everywhere function test_fun(arg)
println(arg);
end
flag = 1;
if flag == 1
asd = 123;
#sync #parallel for i=1:10
test_fun(asd);
end
end
Example 2: It works
#everywhere function test_fun(arg)
println(arg);
end
asd = 123;
#sync #parallel for i=1:10
test_fun(asd);
end
#everywhere asd = 123; fixes the problem for the examples defined above, but it doesn't work when you define a SharedArray:
Example 3: It doesn't work with #everywhere asd
#everywhere function test_fun(arg)
println(arg);
end
flag = 1;
if flag == 1
#everywhere asd = 123;
arr = SharedArray{Float64}(asd);
#sync #parallel for i=1:10
test_fun(arr);
end
end
Any suggestion to solve this issue?
julia 0.5.1
I want to create a function inside a quote that can be used after the specified macro has been used. Here is an example of what I mean
macro wat()
quote
type Foo end
global bar() = begin end
global function bar2()
end
type Baaz end
end
end
#wat
Foo()
Baaz()
bar()
bar2()
Now when I run this the last line crashes, because bar2 is undefined. I do not understand why because in my understanding bar() and bar2() should be equal and bar is just syntactic sugar for bar2. But they are apparently not equal and I do not understand why the one works and other does not.
Secondly is there a way to define bar and bar2 inside that quote without the global-keyword and still being available after the macro has been executed?
My motivation for wanting the bar2 notation is that I can specify a return-type with this syntax.
global bar3()::Void = begin end
Is not allowed syntax.
In the returned expressions of Julia macros, names of local variables are replaced with unique symbols:
julia> macro foo()
quote
x = 1
global y = 2
end
end
#foo (macro with 1 method)
julia> macroexpand(:(#foo))
quote # REPL[1], line 4:
#1#x = 1
global y = 2
end
This feature is called macro hygiene and avoids accidental clashes with variables at the call site.
To escape from this behavior, one has to use esc:
julia> macro bar()
quote
x = 1
end |> esc
end
#bar (macro with 1 method)
julia> macroexpand(:(#bar))
quote # REPL[1], line 3:
x = 1
end
Often, one doesn't want to escape the whole returned expression but only specific parts of it:
julia> macro myshow(expr)
quote
x = $(esc(expr))
println($(string(expr)), " = ", x)
x
end
end
#myshow (macro with 1 method)
julia> x = pi/2
1.5707963267948966
julia> macroexpand(:(#myshow sin(x)))
quote # REPL[1], line 3:
#1#x = sin(x) # REPL[1], line 4:
(Main.println)("sin(x)", " = ", #1#x) # REPL[1], line 5:
#1#x
end
julia> #myshow sin(x)
sin(x) = 1.0
1.0
julia> x
1.5707963267948966
For details, I recommend to read the corresponding section in the manual.
I would like to be able to create a dispatch for a user-defined type which will essentially do an inplace copy. However, I would like to do it in a type-stable manner, and thus I would like to avoid using getfield directly, and instead try to use a generated function. Is it possible for a type like
type UserType{T}
x::Vector{T}
y::Vector{T}
z::T
end
to generate some function
recursivecopy!(A::UserType,B::UserType)
# Do it for x
if typeof(A.x) <: AbstractArray
recursivecopy!(A.x,B.x)
else
A.x = B.x
end
# Now for y
if typeof(A.y) <: AbstractArray
recursivecopy!(A.y,B.y)
else
A.y = B.y
end
# Now for z
if typeof(A.z) <: AbstractArray
recursivecopy!(A.z,B.z)
else
A.z = B.z
end
end
The recursivecopy! in RecursiveArrayTools.jl makes this handle nested (Vector{Vector}) types well, but the only problem is that I do not know the fields the user will have in advance, just at compile-time when this function would be called. Sounds like a job for generated functions, but I'm not quite sure how to generate this.
You don't need to bend over backwards to avoid getfield and setfield. Julia can infer them just fine. The trouble comes when Julia can't figure out which field it's accessing… like in a for loop.
So the only special thing the generated function needs to do is effectively unroll the loop with constant values spliced into getfield:
julia> immutable A
x::Int
y::Float64
end
julia> #generated function f(x)
args = [:(getfield(x, $i)) for i=1:nfields(x)]
:(tuple($(args...)))
end
f (generic function with 1 method)
julia> f(A(1,2.4))
(1,2.4)
julia> #code_warntype f(A(1,2.4))
Variables:
#self#::#f
x::A
Body:
begin # line 2:
return (Main.tuple)((Main.getfield)(x::A,1)::Int64,(Main.getfield)(x::A,2)::Float64)::Tuple{Int64,Float64}
end::Tuple{Int64,Float64}
Just like you can splice in multiple arguments to a function call, you can also directly splice in multiple expressions to the function body.
julia> type B
x::Int
y::Float64
end
julia> #generated function f!{T}(dest::T, src::T)
assignments = [:(setfield!(dest, $i, getfield(src, $i))) for i=1:nfields(T)]
:($(assignments...); dest)
end
f! (generic function with 1 method)
julia> f!(B(0,0), B(1, 2.4))
B(1,2.4)
julia> #code_warntype f!(B(0,0), B(1, 2.4))
Variables:
#self#::#f!
dest::B
src::B
Body:
begin # line 2:
(Main.setfield!)(dest::B,1,(Main.getfield)(src::B,1)::Int64)::Int64
(Main.setfield!)(dest::B,2,(Main.getfield)(src::B,2)::Float64)::Float64
return dest::B
end::B
You can, of course, make the body of that comprehension as complicated as you'd like. That effectively becomes the inside of your for loop. Splatting the array into the body of the function does the unrolling for you.