I am trying a simple Julia code:
using Distributed
addprocs(1)
#everywhere include("count_heads.jl")
a = #spawn count_heads(100000000)
b = #spawn count_heads(100000000)
fetch(a)+fetch(b)
However the count_heads.jl contains the following function:
function count_heads(n)
c::Int = 0
for i = 1:n
c += rand(Bool)
end
c
end
But the above code was showing the error :
On worker 2:
UndefVarError: count_heads not defined
Does anyone know how to resolve this error??
#spawn is deprecated and you should use #spawnat :any instead:
a = #spawnat :any count_heads(100000000)
otherwise your code is perfectly correct and I managed to run it.
Most likely, in your production code you have some typo in the function name or you have run #everywhere include("count_heads.jl") before addprocs
Related
I write some code below and it works correctly.
function value_counter(data, feature_name)
feature_col = unique(data, feature_name)
unique_val = unique!(unique(data, feature_name))
count_val = []
value_counts_dict = Dict()
for val in unique_val
counter = 0
for col_val in feature_col
if val == col_val
counter += 1
end
end
append!(count_val, counter)
value_counts_dict[val] = counter
end
return value_counts_dict
end
But when I run it over three times. It appears a bug 'Stackoverflow' and I think the error from the unique method. How can I free the stack after running the code?
Update: I try to redefine unique method. It is still errors but this time it's in 'in' method
ERROR: LoadError: StackOverflowError:
Stacktrace:
[1] in(x::SubString{String}, itr::Vector{Any})
# Base .\operators.jl:1283
[2] redefine_unique(data::Vector{Any})
# Main E:\Study\3_grade\Csttnt\Project03_decision_tree\Decision_Tree.jl:35
It's probably not the in method or unique method itself that's causing the stack overflow. Rather, you have some large, probably recursive, stack that just happens to hit the stack size limit when it gets to that downstream function. What you should look for in the stack is where the recursive part is. What functions are being called repeatedly? You may be missing a base case somewhere, or perhaps introduced unexpected recursion into an overwritten Base function, as Sundar mentioned.
I have this error
ERROR: MethodError: vcat(::Array{Real,2}, ::TrackedArray{…,Array{Float32,2}}) is ambiguous. Candidates:
vcat(364::AbstractArray, x::Union{TrackedArray, TrackedReal}, xs::Union{Number, AbstractArray}...) in Tracker at C:\Users\Henri\.julia\packages\Tracker\6wcYJ\src\lib\array.jl:167
vcat(A::Union{AbstractArray{T,2}, AbstractArray{T,1}} where T...) in Base at abstractarray.jl:1296
Possible fix, define
vcat(::Union{AbstractArray{T,2}, AbstractArray{T,1}} where T, ::Union{TrackedArray{T,1,A} where A<:AbstractArray{T,1} where T, TrackedArray{T,2,A} where A<:AbstractArray{T,2} where T}, ::Vararg{Union{AbstractArray{T,2}, AbstractArray{T,1}} where T,N} where N)
Telling me that two vcat() functions are ambiguous. I want to use the Base.vcat() function but using it explicitly throws the same error. Why is that ? And what is this "possible fix" proposed by the error throw?
Moreover, when I call manually each line in the REPL no error is thrown. I do not understand this behavior. This only happens when vcat() is in a function called inside another function. Like in my example below.
Here is a code that reproduces the error:
using Flux
function loss(a, b, net, net2)
net2(vcat(net(a),a))
end
function test()
opt = ADAM()
net = Chain(Dense(3,3))
net2 = Chain(Dense(6,1))
L(a, b) = loss(a, b, net, net2)
data = tuple(rand(3,1), rand(3,1))
xs = Flux.params(net)
gs = Tracker.gradient(() -> L(data...), xs)
Tracker.update!(opt, xs, gs)
end
As mentionned in comments with Henri.D, we've managed to fix it by being carreful with the type of a which was an Array of Float64, default type returned by rand whereas net(a) returned a TrackedArray of Float32 and made impossible to vcat it with a.
I've managed to fix vcat by changing your loss function with this: net2(vcat(net(a),Float32.(a))) because vcat couldn't concatenate as net(a) was a Float32 Array and a a Float64 one. Then L(data...) is a TrackedArray of 1 element whereas I think you need a Float32 that's why I finally replace loss function by net2(vcat(net(a),Float32.(a)))[1]
I am trying to write a simple Julia code for parallel computing.
I wrote a simple code based on this doc: https://docs.julialang.org/en/latest/manual/parallel-computing
#everywhere function test(x)
return x * 2.0
end
nprocess = 5
addprocs(nprocess)
responses = Vector{Any}(nworkers())
for i in 1:nworkers()
responses[i] = remotecall(test, i+1, i)
end
for res in responses
wait(res)
end
However, I got this error message.
ERROR: LoadError: On worker 2:
UndefVarError: #test not defined
I think the #everywhere macro doesn't work correctly.
I'm using Julia 0.6.0.
Does anyone know how to fix it?
The #everywhere and the addprocs are in the reverse order (causing the added workers not to know about the function test). The other way around it works and doesn't do the UndefVarError:
nprocess = 5
addprocs(nprocess)
responses = Vector{Any}(nworkers())
#everywhere function test(x)
return x * 2.0
end
for i in 1:nworkers()
responses[i] = remotecall(test, i+1, i)
end
for res in responses
wait(res)
end
Can someone resolve this macro error I'm having, it only started happening in version 0.6:
mutable struct Foo
x::Int
end
macro test(myfoo)
quoteblock =
quote
myfoo.x += 1
end
return quoteblock
end
function func(myfoo)
#test myfoo
println(myfoo.x)
end
foo = Foo(3)
func(foo)
In theory this should just replace the line #test myfoo in the function func with myfoo.x += 1 at compile time, which should work, but instead I get the error:
UndefVarError: myfoo not defined
The corresponding change-notes are listed here:
When a macro is called in the module in which that macro is defined,
global variables in the macro are now correctly resolved in the macro
definition environment. Breakage from this change commonly manifests
as undefined variable errors that do not occur under 0.5. Fixing such
breakage typically requires sprinkling additional escs in the
offending macro (#15850).
so the answer is to escape myfoo:
macro test(myfoo)
quote
$(esc(myfoo)).x += 1
end
end
Here's some Julia code:
addprocs()
#everywhere begin
type Test
values::Array{Any,1}
addValue::Function
function Test()
this = new()
this.values = Any[]
this.addValue = function(v)
push!(this.values,v)
end
return this
end
end
end
#everywhere t = Test()
#everywhere t.addValue(myid())
#spawnat 2 whos()
r = #spawnat 2 t
I believe this code defines a Test type on all processes and then creates an instance of that type stored in a variable, called t, on each process. This variable is local to that process. I then use one of the Test methods run in parallel on each process to mutate the local instance of Test. In the line
#spawnat 2 whos()
we can see that the local t has indeed been updated. However, I get a HUGE error when trying to fetch any of the remote variables t into a RemoteRef. The error message is quite large, but it leads me to believe that the Julia serialization process cannot handle user-defined types. Can anybody lend some insight? Thank You!
Upadate:
A simpler example yielding the same error:
addprocs()
#everywhere begin
type Test
values::Array{Any,1}
addValue::Function
function Test()
this = new()
this.values = Any[]
this.addValue = function(v)
push!(this.values,v)
end
return this
end
end
end
t = Test()
R = RemoteRef(2)
put!(R,t)
I don't know why trying to fetch that composite object with a function field inside, causes the Error..., anyway I encourage you to use Julia way for creating the same functionality:
First of all you don't need to include addValue function inside the Test type, try to separate data storage from functionality:
#everywhere begin
type Test
values::Array{Any,1}
addValue::Function
function Test()
this = new()
this.values = Any[]
return this
end
end
end
#everywhere function addValue(v,t::test)
push!(t.values,v)
end
then initialize t everywhere:
#everywhere t = Test()
and mutate t everywhere:
#everywhere addValue(myid(),t)
Secondly run getfield() command on desired process to get local variable:
r = #spawnat 3 getfield(Main,:t)
check the result:
assert(fetch(r).values==[3])
for more details about how to pass variable between processes check this question.