Consider this toy version of a very real-life problem:
julia> foo(a,b) = sum(a) + b
foo (generic function with 1 method)
julia> foo.([1,2],[3,4,5])
ERROR: DimensionMismatch("arrays could not be broadcast to a common size")
Stacktrace:
[1] _bcs1(::Base.OneTo{Int64}, ::Base.OneTo{Int64}) at ./broadcast.jl:70
[2] _bcs at ./broadcast.jl:63 [inlined]
[3] broadcast_shape at ./broadcast.jl:57 [inlined] (repeats 2 times)
[4] broadcast_indices at ./broadcast.jl:53 [inlined]
[5] broadcast_c at ./broadcast.jl:311 [inlined]
[6] broadcast(::Function, ::Array{Int64,1}, ::Array{Int64,1}) at ./broadcast.jl:434
I want the above code to return [6,7,8], but that's not happening because the broadcast implied by the dot tries to match up the input vectors of length 2 and 3 and feed scalars into both arguments of foo. How can I avoid this?
Simply wrap the arguments you don't want to broadcast on into a Ref:
julia> foo.(Ref([1,2]),[3,4,5])
3-element Array{Int64,1}:
6
7
8
Related
Suppose I make my own custom vector type with it's own custom show method:
struct MyVector{T} <: AbstractVector{T}
v::Vector{T}
end
function Base.show(io::IO, v::MyVector{T}) where {T}
println(io, "My custom vector with eltype $T with elements")
for i in eachindex(v)
println(io, " ", v.v[i])
end
end
If I try making one of these objects at the REPL I get unexpected errors related to functions I never intended to call:
julia> MyVector([1, 2, 3])
Error showing value of type MyVector{Int64}:
ERROR: MethodError: no method matching size(::MyVector{Int64})
Closest candidates are:
size(::AbstractArray{T,N}, ::Any) where {T, N} at abstractarray.jl:38
size(::BitArray{1}) at bitarray.jl:77
size(::BitArray{1}, ::Integer) at bitarray.jl:81
...
Stacktrace:
[1] axes at ./abstractarray.jl:75 [inlined]
[2] summary(::IOContext{REPL.Terminals.TTYTerminal}, ::MyVector{Int64}) at ./show.jl:1877
[3] show(::IOContext{REPL.Terminals.TTYTerminal}, ::MIME{Symbol("text/plain")}, ::MyVector{Int64}) at ./arrayshow.jl:316
[4] display(::REPL.REPLDisplay, ::MIME{Symbol("text/plain")}, ::Any) at /Users/mason/julia/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:132
[5] display(::REPL.REPLDisplay, ::Any) at /Users/mason/julia/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:136
[6] display(::Any) at ./multimedia.jl:323
...
Okay, whatever so I'll implement Base.size so it'll leave me alone:
julia> Base.size(v::MyVector) = size(v.v)
julia> MyVector([1, 2, 3])
3-element MyVector{Int64}:
Error showing value of type MyVector{Int64}:
ERROR: getindex not defined for MyVector{Int64}
Stacktrace:
[1] error(::String, ::Type) at ./error.jl:42
[2] error_if_canonical_getindex(::IndexCartesian, ::MyVector{Int64}, ::Int64) at ./abstractarray.jl:991
[3] _getindex at ./abstractarray.jl:980 [inlined]
[4] getindex at ./abstractarray.jl:981 [inlined]
[5] isassigned(::MyVector{Int64}, ::Int64, ::Int64) at ./abstractarray.jl:405
[6] alignment(::IOContext{REPL.Terminals.TTYTerminal}, ::MyVector{Int64}, ::UnitRange{Int64}, ::UnitRange{Int64}, ::Int64, ::Int64, ::Int64) at ./arrayshow.jl:67
[7] print_matrix(::IOContext{REPL.Terminals.TTYTerminal}, ::MyVector{Int64}, ::String, ::String, ::String, ::String, ::String, ::String, ::Int64, ::Int64) at ./arrayshow.jl:186
[8] print_matrix at ./arrayshow.jl:159 [inlined]
[9] print_array at ./arrayshow.jl:308 [inlined]
[10] show(::IOContext{REPL.Terminals.TTYTerminal}, ::MIME{Symbol("text/plain")}, ::MyVector{Int64}) at ./arrayshow.jl:345
[11] display(::REPL.REPLDisplay, ::MIME{Symbol("text/plain")}, ::Any) at /Users/mason/julia/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:132
[12] display(::REPL.REPLDisplay, ::Any) at /Users/mason/julia/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:136
[13] display(::Any) at ./multimedia.jl:323
...
Hmm, now it wants getindex
julia> Base.getindex(v::MyVector, args...) = getindex(v.v, args...)
julia> MyVector([1, 2, 3])
3-element MyVector{Int64}:
1
2
3
What? That wasn't the print formatting I told it to do! what's going on here?
The problem is that in julia, Base defines a method Base.show(io::IO ::MIME"text/plain", X::AbstractArray) which is actually more specific than the Base.show(io::IO, v::MyVector) for the purposes of display. This section of the julia manual describes the sort of custom printing that AbstractArray uses. So if we want to use our custom show method, we instead need to do
julia> function Base.show(io::IO, ::MIME"text/plain", v::MyVector{T}) where {T}
println(io, "My custom vector with eltype $T and elements")
for i in eachindex(v)
println(io, " ", v.v[i])
end
end
julia> MyVector([1, 2, 3])
My custom vector with eltype Int64 and elements
1
2
3
See also: https://discourse.julialang.org/t/extending-base-show-for-array-of-types/31289
I am having trouble with pmap() throwing a BoundsError when setting the values of array elements - my code works for 1 worker but not >1. I have written a minimum working example which roughly follows the real code flow:
Get source data
Define set of points over which to iterate
Initialise array points to be calculated
Calculate each array point
The main file:
#pmapdemo.jl
using Distributed
#addprocs(length(Sys.cpu_info())) # uncomment this line for error
#everywhere include(joinpath(#__DIR__, "pmapdemo2.jl"))
function main()
# Get source data
source = Dict{String, Any}("t"=>zeros(5),
"x"=>zeros(5,6),
"y"=>zeros(5,3),
"z"=>zeros(5,3))
# Define set of points over which to iterate
iterset = Dict{String, Any}("t"=>source["t"],
"x"=>source["x"],
"y"=>fill(2, size(source["t"])[1], 1),
"z"=>fill(2, size(source["t"])[1], 1))
data = Dict{String, Any}()
# Initialise array points to be calculated
MyMod.initialisearray!(data, iterset)
# Calculate each array point
MyMod.calcarray!(data, iterset, source)
#show data
end
main()
The functionality file:
#pmapdemo2.jl
module MyMod
using Distributed
#everywhere using SharedArrays
# Initialise data array
function initialisearray!(data, fieldset)
zerofield::SharedArray{Float64, 4} = zeros(size(fieldset["t"])[1],
size(fieldset["x"])[2],
size(fieldset["y"])[2],
size(fieldset["z"])[2])
data["field"] = deepcopy(zerofield)
end
# Calculate values of array elements according to values in source
function calcpoint!((data, source, a, b, c, d))
data["field"][a,b,c,d] = rand()
end
# Set values in array
function calcarray!(data, iterset, source)
for a in eachindex(iterset["t"])
# [additional functionality f(a) here]
b = eachindex(iterset["x"][a,:])
c = eachindex(iterset["y"][a,:])
d = eachindex(iterset["z"][a,:])
pmap(calcpoint!, Iterators.product(Iterators.repeated(data,1), Iterators.repeated(source,1), Iterators.repeated(a,1), b, c, d))
end
end
end
The error output:
ERROR: LoadError: On worker 2:
BoundsError: attempt to access 0×0×0×0 Array{Float64,4} at index [1]
setindex! at ./array.jl:767 [inlined]
setindex! at /build/julia/src/julia-1.1.1/usr/share/julia/stdlib/v1.1/SharedArrays/src/SharedArrays.jl:500 [inlined]
_setindex! at ./abstractarray.jl:1043
setindex! at ./abstractarray.jl:1020
calcpoint! at /home/dave/pmapdemo2.jl:25
#112 at /build/julia/src/julia-1.1.1/usr/share/julia/stdlib/v1.1/Distributed/src/process_messages.jl:269
run_work_thunk at /build/julia/src/julia-1.1.1/usr/share/julia/stdlib/v1.1/Distributed/src/process_messages.jl:56
macro expansion at /build/julia/src/julia-1.1.1/usr/share/julia/stdlib/v1.1/Distributed/src/process_messages.jl:269 [inlined]
#111 at ./task.jl:259
Stacktrace:
[1] (::getfield(Base, Symbol("##696#698")))(::Task) at ./asyncmap.jl:178
[2] foreach(::getfield(Base, Symbol("##696#698")), ::Array{Any,1}) at ./abstractarray.jl:1866
[3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::Base.Iterators.ProductIterator{Tuple{Base.Iterators.Take{Base.Iterators.Repeated{Dict{String,Any}}},Base.Iterators.Take{Base.Iterators.Repeated{Dict{String,Any}}},Base.Iterators.Take{Base.Iterators.Repeated{Int64}},Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}}) at ./asyncmap.jl:178
[4] #async_usemap#681 at ./asyncmap.jl:154 [inlined]
[5] #async_usemap at ./none:0 [inlined]
[6] #asyncmap#680 at ./asyncmap.jl:81 [inlined]
[7] #asyncmap at ./none:0 [inlined]
[8] #pmap#213(::Bool, ::Int64, ::Nothing, ::Array{Any,1}, ::Nothing, ::Function, ::Function, ::WorkerPool, ::Base.Iterators.ProductIterator{Tuple{Base.Iterators.Take{Base.Iterators.Repeated{Dict{String,Any}}},Base.Iterators.Take{Base.Iterators.Repeated{Dict{String,Any}}},Base.Iterators.Take{Base.Iterators.Repeated{Int64}},Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}}) at /build/julia/src/julia-1.1.1/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:126
[9] pmap(::Function, ::WorkerPool, ::Base.Iterators.ProductIterator{Tuple{Base.Iterators.Take{Base.Iterators.Repeated{Dict{String,Any}}},Base.Iterators.Take{Base.Iterators.Repeated{Dict{String,Any}}},Base.Iterators.Take{Base.Iterators.Repeated{Int64}},Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}}) at /build/julia/src/julia-1.1.1/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:101
[10] #pmap#223(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Base.Iterators.ProductIterator{Tuple{Base.Iterators.Take{Base.Iterators.Repeated{Dict{String,Any}}},Base.Iterators.Take{Base.Iterators.Repeated{Dict{String,Any}}},Base.Iterators.Take{Base.Iterators.Repeated{Int64}},Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}}) at /build/julia/src/julia-1.1.1/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:156
[11] pmap(::Function, ::Base.Iterators.ProductIterator{Tuple{Base.Iterators.Take{Base.Iterators.Repeated{Dict{String,Any}}},Base.Iterators.Take{Base.Iterators.Repeated{Dict{String,Any}}},Base.Iterators.Take{Base.Iterators.Repeated{Int64}},Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}}}) at /build/julia/src/julia-1.1.1/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:156
[12] calcarray!(::Dict{String,Any}, ::Dict{String,Any}, ::Dict{String,Any}) at /home/dave/pmapdemo2.jl:20
[13] main() at /home/dave/pmapdemo.jl:19
[14] top-level scope at none:0
in expression starting at /home/dave/pmapdemo.jl:23
In pmapdemo2.jl, replacing data["field"][a,b,c,d] = rand() with #show a, b, c, d demonstrates that all workers are running and have full access to the variables being passed, however instead replacing it with #show data["field"] throws the same error. Surely the entire purpose of SharedArrays is to avoid this? Or am I misunderstanding how to use it with pmap?
This is a crosspost from the Julia discourse here.
pmap will do the work of passing the data to the processes, so you don't need to use SharedArrays. Typically, the function provided to pmap (and indeed map) will be a pure function (and therefore doesn't mutate any variable) which returns one element of an output array. That function is mapped across each element of the input array, and the pmap function will construct the output array for you. For example, in your case, the code may look a bit like this
calcpoint(source, (a,b,c,d)) = rand() # Or some function of source and the indices a,b,c,d
field["data"] = pmap(calcpoint, Iterators.repeated(source), Iterators.product(a,b,c,d))
I created a UnitRange in Julia and trying to create a plot of x with respect to x and x ^ 2. I have written the following code.
x = [-10:10]
p1 = plot(x, x)
p2 = plot(x, x.^2)
I am getting the following error:
MethodError: no method matching ^(::UnitRange{Int64}, ::Int64)
Closest candidates are:
^(!Matched::Float16, ::Integer) at math.jl:795
^(!Matched::Missing, ::Integer) at missing.jl:120
^(!Matched::Missing, ::Number) at missing.jl:93
...
Stacktrace:
[1] _broadcast_getindex at ./none:0 [inlined]
[2] getindex at ./broadcast.jl:515 [inlined]
[3] copy at ./broadcast.jl:790 [inlined]
[4] materialize(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(Base.literal_pow),Tuple{Base.RefValue{typeof(^)},Array{UnitRange{Int64},1},Base.RefValue{Val{2}}}}) at ./broadcast.jl:756
[5] top-level scope at In[18]:3
What is the mistake in my code?
You can change:
x.^2
to:
x[1].^2
When I tried to calculate
julia> -2.3^-7.6
-0.0017818389423254909
But the result given by my calculator is
0.0005506 + 0.001694 i
Just to be safe I tried it again and this time it complains. Why does it not complain when I tried it the first time?
julia> a = -2.3; b = -7.6; a^b
ERROR: DomainError with -2.6:
Exponentiation yielding a complex result requires a complex argument.
Replace x^y with (x+0im)^y, Complex(x)^y, or similar.
Stacktrace:
[1] throw_exp_domainerror(::Float64) at ./math.jl:35
[2] ^(::Float64, ::Float64) at ./math.jl:769
[3] top-level scope at none:0
[4] eval at ./boot.jl:319 [inlined]
[5] #85 at /Users/ssiew/.julia/packages/Atom/jodeb/src/repl.jl:129 [inlined]
[6] with_logstate(::getfield(Main, Symbol("##85#87")),::Base.CoreLogging.LogState) at ./logging.jl:397
[7] with_logger(::Function, ::Atom.Progress.JunoProgressLogger) at ./logging.jl:493
[8] top-level scope at /Users/ssiew/.julia/packages/Atom/jodeb/src/repl.jl:128
This is an order of operations issue. You can see how Julia's parsing that expression:
julia> parse("-2.3^-7.6")
:(-(2.3 ^ -7.6))
and so the reason you don't have any problems is because you're actually taking 2.3 ^ (-7.6), which is 0.0017818389423254909, and then flipping the sign.
Your second approach is equivalent to making sure that the "x" in "x^y" is really negative, or:
julia> parse("(-2.3)^-7.6")
:(-2.3 ^ -7.6)
julia> eval(parse("(-2.3)^-7.6"))
ERROR: DomainError:
Exponentiation yielding a complex result requires a complex argument.
Replace x^y with (x+0im)^y, Complex(x)^y, or similar.
Stacktrace:
[1] nan_dom_err at ./math.jl:300 [inlined]
[2] ^(::Float64, ::Float64) at ./math.jl:699
[3] eval(::Module, ::Any) at ./boot.jl:235
[4] eval(::Any) at ./boot.jl:234
And if we follow that instruction, we get what you expect:
julia> Complex(-2.3)^-7.6
0.0005506185144176565 + 0.0016946295370871215im
Dear users of the language julia. I have a problem when using the optimize function of the Optim package. What is the error of the code below?
using Optim
using Distributions
rng = MersenneTwister(1234);
d = Weibull(1,1)
x = rand(d,1000)
function pdf_weibull(x, lambda, k)
k/lambda * (x/lambda).^(k-1) * exp((-x/lambda)^k)
end
function obj(x::Vector, lambda, k)
soma = 0
for i in x
soma = soma + log(pdf_weibull(i,lambda,k))
end
-soma
end
obj(x, pars) = obj(x, pars...)
optimize(vars -> obj(x, vars...), [1.0,1.0])
Output
julia> optimize(vars -> obj(x, vars...), [1.0,1.0])
ERROR: DomainError:
Exponentiation yielding a complex result requires a complex argument.
Replace x^y with (x+0im)^y, Complex(x)^y, or similar.
Stacktrace:
[1] nan_dom_err at ./math.jl:300 [inlined]
[2] ^ at ./math.jl:699 [inlined]
[3] (::##2#4)(::Float64, ::Float64, ::Float64) at ./<missing>:0
[4] pdf_weibull(::Float64, ::Float64, ::Float64) at ./REPL[6]:2
[5] obj(::Array{Float64,1}, ::Float64, ::Float64) at ./REPL[7]:4
[6] (::##5#6)(::Array{Float64,1}) at ./REPL[11]:1
[7] value(::NLSolversBase.NonDifferentiable{Float64,Array{Float64,1},Val{false}}, ::Array{Float64,1}) at /home/pedro/.julia/v0.6/NLSolversBase/src/interface.jl:19
[8] initial_state(::Optim.NelderMead{Optim.AffineSimplexer,Optim.AdaptiveParameters}, ::Optim.Options{Float64,Void}, ::NLSolversBase.NonDifferentiable{Float64,Array{Float64,1},Val{false}}, ::Array{Float64,1}) at /home/pedro/.julia/v0.6/Optim/src/multivariate/solvers/zeroth_order/nelder_mead.jl:139
[9] optimize(::NLSolversBase.NonDifferentiable{Float64,Array{Float64,1},Val{false}}, ::Array{Float64,1}, ::Optim.NelderMead{Optim.AffineSimplexer,Optim.AdaptiveParameters}, ::Optim.Options{Float64,Void}) at /home/pedro/.julia/v0.6/Optim/src/multivariate/optimize/optimize.jl:25
[10] #optimize#151(::Array{Any,1}, ::Function, ::Tuple{##5#6}, ::Array{Float64,1}) at /home/pedro/.julia/v0.6/Optim/src/multivariate/optimize/interface.jl:62
[11] #optimize#148(::Array{Any,1}, ::Function, ::Function, ::Array{Float64,1}) at /home/pedro/.julia/v0.6/Optim/src/multivariate/optimize/interface.jl:52
[12] optimize(::Function, ::Array{Float64,1}) at /home/pedro/.julia/v0.6/Optim/src/multivariate/optimize/interface.jl:52
[13] macro expansion at ./REPL.jl:97 [inlined]
[14] (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:73
It is a simple problem to obtain the maximum likelihood estimates of the parameters that index the weibull distribution.
Best regards.
The reason of your problem is that your definition of pdf_weibull is incorrect. Here is a corrected definition:
function pdf_weibull(x, lambda, k)
k/lambda * (x/lambda)^(k-1) * exp(-(x/lambda)^k)
end
Note that I have moved - sign in exp part of the expression. If you change this all will work as expected.
Now - why does Julia complain with DomainError. The reason is that because of the error in your code you try to calculate the value of something like (-1.0)^0.5. In Julia ^ is implemented in type stable way. This means, in particular, that when it is passed Float64 as both arguments it guarantees to return Float64 or throw an error. Clearly (-1.0)^0.5 cannot be computed in real domain - that is why an error is thrown. If you passed (-1+0im)^0.5 then there would be no error as we are passing a complex number to ^ so the result can also be complex number in a type stable way.