Scalar getindex error when doing optimisation in Flux - julia

I am trying to do optimisation using Flux's logitbinarycrossentropy function on a fully-convolutional network on GPU:
Flux.train!(loss, params(UNet_model), train_batch, optimiser)
Where loss is defined as:
loss(x, y) = mean(logitbinarycrossentropy.(model(x) |> gpu, y))
However, I'm getting a ERROR: LoadError: scalar getindex is disallowed error. I have verified that my model runs correctly (returns right output, etc), and that the loss function is able to compute a value. I tried rewriting logitbinarycrossentropy as a custom loss function but it still fails with the same error in the line update!(opt, ps, gs). I also tried using σ with binarycrossentropy instead of logitbinarycrossentropy, and got the same error.
I am using Julia 1.3.0, Flux 0.10.3, Zygote 0.4.6.
I speculate that this might have something to do with my use of cat in the model definition, is this likely? Here is roughly how I defined my model:
function (t::test)(x)
enc1 = t.conv_block1[1](x)
bn = t.bottle(enc1)
dec1 = t.upconv_block[1](bn)
dec1 = cat(dims=3, dec1, enc1)
dec1 = t.conv_block[2](dec1)
dec1 = t.conv(dec1)
end
Full stacktrace:
ERROR: LoadError: scalar getindex is disallowed
Stacktrace:
[1] error(::String) at .\error.jl:33
[2] assertscalar(::String) at C:\Users\CCL\.julia\packages\GPUArrays\1wgPO\src\indexing.jl:14
[3] getindex at C:\Users\CCL\.julia\packages\GPUArrays\1wgPO\src\indexing.jl:54 [inlined]
[4] _getindex at .\abstractarray.jl:1004 [inlined]
[5] getindex at .\abstractarray.jl:981 [inlined]
[6] hash(::CuArray{Float32,4,Nothing}, ::UInt64) at .\abstractarray.jl:2203
[7] hash at .\hashing.jl:18 [inlined]
[8] hashindex at .\dict.jl:168 [inlined]
[9] ht_keyindex(::Dict{Any,Any}, ::CuArray{Float32,4,Nothing}) at .\dict.jl:282
[10] get(::Dict{Any,Any}, ::CuArray{Float32,4,Nothing}, ::Nothing) at .\dict.jl:500
[11] (::Zygote.var"#876#877"{Zygote.Context,IdDict{Any,Any},CuArray{Float32,4,Nothing}})(::Nothing) at C:\Users\CCL\.julia\packages\Zygote\oMScO\src\lib\base.jl:44
[12] (::Zygote.var"#2375#back#878"{Zygote.var"#876#877"{Zygote.Context,IdDict{Any,Any},CuArray{Float32,4,Nothing}}})(::Nothing) at C:\Users\CCL\.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:49
[13] #fmap#53 at C:\Users\CCL\.julia\packages\Flux\NpkMm\src\functor.jl:37 [inlined]
[14] (::typeof(∂(#fmap#53)))(::CuArray{Float32,4,Nothing}) at C:\Users\CCL\.julia\packages\Zygote\oMScO\src\compiler\interface2.jl:0
[15] fmap at C:\Users\CCL\.julia\packages\Flux\NpkMm\src\functor.jl:36 [inlined]
[16] (::typeof(∂(fmap)))(::CuArray{Float32,4,Nothing}) at C:\Users\CCL\.julia\packages\Zygote\oMScO\src\compiler\interface2.jl:0
[17] gpu at C:\Users\CCL\.julia\packages\Flux\NpkMm\src\functor.jl:108 [inlined]
[18] (::typeof(∂(gpu)))(::CuArray{Float32,4,Nothing}) at C:\Users\CCL\.julia\packages\Zygote\oMScO\src\compiler\interface2.jl:0
[19] |> at .\operators.jl:854 [inlined]
[20] (::typeof(∂(|>)))(::CuArray{Float32,4,Nothing}) at C:\Users\CCL\.julia\packages\Zygote\oMScO\src\compiler\interface2.jl:0
[21] (::typeof(∂(loss)))(::Float32) at C:\Users\CCL\fcn\main_flux2.jl:47
[22] #157 at C:\Users\CCL\.julia\packages\Zygote\oMScO\src\lib\lib.jl:156 [inlined]
[23] #297#back at C:\Users\CCL\.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:49 [inlined]
[24] #17 at C:\Users\CCL\.julia\packages\Flux\NpkMm\src\optimise\train.jl:88 [inlined]
[25] (::typeof(∂(λ)))(::Float32) at C:\Users\CCL\.julia\packages\Zygote\oMScO\src\compiler\interface2.jl:0
[26] (::Zygote.var"#38#39"{Zygote.Params,Zygote.Context,typeof(∂(λ))})(::Float32) at C:\Users\CCL\.julia\packages\Zygote\oMScO\src\compiler\interface.jl:101
[27] gradient(::Function, ::Zygote.Params) at C:\Users\CCL\.julia\packages\Zygote\oMScO\src\compiler\interface.jl:47
[28] macro expansion at C:\Users\CCL\.julia\packages\Flux\NpkMm\src\optimise\train.jl:87 [inlined]
[29] macro expansion at C:\Users\CCL\.julia\packages\Juno\f8hj2\src\progress.jl:134 [inlined]
[30] #train!#12(::Flux.Optimise.var"#18#26", ::typeof(Flux.Optimise.train!), ::typeof(loss), ::Zygote.Params, ::Array{Tuple{CuArray{Float32,4,Nothing},CuArray{Float32,4,Nothing}},1}, ::ADAM) at C:\Users\CCL\.julia\packages\Flux\NpkMm\src\optimise\train.jl:80
[31] train!(::Function, ::Zygote.Params, ::Array{Tuple{CuArray{Float32,4,Nothing},CuArray{Float32,4,Nothing}},1}, ::ADAM) at C:\Users\CCL\.julia\packages\Flux\NpkMm\src\optimise\train.jl:78
[32] top-level scope at C:\Users\CCL\fcn\main_flux2.jl:83
[33] include at .\boot.jl:328 [inlined]
[34] include_relative(::Module, ::String) at .\loading.jl:1105
[35] include(::Module, ::String) at .\Base.jl:31
[36] exec_options(::Base.JLOptions) at .\client.jl:287
[37] _start() at .\client.jl:460
in expression starting at C:\Users\CCL\fcn\main_flux2.jl:74
Thanks!

Related

Julia Plots animate `gif()` hangs indefinitely

I am experiencing slightly strange behaviour from a visualisation routine I have written using Plots.jl. The module I have written uses code of the following form (with some irrelevant details removed). It is intended to produce a 2D animation of the results of a 2D partial differential equation.
anim = #animate for (i,u) in enumerate(sol.u)
heatmap(u)
end
gif(anim,"output/anim_u.gif",fps=5)
If I run my program with this code at the end, the program hangs indefinitely after opening a "gksqt" application in the dock. Killing the program produces the following stacktrace.
Stacktrace:
[1] poptask(W::Base.InvasiveLinkedListSynchronized{Task})
# Base ./task.jl:760
[2] wait
# ./task.jl:768 [inlined]
[3] wait(c::Base.GenericCondition{Base.Threads.SpinLock})
# Base ./condition.jl:106
[4] wait(x::Base.Process)
# Base ./process.jl:621
[5] success
# ./process.jl:483 [inlined]
[6] run(::Cmd; wait::Bool)
# Base ./process.jl:440
[7] run
# ./process.jl:438 [inlined]
[8] (::FFMPEG.var"#4#6"{Cmd})(command_path::String)
# FFMPEG ~/.julia/packages/FFMPEG/OUpap/src/FFMPEG.jl:112
[9] (::JLLWrappers.var"#2#3"{FFMPEG.var"#4#6"{Cmd}, String})()
# JLLWrappers ~/.julia/packages/JLLWrappers/bkwIo/src/runtime.jl:49
[10] withenv(::JLLWrappers.var"#2#3"{FFMPEG.var"#4#6"{Cmd}, String}, ::Pair{String, String}, ::Vararg{Pair{String, String}, N} where N)
# Base ./env.jl:161
[11] withenv_executable_wrapper(f::Function, executable_path::String, PATH::String, LIBPATH::String, adjust_PATH::Bool, adjust_LIBPATH::Bool)
# JLLWrappers ~/.julia/packages/JLLWrappers/bkwIo/src/runtime.jl:48
[12] #invokelatest#2
# ./essentials.jl:708 [inlined]
[13] invokelatest
# ./essentials.jl:706 [inlined]
[14] #ffmpeg#7
# ~/.julia/packages/JLLWrappers/bkwIo/src/products/executable_generators.jl:7 [inlined]
[15] ffmpeg
# ~/.julia/packages/JLLWrappers/bkwIo/src/products/executable_generators.jl:7 [inlined]
[16] #exe#2
# ~/.julia/packages/FFMPEG/OUpap/src/FFMPEG.jl:111 [inlined]
[17] ffmpeg_exe
# ~/.julia/packages/FFMPEG/OUpap/src/FFMPEG.jl:123 [inlined]
[18] buildanimation(anim::Plots.Animation, fn::String, is_animated_gif::Bool; fps::Int64, loop::Int64, variable_palette::Bool, verbose::Bool, show_msg::Bool)
# Plots ~/.julia/packages/Plots/S2aH5/src/animation.jl:96
I am completely baffled by this behaviour - if anyone has any suggestions it would be greatly appreciated.
Running on MacOS 11.2.3 with Julia 1.6.2.

julia: UndefRefError: access to undefined reference

I'm trying to create a POMDP model and solve it. But during the solving process, I get this error:
Got exception outside of a #test
UndefRefError: access to undefined reference
Stacktrace:
[1] getindex
# ./array.jl:801 [inlined]
[2] iterate
# ./array.jl:777 [inlined]
[3] iterate
# ./iterators.jl:159 [inlined]
[4] write(io::IOStream, pomdp::AccPOMDP)
# POMDPFiles ~/.julia/packages/POMDPFiles/vOfxh/src/write.jl:64
[5] #3
# ~/.julia/packages/POMDPSolve/LzAMF/src/solver.jl:175 [inlined]
[6] open(::POMDPSolve.var"#3#4"{AccPOMDP}, ::String, ::Vararg{String, N} where N; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
# Base ./io.jl:330
[7] open
# ./io.jl:328 [inlined]
[8] solve(solver::POMDPSolveSolver, pomdp::AccPOMDP)
# POMDPSolve ~/.julia/packages/POMDPSolve/LzAMF/src/solver.jl:174
[9] macro expansion
# ~/jl/acc/acc_dev.jl:556 [inlined]
[10] macro expansion
# /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
[11] top-level scope
# ~/jl/acc/acc_dev.jl:546
[12] include(mod::Module, _path::String)
# Base ./Base.jl:386
[13] exec_options(opts::Base.JLOptions)
# Base ./client.jl:285
[14] _start()
# Base ./client.jl:485
I'm not so familiar with the Julia Stacktrace, so could you please help me fix this problem?
Caused by stateindex(...) and states(...) wrong implementation in my POMDP model. But after I fix this ans start to solve the problem, the solving just kept run and finally failed. Just want to complain, Julia POMDP is really hard to use in reality...

Why can't I properly install Plots package on Julia?

I'm new to Julia programming and am trying to install the Plots package; this should be rather simple but it has not been so. Every time I try to do it I get the following error message:
ERROR: Pkg.TOML.ParserError(2, 2, "expected a key but found an empty string")
The only code I have run prior to the Pkg.add("Plots") line is import Pkg, so I can't picture how this problem would come about.
Additional information: I'm working on Julia 1.5.4 and running it through the Juno IDE via Atom v1.55.0. Here is the (potentially very uninformative) entire stack trace below:
Stacktrace:
[1] parse(::Base.GenericIOBuffer{Array{UInt8,1}}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\ext\TOML\src\TOML.jl:37
[2] parsefile at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\ext\TOML\src\TOML.jl:50 [inlined]
[3] #parse_toml#10 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Types.jl:154 [inlined]
[4] parse_toml at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Types.jl:153 [inlined]
[5] #load_artifacts_toml#18 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Artifacts.jl:471 [inlined]
[6] ensure_all_artifacts_installed(::String; platform::Pkg.BinaryPlatforms.Platform, pkg_uuid::Nothing, include_lazy::Bool, verbose::Bool, quiet_download::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Artifacts.jl:948
[7] download_artifacts(::Pkg.Types.Context, ::Array{String,1}; platform::Pkg.BinaryPlatforms.Windows, verbose::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:663
[8] download_artifacts(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; platform::Pkg.BinaryPlatforms.Windows, verbose::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:642
[9] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Array{Base.UUID,1}; preserve::Pkg.Types.PreserveLevel, platform::Pkg.BinaryPlatforms.Windows) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1141
[10] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; preserve::Pkg.Types.PreserveLevel, platform::Pkg.BinaryPlatforms.Windows, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:188
[11] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:139
[12] #add#21 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67 [inlined]
[13] add at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67 [inlined]
[14] #add#20 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:66 [inlined]
[15] add at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:66 [inlined]
[16] add(::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:65
[17] add(::String) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:65
[18] top-level scope at none:1
Here is the project.TOML file that was there originally.
[deps]
Atom = "c52e3926-4ff0-5f6e-af25-54175e0327b1"
Colors = "5ae59095-9a9b-59fe-a467-6f913c188581"
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
Juno = "e5e0dc1b-0480-54bc-9374-aad01c23163d"
PDMats = "90014a1f-27ba-587c-ab20-58faa44d9150"
TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
When I renamed it project.toml.backup per request, and tried installing again, it still failed, with the same error message, but a different (longer) stack trace.
Stacktrace:
[1] parse(::Base.GenericIOBuffer{Array{UInt8,1}}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\ext\TOML\src\TOML.jl:37
[2] parsefile at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\ext\TOML\src\TOML.jl:50 [inlined]
[3] #parse_toml#10 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Types.jl:154 [inlined]
[4] parse_toml at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Types.jl:153 [inlined]
[5] #load_artifacts_toml#18 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Artifacts.jl:471 [inlined]
[6] ensure_all_artifacts_installed(::String; platform::Pkg.BinaryPlatforms.Platform, pkg_uuid::Nothing, include_lazy::Bool, verbose::Bool, quiet_download::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Artifacts.jl:948
[7] download_artifacts(::Pkg.Types.Context, ::Array{String,1}; platform::Pkg.BinaryPlatforms.Windows, verbose::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:663
[8] download_artifacts(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; platform::Pkg.BinaryPlatforms.Windows, verbose::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:642
[9] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Array{Base.UUID,1}; preserve::Pkg.Types.PreserveLevel, platform::Pkg.BinaryPlatforms.Windows) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1141
[10] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; preserve::Pkg.Types.PreserveLevel, platform::Pkg.BinaryPlatforms.Windows, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:188
[11] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:139
[12] #add#21 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67 [inlined]
[13] add(::Array{Pkg.Types.PackageSpec,1}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67
[14] do_cmd!(::Pkg.REPLMode.Command, ::REPL.LineEditREPL) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\REPLMode\REPLMode.jl:404
[15] do_cmd(::REPL.LineEditREPL, ::String; do_rethrow::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\REPLMode\REPLMode.jl:382
[16] do_cmd at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\REPLMode\REPLMode.jl:377 [inlined]
[17] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\REPLMode\REPLMode.jl:546
[18] (::Atom.var"#4#15"{Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL,REPL.LineEdit.Prompt}})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at C:\Users\pthom\.julia\packages\Atom\BPtI0\src\Atom.jl:31 [19] #invokelatest#1 at .\essentials.jl:710 [inlined]
[20] invokelatest at .\essentials.jl:709 [inlined]
[21] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\REPL\src\LineEdit.jl:2355
[22] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\REPL\src\REPL.jl:1144
[23] (::REPL.var"#38#42"{REPL.LineEditREPL,REPL.REPLBackendRef})() at .\task.jl:356
Let me know if I've omitted any information or if anyone has any ideas. Thanks very much.

How to resolve duplicate key <PACKAGE NAME> issue of Pkg?

Objective: To get installed package particulars using status command of Pkg module
Procedure:
$ julia
julia> ]
(v1.1) pkg> status
Output:
ERROR: Pkg.TOML.ParserError(258, 263, "duplicate key `Dates`")
Stacktrace:
[1] parse(::IOStream) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/ext/TOML/src/TOML.jl:33
[2] #open#310(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::typeof(Pkg.TOML.parse), ::String, ::Vararg{String,N} where N) at ./iostream.jl:369
[3] open at ./iostream.jl:367 [inlined]
[4] parsefile(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/ext/TOML/src/TOML.jl:46
[5] load_stdlib() at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Types.jl:338
[6] Pkg.Types.Context() at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Types.jl:347
[7] Context!(::Array{Pair{Symbol,Any},1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Types.jl:368
[8] Context! at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/REPLMode.jl:526 [inlined]
[9] do_status!(::Dict{Symbol,Any}, ::Array{Pkg.Types.PackageSpec,1}, ::Dict{Symbol,Any}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/REPLMode.jl:603
[10] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:742
[11] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:741
[12] do_cmd!(::Pkg.REPLMode.PkgCommand, ::REPL.LineEditREPL) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/REPLMode.jl:561
[13] #do_cmd#30(::Bool, ::Function, ::REPL.LineEditREPL, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/REPLMode.jl:536
[14] do_cmd at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/REPLMode.jl:532 [inlined]
[15] (::getfield(Pkg.REPLMode, Symbol("##48#51")){REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/REPLMode.jl:967
[16] #invokelatest#1 at ./essentials.jl:742 [inlined]
[17] invokelatest at ./essentials.jl:741 [inlined]
[18] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/LineEdit.jl:2273
[19] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:1035
[20] run_repl(::REPL.AbstractREPL, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:192
[21] (::getfield(Base, Symbol("##734#736")){Bool,Bool,Bool,Bool})(::Module) at ./client.jl:362
[22] #invokelatest#1 at ./essentials.jl:742 [inlined]
[23] invokelatest at ./essentials.jl:741 [inlined]
[24] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:346
[25] exec_options(::Base.JLOptions) at ./client.jl:284
[26] _start() at ./client.jl:436
Observation:
Dates and its dependent packages have only one uuid as follows:
Contents of "/opt/julia/julia-1.1.0/share/julia/stdlib/v1.1/Dates/Project.toml"
name = "Dates"
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"
...
and for its dependent packages:
Content of "/opt/julia/julia-1.1.0/share/julia/stdlib/v1.1/<PACKAGE_NAME>/Project.toml"
...
[deps]
...
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
...
Please help me in resolving the duplicate key issue of Pkg!

How to fix `LoadError: unrecognized HDF5 datatype class 4` using Julia's JLD to store data from Eirene package?

I am trying to save a large variable in Julia resulting from a long computation as a file, so I can load the variable later on without recomputing it. When I load the variable I get an exception. I'm storing the file using the JLD library for Julia. I compute the variable using the library Eirene, which performs topological data analysis. The resulting file is 173MB.
I've looked this up and I couldn't find anything about this error message online. I was worried at first that the file was corrupted or something due to insufficient space but that's not the case.
This is the offending code:
using Eirene
using JLD
C = load("/path_to_file/file.jld")["filtration"]
And here is the code where I save the file:
using Eirene
using JLD
C = eirene("/path_to_distance_matrix/dmat",maxdim=3)
save("/path_to_file/file.jld", "filtration", C)
eirene basically builds a series of triangular structures based on the distances between points in a distance matrix I give it. The distance matrix in particular is very dense, causing a long triangle-building calculation.
You can access the distance matrix in this gist if you want to reproduce this problem. The calculation takes me 30min on my laptop.
I get the following error message (the file get_class_reps is the one that contains the first code snippet above).
ERROR: LoadError: unrecognized HDF5 datatype class 4
Stacktrace:
[1] jldatatype(::JLD.JldFile, ::HDF5.HDF5Datatype) at /home/curran/.julia/packages/JLD/1BoSz/src/jld_types.jl:738
[2] read(::JLD.JldDataset) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:370
[3] read_ref(::JLD.JldFile, ::HDF5.HDF5ReferenceObj) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:502
[4] read_refs(::JLD.JldDataset, ::Type{Any}, ::Int64, ::Int64, ::Tuple{Int64}) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:475
[5] read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int64, ::Int64, ::Tuple{Int64}) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:409
[6] read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int64, ::Int64) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:408
[7] read(::JLD.JldDataset) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:372
[8] read_ref(::JLD.JldFile, ::HDF5.HDF5ReferenceObj) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:502
[9] jlconvert(::Type{JLD.AssociativeWrapper{String,Any,Dict{String,Any}}}, ::JLD.JldFile, ::Ptr{UInt8}) at /home/curran/.julia/packages/JLD/1BoSz/src/jld_types.jl:434
[10] read_scalar(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Type) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:398
[11] read(::JLD.JldDataset) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:370
[12] read_ref(::JLD.JldFile, ::HDF5.HDF5ReferenceObj) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:502
[13] read_refs(::JLD.JldDataset, ::Type{Any}, ::Int64, ::Int64, ::Tuple{Int64}) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:475
[14] read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int64, ::Int64, ::Tuple{Int64}) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:409
[15] read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int64, ::Int64) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:408
[16] read(::JLD.JldDataset) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:372
[17] read_ref(::JLD.JldFile, ::HDF5.HDF5ReferenceObj) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:502
[18] jlconvert(::Type{JLD.AssociativeWrapper{String,Any,Dict{String,Any}}}, ::JLD.JldFile, ::Ptr{UInt8}) at /home/curran/.julia/packages/JLD/1BoSz/src/jld_types.jl:434
[19] read_scalar(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Type) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:398
[20] read(::JLD.JldDataset) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:370
[21] read(::JLD.JldFile, ::String) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:346
[22] #39 at ./none:0 [inlined]
[23] iterate at ./generator.jl:47 [inlined]
[24] collect(::Base.Generator{Array{String,1},getfield(JLD, Symbol("##39#41")){JLD.JldFile}}) at ./array.jl:606
[25] (::getfield(JLD, Symbol("##38#40")))(::JLD.JldFile) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:1234
[26] #jldopen#14(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(JLD, Symbol("##38#40")), ::String, ::Vararg{String,N} where N) at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:246
[27] load at /home/curran/.julia/packages/JLD/1BoSz/src/JLD.jl:244 [inlined]
[28] #load#13(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::String) at /home/curran/.julia/packages/FileIO/e8FNK/src/loadsave.jl:118
[29] load at /home/curran/.julia/packages/FileIO/e8FNK/src/loadsave.jl:118 [inlined]
[30] main() at /home/curran/dev/honours/q6/get_class_reps.jl:49
[31] top-level scope at none:0
in expression starting at /home/curran/dev/honours/q6/get_class_reps.jl:57
You've probably already fixed this, but if anyone else runs into this problem, switching from JLD to JLD2 fixed this for me. All that needs to be done is change using JLD to using FileIO and change all extensions in the save and load calls from .jld to .jld2.

Resources