Downloading artifact: CompilerSupportLibraries
curl: (52) Empty reply from server
ERROR: LoadError: Unable to automatically install 'CompilerSupportLibraries' from '/home/zyh/.julia/packages/CompilerSupportLibraries_jll/790hI/Artifacts.toml'
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] ensure_artifact_installed(::String, ::Dict{String,Any}, ::String; platform::Pkg.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool) at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/Artifacts.jl:894
[3] ensure_all_artifacts_installed(::String; platform::Pkg.BinaryPlatforms.Platform, pkg_uuid::Nothing, include_lazy::Bool, verbose::Bool, quiet_download::Bool) at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/Artifacts.jl:958
[4] download_artifacts(::Pkg.Types.Context, ::Array{String,1}; platform::Pkg.BinaryPlatforms.Linux, verbose::Bool) at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:616
[5] download_artifacts(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; platform::Pkg.BinaryPlatforms.Linux, verbose::Bool) at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:595
[6] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Array{Base.UUID,1}; preserve::Pkg.Types.PreserveLevel, platform::Pkg.BinaryPlatforms.Linux) at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:1091
[7] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; preserve::Pkg.Types.PreserveLevel, platform::Pkg.BinaryPlatforms.Linux, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:159
[8] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:112
[9] #add#27 at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:109 [inlined]
[10] add at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:109 [inlined]
[11] #add#23 at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:106 [inlined]
[12] add(::Pkg.Types.PackageSpec) at /usr/local/julia/julia-1.4.2/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:106
[13] top-level scope at /home/zyh/DOLPHYN_old/julenv.jl:20
[14] include(::String) at ./client.jl:439
[15] top-level scope at REPL[1]:1
I'm trying to construct a specific environment using JuMP 0.18.6 in Julia 1.4.2. But when downloading JuMP, it seems that this problem got blocking my may. Had anybody ever encountered this before and have a solution?
it's likely due to firewall of some sort, if you're from China, try the mirror source:
https://mirrors.tuna.tsinghua.edu.cn/help/julia/
I work mostly on offline machines and really want to begin to migrate from Python to Julia. Currently the biggest problem I face is how can I setup a package server on my own network that does not have access to the internet. I can copy files to the offline computer/network and want to be able to just cache a good percentage of the Julia Package Ecosystem and copy it to my network, so that I and others can install packages as needed.
I have experimented with PkgSever.jl by using the deployment docker-compose script they have, then just installing a long list of packages so that the PkgServer instance would cache everything. Next took the PkgServer machine offline and attempted to install packages from it. This worked well, however when I restarted the docker container the server was running in, everything fell apart quickly.
It seems that maybe the PkgServer needs to be able to talk to the Storage Server at least once before being able to serve packages. I tried setting:
JULIA_PKG_SERVER_STORAGE_SERVERS from: "https://us-east.storage.juliahub.com,https://kr.storage.juliahub.com" to: "" but that failed miserably.
Can someone please point me in the right direction.
TIA
It looks like the PkgServer is actually trying to contact the Registry before it starts. I don't know enough about the registry stuff enough to know if there is a way to hack this to look locally or just ignore this..
pkgserver_1 | ERROR: LoadError: DNSError: kr.storage.juliahub.com, temporary failure (EAI_AGAIN)
pkgserver_1 | Stacktrace:
pkgserver_1 | [1] getalladdrinfo(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Sockets/src/addrinfo.jl:112
pkgserver_1 | [2] getalladdrinfo at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Sockets/src/addrinfo.jl:121 [inlined]
pkgserver_1 | [3] getconnection(::Type{TCPSocket}, ::SubString{String}, ::String; keepalive::Bool, connect_timeout::Int64, kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},NTuple{4,Symbol},NamedTuple{(:require_ssl_verification, :iofunction, :reached_redirect_limit, :status_exception),Tuple{Bool,Nothing,Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/ConnectionPool.jl:630
pkgserver_1 | [4] #getconnection#29 at /depot/packages/HTTP/IAI92/src/ConnectionPool.jl:682 [inlined]
pkgserver_1 | [5] newconnection(::HTTP.ConnectionPool.Pod, ::Type{T} where T, ::SubString{String}, ::SubString{String}, ::Int64, ::Bool, ::Int64; kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception),Tuple{Nothing,Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/ConnectionPool.jl:597
pkgserver_1 | [6] getconnection(::Type{HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}, ::SubString{String}, ::SubString{String}; connection_limit::Int64, pipeline_limit::Int64, idle_timeout::Int64, reuse_limit::Int64, require_ssl_verification::Bool, kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception),Tuple{Nothing,Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/ConnectionPool.jl:541
pkgserver_1 | [7] request(::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}; proxy::Nothing, socket_type::Type{T} where T, reuse_limit::Int64, kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception),Tuple{Nothing,Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/ConnectionRequest.jl:73
pkgserver_1 | [8] (::Base.var"#56#58"{Base.var"#56#57#59"{ExponentialBackOff,HTTP.RetryRequest.var"#2#3"{Bool,HTTP.Messages.Request},typeof(HTTP.request)}})(::Type{T} where T, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception),Tuple{Nothing,Bool,Bool}}}) at ./error.jl:301
pkgserver_1 | [9] #request#1 at /depot/packages/HTTP/IAI92/src/RetryRequest.jl:44 [inlined]
pkgserver_1 | [10] request(::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol,Symbol},NamedTuple{(:reached_redirect_limit, :status_exception),Tuple{Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/MessageRequest.jl:51
pkgserver_1 | [11] request(::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; kw::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol,Symbol},NamedTuple{(:reached_redirect_limit, :status_exception),Tuple{Bool,Bool}}}) at /depot/packages/HTTP/IAI92/src/BasicAuthRequest.jl:28
pkgserver_1 | [12] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; redirect_limit::Int64, forwardheaders::Bool, kw::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:status_exception,),Tuple{Bool}}}) at /depot/packages/HTTP/IAI92/src/RedirectRequest.jl:24
pkgserver_1 | [13] request(::String, ::String, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; headers::Array{Pair{SubString{String},SubString{String}},1}, body::Array{UInt8,1}, query::Nothing, kw::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:status_exception,),Tuple{Bool}}}) at /depot/packages/HTTP/IAI92/src/HTTP.jl:314
pkgserver_1 | [14] #get#12 at /depot/packages/HTTP/IAI92/src/HTTP.jl:391 [inlined]
pkgserver_1 | [15] get_registries(::String) at /app/src/resource.jl:21
pkgserver_1 | [16] update_registries() at /app/src/resource.jl:130
pkgserver_1 | [17] start(; kwargs::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:listen_addr, :storage_root, :storage_servers),Tuple{Sockets.InetAddr{IPv4},String,Array{SubString{String},1}}}}) at /app/src/PkgServer.jl:88
pkgserver_1 | [18] top-level scope at /app/bin/run_server.jl:43
pkgserver_1 | [19] include(::Function, ::Module, ::String) at ./Base.jl:380
pkgserver_1 | [20] include(::Module, ::String) at ./Base.jl:368
pkgserver_1 | [21] exec_options(::Base.JLOptions) at ./client.jl:296
pkgserver_1 | [22] _start() at ./client.jl:506
pkgserver_1 | in expression starting at /app/bin/run_server.jl:43
This might be helpful but I'm not sure, yet, how to get it started
LocalRegistry.jl
Here is a solution that seems to work, based on LocalPackageServer.
Preliminary steps
Install all required packages. You can either put them in your default environment (e.g. #v1.5) or in a dedicated project.
LocalRegistry
LocalPackageServer
In order to use LocalPackageServer, we'll need to set up a local registry, even though we won't really use it (but it can still be handy if you also have to serve local packages).
Something like this should create an empty local registry as local-registry.gitt in the current folder:
# Create an empty (bare) git repository to host the registry
run(`git init --bare local-registry.git`)
# Populate the repository with a new, empty local registry
using LocalRegistry
Base.Filesystem.mktempdir() do dir
create_registry(joinpath(dir, "local-registry"),
abspath("local-registry.git"),
description = "(unused) local registry",
push=true)
end
Step 1a - run the local package server (online)
A script like the following should run a local package server listening on http://localhost:8000.
#################
# run_server.jl #
#################
using LocalPackageServer
config = LocalPackageServer.Config(Dict(
# Server parameters
"host" => "127.0.0.1",
"port" => 8000,
"pkg_server" => "https://pkg.julialang.org",
# This is where persistent data will be stored
# (I use the current directory here; adjust to your constraints)
"local_registry" => abspath("local-registry.git"), # In accordance with the preliminary step above
"cache_dir" => abspath("cache"),
"git_clones_dir" => abspath("data"),
))
# The tricky part: arrange for the server to never update its registries
# when it is offline
if get(ENV, "LOCAL_PKG_SERVER_OFFLINE", "0") == "1"
#info "Running offline => no registry updates"
config.min_time_between_registry_updates = typemax(Int)
end
# Start the server
LocalPackageServer.start(config)
Use this script to run the server online first:
shell$ julia --project run_server.jl
Step 1b - cache some packages (online)
Configure a Julia process to use your local server, and install the packages you want to cache:
# Take care to specify the "http://" protocol
# (otherwise https might be used by the client, and the server won't answer)
shell$ JULIA_PKG_SERVER=http://localhost:8000 julia
julia> using Pkg
julia> pkg"add Example"
[...]
At this point, the server should be caching things.
Step 2 - run the package server (offline)
When you're offline, simply restart the server, ensuring it won't try to update the registries:
shell$ LOCAL_PKG_SERVER_OFFLINE=1 julia --project run_server.jl
As before, set JULIA_PKG_SERVER as needed for Julia clients to use the local server; they should now be able to install packages, provided that the project environments resolve to the exact same dependency versions that were cached.
(You might want to resolve and instantiate your project environments online, and then transfer the relevant manifests to offline systems: this might help guarantee the consistency between the package server cache and what clients ask for.)
I just installed Julia and would now like to install the IJulia notebook. However, compilation fails:
julia> Pkg.build("IJulia")
Building Conda ──→ `~/.julia/packages/Conda/m7vem/deps/build.log`
Building ZMQ ────→ `~/.julia/packages/ZMQ/yClfT/deps/build.log`
Building MbedTLS → `~/.julia/packages/MbedTLS/Qo8TN/deps/build.log`
Building IJulia ─→ `~/.julia/packages/IJulia/iMVN2/deps/build.log`
┌ Error: Error building `IJulia`:
│ ┌ Warning: Could not execute `jupyter --version`.
│ └ # Main ~/.julia/packages/IJulia/iMVN2/deps/build.jl:38
│ [ Info: Installing Jupyter via the Conda package.
│ [ Info: Downloading miniconda installer ...
│ ERROR: LoadError: IOError: could not spawn `curl -g -L -f -o /home/user001/.julia/packages/Conda/m7vem/deps/usr/installer.sh https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh`: no such file or directory (ENOENT)
│ Stacktrace:
│ [1] _jl_spawn(::String, ::Array{String,1}, ::Cmd, ::Tuple{RawFD,RawFD,RawFD}) at ./process.jl:367
│ [2] (::getfield(Base, Symbol("##495#496")){Cmd})(::Tuple{RawFD,RawFD,RawFD}) at ./process.jl:509
│ [3] setup_stdio(::getfield(Base, Symbol("##495#496")){Cmd}, ::Tuple{RawFD,RawFD,RawFD}) at ./process.jl:490
│ [4] #_spawn#494(::Nothing, ::Function, ::Cmd, ::Tuple{RawFD,RawFD,RawFD}) at ./process.jl:508
│ [5] _spawn at ./process.jl:504 [inlined]
│ [6] #run#505(::Bool, ::Function, ::Cmd) at ./process.jl:652
│ [7] run at ./process.jl:651 [inlined]
│ [8] download(::String, ::String) at ./download.jl:48
│ [9] macro expansion at ./logging.jl:311 [inlined]
│ [10] _install_conda(::String, ::Bool) at /home/user001/.julia/packages/Conda/m7vem/src/Conda.jl:146
│ [11] _install_conda(::String) at /home/user001/.julia/packages/Conda/m7vem/src/Conda.jl:145
│ [12] runconda(::Cmd, ::String) at /home/user001/.julia/packages/Conda/m7vem/src/Conda.jl:111
│ [13] add at /home/user001/.julia/packages/Conda/m7vem/src/Conda.jl:174 [inlined] (repeats 2 times)
│ [14] top-level scope at logging.jl:311
│ [15] top-level scope at /home/user001/.julia/packages/IJulia/iMVN2/deps/build.jl:38
│ [16] include at ./boot.jl:317 [inlined]
│ [17] include_relative(::Module, ::String) at ./loading.jl:1038
│ [18] include(::Module, ::String) at ./sysimg.jl:29
│ [19] include(::String) at ./client.jl:388
│ [20] top-level scope at none:0
│ in expression starting at /home/user001/.julia/packages/IJulia/iMVN2/deps/build.jl:9
└ # Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1068
The file pointed out actually exists and I am able to fetch it:
$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
--2018-08-27 16:29:15-- https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
Resolving repo.continuum.io (repo.continuum.io)... 104.16.18.10, 104.16.19.10, 2400:cb00:2048:1::6810:130a, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.16.18.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 58468498 (56M) [application/x-sh]
Saving to: ‘Miniconda3-latest-Linux-x86_64.sh’
Miniconda3-latest-Linux-x86_64.sh 100%[======================================================================================================>] 55,76M 102MB/s in 0,5s
2018-08-27 16:29:16 (102 MB/s) - ‘Miniconda3-latest-Linux-x86_64.sh’ saved [58468498/58468498]
What's wrong?
Update: regarding the concernes with the .julia/packages/Conda/m7vem/deps/usr/ folder, all appears correct:
$ ll /home/user001/.julia/packages/Conda/m7vem/deps/usr/
total 8
drwxr-xr-x 2 user001 user001 4096 aug 27 15:56 ./
drwxr-xr-x 3 user001 user001 4096 aug 27 15:56 ../
Trying to understand following error in a pmap execution of some embarrassingly parallel tasks. Running on Linux server. might be occurring while writing to HDF (part of parallel call), but I don't think so given stacktrace doesn't point to line in the user-function being executed, and reference to TCP suggests it's part of parallel calls. Has happened in several sequential runs, so not one-time fluke.
Worker 139 terminated.
ERROR (unhandled task failure): EOFError: read end of file
Stacktrace:
[1] unsafe_read(::Base.AbstractIOBuffer{Array{UInt8,1}}, ::Ptr{UInt8}, ::UInt64) at ./iobuffer.jl:105
[2] unsafe_read(::TCPSocket, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:752
[3] unsafe_read(::TCPSocket, ::Base.RefValue{NTuple{4,Int64}}, ::Int64) at ./io.jl:361
[4] read at ./io.jl:363 [inlined]
[5] deserialize_hdr_raw at ./distributed/messages.jl:170 [inlined]
[6] message_handler_loop(::TCPSocket, ::TCPSocket, ::Bool) at ./distributed/process_messages.jl:157
[7] process_tcp_streams(::TCPSocket, ::TCPSocket, ::Bool) at ./distributed/process_messages.jl:118
[8] (::Base.Distributed.##99#100{TCPSocket,TCPSocket,Bool})() at ./event.jl:73
Julia info:
julia> versioninfo()
Julia Version 0.6.0
Commit 9036443 (2017-06-19 13:05 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5620 # 2.40GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Nehalem)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.9.1 (ORCJIT, westmere)
[EDIT: more info] Also, if it's helpful, this appears to be happening well into the run -- output from the first set of parallel runs looks like it's being saved to disk, so this is not an immediate crash, but something that's happening at the end of a run or start of the second set of executions.
OK, so I did finally figure out at a high level what this means:
This is the error you get when one of the parallelized workers hits an error. The specific error language (EOFError: read end of file) doesn't really mean anything. And the references to read and io in the stackoverflow just relate to messaging between the overview task and workers.
In my case, the error was a memory overflow leading to the task manager terminating the worker.
Is there a way to request an URL and get the response as a string?
I've tried to use both the "Requests" package and "LibCURL" but I cant find a way to do this
Fx. if I use requests the error I receive looks like
get(url)
ERROR: MbedTLS error code -30592: SSL - A fatal alert message was received from our peer
in macro expansion at /home/david/.julia/v0.5/MbedTLS/src/error.jl:4 [inlined]
in handshake(::MbedTLS.SSLContext) at /home/david/.julia/v0.5/MbedTLS/src/ ssl.jl:145
in open_stream(::HttpCommon.Request, ::MbedTLS.SSLConfig, ::Float64, ::Nullable{URIParser.URI}, ::Nullable{URIParser.URI}) at /home/david/.julia/ v0.5/Requests/src/streaming.jl:209
in #do_stream_request#23(::Dict{AbstractString,AbstractString}, ::Void, ::Void, ::Void, ::Array{Requests.FileParam,1}, ::Void, ::Dict{Any,Any}, ::Bool, ::Int64, ::Array{HttpCommon.Response,1}, ::MbedTLS.SSLConfig, ::Bool, ::Bool, ::Bool, ::Nullable{URIParser.URI}, ::Nullable{URIParser.URI}, ::Requests.#do_stream_request, ::URIParser.URI, ::String) at /home/david/.julia /v0.5/Requests/src/Requests.jl:361
in do_stream_request(::URIParser.URI, ::String) at /home/david/.julia/v0.5/ Requests/src/Requests.jl:324
in #do_request#22(::Array{Any,1}, ::Function, ::URIParser.URI, ::String) at / home/david/.julia/v0.5/Requests/src/Requests.jl:291
in #get#29(::Array{Any,1}, ::Function, ::URIParser.URI) at /home/david/.julia/ v0.5/Requests/src/Requests.jl:424
in #get#28(::Array{Any,1}, ::Function, ::String) at /home/david/.julia/v0.5/ Requests/src/Requests.jl:423
in get(::String) at /home/david/.julia/v0.5/Requests/src/Requests.jl:423
You're looking for readall with Requests (but in the future it may be moved to readstring since readall has been deprecated in Base Julia):
julia> using Requests
julia> readall(get("http://example.com"))
"<!doctype html>\n<html>\n<head> …