I would like to have .julia under julia-1.0.0 directory so that I can copy entire julia installation to an offline machine.
what are the environment associated variables ? For 0.6 there used to be
JULIA_PKGDIR
JULIA_HOME
LD_LIBRARY_PATH
Tried setting these, still have an issue:
julia> LOAD_PATH
1-element Array{String,1}:
"/share/apps/Julia/julia-1.0.0/.julia/"
julia> DEPOT_PATH
1-element Array{String,1}:
"/share/apps/Julia/julia-1.0.0/.julia/"
julia> using Pkg
ERROR: ArgumentError: Package Pkg not found in current path:
- Run `Pkg.add("Pkg")` to install the Pkg package.
Stacktrace:
[1] require(::Module, ::Symbol) at ./loading.jl:817
JULIA_LOAD_PATH and JULIA_DEPOT_PATH. See the documentation of the new package manager.
your *_PATHs should have more entries:
> LOAD_PATH
3-element Array{String,1}:
"#"
"#v#.#"
"#stdlib"
> DEPOT_PATH
3-element Array{String,1}:
"/home/no/.julia"
"/home/no/.julia/julia-1.0.0/local/share/julia"
"/home/no/.julia/julia-1.0.0/share/julia"
It looks like you overwrote the default values and now julia cant find the other packages.
Also, (at least for my installation on ubuntu/wsl), if you just unpack the julia installation to a folder it'll do its magic there and you can copy that folder, ie .julia.
Related
Info:
$ julia --version
julia version 1.6.0
$ lscpu
~/root/MyPackage$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core(TM) i9-9880H CPU # 2.30GHz
...
Say I want the following package structure, and want to use ReTest's parallel testing (my issue appears to be with how code-loading works in Distributed, so this isn't really a ReTest-specific issue).
| root/
| MyPackage/
| Project.toml
| Manifest.toml
| src/
| MyPackage.jl
| test/
| runtests.jl
| MyPackageTests.jl
I initialised this package in the following way:
$ cd root && julia
(...) pkg> generate MyPackage;
$ cd MyPackage && julia
(...) pkg> activate .
(...) pkg> instantiate
(...) pkg> add ReTest InlineTest Distributed;
...
Fill in MyPackage.jl, runtests.jl, and MyPackageTests.jl with some Julia code.
Not too important what that code is - although I am following the guide from here in ReTest.
Then to set up:
$ julia
(...) pkg> activate .
(...) pkg> instantiate
(MyPackage) pkg> st
Project MyPackage v0.1.0
Status `~/root/MyPackage/Project.toml`
[bd334432] InlineTest v0.2.0
[e0db7c4e] ReTest v0.3.2
[8ba89e20] Distributed
julia> LOAD_PATH
3-element Vector{String}:
"#" # Should be current active environment for MyPackage
"#v#.#" # Should be #v1.6 on my system
"#stdlib" # Should be absolute path of current Julia installation's stdlib
julia> # Should this code be in .jl files? Don't think that should matter.
julia> using Distributed
julia> addprocs(2)
julia> #everywhere include("test/MyPackageTests.jl")
ERROR: On worker 2:
LoadError: ArgumentError: Package MyPackage not found in current path:
- Run `import Pkg; Pkg.add("MyPackage")` to install the MyPackage package.
Stacktrace:
[1] require
# ./loading.jl:871
[2] include
# ./client.jl:444
[3] top-level scope
# none:1
[4] eval
# ./boot.jl:360
[5] #103
# /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:274
[6] run_work_thunk
# /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
[7] run_work_thunk
# /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72
[8] #96
# ./task.jl:406
in expression starting at /path/to/root/MyPackage/test/MyPackageTests.jl:1
...and 2 more exceptions.
Stacktrace:
[1] sync_end(c::Channel{Any})
# Base ./task.jl:364
[2] macro expansion
# ./task.jl:383 [inlined]
[3] remotecall_eval(m::Module, procs::Vector{Int64}, ex::Expr)
# Distributed /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:223
[4] top-level scope
# /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:207
I'd like to understand why this is. On GitHub I see this issue that was supposedly fixed. I can confirm when I run that example that I also get the exact same problem as above with MyPackage, if the using statement involves the package that the environment is for.
Before filing a bug or opening an issue there, I'd like to check here in case this is a process problem on my part. If not, there's evidently something wrong with Distributed/ReTest and I'll open tickets for those. Any help much appreciated.
Following on from what #carstenbauer said, the active Julia environment is not automatically propagated to worker processes by default. The way around this is to set the environment in the arguments to the call to addprocs like so:
julia> using Distributed
julia> addprocs(2, exeflags="--project=$(Base.active_project())")
julia> #everywhere include("test/MyPackageTests.jl")
julia> MyPackageTests.runtests() # runs to completion
I can confirm that this works with both the MyPackage example as well as the one shown in the JuliaLang issue. Thanks to those that contributed towards this answer.
Try the following code:
using Distributed
addprocs(4) # or whatever you need or use the -p parameter
using Pkg
pkg"activate ."
pkg"instantiate" # run this when needed
using MyPackage # first load package only an the master worker
#everywhere pkg"activate ."
#everywhere using MyPackage
Explanation:
each Julia processes is totally separated so it has its own package state, variables, memory etc.
Please note that you will usually prefer to load the package first on the master node as some packages might be performing some actions where loaded for the first time.
I want to use Base.compilecache.
It needs PkgId(UUID). But I don't know how to get installed Pkg.uuid in REPL.
https://docs.julialang.org/en/v1.0.0/base/base/#Base.compilecache
I tried Pkg.PackageSpec but can't get UUID.
https://docs.julialang.org/en/stable/stdlib/Pkg/#Pkg.PackageSpec
julia> using Pkg
julia> Pkg.status()
Status `~/.julia/environments/v1.0/Project.toml`
[8f4d0f93] Conda v1.0.1
[7073ff75] IJulia v1.9.3
julia> Pkg.PackageSpec("Conda")
PackageSpec(name=Conda)
julia> Pkg.PackageSpec("Conda").uuid
UUID("00000000-0000-0000-0000-000000000000")
My goal is to call Base.compilecache at Docker file.
Julia v0.6.4 code
julia -e "Base.compilecache(\"JSON\")"
You can use Base.identify_package("Conda").
I want to help Julia find my .jl file by modifying the LOAD_PATH variable:
julia> readdir()
1-element Array{String,1}:
"test.jl"
shell> cmd /c type test.jl
# Test.jl
module Test
export f
f() = println("hi")
end
julia> push!(LOAD_PATH,pwd());
julia> import Test
ERROR: ArgumentError: Module Test not found in current path.
Run `Pkg.add("Test")` to install the Test package.
in require(::Symbol) at .\loading.jl:365
The first call to readdir() proves that I have a file called test.jl in my current directory. The following shell call shows that this file contains a module called Test. The next call to push!(LOAD_PATH,pwd()); puts the current directory in LOAD_PATH. But even with the current directory in LOAD_PATH, Julia still can't find the Test module in test.jl.
What's wrong?
The error was talking about something concerning require. As the doc says:
Given the statement using Foo, the system looks for Foo within Main. If the module does not exist, the system attempts to require("Foo"), which typically results in loading code from an installed package. ... require is case-sensitive on all platforms, including those with case-insensitive filesystems like macOS and Windows.
and the reason is clear: require couldn't find a file named Test in LOAD_PATH. So we need to make the file name matching the module name, but this is just a convention, not a mandatory rule. What will happen if someone mistakenly runs using test?
julia> push!(LOAD_PATH,pwd())
julia> using test
WARNING: requiring "test" in module "Main" did not define a corresponding module.
julia> whos()
Base 34427 KB Module
Core 12386 KB Module
Main 41296 KB Module
Test 1837 bytes Module
The result shows that we've loaded the file test.jl and the module(Test) in it, but not actually using/import the module. This is a respected behavior since we used a wrong module name, which is also the reason why julia complained in the warning. In this case, using test is equivalent to include("test.jl"), but I highly recommend you to follow the convention and do not use this behavior.
BTW, require became generally case-sensitive after this PR. A side-effect is your LOAD_PATH should also be case-sensitive, this will be fixed by this PR in julia-0.6.
The current version of Julia is 0.4.6. I, however, am running the development version 0.5. Suddenly JLD doesn't work. It's installed and updated. Yesterday I compiled code using JLD but this morning it doesn't work.
julia> using JLD
INFO: Precompiling module JLD...
WARNING: Method definition convert(Type{#T<:AbstractString}, AbstractArray{#S<:Union{Char, Int32, UInt32}, 1}) in module Base at unicode/utf32.jl:131 overwritten in module LegacyStrings at /root/.julia/v0.5/LegacyStrings/src/utf32.jl:133.
WARNING: Method definition isvalid(Array{Char, 1}) in module Base at unicode/utf32.jl:177 overwritten in module LegacyStrings at /root/.julia/v0.5/LegacyStrings/src/utf32.jl:179.
WARNING: New definition
string(Union{Char, LegacyStrings.UTF8String, LegacyStrings.ASCIIString}...) at /root/.julia/v0.5/LegacyStrings/src/utf8.jl:161
is ambiguous with:
string(Union{Char, UTF8String, ASCIIString}...) at unicode/utf8.jl:166.
To fix, define
string(Char...)
before the new definition.
WARNING: both LegacyStrings and Base export "UTF16String"; uses of it in module JLD must be qualified
ERROR: LoadError: LoadError: UndefVarError: UTF16String not defined
in include(::ASCIIString) at ./boot.jl:234
in include_from_node1(::ASCIIString) at ./loading.jl:417
in include(::ASCIIString) at ./boot.jl:234
in include_from_node1(::ASCIIString) at ./loading.jl:417
[inlined code] from ./boot.jl:237
in anonymous at ./<no file>:4294967295
in eval(::Module, ::Any) at ./boot.jl:237
[inlined code] from ./sysimg.jl:11
in process_options(::Base.JLOptions) at ./client.jl:239
in _start() at ./client.jl:318
while loading /root/.julia/v0.5/JLD/src/jld_types.jl, in expression starting on line 11
while loading /root/.julia/v0.5/JLD/src/JLD.jl, in expression starting on line 130
ERROR: Failed to precompile JLD to /root/.julia/lib/v0.5/JLD.ji
in error(::ASCIIString) at ./error.jl:21
in compilecache(::ASCIIString) at ./loading.jl:496
in compilecache(::Symbol) at ./loading.jl:485
in require(::Symbol) at ./loading.jl:355
in eval(::Module, ::Any) at ./boot.jl:237
When using the development version of Julia, you need to use the development versions of the packages (which works for packages where the developer keeps master up-to-date but hasn't tagged yet). If you run Pkg.checkout("JLD") to checkout master, JLD should work (works on my machine. Note you may need to Pkg.update() before checking out, and you may need to quit Julia and re-open it to recompile the new version).
But as a word of caution, don't use the development versions of Julia as a way to test things out. Remember, the language is still in alpha and there is no guarantee that the package ecosystem or Julia itself will work with the daily master. The dailies are good for working on the language and preparing packages for the next version (and being ballsy I guess).
Well, you're using Julia 0.5 which is still in development. If you switch to 0.4.6 then it should be fine. I'm using that and JLD works fine for me. You could also trying running Pkg.update() closing Julia, then reopening, to see if that helps.
I want to change the Package directory in Julia. The default is
"~/.julia/v0.4"
I want to move it to /opt/julia/v0.4/. Ideally I want to move the packages that are already installed in ~/.julia/v0.4 to the new location. But if that is not possible I can reinstall them.
What do I have to do?
Julia-v0.6 and before
one can change julia's package directory by following these steps:
run export JULIA_PKGDIR=/your/directory in shell(or manually add a new environment variable JULIA_PKGDIR on windows)
run Pkg.init() in julia to initialize a new package system
copy REQUIRE from old directory to the new one
run Pkg.resolve() in julia
Julia-v0.7+
The "Package directory" in the new package manager is called DEPOT_PATH, it can be changed by adding an environment variable JULIA_DEPOT_PATH:
JULIA_DEPOT_PATH=./test julia
julia> DEPOT_PATH
1-element Array{String,1}:
"./test"
(v0.7) pkg> add JSON2
Cloning default registries into /Users/gnimuc/test/registries
With the new package manager, we are able to create isolated projects in any directory we want instead of having a single giant package directory. Every project contains a Project.toml and a Manifest.toml if it has some dependencies. These two files record and keep tracking the environment of the project.
UPDATE
The following info might be obsoleted. I highly recommend to use PkgTemplates.jl for generating projects in Julia-v1.0+.
Generate a new project
We can generate a new project in any folder, but we must cd to the project folder to using the project. The (v0.7) below shows we're still in the default environment, so we cannot use the newly generated project:
(v0.7) pkg> generate ./MyNewProject
Generating project MyNewProject:
./MyNewProject/Project.toml
./MyNewProject/src/MyNewProject.jl
julia> using MyNewProject
ERROR: ArgumentError: Module MyNewProject not found in current path.
Run `Pkg.add("MyNewProject")` to install the MyNewProject package.
Stacktrace:
[1] require(::Module, ::Symbol) at ./loading.jl:868
If we cd to the project folder and activate the environment, then we can using our new project without any problems:
shell> cd MyNewProject/
/Users/gnimuc/MyNewProject
(v0.7) pkg> activate .
(MyNewProject) pkg>
julia> using MyNewProject
I think that's the big difference between the new package manager and the old one. In short, we need to explicitly activate our unregistered project/package.
Download and init someone else's project
According to the doc, we can add an unregistered package/project via add command:
(HelloWorld) pkg> add https://github.com/fredrikekre/ImportMacros.jl
This command adds the target package as a dependency of our current project. In this example, we added ImportMacros in HelloWorld's Manifest.toml. What if we just use it as a top-level project? To add it to the default environment (v0.7)? no, we don't need to. The answer is we can directly download the code, cd to the folder and run instantiate in the pkg mode:
shell> git clone https://github.com/Gnimuc/GLTF.jl GLTF
Cloning into 'GLTF'...
remote: Counting objects: 286, done.
remote: Compressing objects: 100% (56/56), done.
remote: Total 286 (delta 73), reused 103 (delta 59), pack-reused 167
Receiving objects: 100% (286/286), 62.16 KiB | 46.00 KiB/s, done.
Resolving deltas: 100% (135/135), done.
shell> cd GLTF
pkg> activate .
(GLTF) pkg> instantiate
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
The new package manager is great! We neither need "include before using" nor make everything as a package just for using it. We have full-featured "Project" now!
Julia only way:
julia> ENV["JULIA_PKGDIR"] = "E:\\Julia-0.6.0\\portable"
"E:\\Julia-0.6.0\\portable"
julia> ENV["JULIA_PKGDIR"]
"E:\\Julia-0.6.0\\portable"
julia> Pkg.init()
INFO: Initializing package repository E:\Julia-0.6.0\portable\v0.6
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
However, the cache dir is still pointing to the old folder, so I checked why that is and figured it out:
julia> Base.LOAD_CACHE_PATH
1-element Array{String,1}:
"C:\\Users\\kung\\.julia\\lib\\v0.6"
julia> Pkg.__init__()
2-element Array{String,1}:
"E:\\Julia-0.6.0\\portable\\lib\\v0.6"
"C:\\Users\\kung\\.julia\\lib\\v0.6"
julia> pop!(Base.LOAD_CACHE_PATH)
"C:\\Users\\kung\\.julia\\lib\\v0.6"
julia> Base.LOAD_CACHE_PATH
1-element Array{String,1}:
"E:\\Julia-0.6.0\\portable\\lib\\v0.6"
As simple function:
function set_julia_dir(dir::String)
ENV["JULIA_PKGDIR"] = dir
Pkg.init()
Pkg.__init__()
pop!(Base.LOAD_CACHE_PATH)
end
set_julia_dir("E:\\Julia-0.6.0\\portable")