How to run a Julia project? - julia

Julia is not in my wheelhouse yet, but I have been handed a Julia project to run the code within.
This consists of a directory containing a main.jl, a Project.toml and a Manifest.toml.
I've read up a little on what the TOML files are for; to summarise my current understanding, they form a project or environment (not sure which, or what the real difference is).
I have installed Julia v1.3.1 at the command line by downloading the tar, decompressing and placing in my path. Typing julia at the command line opens the Julia CLI REPL as expected.
I have tried to run the code by using julia main.jl, this results in complaints about the required packages not being present, e.g.:
julia main.jl
ERROR: LoadError: ArgumentError: Package JSON not found in current path:
- Run `import Pkg; Pkg.add("JSON")` to install the JSON package.
Stacktrace:
[1] require(::Module, ::Symbol) at ./loading.jl:887
[2] include at ./boot.jl:328 [inlined]
[3] include_relative(::Module, ::String) at ./loading.jl:1105
[4] include(::Module, ::String) at ./Base.jl:31
[5] exec_options(::Base.JLOptions) at ./client.jl:287
[6] _start() at ./client.jl:460
in expression starting at /home/<user>/myproject/main.jl:3
I can follow the instructions here and load the required packages, but surely I shouldn't do this manually for every package?
As every package required is listed in the Project.toml I guess there should be some way to tell Julia to make sure the packages in the project are made available (I'm thinking something along the lines of Python's requirements file).
I have tried julia --project=main.jl, but this just results in the REPL loading again with nothing happening (not sure if any project or environment is loaded or not).
How can I tell Julia to run the script in this project while taking note of the requirements and other information in the TOML files?
Update:
Have figured out to enter ] at the REPL to enter the pkg package manager. Then I can:
(v1.3) pkg> activate .
Activating environment at `~/myproject/Project.toml`
(myproject) pkg> instantiate
(myproject) pkg>
Then leave the manager by pressing backspace.
Still not sure how to "run" everything though.

You’re very close to the solution! If the files are all in a directory dir then the command would be
julia --project=dir main.jl
You could also start an interactive session in that environment and then run the code in the file via
julia --project=dir
julia> include(“main.jl”)
Edit: If the directory is the current working directory, then you can just use --project=.

The error message Package JSON not found in current path imply that you don't have JSON installed.
You can check this by starting Julia and type using JSON
To install JSON all you have to do is writing import Pkg; Pkg.add("JSON")
See this output for example:
$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.5.2 (2020-09-23)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using JSON
ERROR: ArgumentError: Package JSON not found in current path:
- Run `import Pkg; Pkg.add("JSON")` to install the JSON package.
Stacktrace:
[1] require(::Module, ::Symbol) at ./loading.jl:893
julia> import Pkg; Pkg.add("JSON")
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
Updating `~/.julia/environments/v1.5/Project.toml`
[682c06a0] + JSON v0.21.1
Updating `~/.julia/environments/v1.5/Manifest.toml`
[682c06a0] + JSON v0.21.1
[69de0a69] + Parsers v1.1.0
[ade2ca70] + Dates
[a63ad114] + Mmap
[de0858da] + Printf
[4ec0a83e] + Unicode
julia> using JSON
[ Info: Precompiling JSON [682c06a0-de6a-54ab-a142-c8b1cf79cde6]
julia>

Related

How include a file module in Julia 1.7?

My goal is: in Julia 1.7 on Mac with M1 processor I would include a module file with many functions inside.
I've tried to follow this thread to generate my own package
but it's generated an error.
I followed the answer but when i try importing MyPackage, Julia says: ArgumentError: Package MyPackage not found in current path.
With pwd() current path is Users/myname and in this folder MyPackage exists.
With command "import MyPackage" where i can see default folder for packages import ?
Where I get wrong ?
Sorry for my English.
I know two options here. Suppose I generated a package with the following file structure
MyPackage/
src/
MyPackage.jl
Project.toml
and MyPackage.jl is
module MyPackage
export greet
greet() = print("Hello World!")
end # module
Then there are two options, I know
Including of module MyPackage;
Embedding of package MyPackage into Julia's ecosystem.
Suppose, that pwd is MyPackage/
First option is
julia> include("src/MyPackage.jl")
Main.MyPackage
julia> using .MyPackage
julia> greet()
Hello World!
Notice the dot in using statement.
Second option
Start Julia REPL and enter pkg mode, then
(#v1.6) pkg> dev MyPackage/
Resolving package versions...
Updating `~/.julia/environments/v1.6/Project.toml`
[88e94d31] + MyPackage v0.1.0 `foo/bar/MyPackage`
Updating `~/.julia/environments/v1.6/Manifest.toml`
[88e94d31] + MyPackage v0.1.0 `foo/bar/MyPackage`
(#v1.6) pkg> st
Status `~/.julia/environments/v1.6/Project.toml`
[6e4b80f9] BenchmarkTools v1.3.1
[0772a1fa] CubicEoS v0.2.0
..........
[88e94d31] MyPackage v0.1.0 `foo/bar/MyPackage`
..........
[09ab397b] StructArrays v0.6.5
[a759f4b9] TimerOutputs v0.5.15
st command says that MyPackage became available in default environment.
Then you should be able to import/using the package. For the first time you should see precompiling message.
julia> using MyPackage
[ Info: Precompiling MyPackage [88e94d31-ecaf-41ca-ae10-053d89a189ff]
julia> greet()
Hello World!
P.S. The best place for local packages is .julia/dev/.

Inheriting dependencies when running unit tests from command line

I am trying to run Julia unit tests from the command line but the unit tests fail to run because they cannot find a dependency that I am using in my main project. How can I make this work? The actual command that I try to execute is julia test/test_blueprint.jl from the project root. Here follows more details.
Details about the setup
My project is located at the path /home/jonas/prog/julia/blueprint. In that directory, I have a Project.toml file containing these lines:
name = "blueprint"
uuid = "c1615a0c-c255-402d-ae34-0b88819b43c6"
authors = [""]
version = "0.1.0"
[deps]
FunctionalCollections = "de31a74c-ac4f-5751-b3fd-e18cd04993ca"
Setfield = "efcf1570-3423-57d1-acb7-fd33fddbac46"
along with the Manifest.toml file.
I have a subdirectory at test/ with unit tests that I created following this guide and that directory contains another Project.toml file containing
[deps]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
There is a file test/test_blueprint.jl with unit tests and that file starts with
using Test
include("../src/blueprint.jl") # Alternative 1
#using blueprint # Alternative 2
using FunctionalCollections
using LinearAlgebra
...
The actual code being tested is in the file src/blueprint.jl.
Details about the problem
From the project root, I attempt to run the unit tests using the command julia test/test_blueprint.jl. When I run that command it produces the following output:
ERROR: LoadError: ArgumentError: Package Setfield not found in current path:
- Run `import Pkg; Pkg.add("Setfield")` to install the Setfield package.
Stacktrace:
[1] require(into::Module, mod::Symbol)
# Base ./loading.jl:967
[2] include(fname::String)
# Base.MainInclude ./client.jl:451
[3] top-level scope
# ~/prog/julia/blueprint/test/test_blueprint.jl:8
in expression starting at /home/jonas/prog/julia/blueprint/src/blueprint.jl:1
in expression starting at /home/jonas/prog/julia/blueprint/test/test_blueprint.jl:8
suggesting that it cannot find the dependency Setfield. If I edit the top of the file test/test_blueprint.jl slightly from
include("../src/blueprint.jl") # Alternative 1
#using blueprint # Alternative 2
to
#include("../src/blueprint.jl") # Alternative 1
using blueprint # Alternative 2
it still fails, but with a different error:
ERROR: LoadError: ArgumentError: Package blueprint not found in current path:
- Run `import Pkg; Pkg.add("blueprint")` to install the blueprint package.
Stacktrace:
[1] require(into::Module, mod::Symbol)
# Base ./loading.jl:967
in expression starting at /home/jonas/prog/julia/blueprint/test/test_blueprint.jl:9
Question: How can I make the unit tests run from the command line?
Note that I can run the unit tests from within the Julia REPL in Emacs by activating the project using C-c C-a at the src/blueprint.jl file and calling C-c C-b at the unit test file test/test_blueprint.jl. My Julia version is 1.7.0 (2021-11-30). Don't hesitate to ask for more clarifications.
First, a few naming conventions that are probably not (but may be) contributing to the issues here:
By convention, package names begin with a single capital, so I would recommend changing the name to Blueprint everywhere
By default, ] test runs tests found in the test/runtests.jl, so I would recommend naming your top-level testing script runtests.jl to avoid confusion, even though it does seem from the errors here that test is finding your test_blueprint.jl file one way or another.
Now, while I can't test this without the full code of your package, what I suspect is happening here is the following:
Normally, dependencies of the package you are testing (let's say MyPackage) are not required in test/Project.toml because they are implicit in MyPackage. So after a successful using MyPackage, while they will still not be available to any functions written in your test scripts (test/runtests.jl), will be available to the functions written in MyPackage -- just as if you had typed ]using MyPackage at the REPL and then run your test code there. This is the only reason you don't normally need to duplicate all the deps from the main Project.toml in test/Project.toml.
Since the using Blueprint approach is failing here for other reasons, when you simply include the code from src/blueprint.jl, the usings within that file will in turn fail because those packages are not present in the active environment at test/Project.toml (even if they are present on your system elsewhere).
Consequently, one quick fix to your problem with the current include("../src/blueprint.jl") approach would be to simply add those dependencies to your test/Project.toml
However, it would be more satisfying to fix the problem you are having with using Blueprint. I don't have enough information to debug this without seeing the full structure of your packages, but I would suggest as a start
making sure that your code is properly structured as a package
testing that, even if unregistered, you can ] add your package from the REPL by git repo URL (i.e. ] add https://some_website.com/you/Blueprint.jl)
EDIT:
Upon inspection of the code linked in the comments (https://github.com/jonasseglare/Blueprint), a few other issues:
Although they are already installed by default, standard libraries these days do need to be included in [deps]. In this case, that means the LinearAlgebra stdlib
Any packages you are explicitly using in your test scripts, other than your package itself, do need to be added to test/Project.toml. I.e., any packages that you are directly using functions from in your test scripts (rather than just indirectly using via the exported functions of your package) do need to be included in test/Project.toml.
In your case, the latter would appear to mean LinearAlgebra and FunctionalCollections, but not Setfield (that one only needs to be included in the regular Project.toml, since it's not being directly used in runtests.jl).
Consequently, with a few minor changes to your repo we are able to simply
] add https://github.com/brenhinkeller/Blueprint
] test Blueprint
or, since you preferred at the command line
user$ julia -e "using Pkg; Pkg.add(url=\"https://github.com/brenhinkeller/Blueprint\")
user$ julia -e "using Pkg; Pkg.test(\"Blueprint\")"
Testing Blueprint
Status `/private/var/folders/qk/2qyrdb854mvd2tn4crc802lw0000gn/T/jl_fSypP7/Project.toml`
[c1615a0c] Blueprint v0.1.0 `https://github.com/brenhinkeller/Blueprint#master`
[de31a74c] FunctionalCollections v0.5.0
[37e2e46d] LinearAlgebra `#stdlib/LinearAlgebra`
[8dfed614] Test `#stdlib/Test`
Status `/private/var/folders/qk/2qyrdb854mvd2tn4crc802lw0000gn/T/jl_fSypP7/Manifest.toml`
[c1615a0c] Blueprint v0.1.0 `https://github.com/brenhinkeller/Blueprint#master`
[187b0558] ConstructionBase v1.3.0
[de31a74c] FunctionalCollections v0.5.0
[1914dd2f] MacroTools v0.5.9
[ae029012] Requires v1.3.0
[efcf1570] Setfield v0.8.1
[56f22d72] Artifacts `#stdlib/Artifacts`
[2a0f44e3] Base64 `#stdlib/Base64`
[9fa8497b] Future `#stdlib/Future`
[b77e0a4c] InteractiveUtils `#stdlib/InteractiveUtils`
[8f399da3] Libdl `#stdlib/Libdl`
[37e2e46d] LinearAlgebra `#stdlib/LinearAlgebra`
[56ddb016] Logging `#stdlib/Logging`
[d6f4376e] Markdown `#stdlib/Markdown`
[9a3f8284] Random `#stdlib/Random`
[ea8e919c] SHA `#stdlib/SHA`
[9e88b42a] Serialization `#stdlib/Serialization`
[8dfed614] Test `#stdlib/Test`
[cf7118a7] UUIDs `#stdlib/UUIDs`
[e66e0078] CompilerSupportLibraries_jll `#stdlib/CompilerSupportLibraries_jll`
[4536629a] OpenBLAS_jll `#stdlib/OpenBLAS_jll`
[8e850b90] libblastrampoline_jll `#stdlib/libblastrampoline_jll`
Testing Running tests...
Test Summary: | Pass Total
Plane tests | 7 7
Test Summary: | Pass Total
Plane intersection | 2 2
Test Summary: | Pass Total
Plane intersection 2 | 4 4
Test Summary: | Pass Total
Plane shadowing | 3 3
Test Summary: | Pass Total
Polyhedron tests | 3 3
Test Summary: | Pass Total
Polyhedron tests 2 | 5 5
Test Summary: | Pass Total
Beam tests | 2 2
Test Summary: | Pass Total
Half-space test | 2 2
Test Summary: | Pass Total
Ordered pair test | 2 2
Test Summary: | Pass Total
Test plane/line intersection | 2 2
Test Summary: | Pass Total
Update line bounds test | 21 21
Testing Blueprint tests passed
FWIW, you should also be able to mix and match those command-line and REPL approaches (i.e., install in repl, test via command line or vice versa).
While I had not originally considered this case, one additional possibility discussed in the comments is where one wishes to test the local state of a package without, or without relying upon, a git remote; in this case #Rulle reports that activating the package directory, i.e,
julia -e "using Pkg; Pkg.activate(\".\"); Pkg.test(\"Blueprint\")"
or
julia --project=. -e "using Pkg; Pkg.test(\"Blueprint\")"
or equivalently in the REPL
] activate .
] test Blueprint
will work assuming the package directory is currently the local directory .
Possible answer to my own question:
To make it work, specify the main project root directory on the command line when calling the script using --project. In this case, we would call
julia --project=/home/jonas/prog/julia/blueprint test/test_blueprint.jl
However, there seems to be some hidden state that I don't understand, because after this command has been run once, it seems as if the --project option can be omitted. On the other hand, I have also tried to provide a nonsense project directory, e.g. /tmp:
julia --project=/tmp test/test_blueprint.jl
and sometimes it will still run the unit tests (!) and sometimes it won't. But when it fails to run the unit tests, it will succeed again as soon as I specify the correct path, that is /home/jonas/prog/julia/blueprint. I don't understand also how this interacts with whether I use using blueprint or include('../src/blueprint.jl') but it seems as if, when I use using, it works only iff the --project path is set correctly. But I am still not sure.

Package set-up not propagating to workers with Distributed

Info:
$ julia --version
julia version 1.6.0
$ lscpu
~/root/MyPackage$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core(TM) i9-9880H CPU # 2.30GHz
...
Say I want the following package structure, and want to use ReTest's parallel testing (my issue appears to be with how code-loading works in Distributed, so this isn't really a ReTest-specific issue).
| root/
| MyPackage/
| Project.toml
| Manifest.toml
| src/
| MyPackage.jl
| test/
| runtests.jl
| MyPackageTests.jl
I initialised this package in the following way:
$ cd root && julia
(...) pkg> generate MyPackage;
$ cd MyPackage && julia
(...) pkg> activate .
(...) pkg> instantiate
(...) pkg> add ReTest InlineTest Distributed;
...
Fill in MyPackage.jl, runtests.jl, and MyPackageTests.jl with some Julia code.
Not too important what that code is - although I am following the guide from here in ReTest.
Then to set up:
$ julia
(...) pkg> activate .
(...) pkg> instantiate
(MyPackage) pkg> st
Project MyPackage v0.1.0
Status `~/root/MyPackage/Project.toml`
[bd334432] InlineTest v0.2.0
[e0db7c4e] ReTest v0.3.2
[8ba89e20] Distributed
julia> LOAD_PATH
3-element Vector{String}:
"#" # Should be current active environment for MyPackage
"#v#.#" # Should be #v1.6 on my system
"#stdlib" # Should be absolute path of current Julia installation's stdlib
julia> # Should this code be in .jl files? Don't think that should matter.
julia> using Distributed
julia> addprocs(2)
julia> #everywhere include("test/MyPackageTests.jl")
ERROR: On worker 2:
LoadError: ArgumentError: Package MyPackage not found in current path:
- Run `import Pkg; Pkg.add("MyPackage")` to install the MyPackage package.
Stacktrace:
[1] require
# ./loading.jl:871
[2] include
# ./client.jl:444
[3] top-level scope
# none:1
[4] eval
# ./boot.jl:360
[5] #103
# /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:274
[6] run_work_thunk
# /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
[7] run_work_thunk
# /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72
[8] #96
# ./task.jl:406
in expression starting at /path/to/root/MyPackage/test/MyPackageTests.jl:1
...and 2 more exceptions.
Stacktrace:
[1] sync_end(c::Channel{Any})
# Base ./task.jl:364
[2] macro expansion
# ./task.jl:383 [inlined]
[3] remotecall_eval(m::Module, procs::Vector{Int64}, ex::Expr)
# Distributed /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:223
[4] top-level scope
# /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:207
I'd like to understand why this is. On GitHub I see this issue that was supposedly fixed. I can confirm when I run that example that I also get the exact same problem as above with MyPackage, if the using statement involves the package that the environment is for.
Before filing a bug or opening an issue there, I'd like to check here in case this is a process problem on my part. If not, there's evidently something wrong with Distributed/ReTest and I'll open tickets for those. Any help much appreciated.
Following on from what #carstenbauer said, the active Julia environment is not automatically propagated to worker processes by default. The way around this is to set the environment in the arguments to the call to addprocs like so:
julia> using Distributed
julia> addprocs(2, exeflags="--project=$(Base.active_project())")
julia> #everywhere include("test/MyPackageTests.jl")
julia> MyPackageTests.runtests() # runs to completion
I can confirm that this works with both the MyPackage example as well as the one shown in the JuliaLang issue. Thanks to those that contributed towards this answer.
Try the following code:
using Distributed
addprocs(4) # or whatever you need or use the -p parameter
using Pkg
pkg"activate ."
pkg"instantiate" # run this when needed
using MyPackage # first load package only an the master worker
#everywhere pkg"activate ."
#everywhere using MyPackage
Explanation:
each Julia processes is totally separated so it has its own package state, variables, memory etc.
Please note that you will usually prefer to load the package first on the master node as some packages might be performing some actions where loaded for the first time.

Haskell: Could not find module ‘Network.HTTP’

I am trying to write a simple script that takes as input a URL (or set of URLs) and as output it downloads the contents of that page to a file (in particular I am trying to download hundreds of JSON files, which ultimately I wish to diff against other JSON files).
In a file, download.hs, I have import "HTTP" Network.HTTP.
When I run: $ ghc -o download download.hs
I get the following error:
download.hs:24:1: error:
Could not find module ‘Network.HTTP’
Perhaps you meant Network.TLS (needs flag -package-key tls-1.5.2)
|
24 | import "HTTP" Network.HTTP
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
My GHC version is:
$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 8.6.5
I also get errors like:
download.hs:22:1: error:
Could not load module ‘Control.Concurrent.Async’
It is a member of the hidden package ‘async-2.2.2’.
You can run ‘:set -package async’ to expose it.
(Note: this unloads all the modules in the current scope.)
|
22 | import "async" Control.Concurrent.Async (mapConcurrently)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I think it's possible there have been breaking changes between the ghc versions, and the examples I am finding online to start with may be outdated.
Any pointers on doing started with Haskell, and particularly easy ways to download and diff JSON files in Haskell?
I have been following this example: Running parallel URL downloads in Haskell, this is where I got the code that is erroring now.

How to use packages in Julia Studio

I can't get the package system to work in Julia Studio. For example if I want to plot a simple graph I've tried double clicking the Winston package which seems to install from the Git repo, then:
using Winston
plot([1 2 3],[3 2 6])
But I get the error:
could not open file /Applications/JuliaStudio.app/Contents/Resources/juliaengine/Winston.jl
Which looks like Julia is looking in the wrong directory.
How should I set up Julia Studio to correctly work with the packages?
Response to Adam: thanks, unfortunately there seem to be a few issues. When I try to remove/add the Winston package I get a message like:
julia> Pkg.rm("Winston")
ERROR: Unknown dependency for ODBC: julia
in dependencies at pkg/metadata.jl:156
in ReqsStruct at pkg/resolve.jl:65
in resolve at pkg/resolve.jl:1162
in _resolve at pkg.jl:230
in anonymous at no file:163
in cd at file.jl:26
in cd_pkgdir at pkg.jl:34
in rm at pkg.jl:141
in rm at pkg.jl:165
I'll spend some more time on this and try and work out what's going on. I'll post an update for completeness if I get anywhere.
UPDATE
I'm now up to Julia Studio version 0.4.4 and after updating the packages the original example works. Unfortunately I can't determine the original problem but it looks like a complex dependency or version issue.
I think it's related to this issue:
https://github.com/forio/julia-studio/issues/83
The Winston installation requires external dependencies and prompts you for your input on how you want to install them. Julia Studio doesn't allow you to respond to this input.
Here's the workaround:
In your console, enter:
/Applications/JuliaStudio.app/julia/bin/julia-release-readline
Then
Pkg.rm("Winston")
Pkg.add("Winston")
Follow the prompts and when it's done close the process and return to Julia Studio.
Winston should now be working.
This is what I did:
Remove the $HOME/.julia folder (this will also erase all previously installed packages)
Run from a terminal/console
Last login: Sat Jul 27 02:58:06 on ttys001
~ ᐅ /Applications/JuliaStudio.app/julia/bin/julia-release-readline
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "help()" to list help topics
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.1.2+111981303.ra703.dirty
_/ |\__'_|_|_|\__'_| | Commit a703335d02 (2013-03-10 22:34:09)*
|__/ |
julia>
Install the package
julia> Pkg.add("Winston")
MESSAGE: Auto-initializing default package repository /Users/elyase/.julia.
...
It works!

Resources