I'm trying to create a package with the following layout:
MyPkg
├── MyPkg.jl
├── Foo
│ ├── Foo.jl
│ └── another_file.jl
└── Bar
├── Bar.jl
└── yet_another_file.jl
My main package module looks something like this:
# MyPkg.jl
module Pkg
include("./Foo/Foo.jl")
using .Foo: FooStuffA, FooStuffB
export FooStuffA, FooStuffB
include("./Bar/Bar.jl")
using .Bar: BarStruct, BarStuffC, BarStuffD
export BarStruct, BarStuffC, BarStuffD
end
The problem arises when Foo needs a type (specifically a struct) defined in Bar in some function arguments. I'm not sure how to import this type. I've tried seemingly all combinations of include("../Bar/Bar.jl"), using Bar/.Bar/..Bar, inside the Foo submodule, outside the submodule, etc.
# Foo.jl
module Foo
# what am I missing here?
function print_bar_struct(bar::BarStruct)
#show bar
end
end
Any advice?
This should work
# MyPkg.jl
module MyPkg
include("./Bar/Bar.jl")
using .Bar: BarStruct, BarStuffC, BarStuffD
export BarStruct, BarStuffC, BarStuffD
include("./Foo/Foo.jl")
using .Foo: FooStuffA, FooStuffB
export FooStuffA, FooStuffB
end
# Foo.jl
module Foo
using ..MyPkg: BarStruct
function print_bar_struct(bar::BarStruct)
#show bar
end
end
Explanation: Remember that include statements are essentially copying+pasting the code from the source file into the module at the given line. So by the time the compiler is looking at the references for all of the symbols (reading from the top of the file to the bottom), at the point where include("./Foo/Foo.jl") occurs, it needs to know that BarStruct exists and is accessible in the current module (i.e., MyPkg), which it is in this rearranged layout.
So by looking just at this first half of MyPkg
# MyPkg.jl
module MyPkg
include("./Bar/Bar.jl")
using .Bar: BarStruct, BarStuffC, BarStuffD
export BarStruct, BarStuffC, BarStuffD
by the time the compiler reaches the last line here, BarStruct, BarStuffC, BarStuffD are the symbols brought into the MyPkg namespace (https://docs.julialang.org/en/v1/manual/modules/#Summary-of-module-usage-1).
When we reach the include("./Foo/Foo.jl") line (aka copying + pasting this source file into the current module at this point), we need to reference BarStruct in the parent namespace of this module, i.e., ..BarStruct
Have you tried referencing the struct with its module.struct full name, as in Bar.BarStruct?
With structures and enums, the export function seems to not work as well as with function names, but using the Module.Struct type syntax often works.
Related
I have 2 modules and a .jl file in my folder.
module TreasureHuntEnv
export MapInfo
struct MapInfo
end
end
module DQNModule
include("./TreasureHuntEnv.jl")
using .TreasureHuntEnv
function runIters(mapInfo::MapInfo)
end
end
include("./TreasureHuntEnv.jl")
include("./DQN.jl")
using .TreasureHuntEnv
using .DQNModule
function comparisonExperiment(mapInfo::MapInfo)
end
mapInfo = MapInfo()
comparisonExperiment(mapInfo)
But when I run the 3rd file, it returns me the following error:
ERROR: MethodError: no method matching runIters(::MapInfo, ::DQN{:dqn}; gameNum=1000, displayInterval=2000, earlyStop=5000, recordQvalues=true)
Closest candidates are:
runIters(::Main.DQNModule.TreasureHuntEnv.MapInfo, ::DQN; gameNum, stepBeforeLearing, learnInterval, displayInterval, earlyStop, recordQvalues) at e:\Master Thesis\lu_jizhou\Learning\DQN.jl:253
Stacktrace:
[1] comparisonExperiment(mapInfo::MapInfo)
# Main e:\Master Thesis\lu_jizhou\Learning\Experiments.jl:14
[2] top-level scope
# e:\Master Thesis\lu_jizhou\Learning\Experiments.jl:27
Why is that and how should I solve the problem?
I'm assuming comparisonExperiment is meant to have a call to runIters, since as is the code you posted works.
The issue is that include in the second module is essentially copying all the code for the first module, which then produces a submodule DQNModule.TreasureHuntEnv, which explains your error message.
One way to avoid the problem is to export DQNModule and this submodule in your script:
include("DQN.jl")
using .DQNModule
using .DQNModule.TreasureHuntEnv
function comparisonExperiment(mapInfo::MapInfo)
DQNModule.runIters(mapInfo)
end
mapInfo = MapInfo()
comparisonExperiment(mapInfo)
Personally I would recommend sticking to include statements rather than formal modules at first, then creating packages for your modules once you are ready
I am working on a julia code where I have several files and call functions from these files to a main function called run.jl. Every time I make changes to any one of these files I need to restart the julia REPL which is a bit annoying. Anyway to work around this.
For example
# --------------------------------- run.jl
include("agent_types.jl")
include("resc_funcs.jl")
using .AgentTypes: Cas, Resc
using .RescFuncs: update_rescuers_at_pma!, travel_to_loc
# ... some code
for i = 1:500
update_rescuers_at_pma!(model)
travel_to_loc(model)
end
# ------------------------------ resc_funcs.jl
function travel_to_loc(model)
println(get_resc_by_prop(model, :on_way_to_iz))
for resc_id in get_resc_by_prop(model,:on_way_to_iz)
push!(model[resc_id].dist_traject, model[resc_id].dist_to_agent) #This is just for checking purposes
model[resc_id].dist_to_agent -= model.dist_per_step
push!(model[resc_id].loc_traject, "on_way_to_iz")
end
#If we add new print statement here and execute run.jl it wont print
println(model[resc_id].loc_traject) #new amendment to code
end
But now when I go and update travel_to_loc function for example. I need to restart the julia repl before those changes are reflected. I am wondering if there is a way after you save the file (resc_funcs.jl in this case) those amendments are reflected when you execute run.jl.
The simplest workflow could be the following:
Select some folder to be your current working folder in Julia (eg. by using cd() command from Julia)
Create sample file MyMod.jl
module MyMod
export f1
function f1(x)
x+1
end
end
Add the current folder to LOAD_PATH and load Revise.jl
push!(LOAD_PATH, ".")
using Revise
Load the module and test it:
julia> using MyMod
[ Info: Precompiling MyMod [top-level]
julia> f1(3)
4
Try to edit the file eg. change x+1 to x+3
Run again
julia> f1(3)
6
Notes:
you will still need to restart REPL when your data structures change (you modify a definition of a struct object)
you could generate a full module package using Pkg.generate but I wanted to make things simplified.
It took me hours to figure out how to patch the below code. The path to it was very much unexpected.
Depending on how I run the tests, and which dir I am in, I find the dotted path to the module to patch changes. This is really bad for unittesting. Which makes me think I am doing it wrong.
The file structure related to the code is:
loaders.py <-- Has a load_palette() func required to be patched
typers.py <-- Has `from . loaders import load_palette`, and calls load_palette()
render.py <-- Has a func that calls the typers func
tests/test_render.py <-- Tests for render which calls a func in render, which calls a func in typers, which calls load_palette()
In the code below __package__.replace('.tests', '.typers.load_palette') takes the current path to the current package which could be:
bar.tests or
foo.bar.tests
or something else
and builds the dotted path relatively so that is is correct. This seems very hackish. How is one supposed to safe guard against these kind of issues?
Ideally the dotted path would be ..typers.load_palette but it did not accept the relative dotted path.
Heres the actual code:
# file: test_render.py
# Depending where one runs the test from, the path is different, so generate it dynamically
#mock.patch(__package__.replace('.tests', '.typers.load_palette'), return_value=mocks.palette)
class render_rule_Tests(SimpleTestCase):
def test_render_preset_rule(self, _): # _ = mocked_load_palette
...
files layout as following:
$ tree issue
issue
├── __init__.py
├── loaders.py
├── renders.py
├── tests
│ ├── __init__.py
│ └── test_render.py
├── run_tests.sh
└── typers.py
1 directory, 7 files
the root package is issue, you should always import modules from issue, and patch issue.xxx.yyy.
then run pytest (or some other unittest tools) from the same path as tests resident.
for example, run_tests.sh is a shell script to run all test cases under tests.
and test_render may be like this
# file: test_render.py
# Depending where one runs the test from, the path is different, so generate it dynamically
#mock.patch('issue.typers.load_palette', return_value=mocks.palette)
class render_rule_Tests(SimpleTestCase):
def test_render_preset_rule(self, _): # _ = mocked_load_palette
...
You can add the path of the "tests" directory using sys.path.insert.
In the top of "tests/test_render.py" add:
import sys
sys.path.insert(0, "<path/to/the/folder/tests/>")
# Depending where one runs the test from, the path is different, so generate it dynamically
#mock.patch(__package__.replace('.tests', '.typers.load_palette'), return_value=mocks.palette)
class render_rule_Tests(SimpleTestCase):
def test_render_preset_rule(self, _): # _ = mocked_load_palette
...
This will add the path in system paths where python interpreter. From there, the python interpreter can locate the relative imports.
Note: The safest option would be to add the absolute path to the tests folder. However, if it's not possible, add the shortest relative path possible.
I would like to know how in the Julia language, I can determine if a file.jl is run as script, such as in the call:
bash$ julia file.jl
It must only in this case start a function main, for example. Thus I could use include('file.jl'), without actually executing the function.
To be specific, I am looking for something similar answered already in a python question:
def main():
# does something
if __name__ == '__main__':
main()
Edit:
To be more specific, the method Base.isinteractive (see here) is not solving the problem, when using include('file.jl') from within a non-interactive (e.g. script) environment.
The global constant PROGRAM_FILE contains the script name passed to Julia from the command line (it does not change when include is called).
On the other hand #__FILE__ macro gives you a name of the file where it is present.
For instance if you have a files:
a.jl
println(PROGRAM_FILE)
println(#__FILE__)
include("b.jl")
b.jl
println(PROGRAM_FILE)
println(#__FILE__)
You have the following behavior:
$ julia a.jl
a.jl
D:\a.jl
a.jl
D:\b.jl
$ julia b.jl
b.jl
D:\b.jl
In summary:
PROGRAM_FILE tells you what is the file name that Julia was started with;
#__FILE__ tells you in what file actually the macro was called.
tl;dr version:
if !isdefined(:__init__) || Base.function_module(__init__) != MyModule
main()
end
Explanation:
There seems to be some confusion. Python and Julia work very differently in terms of their "modules" (even though the two use the same term, in principle they are different).
In python, a source file is either a module or a script, depending on how you chose to "load" / "run" it: the boilerplate exists to detect the environment in which the source code was run, by querying the __name__ of the embedding module at the time of execution. E.g. if you have a file called mymodule.py, it you import it normally, then within the module definition the variable __name__ automatically gets set to the value mymodule; but if you ran it as a standalone script (effectively "dumping" the code into the "main" module), the __name__ variable is that of the global scope, namely __main__. This difference gives you the ability to detect how a python file was ran, so you could act slightly differently in each case, and this is exactly what the boilerplate does.
In julia, however, a module is defined explicitly as code. Running a file that contains a module declaration will load that module regardless of whether you did using or include; however in the former case, the module will not be reloaded if it's already on the workspace, whereas in the latter case it's as if you "redefined" it.
Modules can have initialisation code via the special __init__() function, whose job is to only run the first time a module is loaded (e.g. when imported via a using statement). So one thing you could do is have a standalone script, which you could either include directly to run as a standalone script, or include it within the scope of a module definition, and have it detect the presence of module-specific variables such that it behaves differently in each case. But it would still have to be a standalone file, separate from the main module definition.
If you want the module to do stuff, that the standalone script shouldn't, this is easy: you just have something like this:
module MyModule
__init__() = # do module specific initialisation stuff here
include("MyModule_Implementation.jl")
end
If you want the reverse situation, you need a way to detect whether you're running inside the module or not. You could do this, e.g. by detecting the presence of a suitable __init__() function, belonging to that particular module. For example:
### in file "MyModule.jl"
module MyModule
export fun1, fun2;
__init__() = print("Initialising module ...");
include("MyModuleImplementation.jl");
end
### in file "MyModuleImplementation.jl"
fun1(a,b) = a + b;
fun2(a,b) = a * b;
main() = print("Demo of fun1 and fun2. \n" *
" fun1(1,2) = $(fun1(1,2)) \n" *
" fun2(1,2) = $(fun2(1,2)) \n");
if !isdefined(:__init__) || Base.function_module(__init__) != MyModule
main()
end
If MyModule is loaded as a module, the main function in MyModuleImplementation.jl will not run.
If you run MyModuleImplementation.jl as a standalone script, the main function will run.
So this is a way to achieve something close to the effect you want; but it's very different to saying running a module-defining file as either a module or a standalone script; I don't think you can simply "strip" the module instruction from the code and run the module's "contents" in such a manner in julia.
The answer is available at the official Julia docs FAQ. I am copy/pasting it here because this question comes up as the first hit on some search engines. It would be nice if people found the answer on the first-hit site.
How do I check if the current file is being run as the main script?
When a file is run as the main script using julia file.jl one might want to activate extra functionality like command line argument handling. A way to determine that a file is run in this fashion is to check if abspath(PROGRAM_FILE) == #__FILE__ is true.
I wrote
let fact x =
let result = ref 1 in
for i = 1 to x do
result := !result * i;
Printf.printf "%d %d %d\n" x i !result;
done;
!result;;
in a file named "Moduletest.ml", and
val fact : int -> int
in a file named "Moduletest.mli".
But, why don't they work?
When I tried to use in ocaml,
Moduletest.fact 3
it told me:
Error: Reference to undefined global `Moduletest'
What's happening?
Thanks.
OCaml toplevel is linked only with a standard library. There're several options on how to make other code visible to it:
copy-pasting
evaluating from the editor
loading files #use directive
making custom toplevel
loading with ocamlfind
Copy-pasting
This self-describing, you just copy code from some source and paste it into toplevel. Don't forget that toplevel won't evaluate your code until you add ;;
Evaluating from the editor
Where the editor is of course Emacs... Well, indeed it can be any other capable editor, like vim for example. This method is an elaboration of the previous, where the editor is actually responsible for copying and pasting the code for you. In Emacs you can evaluate the whole file with C-c C-b command, or you can narrow it to a selected area with C-c C-r, and the most granular is to use C-c C-e, i.e., evaluate an expression. Although it is slightly buggy.
Loading with #use directive.
This directive accepts a filename, and it will essentially copy and paste the code from the file. Notice, that it won't create a file-module for you/ For example, if you have file test.ml with this contents:
(* file test.ml *)
let sum x y = x + y
then loading it with the #use directive, will actually bring to your scope, sum value:
# #use "test.ml";;
# let z = sum 2 2
You mustn't to qualify sum with Test., because no Test module is actually created. #use directive merely copies the contents of the file to the toplevel. Nothing more.
Making custom toplevels
You can create your own toplevel with your code compiled in. It is an advanced theme, so I will skip it.
Loading libraries with ocamlfind
ocamlfind is a tool that allows you to find and load libraries, installed on your system, into your toplevel. By default, toplevel is not linked with any code except standard library. Even, not all parts of the library are actually linked, e.g., Unix module is not available, and needed to be loaded explicitly. There're primitive directives that can load any library, like #load and #include, but they are not for a casual user, especially if you have excellent ocamlfind at your disposal. Before using it, you need to load it, since it is also not available by default. The following command, will load ocamlfind and add few new directives:
# #use "topfind";;
In a process of loading it will show you a little hint on how to use it. The most interesting directive, that is added is #require. It accepts a library name, and loads (i.e., links) its code into toplevel:
# #require "unix";;
This will load a unix library. If you're not sure, about the name of the library you can always view all libraries with a #list command. The #require directive is clever and it will automatically load all dependencies of the library.
If you do not want to type all this directives every time you start OCaml top-level, then you cam create .ocamlinit file in your home directory, and put them there. This file will be loaded automatically on a top-level startup.
I have tested your code and it looks fine. You should "load" it from the OCaml toplevel (launched from the same directory as your .ml and .mli files) in the following way:
# #use "Moduletest.ml";;
val fact : int -> int = <fun>
# fact 4;;
4 1 1
4 2 2
4 3 6
4 4 24
- : int = 24