Goal:
To have possibility to reload a whole module and use its exported functions and types in tasks without restarting them.
Problem:
I have a problem with applying a new function definitions while the task, which uses those, is running. The idea is to reload a module, not to include a file again, but further in the post I show the simplified problem version.
A simplified example:
Let me explain the problem using one file defining only one function f, as follows:
#sample_file.jl
f() = info("f version 01")
Run f every 10 seconds from a task:
julia> include("sample_file.jl")
julia> function call_f()
while (true)
f()
sleep(10)
end
end
julia> task = #async call_f()
Then in a REPL every 10 seconds we see:
julia> INFO: f version v01
INFO: f version 01
INFO: f version 01
INFO: f version 01
Now try to change definition in the sample_file.jl, e.g.
#sample_file.jl
f() = info("f version 02")
In the REPL:
julia> reload("sample_file")
julia> f()
INFO: f version 02
...but the infos from the task still give:
julia> INFO: f version 01
INFO: f version 01
INFO: f version 01
INFO: f version 01
INFO: f version 01
...
Question:
Do you have any idea to deal with that?
In your simplified example, this is https://github.com/JuliaLang/julia/issues/265. The function call_f gets compiled with the original definition of f, and does currently not get recompiled when f is changed.
In general, I think that you need to consider what you want to happen when f is changed. Do you want call_f to be recompiled? The simple solution, which doesn't need to recompile call_f, is to store the current function f in a non-const variable (f becomes const when you define your function). Then the jit compiler will know that the function can change and will generate an indirect call.
The core of your problem as you describe it is to share data in parallel computing, which is always a reason to seat down and ponder about the options available, restrictions, etc.
You can just call #everywhere which runs a command on all processes, but I'd say this is a bad idea because you will probably bump into another data sharing/synchronization issue.
My best bet, considering the short description, would be to use a "get an update on global state" approach:
# main process
# ...
#
if should_update
current_state.updateSomeParameter(newValue)
end
# state is always `spawn`ed
#spawn current_state
# continue doing main process stuff
# on remote process
while do_stuff
# do stuff
fetch(current_state)
updateSelfTo(current_state)
#continue doing my remote stuff
end
Related
I frequently need to debug a function in one of my R packages, and find it convenient to add test code and print statements throughout the code. Then the function is a method inside a package, running tests from the console will use the package-stored old version of the function and not the new test version. I often resort to something like cat *.r > /tmp/package.r and then source('/tmp/package.r') to override all functions, which allows the test function to be prioritized. But this doesn't work when I have .Fortran or similar calls within the package.
Is there an elegant way to debug with function overrides within the correct version of a local package?
Regardless of your IDE, you can reload your package under development with devtools:
devtools::load_all("path/to/your/package/directory")
This should load it to your R session (RStudio has buttons and keyboard shortcuts for this too)
This is an extension of my comment above. As said in the comment checkout this guide for more detail.
To inspect calls dynamically during calls, editing functions and adding code can be done using the trace(func, edit = TRUE) method. However it is not the recommended methodology for doing so in any programming language. It is however recommended to instead perform debugging. Luckily debugging in R is simpler than many other languages. Some of the most noticeable methods for debugging in R is obtained using
debug(func) or debugonce(func)
trace
browser
I'll ignore trace's main usage, and only pass over it briefly in conjunction with browser.
Take this small example code
x <- 3
f <- function(x){
x <- x ** 2
x
}
Now if we wanted to "go into" this function and inspect what happens we can use the debug method simply by doing
debug(f) # alt: debugonce(f)
f(x)
and the following shows in the console
debugging in: f(x)
debug at #1: {
x <- x^2
x
}
We see 2 things: The original call in line 1, the function debugged including a function-line-number (#1) with the body in the function (potentially truncated). In addition the command has changed to Browse[n] (where n is a number), indicating we are in the debugger.
At this point nothing has run so in the "environment" tab in Rstudio we can see x | x. Note that x is still a promise (its value is delayed until used or changed). If we execute x we get the value [1] 3 and we see the environment change to x | 3 (it has been forced, and is no longer a promise).
Inside the debugger we can use the following commands to "walk through" the function
n to move to the "next line"
s to move to the "next line", or if the current call is a function, "step" into the function (debug this function)
f to move forward until the next break (with the added utility that if you are in a loop you stop at "loop end".
c to move until next break point or end of function (without breaking at end of loops).
Q to exit immediately
It you click n for example you will see
debug at #2: x <- x^2
printed in the console. This indicates the line that is executed next. Notice the value of x in the environment and run n again, notice the value changed from x | 3 to x | 9 and
debug at #3: x
is printed. This being the last line pressing n again will exit the function and print
exiting from: f(x)
[1] 9
Once you're done debugging you can run undebug(f) to remove the breakpoint and stop the debugger from activating.
This is a simple function, easy to debug, but the idea for more complex functions are the same. If you are in a long loop you can use f to skip to the end of the loop, similar to pressing n a bunch of times. Note that if you hit an error at any point it will exit automatically when the error occurs and you'll have to walk back to the point again or alternatively use browser.
In the case where you have a function like
f2 <- function(x){
x <- x + 2
f(x)
}
you can further step into the nested function call f(x) by using the s command while the line is printing
debug at #3: f(x)
or by using debug(f2) and debug(f) in conjunction. Both will give the same result (try it out!).
Now in many cases you might hit a bug or debug many lines (potentially thousands). In this case, you might have some "idea" where you want to start, and this might not be the start of the function. In this case you can use browser(). This basically sets a breakpoint. Whenever browser() is hit, it will stop and start the debugger (similar to debug(f) and calling f(x) but at a specific point). Try for example
f3 <- function(x){
x1 <- f(x)
browser()
x2 <- f2(x)
c(x1, x2)
}
f3(x)
and you'll notice see
Called from: f3(x)
printed (if you have run undebug(f2) and undebug(f) first).
Lets say it is not your function but a function within a namespace, well then we can even add the breakpoint ourself at run-time. Try for example calling
trace(f3, edit = TRUE)
and you will see an editing window pop up. Simply add browser() at the desired spot and click save. This edits the function within the namespace. It will be reverted once R is closed or alternatively you can remove it with another call to trace(f3, edit = TRUE).
(Question refers to Julia version v1.5)
I'm trying to understand how the #deprecate macro works in Julia. The documentation is unfortunately not clear to me:
#deprecate old new [ex=true]
Deprecate method old and specify the replacement call new. Prevent
#deprecate from exporting old by setting ex to false. #deprecate
defines a new method with the same signature as old.
Warning:
As of Julia 1.5, functions defined by #deprecate do not print warning when julia is run without the --depwarn=yes flag set, as the default value of --depwarn option is no. The warnings are printed from tests run by Pkg.test().
Examples
julia> #deprecate old(x) new(x)
old (generic function with 1 method)
julia> #deprecate old(x) new(x)
false old (generic function with 1 method)
So what do I do?
function old(x::Int)
print("Old behavior")
end
function new(x::Int)
print("New behavior")
end
# Adding true/false argument doesn't change my observations.
#deprecate old(x) new(x) # false
old(3)
# Prints "Old behaviour". No warning.
# Also: The deprecation is not mentioned in the help (julia>? old)
The aim of this #deprecate macro seems to be replacing functions? I find that counterintuitive. How can mark a function as deprecated (i.e. users should receive a warning and a hint what to use as a replacement, also it should be in the documentation)?
edit: I noticed my error. The signatures (in my case the ::Int) have to be identical for this to work. However, how do I get a warning?
Imagine you have this method as part of the public API of your library in version 1:
# v1.0.0
mult3(x::Int) = 3x
In version 2, you'd like to stop supporting mult3 (which is a breaking change). But the same feature will still be available using a more general method:
# v2.0.0
mult(x, y) = x * y
Users of version 1 are used to using mult3, which means that their code will break when they will update to v2. Therefore, you might want to release an intermediate version in the v1.x family, where mult3 exists but is deprecated and implemented in terms of mult:
# v1.1 - transition
# This is the new API for v2
mult(x, y) = x*y
# The old API is still supported, but deprecated and implemented using the old one
#deprecate mult3(x::Int) mult(3, x)
# The above is more or less equivalent to defining
# function mult3(x::Int)
# # print an error message is `--depwarn` has been set
# return mult(3, x)
# end
The v1 API is not broken in late v1.x versions, but users calling deprecated methods will see the following kind of messages to help them transition to the newer v2 API:
julia> mult3(14)
┌ Warning: `mult3(x::Int)` is deprecated, use `mult(3, x)` instead.
│ caller = top-level scope at REPL[3]:1
└ # Core REPL[3]:1
42
(but starting with Julia 1.5, the warning will only be shown if --depwarn=yes has been provided in Julia's command line or if it appears in a test suite run by Pkg.test())
Alternatively, and as mentioned in comments, you may want to leave the old implementation around, simply warning users when they call it. To this end, you can use Base.depwarn directly:
# v1.1 - transition
# This is the new API for v2
mult(x, y) = x*y
# The old API is still supported, but deprecated
# It is implemented by itself:
function mult3(x)
Base.depwarn("`mult3(x)` is deprecated, use `mult(3,x)` instead.", :mult3)
return 3x
end
When --depwarn=yes has been provided in Julia's command line, this produces the same kind of warning as #deprecate:
julia> mult3(14)
┌ Warning: `mult3(x)` is deprecated, use `mult(3,x)` instead.
│ caller = top-level scope at REPL[4]:1
└ # Core REPL[4]:1
42
Starting with Julia 1.6, depwarn will accept a keyword argument to force warning emission even when users haven't asked for them with --depwarn=yes:
julia> Base.depwarn("Foo is deprecated", :foo, force=true)
┌ Warning: Foo is deprecated
│ caller = ip:0x0
└ # Core :-1
Consider the following code:
File C.jl
module C
export printLength
printLength = function(arr)
println(lentgh(arr))
end
end #module
File Main.jl
using C
main = function()
arr = Array(Int64, 4)
printLength(arr)
end
main()
Let's try to execute it.
$ julia Main.jl
ERROR: lentgh not defined
in include at /usr/bin/../lib64/julia/sys.so
in process_options at /usr/bin/../lib64/julia/sys.so
in _start at /usr/bin/../lib64/julia/sys.so
while loading /home/grzes/julia_sucks/Main.jl, in expression starting on line 8
Obviously, it doesn't compile, because lentgh is misspelled. The problem is the message I received. expression starting on line 8 is simply main(). Julia hopelessly fails to point the invalid code fragment -- it just points to the invocation of main, but the erroneous line is not even in that file! Now imagine a real project where an error hides really deep in the call stack. Julia still wouldn't tell anything more than that the problem started on the entry point of the execution. It is impossible to work like that...
Is there a way to force Julia to give a little more precise messages?
In this case it's almost certainly a consequence of inlining: your printLength function is so short, it's almost certainly inlined into the call site, which is why you get the line number 8.
Eventually, it is expected that inlining won't cause problems for backtraces. At the moment, your best bet---if you're running julia's pre-release 0.4 version---is to start julia as julia --inline=no and run your tests again.
This is just a convenience but I think useful. Note that IPython allows a pure quit as does Matlab. Thus it would be reasonble in Julia to allow aliasing.
Thanks for any ideas as to how to do this.
Quitting in Julia
If you are using Julia from the command line then ctrl-d works. But if your intention is to quit by typing a command this is not possible exactly the way you want it because typing quit in the REPL already has a meaning which is return the value associated with quit, which is the function quit.
julia> quit
quit (generic function with 1 method)
julia> typeof(quit)
Function
Also Python
But that's not rare, for example Python has similar behavior.
>>> quit
Use quit() or Ctrl-D (i.e. EOF) to exit
Using a macro
Using \q might be nice in the Julia REPL like in postgres REPL, but unfortunately \ also already has a meaning. However, if you were seeking a simple way to do this, how about a macro
julia> macro q() quit() end
julia> #q
Causes Julia to Quit
If you place the macro definition in a .juliarc.jl file, it will be available every time you run the interpreter.
As waTeim notes, when you type quit into the REPL, it simply shows the function itself… and there's no way to change this behavior. You cannot execute a function without calling it, and there are a limited number of ways to call functions in Julia's syntax.
What you can do, however, is change how the Functions are displayed. This is extremely hacky and is not guaranteed to work, but if you want this behavior badly enough, here's what you can do: hack this behavior into the display method.
julia> function Base.writemime(io::IO, ::MIME"text/plain", f::Function)
f == quit && quit()
if isgeneric(f)
n = length(f.env)
m = n==1 ? "method" : "methods"
print(io, "$(f.env.name) (generic function with $n $m)")
else
show(io, f)
end
end
Warning: Method definition writemime(IO,MIME{symbol("text/plain")},Function) in module Base at replutil.jl:5 overwritten in module Main at none:2.
writemime (generic function with 34 methods)
julia> print # other functions still display normally
print (generic function with 22 methods)
julia> quit # but when quit is displayed, it actually quits!
$
Unfortunately there's no type more specific than ::Function, so you must completely overwrite the writemime(::IO,::MIME"text/plain",::Function) definition, copying its implementation.
Also note that this is pretty unexpected and somewhat dangerous. Some library may actually end up trying to display the function quit… causing you to lose your work from that session.
Related to Quitting in Julia
I was searching for something simple. This question hasn't been updated since 2017, as I try to learn Julia now, and spend some time googling for something simple and similar to python. Here, what I found:
You can use:
exit()
Note
I use julia 1.53
I am new to Julia Lang. I am coming from the background of Matlab.
In Matlab, when pressing whos command I will get all variables in the current scope; and also, I can store them in another variable like x=whos; Is there such commands exists in Julia?
Example code in Matlab:
>> a=3;
>> b=4;
>> whos
Variables in the current scope:
Attr Name Size Bytes Class
==== ==== ==== ===== =====
a 1x1 8 double
b 1x1 8 double
prefix 1x16 16 char
Total is 18 elements using 32 bytes.
An Update:
whos()
... is not working either in iJulia or at the command prompt in Julia-1.0.0.
It is working in Julia-0.6.4, though.
On the other hand,
varinfo()
....prints information about the exported global variables in a module. For Example,
julia-1.0> varinfo()
name size summary
–––––––––––––––– ––––––––––– –––––––––––––––––––––––––––––––
Base Module
Core Module
InteractiveUtils 154.271 KiB Module
Main Module
PyPlot 781.872 KiB Module
ans 50.323 KiB Plots.Plot{Plots.PyPlotBackend}
myrepl 0 bytes typeof(myrepl)
x 88 bytes 1×6 Array{Int64,2}
y 0 bytes typeof(y)
Hope, this is found useful.
You can use Julia's whos functions just like that Matlab command.
julia> whos()
Base Module
Core Module
Main Module
ans Nothing
julia> x = 5
5
julia> whos()
Base Module
Core Module
Main Module
ans Int64
x Int64
Any modules (packages/libraries) you import into your local scope (using using) will also show up in the list (as Modules, like Base, Core, and Main above).
Additionally, you can ask about names exported by Modules. Base is the module containing the standard library.
julia> whos(Base)
! Function
!= Function
!== Function
$ Function
% Function
& Function
* Function
+ Function
.... (lots and lots more)
Considering that that result scrolls way off my screen, you can understand why you'd want to filter the results. For that you can use Regexes. (For more info on Julia's regexes, see this manual section)
julia> whos(r"M")
Main Module
julia> whos(Base, r"Match"i)
DimensionMismatch DataType
RegexMatch DataType
each_match Function
eachmatch Function
ismatch Function
match Function
matchall Function
I wasn't aware of the whos function before you asked, so thanks for helping me learn something new too. :)
Julia issue #3393 on github is about adding memory sizes to the whos output. It also references making whos return a value rather than just printing the information out.
Not sure if there is something better, but
names(Main)[4:end]
seems to work. The [4:end] part is because it includes :Main, :Core and :Base which I think you would not want. I hope they will always be at the beginning.
whos() is not available in newer versions of Julia (1.0 onward). Use varinfo() instead. For example, varinfo(Core,r".*field.*")
As of version 1.1 there is also the #locals macro
The experimental macro Base.#locals returns a dictionary of current local variable names and values
Release notes