Consider the following code:
File C.jl
module C
export printLength
printLength = function(arr)
println(lentgh(arr))
end
end #module
File Main.jl
using C
main = function()
arr = Array(Int64, 4)
printLength(arr)
end
main()
Let's try to execute it.
$ julia Main.jl
ERROR: lentgh not defined
in include at /usr/bin/../lib64/julia/sys.so
in process_options at /usr/bin/../lib64/julia/sys.so
in _start at /usr/bin/../lib64/julia/sys.so
while loading /home/grzes/julia_sucks/Main.jl, in expression starting on line 8
Obviously, it doesn't compile, because lentgh is misspelled. The problem is the message I received. expression starting on line 8 is simply main(). Julia hopelessly fails to point the invalid code fragment -- it just points to the invocation of main, but the erroneous line is not even in that file! Now imagine a real project where an error hides really deep in the call stack. Julia still wouldn't tell anything more than that the problem started on the entry point of the execution. It is impossible to work like that...
Is there a way to force Julia to give a little more precise messages?
In this case it's almost certainly a consequence of inlining: your printLength function is so short, it's almost certainly inlined into the call site, which is why you get the line number 8.
Eventually, it is expected that inlining won't cause problems for backtraces. At the moment, your best bet---if you're running julia's pre-release 0.4 version---is to start julia as julia --inline=no and run your tests again.
Related
On pp. 260-263 of Programming in Lua (4th ed.), the author discusses how to implement "sandboxing" (i.e. the running of untrusted code) in Lua.
When it comes to imposing limiting the functions that untrusted code can run, he recommends a "whitelist approach":
We should never think in terms of what functions to remove, but what functions to add.
This question is about tools and techniques for putting this suggestion into practice. (I expect there will be confusion on this point I want to emphasize it upfront.)
The author gives the following code as an illustration of a sandbox program based on a whitelist of allowed functions. (I have added or moved around some comments, and removed some blank lines, but I've copied the executable content verbatim from the book).
-- From p. 263 of *Programming in Lua* (4th ed.)
-- Listing 25.6. Using hooks to bar calls to unauthorized functions
local debug = require "debug"
local steplimit = 1000 -- maximum "steps" that can be performed
local count = 0 -- counter for steps
local validfunc = { -- set of authorized functions
[string.upper] = true,
[string.lower] = true,
... -- other authorized functions
}
local function hook (event)
if event == "call" then
local info = debug.getinfo(2, "fn")
if not validfunc[info.func] then
error("calling bad function: " .. (info.name or "?"))
end
end
count = count + 1
if count > steplimit then
error("script uses too much CPU")
end
end
local f = assert(loadfile(arg[1], "t", {})) -- load chunk
debug.sethook(hook, "", 100) -- set hook
f() -- run chunk
Right off the bat I am puzzled by this code, since the hook tests for event type (if event == "call" then...), and yet, when the hook is set, only count events are requested (debug.sethook(hook, "", 100)). Therefore, the whole song-and-dance with validfunc is for naught.
Maybe it is a typo. So I tried experimenting with this code, but I found it very difficult to put the whitelist technique in practice. The example below is a very simplified illustration of the type of problems I ran into.
First, here is a slightly modified version of the author's code.
#!/usr/bin/env lua5.3
-- Filename: sandbox
-- ----------------------------------------------------------------------------
local debug = require "debug"
local steplimit = 1000 -- maximum "steps" that can be performed
local count = 0 -- counter for steps
local validfunc = { -- set of authorized functions
[string.upper] = true,
[string.lower] = true,
[io.stdout.write] = true,
-- ... -- other authorized functions
}
local function hook (event)
if event == "call" then
local info = debug.getinfo(2, "fnS")
if not validfunc[info.func] then
error(string.format("calling bad function (%s:%d): %s",
info.short_src, info.linedefined, (info.name or "?")))
end
end
count = count + 1
if count > steplimit then
error("script uses too much CPU")
end
end
local f = assert(loadfile(arg[1], "t", {})) -- load chunk
validfunc[f] = true
debug.sethook(hook, "c", 100) -- set hook
f() -- run chunk
The most significant differences in the second snippet relative to the first one are:
the call to debug.sethook has "c" as mask;
the f function for the loaded chunk gets added to the validfunc whitelist;
io.stdout.write is added to the validfunc whitelist;
When I use this sandbox program to run the one-line script shown below:
# Filename: helloworld.lua
io.stdout:write("Hello, World!\n")
...I get the following error:
% ./sandbox helloworld.lua
lua5.3: ./sandbox:20: calling bad function ([C]:-1): __index
stack traceback:
[C]: in function 'error'
./sandbox:20: in function <./sandbox:16>
[C]: in metamethod '__index'
helloworld.lua:3: in local 'f'
./sandbox:34: in main chunk
[C]: in ?
I tried to fix this by adding the following to validfunc:
[getmetatable(io.stdout).__index] = true,
...but I still get pretty much the same error. I could go on guessing and trying more things to add, but this is what I would like to avoid.
I have two related questions:
What can I add to validfunc so that sandbox will run helloworld (as is) to completion?
More importantly, what is a systematic way to find determine what to add to a whitelist table?
Part (2) is the heart of this post. I am looking for tools/techniques that remove the guesswork from the problem of populating a whitelist table.
(I know that I can get helloworld to work if I replace io.stdout:write with print, register print in sandbox's validfunc, and pass {print = print} as the last argument to loadfile, but doing this does not answer the general question of how to systematically determine what needs to be added to the whitelist to allow some specific code to work in the sandbox.)
EDIT: Ask #DarkWiiPlayer pointed out, the calling bad function error is being triggered by the calling of an unregistered function (__index?), which happened as part of the response to an earlier attempt to index a nil value error. So, this post's questions are all about systematically determining what to add to validfunc to allow Lua to emit the attempt to index a nil value error normally.
I should add that the question of which function's call triggered the hook's execution responsible for the calling bad function error message is at the moment completely unclear. This error message blames the error on __index, but I suspect that this may be a red herring, possibly due to a bug in Lua.
Why suspect a bug in Lua? If I change the error call in sandbox slightly to
error(string.format("calling bad function (%s:%d): %s (%s)",
info.short_src, info.linedefined, (info.name or "?"),
info.func))
...then the error message looks like this:
lua5.3: ./sandbox:20: calling bad function ([C]:-1): __index (function: 0x55b391b79ef0)
stack traceback:
[C]: in function 'error'
./sandbox:20: in function <./sandbox:16>
[C]: in metamethod '__index'
helloworld.lua:3: in local 'f'
./sandbox:34: in main chunk
[C]: in ?
Nothing surprising there, but if now I change helloworld.lua to
# Filename: helloworld.lua
nonexistent()
io.stdout:write("Hello, World!\n")
...and run it under sandbox, the error message becomes
lua5.3: ./sandbox:20: calling bad function ([C]:-1): nonexistent (function: 0x556a161cdef0)
stack traceback:
[C]: in function 'error'
./sandbox:20: in function <./sandbox:16>
[C]: in global 'nonexistent'
helloworld.lua:3: in local 'f'
./sandbox:34: in main chunk
[C]: in ?
From this error message, one may conclude that nonexistent is a real function; after all, it's sitting right there at 0x556a161cdef0! But we know that nonexistent lives up to its name: it doesn't exist!
The whiff of a bug is definitely in the air. It could be that the function that is triggering the hook should really be excluded from those that trigger such "c"-masked hooks? Be that as it may, it appears that, in this particular situation, the call to debug.info is returning inconsistent information (since the name of the function [e.g. nonexistent] clearly does not correspond at all to the actual function object [e.g. function: 0x556a161cdef0] that is supposedly triggering the hook).
(Final answer at the bottom, feel free to skip until the <hr> line)
I'll explain my debugging step by step.
This is a really weird phenomenon. After some testing, I've managed to narrow it down a bit:
Since you pass {} to load, the function runs with an empty environment, so io is, in fact, nil (and io.stdout would error anyway)
The error happens directly when attempting to index io (which is a nil value)
The functio __index is a C function (see error message)
My first intuition was that __index was called somewhere internally. Thus, to find out what it does, I decided to look at its locals in hopes of guessing what it does.
A quick helper function I threw together:
local function locals(f)
return function(f, n)
local name, value = debug.getlocal(f+1, n)
if name then
return n+1, name, value
end
end, f, 1
end
Insert that right before the line where the error is raised:
for idx, name, value in locals(2) do
print(name, value)
end
error(string.format("calling bad function (%s:%d): %s", info.short_src, info.linedefined, (info.name or "?")))
This led to an interesting result:
(*temporary) stdin:43: attempt to index a nil value (global 'io')
(*temporary) table: 0x563cef2fd170
lua: stdin:29: calling bad function ([C]:-1): __index
stack traceback:
[C]: in function 'error'
stdin:29: in function <stdin:21>
[C]: in metamethod '__index'
stdin:43: in function 'f'
stdin:49: in main chunk
[C]: in ?
shell returned 1
Why is there a temporary string value with a completely different error message?
By the way, this error makes total sense; io does not exist because of the empty environment, so indexing it should obviously raise just that error.
It's honestly a very interesting error, but I'll leave it at this, as you're learning the language and this hint might be enough for you to figure it out on your own. It's also a very nice chance to actually use (and get to know) the debug module in a more practical context.
Actual Solution
After some time has now passed, I came back to add a proper solution to this problem, but I really already did just that. The weird error reporting is just Lua being weird. The real error is the empty environment that's set when loading the chunk, as I mentioned a few paragraphs above.
From the manual:
load (chunk [, chunkname [, mode [, env]]])
Loads a chunk.
[...]
If the resulting function has upvalues, the first upvalue is set to the value of env, if that parameter is given, or to the value of the global environment. Other upvalues are initialized with nil. (When you load a main chunk, the resulting function will always have exactly one upvalue, the _ENV variable (see §2.2). However, when you load a binary chunk created from a function (see string.dump), the resulting function can have an arbitrary number of upvalues.) All upvalues are fresh, that is, they are not shared with any other function.
[...]
Now, in a "main chunk", i.e. one loaded from a text Lua file, the first (and only) upvalue is always the environment of the chunk, so where it will look for "globals" (this is slightly different in Lua 5.1). Since an empty table is passed in, the chunk has no access to any of the global variables like string or io.
Therefore, when the function f() tries to index io, Lua throws an error "attempt to index a nil value", because io is nil. For whatever reason Lua then makes some internal function calls that end up triggering the blacklist, causing a new error that shadows the previous one; this makes debugging this error extremely inconvenient and almost impossible without using the debug library to get additional information about the call stack.
I ultimately only realized this myself after I noticed the original error message while looking at the locals of the function that made the blocked call.
I hope this solves the problem :)
I am writing a tool in Julia that requires a package with a deprecated function.
My script is called from the command line and takes many arguments so I would like to avoid using --depwarn=no to suppress deprecation warnings.
Instead, I'd like to embed this --depwarn=no or somehow signal this into my script so the user doesn't have to type it in, or worry about it whenever they run the script.
Does anyone know how can I do this using only Base Julia without installing any another package like Suppressor.jl?
I wrote Suppressor initially, as far as I know there is no other way right now, which is why I started Suppressor.
You could always copy paste verbatim the suppress* macro you need, into your code (but I would advise just to use Suppressor honestly, in case of updates), all the Suppressor macros are self contained and require only Base (if you are on 0.6.x this macros shouldn't need Compat).
#suppress_err (latest version):
"""
#suppress_err expr
Suppress the STDERR stream for the given expression.
"""
macro suppress_err(block)
quote
if ccall(:jl_generating_output, Cint, ()) == 0
ORIGINAL_STDERR = STDERR
err_rd, err_wr = redirect_stderr()
err_reader = #async read(err_rd, String)
end
value = $(esc(block))
if ccall(:jl_generating_output, Cint, ()) == 0
redirect_stderr(ORIGINAL_STDERR)
close(err_wr)
end
value
end
end
If you just want to get rid of the deprecation warnings, then #suppress_err is all you need. There have been improvements recently on the current Julia master branch related to logging, but I haven't checked out those yet.
What can I do within a file "example.jl" to exit/return from a call to include() in the command line
julia> include("example.jl")
without existing julia itself. quit() will just terminate julia itself.
Edit: For me this would be useful while interactively developing code, for example to include a test file and return from the execution to the julia prompt when a certain condition is met or do only compile the tests I am currently working on without reorganizing the code to much.
I'm not quite sure what you're looking to do, but it sounds like you might be better off writing your code as a function, and use a return to exit. You could even call the function in the include.
Kristoffer will not love it, but
stop(text="Stop.") = throw(StopException(text))
struct StopException{T}
S::T
end
function Base.showerror(io::IO, ex::StopException, bt; backtrace=true)
Base.with_output_color(get(io, :color, false) ? :green : :nothing, io) do io
showerror(io, ex.S)
end
end
will give a nice, less alarming message than just throwing an error.
julia> stop("Stopped. Reason: Converged.")
ERROR: "Stopped. Reason: Converged."
Source: https://discourse.julialang.org/t/a-julia-equivalent-to-rs-stop/36568/12
You have a latent need for a debugging workflow in Julia. If you use Revise.jl and Rebugger.jl you can do exactly what you are asking for.
You can put in a breakpoint and step into code that is in an included file.
If you include a file from the julia prompt that you want tracked by Revise.jl, you need to use includet(.
The keyboard shortcuts in Rebugger let you iterate and inspect variables and modify code and rerun it from within an included file with real values.
Revise lets you reload functions and modules without needing to restart a julia session to pick up the changes.
https://timholy.github.io/Rebugger.jl/stable/
https://timholy.github.io/Revise.jl/stable/
The combination is very powerful and is described deeply by Tim Holy.
https://www.youtube.com/watch?v=SU0SmQnnGys
https://youtu.be/KuM0AGaN09s?t=515
Note that there are some limitations with Revise, such as it doesn't reset global variables, so if you are using some global count or something, it won't reset it for the next run through or when you go back into it. Also it isn't great with runtests.jl and the Test package. So as you develop with Revise, when you are done, you move it into your runtests.jl.
Also the Juno IDE (Atom + uber-juno package) has good support for code inspection and running line by line and the debugging has gotten some good support lately. I've used Rebugger from the julia prompt more than from the Juno IDE.
Hope that helps.
#DanielArndt is right.
It's just create a dummy function in your include file and put all the code inside (except other functions and variable declaration part that will be place before). So you can use return where you wish. The variables that only are used in the local context can stay inside dummy function. Then it's just call the new function in the end.
Suppose that the previous code is:
function func1(...)
....
end
function func2(...)
....
end
var1 = valor1
var2 = valor2
localVar = valor3
1st code part
# I want exit here!
2nd code part
Your code will look like this:
var1 = valor1
var2 = valor2
function func1(...)
....
end
function func2(...)
....
end
function dummy()
localVar = valor3
1st code part
return # it's the last running line!
2nd code part
end
dummy()
Other possibility is placing the top variables inside a function with a global prefix.
function dummy()
global var1 = valor1
global var2 = valor2
...
end
That global variables can be used inside auxiliary function (static scope) and outside in the REPL
Another variant only declares the variables and its posterior use is free
function dummy()
global var1, var2
...
end
This is just a convenience but I think useful. Note that IPython allows a pure quit as does Matlab. Thus it would be reasonble in Julia to allow aliasing.
Thanks for any ideas as to how to do this.
Quitting in Julia
If you are using Julia from the command line then ctrl-d works. But if your intention is to quit by typing a command this is not possible exactly the way you want it because typing quit in the REPL already has a meaning which is return the value associated with quit, which is the function quit.
julia> quit
quit (generic function with 1 method)
julia> typeof(quit)
Function
Also Python
But that's not rare, for example Python has similar behavior.
>>> quit
Use quit() or Ctrl-D (i.e. EOF) to exit
Using a macro
Using \q might be nice in the Julia REPL like in postgres REPL, but unfortunately \ also already has a meaning. However, if you were seeking a simple way to do this, how about a macro
julia> macro q() quit() end
julia> #q
Causes Julia to Quit
If you place the macro definition in a .juliarc.jl file, it will be available every time you run the interpreter.
As waTeim notes, when you type quit into the REPL, it simply shows the function itself… and there's no way to change this behavior. You cannot execute a function without calling it, and there are a limited number of ways to call functions in Julia's syntax.
What you can do, however, is change how the Functions are displayed. This is extremely hacky and is not guaranteed to work, but if you want this behavior badly enough, here's what you can do: hack this behavior into the display method.
julia> function Base.writemime(io::IO, ::MIME"text/plain", f::Function)
f == quit && quit()
if isgeneric(f)
n = length(f.env)
m = n==1 ? "method" : "methods"
print(io, "$(f.env.name) (generic function with $n $m)")
else
show(io, f)
end
end
Warning: Method definition writemime(IO,MIME{symbol("text/plain")},Function) in module Base at replutil.jl:5 overwritten in module Main at none:2.
writemime (generic function with 34 methods)
julia> print # other functions still display normally
print (generic function with 22 methods)
julia> quit # but when quit is displayed, it actually quits!
$
Unfortunately there's no type more specific than ::Function, so you must completely overwrite the writemime(::IO,::MIME"text/plain",::Function) definition, copying its implementation.
Also note that this is pretty unexpected and somewhat dangerous. Some library may actually end up trying to display the function quit… causing you to lose your work from that session.
Related to Quitting in Julia
I was searching for something simple. This question hasn't been updated since 2017, as I try to learn Julia now, and spend some time googling for something simple and similar to python. Here, what I found:
You can use:
exit()
Note
I use julia 1.53
I am using the Kernel Density Estimator toolbox form http://www.ics.uci.edu/~ihler/code/kde.html . But I am getting the following error when I try to execute the demo files -
>> demo_kde_3
KDE Example #3 : Product sampling methods (single, anecdotal run)
Attempt to reference field of non-structure array.
Error in double (line 10)
if (npd.N > 0) d = 1; % return 1 if the density exists
Error in repmat (line 49)
nelems = prod(double(siz));
Error in kde (line 39)
if (size(ks,1) == 1) ks = repmat(ks,[size(points,1),1]); end;
Error in demo_kde_3 (line 8)
p = kde([.1,.45,.55,.8],.05); % create a mixture of 4 gaussians for
testing
Can anyone suggest what might be wrong? I am new to Matlab and having a hard time to figure out the problem.
Thank You,
Try changing your current directory away from the #kde folder; you may have to add the #kde folder to your path when you do this. For example run:
cd('c:\');
addpath('full\path\to\the\folder\#kde');
You may also need to add
addpath('full\path\to\the\folder\#kde\examples');
Then see if it works.
It looks like function repmat (a mathworks function) is picking up the #kde class's version of the double function, causing an error. Usually, only objects of the class #kde can invoke that functions which are in the #kde folder.
I rarely use the #folder form of class definitions, so I'm not completely sure of the semantics; I'm curious if this has any effect on the error.
In general, I would not recommend using the #folder class format for any development that you do. The mathworks overhauled their OO paradigm a few versions ago to a much more familiar (and useful) format. Use help classdef to see more. This #kde code seems to predate this upgrade.
MATLAB gives you the code line where the error occurs. As double and repmat belong to MATLAB, the bug probably is in kde.m line 39. Open that file in MATLAB debugger, set a breakpoint on that line (so the execution stops immediately before the execution of that specific line), and then when the code is stopped there, check the situation. Try the entire code line in console (copy-paste or type it, do not single-step, as causing an uncatched error while single-stepping ends the execution of code in debugger), it should give you an error (but doesn't stop execution). Then try pieces of the code of that code line, what works as it should and what not, eg. does the result of size(points, 1) make any sense.
However, debugging unfamiliar code is not an easy task, especially if you're a beginner in MATLAB. But if you learn and understand the essential datatypes of MATLAB (arrays, cell arrays and structs) and the different ways they can be addressed, and apply that knowledge to the situation on the line 39 of kde.m, hopefully you can fix the bug.
Repmat calls double and expects the built-in double to be called.
However I would guess that this is not part of that code:
if (npd.N > 0) d = 1; % return 1 if the density exists
So if all is correct this means that the buil-tin function double has been overloaded, and that this is the reason why the code crashes.
EDIT:
I see that #Pursuit has already addressed the issue but I will leave my answer in place as it describes the method of detection a bit more.