Parameter file in ArgParse.jl? - julia

Python's argparse has a simple way to read parameters from a file:
https://docs.python.org/2/library/argparse.html#fromfile-prefix-chars
Instead of passing your arguments one by one:
python script.py --arg1 val1 --arg2 val2 ...
You can say:
python script.py #args.txt
and then the arguments are read from args.txt.
Is there a way to do this in ArgParse.jl?
P.S.: If there is no "default" way of doing this, maybe I can do it by hand, by calling parse_args on a list of arguments read from a file. I know how to do this in a dirty way, but it gets messy if I want to replicate the behavior of argparse in Python, where I can pass multiple files with #, as well as arguments in the command line, and then the value of a parameter is simply the last value passed to this parameter. What's the best way of doing this?

This feature is not currently present in ArgParse.jl, although it would not be difficult to add. I have prepared a pull request.
In the interim, the following code suffices for what you need:
# faithful reproduction of Python 3.5.1 argparse.py
# partial copyright Python Software Foundation
function read_args_from_files(arg_strings, prefixes)
new_arg_strings = AbstractString[]
for arg_string in arg_strings
if isempty(arg_string) || arg_string[1] ∉ prefixes
# for regular arguments, just add them back into the list
push!(new_arg_strings, arg_string)
else
# replace arguments referencing files with the file content
open(arg_string[2:end]) do args_file
arg_strings = AbstractString[]
for arg_line in readlines(args_file)
push!(arg_strings, rstrip(arg_line, '\n'))
end
arg_strings = read_args_from_files(arg_strings, prefixes)
append!(new_arg_strings, arg_strings)
end
end
end
# return the modified argument list
return new_arg_strings
end
# preprocess args, then parse as usual
ARGS = read_args_from_files(ARGS, ['#'])
args = parse_args(ARGS, s)

Related

Julia scoping issue when creating function from string

I would like to build a Julia application where a user can specify a function using a configuration file (and therefore as a string). The configuration file then needs to be parsed before the function is evaluated in the program.
The problem is that while the function name is known locally, it is not known in the module containing the parser. One solution I have come up with is to pass the local eval function to the parsing function but that does not seem very elegant.
I have tried to come up with a minimal working example here, where instead of parsing a configuration file, the function name is already contained in a string:
module MyFuns
function myfun(a)
return a+2
end
end
module MyUtil
# in a real application, parseconfig would parse the configuration file to extract funstr
function parseconfig(funstr)
return eval(Meta.parse(funstr))
end
function parseconfig(funstr, myeval)
return myeval(Meta.parse(funstr))
end
end
# test 1 -- succeeds
f1 = MyFuns.myfun
println("test1: $(f1(1))")
# test 2 -- succeeds
f2 = MyUtil.parseconfig("MyFuns.myfun", eval)
println("test2: $(f2(1))")
# test 3 -- fails
f3 = MyUtil.parseconfig("MyFuns.myfun")
println("test3: $(f3(1))")
The output is:
test1: 3
test2: 3
ERROR: LoadError: UndefVarError: MyFuns not defined
So, the second approach works but is there a better way to achieve the goal?
Meta.parse() will transform your string to an AST. What MyFuns.myfun refers to depends on the scope provided by the eval() you use.
The issue with your example is that the eval() inside MyUtil will evaluate in the context of that module. If that is the desired behavior, you simply miss using MyFuns inside MyUtil.
But what you really want to do is write a macro. This allows the code to be included when parsing your program, before running it. The macro will have access to a special argument __module__, which is the context where the macro is used. So __module__.eval() will execute an expression in that very scope.
foo = "outside"
module MyMod
foo = "inside"
macro eval(string)
expr = Meta.parse(string)
__module__.eval(expr)
end
end
MyMod.#eval "foo"
# Output is "outside"
See also this explanation on macros:
https://docs.julialang.org/en/v1/manual/metaprogramming/index.html#man-macros-1
And for the sake of transforming the answer of #MauricevanLeeuwen into the framework of my question, this code will work:
module MyFuns
function myfun(a)
return a+2
end
end
module MyUtil
macro parseconfig(funstr)
__module__.eval(Meta.parse(funstr))
end
end
f4 = MyUtil.#parseconfig "MyFuns.myfun"
println("test4: $(f4(1))")

Finding a Module's path, using the Module object

What is the sane way to go from a Module object to a path to the file in which it was declared?
To be precise, I am looking for the file where the keyword module occurs.
The indirect method is to find the location of the automatically defined eval method in each module.
moduleloc(mm::Module) = first(functionloc(mm.eval, (Symbol,)))
for example
moduleloc(mm::Module) = first(functionloc(mm.eval, (Symbol,)))
using DataStructures
moduleloc(DataStructures)
Outputs:
/home/oxinabox/.julia/v0.6/DataStructures/src/DataStructures.jl
This indirect method works, but it feels like a bit of a kludge.
Have I missed some inbuilt function to do this?
I will remind answered that Modules are not the same thing as packages.
Consider the existence of submodules, or even modules that are being loaded via includeing some abolute path that is outside the package directory or loadpath.
Modules simply do not store the file location where they were defined. You can see that for yourself in their definition in C. Your only hope is to look through the bindings they hold.
Methods, on the other hand, do store their file location. And eval is the one function that is defined in every single module (although not baremodules). Slightly more correct might be:
moduleloc(mm::Module) = first(functionloc(mm.eval, (Any,)))
as that more precisely mirrors the auto-defined eval method.
If you aren't looking for a programmatic way of doing it you can use the methods function.
using DataFrames
locations = methods(DataFrames.readtable).ms
It's for all methods but it's hardly difficult to find the right one unless you have an enormous number of methods that differ only in small ways.
There is now pathof:
using DataStructures
pathof(DataStructures)
"/home/ederag/.julia/packages/DataStructures/59MD0/src/DataStructures.jl"
See also: pkgdir.
pkgdir(DataStructures)
"/home/ederag/.julia/packages/DataStructures/59MD0"
Tested with julia-1.7.3
require obviously needs to perform that operation. Looking into loading.jl, I found that finding the module path has changed a bit recently: in v0.6.0, there is a function
load_hook(prefix::String, name::String, ::Void)
which you can call "manually":
julia> Base.load_hook(Pkg.dir(), "DataFrames", nothing)
"/home/philipp/.julia/v0.6/DataFrames/src/DataFrames.jl"
However, this has changed to the better in the current master; there's now a function find_package, which we can copy:
macro return_if_file(path)
quote
path = $(esc(path))
isfile(path) && return path
end
end
function find_package(name::String)
endswith(name, ".jl") && (name = chop(name, 0, 3))
for dir in [Pkg.dir(); LOAD_PATH]
dir = abspath(dir)
#return_if_file joinpath(dir, "$name.jl")
#return_if_file joinpath(dir, "$name.jl", "src", "$name.jl")
#return_if_file joinpath(dir, name, "src", "$name.jl")
end
return nothing
end
and add a little helper:
find_package(m::Module) = find_package(string(module_name(m)))
Basically, this takes Pkg.dir() and looks in the "usual locations".
Additionally, chop in v0.6.0 doesn't take these additional arguments, which we can fix by adding
chop(s::AbstractString, m, n) = SubString(s, m, endof(s)-n)
Also, if you're not on Unix, you might want to care about the definitions of isfile_casesensitive above the linked code.
And if you're not so concerned about corner cases, maybe this is enough or can serve as a basis:
function modulepath(m::Module)
name = string(module_name(m))
Pkg.dir(name, "src", "$name.jl")
end
julia> Pkg.dir("DataStructures")
"/home/liso/.julia/v0.7/DataStructures"
Edit: I now realized that you want to use Module object!
julia> m = DataStructures
julia> Pkg.dir(repr(m))
"/home/liso/.julia/v0.7/DataStructures"
Edit2: I am not sure if you are trying to find path to module or to object defined in module (I hope that parsing path from next result is easy):
julia> repr(which(DataStructures.eval, (String,)))
"eval(x) in DataStructures at /home/liso/.julia/v0.7/DataStructures/src/DataStructures.jl:3"

How to quit/exit from file included in the terminal

What can I do within a file "example.jl" to exit/return from a call to include() in the command line
julia> include("example.jl")
without existing julia itself. quit() will just terminate julia itself.
Edit: For me this would be useful while interactively developing code, for example to include a test file and return from the execution to the julia prompt when a certain condition is met or do only compile the tests I am currently working on without reorganizing the code to much.
I'm not quite sure what you're looking to do, but it sounds like you might be better off writing your code as a function, and use a return to exit. You could even call the function in the include.
Kristoffer will not love it, but
stop(text="Stop.") = throw(StopException(text))
struct StopException{T}
S::T
end
function Base.showerror(io::IO, ex::StopException, bt; backtrace=true)
Base.with_output_color(get(io, :color, false) ? :green : :nothing, io) do io
showerror(io, ex.S)
end
end
will give a nice, less alarming message than just throwing an error.
julia> stop("Stopped. Reason: Converged.")
ERROR: "Stopped. Reason: Converged."
Source: https://discourse.julialang.org/t/a-julia-equivalent-to-rs-stop/36568/12
You have a latent need for a debugging workflow in Julia. If you use Revise.jl and Rebugger.jl you can do exactly what you are asking for.
You can put in a breakpoint and step into code that is in an included file.
If you include a file from the julia prompt that you want tracked by Revise.jl, you need to use includet(.
The keyboard shortcuts in Rebugger let you iterate and inspect variables and modify code and rerun it from within an included file with real values.
Revise lets you reload functions and modules without needing to restart a julia session to pick up the changes.
https://timholy.github.io/Rebugger.jl/stable/
https://timholy.github.io/Revise.jl/stable/
The combination is very powerful and is described deeply by Tim Holy.
https://www.youtube.com/watch?v=SU0SmQnnGys
https://youtu.be/KuM0AGaN09s?t=515
Note that there are some limitations with Revise, such as it doesn't reset global variables, so if you are using some global count or something, it won't reset it for the next run through or when you go back into it. Also it isn't great with runtests.jl and the Test package. So as you develop with Revise, when you are done, you move it into your runtests.jl.
Also the Juno IDE (Atom + uber-juno package) has good support for code inspection and running line by line and the debugging has gotten some good support lately. I've used Rebugger from the julia prompt more than from the Juno IDE.
Hope that helps.
#DanielArndt is right.
It's just create a dummy function in your include file and put all the code inside (except other functions and variable declaration part that will be place before). So you can use return where you wish. The variables that only are used in the local context can stay inside dummy function. Then it's just call the new function in the end.
Suppose that the previous code is:
function func1(...)
....
end
function func2(...)
....
end
var1 = valor1
var2 = valor2
localVar = valor3
1st code part
# I want exit here!
2nd code part
Your code will look like this:
var1 = valor1
var2 = valor2
function func1(...)
....
end
function func2(...)
....
end
function dummy()
localVar = valor3
1st code part
return # it's the last running line!
2nd code part
end
dummy()
Other possibility is placing the top variables inside a function with a global prefix.
function dummy()
global var1 = valor1
global var2 = valor2
...
end
That global variables can be used inside auxiliary function (static scope) and outside in the REPL
Another variant only declares the variables and its posterior use is free
function dummy()
global var1, var2
...
end

How do I make a `Pipe` or `TTY` with a custom callback in Julia?

I'd like to fancy up my embedding of Julia in a MATLAB mex function by hooking up Julia's STDIN, STDOUT, and STDERR to the MATLAB terminal. The documentation for redirect_std[in|out|err] says that the stream that I pass in as the argument needs to be a TTY or a Pipe (or a TcpSocket, which wouldn't seem to apply).
I know how I will define the right callbacks for each stream (basically, wrappers around calls to MATLAB's input and fprintf), but I'm not sure how to construct the required stream.
Pipe was renamed PipeEndpoint in https://github.com/JuliaLang/julia/pull/12739, but the corresponding documentation was not updated and PipeEndpoint is now considered internal. Even so, creating the pipe up front is still doable:
pipe = Pipe()
Base.link_pipe(pipe)
redirect_stdout(pipe.in)
#async while !eof(pipe)
data = readavailable(pipe)
# Pass data to whatever function handles display here
end
Furthermore, the no-argument version of these functions already create a pipe object, so the recommended way to do this would be:
(rd,wr) = redirect_stdout()
#async while !eof(rd)
data = readavailable(rd)
# Pass data to whatever function handles display here
end
Nevertheless, all of this is less clear than it could be, so I have created a pull request to clean up this API: https://github.com/JuliaLang/julia/pull/18253. Once that pull request is merged, the link_pipe call will become unnecessary and pipe can be passed directly into redirect_stdout. Further, the return value from the no-argument version will become a regular Pipe.

Run Rscript.exe on batch file with optional arguments

I run a Windows batchfile (.bat)
path_to_Rscript.exe file.R parameter1 parameter2 parameter3
on a file called file.R
In file.R, I read the arguments using
commandArgs(trailingOnly = TRUE)
ABC <- args[1]
DEF <- args[2]
GHI <- args[3]
In case there is no parameter2, calling only
path_to_Rscript.exe file.R parameter1 parameter3
how can I make sure that parameter3 is not assigned to DEF?
So far, I used %% as a placeholder, but I am not sure whether this is a common approach.
Is there a general placeholder for empty parameters?
There is no general "no arg" placeholder, it depends on the progam.
The answer is that you can't make sure p3 is not assigned to DEF without some assumptions, i.e., if you only have two arguments then they are always going to be ABC and GHI. In that case, you check for the length of commandArgs and adjust accordingly - that is a very narrow solution.
If you want to use optional positional arguments (which is generally a bad idea), they'd have to be at the end and then you really only get one for the same reason your facing now.
The best way out of this conundrum is to use docopt. If you can't use docopt, then your stuck implementing pieces of a command-line parser which is a generally solved problem.

Resources