How to parse multiline string in Julia? - julia

Question

How can I parse more lines of code?
This is working:
julia> eval(parse("""print("O");print("K")"""))
OK
This is not working:
julia> eval(parse("""print("N");
print("O")"""))
ERROR: ParseError("extra token after end of expression")
Stacktrace:
[1] #parse#235(::Bool, ::Function, ::String) at ./parse.jl:237
[2] parse(::String) at ./parse.jl:232
BTW if I try line by line I have other problems. For example:
julia> parse("""for i in 1:3""")
:($(Expr(:incomplete, "incomplete: premature end of input")))
although:
julia> eval(parse("""for i in 1:2
println(i)
end"""))
1
2

Answer 1

It's a bit dopey, but this will do the trick!:
function parseall(str)
return Meta.parse("begin $str end").args
end
Explanation:
As #Alexander Morley pointed out above, parse is designed to only parse a single expression. So if you just make that single expression able to contain all the expressions in your original string, then it will parse it just fine! :)
The simplest way to do that is to wrap the string in a block, which makes it a valid julia expression. You can see that such an expression parses just fine:
julia> Meta.parse("""
begin
function f3(x)
x + 2
end
print("N")
print(f3(5))
end
""")
>> quote
#= none:2 =#
function f3(x)
#= none:3 =#
x + 2
end
#= none:5 =#
print("N")
#= none:6 =#
print(f3(5))
end
This is in fact the structure that the code in #Alexander Morley's is building. (And that's what made me think of doing it this way! Thanks Alexander!)
However, do note that if you are trying to parse a file, you actually don't want to return a single block expression like that returns, because you can't eval it as a single block: Julia doesn't allow you to nest some "top-level statements" (Modules) inside of anything. Instead you want to return an array of top-level expressions. That's why we return the args out of the Block expression.
With the above, you can eval each of those top-level expressions, and it will work correctly! :)
julia> for expr in parseall("module M f() = 1 end"); Core.eval(Main, expr); end
julia> M.f()
1
This part was figured out together with #misakawa (#thautwarm on Github).

Answer 2

parse is designed to parse a single expression (at least that's what the docs say: given this I'm actually a bit surprised your first example works , without throwing an error...).
If you want to parse mutliple expressions then you can take advantage of the fact that:
parse can take a second argument start that tells it where to
start parsing from.
If you provide this start argument then it returns a tuple containing the expression, and where the expression finished.
to define a parseall function yourself. There used to be one in base but I'm not sure there is anymore. Edit: there is still in the tests see below
# modified from the julia source ./test/parse.jl
function parseall(str)
pos = start(str)
exs = []
while !done(str, pos)
ex, pos = parse(str, pos) # returns next starting point as well as expr
ex.head == :toplevel ? append!(exs, ex.args) : push!(exs, ex) #see comments for info
end
if length(exs) == 0
throw(ParseError("end of input"))
elseif length(exs) == 1
return exs[1]
else
return Expr(:block, exs...) # convert the array of expressions
# back to a single expression
end
end