I want to read data from a file with a do-block. Using a do-block it is quite important because I need to be sure that the file gets properly closed even if an error occurs during the reading of the file. The data in the file should be converted to the a type provided as argument. My real use case is to use NetCDF files, but I can reproduce the type stability issue with plain text file.
Assume, there is a file file.txt with the content 123, which can be created with:
write("file.txt","123")
When I load the file without a do-block as follows, the result is type stable:
function loadfile1(T)
f = open("file.txt")
data = parse(T,readline(f))
close(f)
return data
end
#code_warntype infers correctly that I get Float32 as a result:
#code_warntype loadfile1(Float32)
Body::Float32
[...]
However, the following variant:
function loadfile2(T)
f = open("file.txt") do f
return parse(T,readline(f))
end
end
produces a type-unstable code:
#code_warntype loadfile2(Float32)
Body::Any
9 1 ─ %1 = %new(getfield(Main, Symbol("##842#843")){DataType}, T)::getfield(Main, Symbol("##842#843")){DataType} │
│ %2 = Main.open::Core.Compiler.Const(open, false) │
│ %3 = invoke Base.:(#open#294)($(QuoteNode(Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}()))::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, %2::Function, %1::getfield(Main, Symbol("##842#843")){DataType}, "file.txt"::String)::Any
└── return %3
How could I modify the function loadfile2 (with a do-block) and still get a type-stable result?
It works to move T to a type specification in the signature:
function loadfile2(::Type{T}) where T
f = open("file.txt") do f
return parse(T,readline(f))
end
end
This way T is known at compile time.
Related
I want to import a function present in a julia file somewhere during runtime
just like in python we have importlib.import_module to import module is there something present in julia
I'm new to julia and I'm not sure how to do that.
I have to import a function main from another julia file and want to run it but I have to also check a condition before that if the condition is true then I want to import the function.
EDIT
I have a file
main.jl
function myMain()
s1 = "Hello"
s2 = "World!"
include("functions/hello.jl")
say(s1, s2)
end
myMain()
hello.jl
function say(s1, s2)
print(s1, s2)
end
Error
ERROR: LoadError: MethodError: no method matching say(::String, ::String)
The applicable method may be too new: running in world age 32378, while current world is 32379.
Closest candidates are:
say(::Any, ::Any) at ~/Desktop/julia_including/functions/hello.jl:1 (method too new to be called from this world context.)
Stacktrace:
[1] myMain()
# Main ~/Desktop/julia_including/main.jl:5
[2] top-level scope
# ~/Desktop/julia_including/main.jl:8
in expression starting at /home/shivansh/Desktop/julia_including/main.jl:8
It works fine when I don't use include inside the myMain() function in main.jl
Julia contrary to Python is a compiled language.
Hence for maximum performance all functions should be known at the compilation time so efficient assembly code can be generated.
Hence generally you want to avoid syntax like the one you are proposing and have the inlude outside of the function.
However, if you really know what you are doing, why you are doing this and need such functionality then comes the world of Julia metaprogramming.
The code is the following:
function myMain()
s1 = "Hello"
s2 = "World!"
include("say.jl")
Base.invokelatest(say, s1, s2)
end
Now you can do:
julia> myMain()
HelloWorld!
Honestly, it just sounds like you wanna do this
if some_condition
include("/path/to/some/file/that/you/need.jl")
else
include("/path/to/some/OTHER/file/that/you/need.jl")
end
EDIT
I think though, after seeing the error that you added, what you wanna do is define your function multiple times instead, and with different argument types:
function say(x::Any, y::Any)
println("Default")
end
function say(x::String, y::Any)
println("String in arg1: $x")
end
function say(x::Any, y::String)
println("String in arg2: $y")
end
function say(x::String, y::String)
println("String in args 1 and 2: $x $y")
end
That way using say differs based on the datatype of the arguments:
julia> say("Foo", "Bar")
String in args 1 and 2: Foo Bar
julia> say("Foo", 2)
String in arg1: Foo
julia> say(0, "Bar")
String in arg2: Bar
and if you have two different functions that act on 2 strings you can do this instead:
struct MySayFunctionType{Symbol}
end
function say(x::String, y::String, ::MySayFunctionType{:Type1})
println("Type1 function is being called: $x $y")
end
function say(x::String, y::String, ::MySayFunctionType{:Type2})
println("Type2 function is being called: $x $y")
end
and then use these like this
if condition
say("Foo", "Bar", MySayFunctionType{Symbol("Type1")}() )
else
say("Foo", "Bar", MySayFunctionType{Symbol("Type2")}() )
end
and if you want a default version of this function to be called -- say Type2 -- you can just add a default value for the third argument like this:
function say(x::String, y::String, z::MySayFunctionType{:Type2}=MySayFunctionType{Symbol("Type2")}())
println("Type2 function is being called: $x $y")
end
The reason this works is because "MySayFunctionType{:Type1}" is considered to be a different type than "MySayFunctionType{:Type2}" so multiple dispatch would consider that as well.
This expression eval(Meta.parse("begin $(code)\nend")) would eval Julia code with include resolved relative to the file eval... is defined in.
How to change it so that it would use another directory? Something like
eval(Meta.parse("begin $(code)\nend"), resolve_include_relative_to=somepath)
Or, if that's not possible - relative to current directory (like REPL)?
UPDATE
Possible solution - replacing relative paths with absolute
function fix_include(code::String, relative_path::String)::String
code = replace(code, r"include\(\"\./(.*?)\"\)" => s"include(\"__relative_path__/\1\")")
code = replace(code, r"__relative_path__" => relative_path)
code
end
eval(Meta.parse("begin $(fix_include(code, relative_path))\nend")
Use case:
I'm evaluating snippets of string code, sometimes they contain include statement with relative paths and they resolved against wrong path. I want to explicitly specify tell it what path should be used for resolution. Or at the very least always use the current directory '.', not the directory where the file with the eval(xxx) line defined ./lib/runner.jl.
This function should do the trick (include is relative to the path in task-local storage, as kinda indicated by the docstring):
function eval_at(code; path = "none", mod = Main)
tls = task_local_storage()
hassource = haskey(tls, :SOURCE_PATH)
hassource && (path′ = tls[:SOURCE_PATH])
# setting this is enough for `include` to be correct
tls[:SOURCE_PATH] = path
try
# let's use the three-arg `include_string` here to make sure `#__FILE__`
# etc resolve correctly
return include_string(mod, code, path)
finally
hassource ?
(tls[:SOURCE_PATH] = path′) :
delete!(tls, :SOURCE_PATH)
end
end
Example usage:
julia> pwd()
"/home/pfitzseb/Documents"
julia> isfile("test.jl")
false
julia> include("test.jl")
ERROR: could not open file /home/pfitzseb/Documents/test.jl
julia> eval_at("""include("test.jl")""", path = "/home/pfitzseb/foo.jl")
Main.LogT
julia> eval_at("""#__FILE__""", path = "/home/pfitzseb/foo.jl")
"/home/pfitzseb/foo.jl"
It is not clear what exactly you want to do but for "do something in a folder" situations usually cd() do ... end syntax works great.
code = quote
cd("c:/temp") do
println("do something in $(pwd())")
#do something more
end
end
And now use it
julia> eval(code)
do something in c:\temp
Depending in your scenario you might consider using macros to manipulate code blocks that do something in a directory. Below is a simple example not offering more functionality than cd ... do ... end statement but of course it can be extended:
macro doinfolder(folder, what)
isa(what, Expr) || error("what needs to be some expression")
quote
cd($folder) do
$what
#other useful code injections can occur here...
end
end
end
And now use it
julia> #doinfolder "C:\\temp" pwd()
"C:\\temp"
Will also work with more complex code structures
julia> #doinfolder "C:\\temp" begin
pwd()
end
"C:\\temp"
I'm trying to write function that receive [String] which are names of files, String which is the name of the files directory and *f. The function will append to each file an integer in the end.
Here is what I got so far:
import StdEnv
import StdFile
import FileManipulation
appendNumInEndOfVmFiles :: [String] String *f -> String
appendNumInEndOfVmFiles [] dirname w = "finished"
appendNumInEndOfVmFiles [x:xs] dirname w
# path = dirname +++ "\\\\" +++ x
# (ok,file,files) = fopen path FAppendText w
# file = fwritei 12 file
# (ok2,_) = fclose file w
= appendNumInEndOfVmFiles xs dirname w
Start w
// 1. Receive name of directory from the user.
# (io,w) = stdio w // open stdio
# io = fwrites "Enter name of directory:\n" io // ask for name
# (name,io) = freadline io // read in name
# name = name % (0, size name - 2) // remove \n from name
# (ok,w) = fclose io w // close stdio
| not ok = abort "Couldn't close stdio" // abort in case of failure
// 2. Get a list of all file names in that directory.
# (dir,w) = getDirectoryContents (RelativePath [PathDown name]) w
# fileList = getNamesOfFilesInDirectory (getEntriesList dir)
= appendNumInEndOfVmFiles (getVmFiles fileList) name w
Assume that getVmFiles is defined in my FileManipulation.dcl file and in the context of this problem name is "myDir" and file list is ["hello.vm","Wiki.vm"]
For some reason, even that I got "finished" message on the screen, the files aren't modified. No matter what kind of integer I give to fopen, even if its FWriteText or FWriteData its still doing nothing... also even if I'm using fwritec or fwrites with characters nothing happened.
What I'm missing here? Thanks a lot!
For some reason, even that I got "finished" message on the screen, the files aren't modified.
This is due to lazy evaluation. In appendNumInEndOfVmFiles, the result of fclose is not used, so fclose is not evaluated. Because of this, fwritei does not need to be evaluated either. You can fix this by adding a guard on ok2:
# (ok2,_) = fclose file w
| not ok2 = abort "fclose failed\n"
= appendNumInEndOfVmFiles xs dirname w
However, the typical way to do this would be to rewrite the function to return a *f instead of a String, so that this unique value is not lost. As long as the result is used, then, the fwritei is evaluated. You can potentially make the *f argument strict (i.e. add a ! in front). This would make sure that it is evaluated before entering the function, so that all lingering file closes have been performed.
There are some more issues with your code:
Here, w is used twice, which is illegal because it is of a strict type. You should use (ok2,w) in the guard to continue with the same environment.
# (ok2,_) = fclose file w
= appendNumInEndOfVmFiles xs dirname w
The appendNumInEndOfVmFiles needs to have a type context | FileSystem f to resolve overloading of fopen and fclose.
Lastly:
... even if its FWriteText or FWriteData ...
Just so you know: the difference would be that the first would write the integer in an ASCII representation whereas the second would write it binary as 4 or 8 bytes (depending on the bitwidth of your system).
Is there a way to get the file/name/line info for the caller of a Julia function?
I found this way to get some stacktrace info, and if the caller is another function (but not the main context) I get the file:line info:
module pd
global g_bTraceOn = true
export callerOfTraceIt
function callerOfTraceIt()
traceit( "hi" ) ;
end
function traceit( msg )
global g_bTraceOn
if ( g_bTraceOn )
bt = backtrace() ;
s = sprint(io->Base.show_backtrace(io, bt))
println( "debug: $s: $msg" )
end
end
end
using pd
callerOfTraceIt( )
This shows:
$ julia bt.jl
debug:
in traceit at C:\cygwin64\home\Peeter\julia\HarmonicBalance\bt.jl:15
in callerOfTraceIt at C:\cygwin64\home\Peeter\julia\HarmonicBalance\bt.jl:8
in include at boot.jl:245
in include_from_node1 at loading.jl:128
in process_options at client.jl:285
in _start at client.jl:354: hi
I'd really like just the second frame (caller of traceit()), and would also like the function name if it's available.
If you do #show bt inside traceit, you'll discover it's just a list of pointers, each corresponding to a single stack frame. Those stack frames that come from julia code (rather than C) are displayed by show_backtrace.
You can call Profile.lookup(uint(bt[1])) to extract file/function/line information from each element:
julia> Profile.lookup(uint(bt[1]))
LineInfo("rec_backtrace","/home/tim/src/julia-old/usr/bin/../lib/libjulia.so",-1,true,140293228378757)
julia> names(Profile.LineInfo)
5-element Array{Symbol,1}:
:func
:file
:line
:fromC
:ip
You likely want to ignore all elements with fromC == true.
I was wondering if there was a way to programmatically or dynamically import a set of modules into Julia? For example, if I have a list of files that conform to some naming convention that are present at startup that I can grab using something like:
module_files = filter(r"^mod[0-9][0-9].jl$", readdir())
which might return a list of files ["mod00.jl", "mod02.jl", "mod05.jl"], is there a way to then import each of the modules in those files. This would be equivalent to having:
import mod00
import mod02
import mod05
in the code if I knew that those modules were available when I wrote the code. Or maybe there is some other approach to doing this that is better. Any suggestions would be much appreciated.
Update
I attempted to do this via a macro, but with no luck. For example:
macro import_mod(modn)
quote
import $modn
end
end
function test_mimport()
module_files = filter(r"^mod[0-9][0-9].jl$", readdir())
println(module_files)
for modname in factor_files
modn = modname[1:end-3]
println(modn)
#import_mod modn
end
end
When I run this, I get ERROR: syntax: invalid "import" statement. I tried various escaping strategies but all failed similarly.
NOTE: Please refer to user1712368's (Jameson Nash, Julia dev) answer, this discussion at the julia-users mailing list and this entry of the Julia manual, to know about why this is not the correct answer.
This is how a multiple import expression looks like, using Julia version 0.3.4:
julia> parse("import Foo, Bar")
:($(Expr(:toplevel, :($(Expr(:import, :Foo))), :($(Expr(:import, :Bar))))))
julia> dump(ans)
Expr
head: Symbol toplevel
args: Array(Any,(2,))
1: Expr
head: Symbol import
args: Array(Any,(1,))
1: Symbol Foo
typ: Any
2: Expr
head: Symbol import
args: Array(Any,(1,))
1: Symbol Bar
typ: Any
typ: Any
Here is a macro which does this programmatically, it takes a modules argument that could be a :call or :vcat Expr or a Symbol, which has to evaluate to a Vector{Symbol}:
julia> macro dynamic_import(modules)
(modules = eval(modules))::Vector{Symbol}
ex = Expr(:toplevel)
for m in modules
push!(ex.args, Expr(:import, m))
end
return ex
end
You could also generalize this line:
module_files = filter(r"^mod[0-9][0-9].jl$", readdir())
By abstracting it into a function that takes a Regex and a String directory path as arguments and returns a Vector{Symbol}:
julia> function needed_modules(rx::Regex, dir::String=".")
module_files = filter(rx, readdir(dir))
module_syms = map(m -> symbol(split(m, '.')[1]), module_files)
end
needed_modules (generic function with 2 methods)
So you may use it like this:
julia> #dynamic_import [:Mod01, :Mod02] # :vcat expression
julia> rx = r"^Mod[0-9][0-9].jl$";
julia> #dynamic_import needed_modules(rx) # :call expression
julia> modules = needed_modules(rx)
2-element Array{Symbol,1}:
:Mod01
:Mod02
julia> #dynamic_import modules # Symbol
Finally you could wrap it all into a module so you may use using DynamicImport:
Note: currently I get this when trying to run the same examples from a module:
julia> using DynamicImport
julia> #dynamic_import [:mod01, :mod02]
julia> rx = r"^mod[0-9][0-9].jl$";
julia> #dynamic_import needed_modules(rx)
ERROR: rx not defined
julia> modules = needed_modules(rx)
2-element Array{Symbol,1}:
:mod01
:mod02
julia> #dynamic_import modules
ERROR: modules not defined
But it works fine If I define the objects inside the REPL, I guess this is an issue involving hygiene, which is something I'm not experienced with, so I'll ask at the julia-users mailing list.
The function version of import X is require("X"), so you should be able to do:
function test_mimport()
module_files = filter(r"^mod[0-9][0-9].jl$", readdir())
println(module_files)
for modname in factor_files
modn = modname[1:end-3]
println(modn)
require(modn)
end
end
Then, assuming each of these defined a module of the same name, you can collect them into an array:
modules = Module[]
...
if isdefined(Main, symbol(modn))
push!(modules, getfield(Main, symbol(modn))
else
warn("importing $modn did not defined a module")
end
...
return modules
I'm not thrilled with this solution, but it seems to work. Basically I dynamically create a temporary file contains the import statements that I then include it. So:
function test_mimport()
module_files = filter(r"^mod[0-9][0-9].jl$", readdir())
path, f = mktemp()
for modname in module_files
modn = splitext(modname)[1]
write(f, "import $modn\n")
end
close(f)
include(path)
rm(path)
end
I would be curious to know if there is a more idiomatic strategy since I'm new to Julia and just wadding through its capabilities.
Edit
The strategy of setting up a temporary file is likely unnecessary since Julia also provides an include_string method.
On the Julia-user list, #Isaiah also suggests using eval(Expr(:import, symbol(modn)))