I am trying to write a program in Julia that given a starting folder, will loop through all sub folders such that I can open and get the contents out of all the files in the sub folders. How can I do this in Julia ?
Ideally the code would allow for an unspecified folder depth in case I don’t know it ahead of time.
You can use walkdir, like so:
for (root, dirs, files) in walkdir("mydir")
operate_on_files(joinpath.(root, files)) # files is a Vector{String}, can be empty
end
https://docs.julialang.org/en/v1/base/file/#Base.Filesystem.walkdir
Edit: A good thing to do here is to broadcast across the array of file paths, so that you don't need to special-case an empty array.
contents = String[]
for (root, dirs, files) in walkdir("mydir")
# global contents # if in REPL
push!.(Ref(contents), read.(joinpath.(root, files), String))
end
Maybe write a recursive function that lists all folders and files in the dir, pushes the contents of each file to a higher-scope Array, then calls itself on each of the folders?
Sth like (untested code):
function outerfun(dir)
function innerfun!(dir, filecontents)
for name in readdir(dir)
if isdir(name)
innerfun!(name, filecontents)
else
push!(readlines(name), filecontents)
end
end
end
filecontents = Array{String}[]
innerfun!(dir, filecontents)
filecontents
end
Related
The following function prints the directories and files within a path, I want it to go into the directories and sub directories by using a recursive function. This causes a stackoverflow error. It only works if doesn't call "RecursiveSearch" func but only prints out directories and files but not subdirectories.
extends Node
func RecursiveSearch(dir):
var path = ("res://")
var err = dir.open(path)
if err != OK:
print("error occurred")
return
dir.list_dir_begin() # points to the first one, true ignores special directory entries (.) and (..)
var name = dir.get_next() # retrieves the firdt file or dir
while name != "":
if dir.current_is_dir(): # test if it's a dir type
print("dir : " , name)
RecursiveSearch(dir)
elif !dir.current_is_dir():
print("file : " , name)
else:
break
name = dir.get_next() # points to the next dir or file
dir.list_dir_end()
func _ready():
var dir = Directory.new()
RecursiveSearch(dir)
Directory.list_dir_begin() only lists the immediate children of a directory. To recurse into a subdirectory, you will need to create a new Directory object for the subdirectory, and call your RecursiveSearch on that.
As it stands, your function calls itself with its own argument. It will do the exact same thing--call itself with the same argument--again and again until it hits the recursion limit.
This expression eval(Meta.parse("begin $(code)\nend")) would eval Julia code with include resolved relative to the file eval... is defined in.
How to change it so that it would use another directory? Something like
eval(Meta.parse("begin $(code)\nend"), resolve_include_relative_to=somepath)
Or, if that's not possible - relative to current directory (like REPL)?
UPDATE
Possible solution - replacing relative paths with absolute
function fix_include(code::String, relative_path::String)::String
code = replace(code, r"include\(\"\./(.*?)\"\)" => s"include(\"__relative_path__/\1\")")
code = replace(code, r"__relative_path__" => relative_path)
code
end
eval(Meta.parse("begin $(fix_include(code, relative_path))\nend")
Use case:
I'm evaluating snippets of string code, sometimes they contain include statement with relative paths and they resolved against wrong path. I want to explicitly specify tell it what path should be used for resolution. Or at the very least always use the current directory '.', not the directory where the file with the eval(xxx) line defined ./lib/runner.jl.
This function should do the trick (include is relative to the path in task-local storage, as kinda indicated by the docstring):
function eval_at(code; path = "none", mod = Main)
tls = task_local_storage()
hassource = haskey(tls, :SOURCE_PATH)
hassource && (path′ = tls[:SOURCE_PATH])
# setting this is enough for `include` to be correct
tls[:SOURCE_PATH] = path
try
# let's use the three-arg `include_string` here to make sure `#__FILE__`
# etc resolve correctly
return include_string(mod, code, path)
finally
hassource ?
(tls[:SOURCE_PATH] = path′) :
delete!(tls, :SOURCE_PATH)
end
end
Example usage:
julia> pwd()
"/home/pfitzseb/Documents"
julia> isfile("test.jl")
false
julia> include("test.jl")
ERROR: could not open file /home/pfitzseb/Documents/test.jl
julia> eval_at("""include("test.jl")""", path = "/home/pfitzseb/foo.jl")
Main.LogT
julia> eval_at("""#__FILE__""", path = "/home/pfitzseb/foo.jl")
"/home/pfitzseb/foo.jl"
It is not clear what exactly you want to do but for "do something in a folder" situations usually cd() do ... end syntax works great.
code = quote
cd("c:/temp") do
println("do something in $(pwd())")
#do something more
end
end
And now use it
julia> eval(code)
do something in c:\temp
Depending in your scenario you might consider using macros to manipulate code blocks that do something in a directory. Below is a simple example not offering more functionality than cd ... do ... end statement but of course it can be extended:
macro doinfolder(folder, what)
isa(what, Expr) || error("what needs to be some expression")
quote
cd($folder) do
$what
#other useful code injections can occur here...
end
end
end
And now use it
julia> #doinfolder "C:\\temp" pwd()
"C:\\temp"
Will also work with more complex code structures
julia> #doinfolder "C:\\temp" begin
pwd()
end
"C:\\temp"
I have an AppleScript that I'm trying to modularize for future use. Working fine: a script that steps through nested Finder folders recursively. What I want: that script to call another subroutine with its own parameters.
Sample code (with a working recurse function), where I'm using commented pseudocode for the parts I need help with:
on recurseFolders_runSubroutine(thisItem, subName, subParameters, notify)
global itemcount
try
set itemcount to itemcount + 1
on error
set itemcount to 1
end try
tell application "Finder"
if notify then display notification ("Recursing into folder ") & name of thisItem with title "Entering Folder"
if kind of thisItem is "Folder" then
set subItems to every item in thisItem
set countItems to count subItems
repeat with i from 1 to countItems
set subItem to item i of subItems
my recurseFolders_runSubroutine(subItem, subName, subParameters, notify, itemcount)
end repeat
else
-- tell script ("subName" & .scpt) to [call subroutine named subName with the parameters, presumably passed in a list]
end if
end tell
end recurseFolders_runSubroutine
I tried it with a few variations on run script with a text string to call the function, but then I can't pass the list of subparameters as a list. Am I missing something?
When I run the following code, I get a deprecation saying produce has been replace with channels.
function source(dir)
filelist = readdir(dir)
for filename in filelist
name,ext = splitext(filename)
if ext == ".jld"
produce(filename)
end
end
end
path = "somepathdirectoryhere"
for fname in Task(source(path))
println(fname)
end
I cannot find an example on how to do this with channels. I've tried creating a global channel and using put! instead of produce with no luck.
Any ideas?
Here's one way. Modify your function to accept a channel argument, and put! data in it:
function source(dir, chnl)
filelist = readdir(dir)
for filename in filelist
name, ext = splitext(filename)
if ext == ".jld"
put!(chnl, filename) % this blocks until "take!" is used elsewhere
end
end
end
Then create your task implicitly using the Channel constructor (which takes a function with a single argument only representing the channel, so we need to wrap the source function around an anonymous function):
my_channel = Channel( (channel_arg) -> source( pwd(), channel_arg) )
Then, either check the channel is still open (i.e. task hasn't finished) and if so take an argument:
julia> while isopen( my_channel)
take!( my_channel) |> println;
end
no.jld
yes.jld
or, use the channel itself as an iterator (iterating over Tasks is becoming deprecated, along with the produce / consume functionality)
julia> for i in my_channel
i |> println
end
no.jld
yes.jld
Alternatively you can use #schedule with bind etc as per the documentation, but it seems like the above is the most straightforward way.
So what exactly does Julia do with the statement using Foo if you don't have package Foo installed? As I understood Julia starts searching JULIA_LOAD_PATH, but how?
At the root level of JULIA_LOAD_PATH there must be a directory named Foo.jl where the Foo part may be case insensitive and the .jl suffix is optional?
And within this Foo.jl directory there must be a source file name Foo.jl with a module Foo?
using implicitly calls require which indirectly calls find_in_path:
function find_in_path(name::AbstractString, wd = pwd())
isabspath(name) && return name
base = name
# this is why `.jl` suffix is optional
if endswith(name,".jl")
base = name[1:end-3]
else
name = string(base,".jl")
end
if wd !== nothing
isfile(joinpath(wd,name)) && return joinpath(wd,name)
end
for prefix in [Pkg.dir(); LOAD_PATH]
path = joinpath(prefix, name)
isfile(path) && return abspath(path)
path = joinpath(prefix, base, "src", name)
isfile(path) && return abspath(path)
path = joinpath(prefix, name, "src", name)
isfile(path) && return abspath(path)
end
return nothing
end
The source code above shows that there is no additional manipulation on name, which means the Foo part should be case sensitive(Currently depend on the filesystem, see the comment below). And the directory name is unnecessary to be compatible with your file name, it can be anything as long as the directory is in your LOAD_PATH.