I have a pretty simple question but I can't seem to find an answer anywhere. I want to run two .jl files parallel using the Julia Terminal.
I tried include("file1.jl" & "file2.jl") and include("file1.jl") & include("file2.jl") but this doesn't work.
I'm not sure exactly what you want to do but if you wanted to run these two files on two different workers from the julia terminal you could for e.g.
addprocs(1) # add a worker
pmap(include,["file1.jl", "file2.jl"]) # apply include to each element
# of the array in parallel
But I'm pretty sure there will be a better way of doing whatever you want to accomplish.
While you can probably wrangle your code into the Julia parallel computing paradigms, it seems like the simplest solution is to execute your Julia scripts from the command line. Here I assume that you are comfortable allowing your CPU to handle the task scheduling, which may or may not result in parallel execution.
What follows below is a skeleton pipeline to get you started. Replace task.jl with your file1.jl, file2.jl, etc.
task.jl
println("running like a cheetah")
run_script.sh
echo `date`
julia task.jl
julia task.jl
echo `date`
run_script_parallel.sh
echo `date`
julia task.jl &
julia task.jl &
wait # do not return before background tasks are complete
echo `date`
From the command line, ensure that your BASH scripts are executable:
chmod +rwx run_script.sh run_script_parallel.sh
Try running the scripts now. Note that my example Julia script task.jl returns almost immediately, so this particular comparison is a little silly:
./run_script.sh
./run_script_parallel.sh
My output
Thu Jan 5 14:24:57 PST 2017
running like a cheetah
running like a cheetah
Thu Jan 5 14:24:57 PST 2017
Thu Jan 5 14:25:05 PST 2017
running like a cheetahrunning like a cheetah
Thu Jan 5 14:25:06 PST 2017
The first output orders the print statements in a clean serial order. But observe in the second case that the text is running together. That is common behavior for parallel print statements.
Related
In REPL mode, Julia lets you type a semicolon and run shell commands, i.e.
;
cd ~
And then to return to Julian REPL
Is there a way to do something similar in a .jl file? The closest I found was run(…) and that has many caveats. This is a Linux environment, so I'm not concerned about the caveats of shell mode on Windows machines.
The broader topic of interest is in doing this for other REPL modes, like the R one provided by using RCall
As you mentioned, the default way to do is via the run command. If you have not already, check out the docs on this https://docs.julialang.org/en/v1/manual/running-external-programs/#Running-External-Programs which go into some of the caveats.
I am not sure I follow what you are getting at with RCall but it may perhaps be worth opening a separate question for that.
You can find the code for this at https://github.com/JuliaLang/julia/tree/master/stdlib/REPL/test.
Seems there is no API, just lots of typing.
Here is a minimal working example (the codes are mostly copied from different places from the folder above):
using REPL
mutable struct FakeTerminal <: REPL.Terminals.UnixTerminal
in_stream::Base.IO
out_stream::Base.IO
err_stream::Base.IO
hascolor::Bool
raw::Bool
FakeTerminal(stdin,stdout,stderr,hascolor=true) =
new(stdin,stdout,stderr,hascolor,false)
end
REPL.Terminals.hascolor(t::FakeTerminal) = t.hascolor
REPL.Terminals.raw!(t::FakeTerminal, raw::Bool) = t.raw = raw
REPL.Terminals.size(t::FakeTerminal) = (24, 80)
input = Pipe()
output = Pipe()
err = Pipe()
Base.link_pipe!(input, reader_supports_async=true, writer_supports_async=true)
Base.link_pipe!(output, reader_supports_async=true, writer_supports_async=true)
Base.link_pipe!(err, reader_supports_async=true, writer_supports_async=true)
repl = REPL.LineEditREPL(FakeTerminal(input.out, output.in, err.in, false), false)
repltask = #async REPL.run_repl(repl)
Now you can do:
julia> println(input,";ls -la *.jld2")
-rw-r--r-- 1 pszufe 197121 5506 Jul 5 2020 file.jld2
-rw-r--r-- 1 pszufe 197121 5506 Jul 5 2020 myfile.jld2
I ran
mpiexec -n $nprocs julia --project myfile.jl
on a cluster, where myfile.jl has the following form
using Distributed; using Dates; using JLD2; using LaTeXStrings
#everywhere begin
using SharedArrays; using QuantumOptics; using LinearAlgebra; using Plots; using Statistics; using DifferentialEquations; using StaticArrays
#Defining some other functions and SharedArrays to be used later e.g.
MySharedArray=SharedArray{SVector{Nt,Float64}}(Np,Np)
end
#sync #distributed for pp in 1:Np^2
for jj in 1:Nj
#do some stuff with local variables
for tt in 1:Nt
#do some stuff with local variables
end
end
MySharedArray[pp]=... #using linear indexing
println("$pp finished")
end
timestr=Dates.format(Dates.now(), "yyyy-mm-dd-HH:MM:SS")
filename="MyName"*timestr
#save filename*".jld2"
#later on, some other small stuff like making and saving a figure. (This does give an error "no method matching heatmap_edges(::Surface{Array{Float64,2}}, ::Symbol)" but I think that this is a technical thing about Plots so not very related to the bigger issue here)
However, when looking at the output, there are a few issues that make me conclude that something is wrong
The "$pp finished" output is repeated many times for each value of pp. It seems that this amount is actually equal to 32=$nprocs
Despite the code not being finished, "MyName" files are generated. It should be one, but I get a dozen of them with different timestr component
EDIT: two more things that I can add
the output of the different "MyName" files is not identical, but this is expected since random numbers are used in the inner loops. There are 28 of them, a number that I don't easily recognize except that its again close to the 32 $nprocs
earlier, I wrote that the walltime was exceeded, but this turns out not to be true. The .o file ends with "BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES ... EXIT CODE :9", pretty shortly after the last output file.
$nprocs is obtained in the pbs script through
#PBS -l select=1:ncpus=32:mpiprocs=32
nprocs= `cat $PBS_NODEFILE|wc -l`
As pointed out by adamslc on the Julia discourse, the proper way to use Julia on a cluster is to either
Start a session with one core from the job script, add more with addprocs() in the Julia script itself
Use more specialized Julia packages
https://discourse.julialang.org/t/julia-distributed-redundant-iterations-appearing/57682/3
I have a series of sequential directories to gather files from on a linux server that I am logging into remotely and processing from an R terminal.
/r18_060, /r18_061, ... /r18_118, /r18_119
Each directory is for the day of year the data was logged on, and it contains a series of files with standard prefix such as "fl.060.gz"
I have to supply a function that contains multiple system() commands with a linux glob for the day. I want to divide the year into 60-day intervals to make the QA/QC more manageable. Since I'm crossing from 099 - 100 in the glob, I have to use brace expansion to match the correct sequence of days.
ls -d /root_driectory/r18_{0[6-9]?,1[0-1]?}
ls -d /root_driectory/r18_{060..119}
All of these work fine when I manually input these globs into my bash shell, but I get an error when the system() function provides a similar command through R.
day_glob <- {060..119}
system(paste("zcat /root_directory/r_18.", day_glob, "/fl.???.gz > tmpfile", sep = "")
>gzip: cannot access '/root_directory/r18_{060..119}': No such file or directory
I know that this could be an error in the shell that the system() function operates in, but when I query that it gives the correct environment and user name
system("env | grep ^SHELL=")
>SHELL=/bin/bash
system("echo $USER")
>tgw
Does anyone know why this fails when it is passed through R's system() command? What can I do to get around this problem without removing the system call altogether? There are many scripts that rely on these functions, and re-writing the entire family of R scripts would be time prohibitive.
Previously I had been using 50-day intervals which avoids this problem, but I thought this should be something easy to change, and make one less iteration of my QA/QC scripts per year. I'm new to the linux OS so I figured I might just be missing something obvious.
I have a script test.R that takes arguments arg1, arg2 and outputs a arg1-arg2.csv file.
I would like to run the test.R in 6 parallel sessions (i am on a 6 core CPU) and in the background. How can I do it?
I am on linux
I suggest using the doParallel backend for the foreach package. The foreach package provides a nice syntax to write loops and takes care of combining the results. doParallel connects it to the parallel package included since R 2.14. On other setups (older versions of R, clusters, whatever) you could simply change the backend without touching any of your foreach loops. The foreach package in particular has excellent documentation, so it is really easy to use.
If you are going to write the results to individual files, then the result-combining features of foreach won't be of much use to you. So people might argue that direct use of parallel would be better suited to your application. Personally I find the way foreach expresses looping concepts much easier to use.
You did not provide a reproducible example so I am making one up. As you are on Linux, I am also swicthing to littler which was after all writtten for the very purpose of scripting with R.
#!/usr/bin/env r
#
# a simple example to install one or more packages
if (is.null(argv) | length(argv) != 2) {
cat("Usage: myscript.r arg1 arg2\n")
q()
}
filename <- sprintf("%s-%s.csv", argv[1], argv[2])
Sys.sleep(60) # do some real work here instead
write.csv(matrix(rnorm(9), 3, 3), file=filename)
and you can then lauch this either from the command-line as I do here, or from another (shell) script. The key is the & at the end to send it in the background:
edd#max:/tmp/tempdir$ ../myscript.r a b &
[1] 19575
edd#max:/tmp/tempdir$ ../myscript.r c d &
[2] 19590
edd#max:/tmp/tempdir$ ../myscript.r e f &
[3] 19607
edd#max:/tmp/tempdir$
The [$n] indicates how process how been launched in the background, the number that follows is the process id which you can use to monitor or kill. After a little while we get the results:
edd#max:/tmp/tempdir$
[1] Done ../myscript.r a b
[2]- Done ../myscript.r c d
[3]+ Done ../myscript.r e f
edd#max:/tmp/tempdir$ ls -ltr
total 12
-rw-rw-r-- 1 edd edd 192 Jun 24 09:39 a-b.csv
-rw-rw-r-- 1 edd edd 193 Jun 24 09:40 c-d.csv
-rw-rw-r-- 1 edd edd 193 Jun 24 09:40 e-f.csv
edd#max:/tmp/tempdir$
You may want to read up on Unix shells to learn more about &m the fg and bg background s etc.
Lastly, all this can a) also be done with Rscript though picking arguments is slightly different and b) there are CRAN packages getopt and optparse to facilitate working with command-line arguments.
The state of art would be to use the parallel package, but when I am lazy, I simply start 6 batch (cmd, assuming Windows) files with rscript.
You can set parameters in the cmd-file
SET ARG1="myfile"
rscript rest.r
and read it from
Sys.getenv("ARG")
Using 6 batch files, I can also append multiple runs in one batch to be sure that the cores are always busy.
I am debugging a program in MacOSX,
and I need that this program thinks we are one year later than the one given by the operating system.
I cannot change the time of the operation system, because I need to run a second program concurrently with the correct time.
I could modify the code of the first program to add one year each time it gets the time from the operation system, but the code is too big to do that; I prefer not to use this solution.
I heard once that there is a command in Unix to run a program with a fake/mocked time.
Do you know about it?
I haven't tried it, but libfaketime claims to do what you need.
Quoted from the website:
As an example, we want the "date" command to report our faked time. To do so, we could use the following command line:
user#host> date
Tue Nov 23 12:01:05 CEST 2007
user#host> LD_PRELOAD=/usr/local/lib/libfaketime.so.1 FAKETIME="-15d" date
Mon Nov 8 12:01:12 CEST 2007
Assuming the lib works as advertised, you can trick your program into thinking it was running a year ahead using:
LD_PRELOAD=/usr/local/lib/libfaketime.so.1 FAKETIME="+1y" ./your_program