I am running a long job using a cluster of computers. On occasion, the process is interrupted and I have to manually restart. There is considerable downtime when the interruptions occur overnight. I was wondering if there is a way run a supervisor script in Julia that monitors whether the job running in another instance of Julia. It would restart the process if it is interrupted and would terminate once the job is finished. Unfortunately, I do not know exactly how to check that the process is running and how to restart the process. Here is the rough idea I have:
state = true
while state == true
#check every minute
sleep(60)
data = readcsv("outputfile.csv")
#read file to check if process is finished
if size(data,1) < N
#some function to check if the process is running
if isrunning() == true
#Do nothing.Keep running
else
#some function to spawn new instance of julia
#run the code
include("myscript.jl")
end
else
#Job finished, exit while loop
state = false
end
end
Right tool for the right Job.
Use your commandline shell.
If something it untimely terminated, it will give a error status code.
Eg Bash
until julia myscript.jl;
do echo "Failed/Interrupted. Restarting in 5s. Press Ctrl-C now to interrupt.";
sleep 5;
done`
Because Julia is not unuable as a commandline runner you could do, in julia:
while true
try
run(`julia myscript.jl`) #Run a separate process
break
catch
println("Failed/Interrupted. Restarting in 5s. Press Ctrl-C now to interrupt.")
sleep(5)
end
end
Related
I am writing an R code on a Linux system using RStudio. At some point in the code, I need to use a system call to a command that will download a few thousand of files from the lines of a text file:
down.command <- paste0("parallel --gnu -a links.txt wget")
system(down.command)
However, this command takes a little while to run (a couple of hours), and the R prompt stays locked while the command runs. I would like to keep using R while the command runs on the background.
I tried to use nohup like this:
down.command <- paste0("nohup parallel --gnu -a links.txt wget > ~/down.log 2>&1")
system(down.command)
but the R prompt still gets "locked" waiting for the end of the command.
Is there any way to circumvent this? Is there a way to submit system commands from R and keep them running on the background?
Using ‘processx’, here’s how to create a new process that redirects both stdout and stderr to the same file:
args = c('--gnu', '-a', 'links.txt', 'wget')
p = processx::process$new('parallel', args, stdout = '~/down.log', stderr = '2>&1')
This launches the process and resumes the execution of the R script. You can then interact with the running process via the p name. Notably you can signal to it, you can query its status (e.g. is_alive()), and you can synchronously wait for its completion (optionally with a timeout after which to kill it):
p$wait()
result = p$get_exit_status()
Based on the comment by #KonradRudolph, I became aware of the processx R package that very smartly deals with system process submissions from within R.
All I had to do was:
library(processx)
down.command <- c("parallel","--gnu", "-a", "links.txt", "wget", ">", "~/down.log", "2>&1")
processx::process$new("nohup", down.comm, cleanup=FALSE)
As simple as that, and very effective.
I was wondering if in R there is a way/keyboard shortcut/command/processing-thread-killing-alternative. For instance, using RStudio as IDE, you got a little stop button to kill the current processing thread(s), but most of the times it ends up in the prompt to end the session and restart the environment. Suggestions?
Ctrl + C <- will stop the R process without exiting the R session
and not
Ctrl + Z <- will stop the R process and exit the R session
When I run Pkg.update(), Pkg.status(), sometimes, Julia hangs there. Is there a way to stop the Pkg-related command? I tried C-c, C-c, C-d, C-q, none of them works.
Is there a way to stop those command?
Pkg.xxx() starts a number of detached tasks which are not killed even when the Pkg command is interrupted using Ctrl-C. To kill these tasks, use interrupt(workers()).
It looks like the behaviour of Ctrl-C might be improved in a future version of Julia with RFC14032.
I want R to shutdown my computer after my (extensive) simulation and saving results, is this possible?
Yes, look at the function shutdown in the package fun.
The flags for system command shutdown depends on your operating system, the function simply calls the appropriately flagged command.
fun::shutdown
function (wait = 0)
{
Sys.sleep(wait)
ifelse(.Platform$OS.type == "windows", shell("shutdown -s -t 0"),
system("shutdown -h now"))
}
R can send commands to the system with ?system, and so whatever is required for Windows can be done with that:
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/shutdown.mspx?mfr=true
R has a .Last() function controlled by quit() (or q()) with its runLast argument, so this is where you would send the shutdown commands via system, so that it occurs after quitting R. Saving objects with R is done with save or save.image, though there is a default to save as well with quit().
Executing
system("shutdown -s")
will shut your computer down. Just add it at the end of your script
I want R to shutdown my computer after my (extensive) simulation and saving results, is this possible?
Yes, look at the function shutdown in the package fun.
The flags for system command shutdown depends on your operating system, the function simply calls the appropriately flagged command.
fun::shutdown
function (wait = 0)
{
Sys.sleep(wait)
ifelse(.Platform$OS.type == "windows", shell("shutdown -s -t 0"),
system("shutdown -h now"))
}
R can send commands to the system with ?system, and so whatever is required for Windows can be done with that:
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/shutdown.mspx?mfr=true
R has a .Last() function controlled by quit() (or q()) with its runLast argument, so this is where you would send the shutdown commands via system, so that it occurs after quitting R. Saving objects with R is done with save or save.image, though there is a default to save as well with quit().
Executing
system("shutdown -s")
will shut your computer down. Just add it at the end of your script