I'm trying to run a SLURM sbatch command with various parameters that I can read in an R script. When using PBS system, I used to write qsub -v param1=x,param2=y (+ other system parameters like the memory requirements etc and the script name to be read by PBS) and then in the R script read it with x = Sys.getenv(‘param1’).
Now I tried
sbatch run.sh --export=basePath=‘a’
With run.sh:
#!/bin/bash
cd $SLURM_SUBMIT_DIR
echo $PWD
module load R/common/3.3.3
R CMD BATCH --quiet --no-restore --no-save runDo.R output.txt
And runDo.R:
base.path = Sys.getenv('basePath')
print(base.path)
The script is running but the argument value is not assigned to base.path variable (it prints an empty string).
The export parameter has to be passed to sbatch not to the run.sh script.
It should be like this:
sbatch --export=basePath=‘a’ run.sh
Related
Does tcsh support launching itself in a remote directory via an argument?
The setup I am dealing with does not allow me to chdir to the remote directory before invoking tcsh, and I'd like to avoid having to create a .sh file for this workflow.
Here are the available arguments I see for v6.19:
> tcsh --help
tcsh 6.19.00 (Astron) 2015-05-21 (x86_64-unknown-Linux) options wide,nls,dl,al,kan,rh,color,filec
-b file batch mode, read and execute commands from 'file'
-c command run 'command' from next argument
-d load directory stack from '~/.cshdirs'
-Dname[=value] define environment variable `name' to `value' (DomainOS only)
-e exit on any error
-f start faster by ignoring the start-up file
-F use fork() instead of vfork() when spawning (ConvexOS only)
-i interactive, even when input is not from a terminal
-l act as a login shell, must be the only option specified
-m load the start-up file, whether or not owned by effective user
-n file no execute mode, just check syntax of the following `file'
-q accept SIGQUIT for running under a debugger
-s read commands from standard input
-t read one line from standard input
-v echo commands after history substitution
-V like -v but including commands read from the start-up file
-x echo commands immediately before execution
-X like -x but including commands read from the start-up file
--help print this message and exit
--version print the version shell variable and exit
This works, but is suboptimal because it launches two instances of tcsh:
tcsh -c 'cd /tmp && tcsh'
I have to run a shell script inside R. I've considered using R's system function.
However, my script involves source activate and other commands that are not available in /bin/sh shell. Is there a way I can use /bin/bash instead?
Thanks!
Invoke /bin/bash, and pass the commands via -c option in one of the following ways:
system(paste("/bin/bash -c", shQuote("Bash commands")))
system2("/bin/bash", args = c("-c", shQuote("Bash commands")))
If you only want to run a Bash file, supply it with a shebang, e.g.:
#!/bin/bash -
builtin printf %q "/tmp/a b c"
and call it by passing script's path to the system function:
system("/path/to/script.sh")
It is implied that the current user/group has sufficient permissions to execute the script.
Rationale
Previously I suggested to set the SHELL environment variable. But it probably won't work, since the implementation of the system function in R calls the C function with the same name (see src/main/sysutils.c):
int R_system(const char *command)
{
/*... */
res = system(command);
And
The system() library function uses fork(2) to create a child process that executes the shell command specified in command using execl(3) as follows:
execl("/bin/sh", "sh", "-c", command, (char *) 0);
(see man 3 system)
Thus, you should invoke /bin/bash, and pass the script body via the -c option.
Testing
Let's list the top-level directories in /tmp using the Bash-specific mapfile:
test.R
script <- '
mapfile -t dir < <(find /tmp -mindepth 1 -maxdepth 1 -type d)
for d in "${dir[#]}"
do
builtin printf "%s\n" "$d"
done > /tmp/out'
system2("/bin/bash", args = c("-c", shQuote(script)))
test.sh
Rscript test.R && cat /tmp/out
Sample Output
/tmp/RtmpjJpuzr
/tmp/fish.ruslan
...
Original Answer
Try to set the SHELL environment variable:
Sys.setenv(SHELL = "/bin/bash")
system("command")
Then the commands passed to system or system2 functions should be invoked using the specified shell.
please could you advise me on the following : I've written a R script that reads 3 arguments from the command line, i.e. :
args <- commandArgs(TRUE)
TUMOR <- args[1]
GERMLINE <- args[2]
CHR <- args[3]
when I submit the R script to a PBS HPC scheduler, I do the following (below), but ... I am getting an error message.
(I am not posting the error message, because the R script I wrote works fine when it is run from a regular terminal ..)
Please may I ask, how do you usually submit the R scripts with command line arguments to PBS HPC schedulers ?
qsub -d $PWD -l nodes=1:ppn=4 -l vmem=10gb -m bea -M tanasa#gmail.com \
-v TUMOR="tumor.bam",GERMLINE="germline.bam",CHR="chr22" \
-e script.efile.chr22 \
-o script.ofile.chr22 \
script.R
I have to run a shell script inside R. I've considered using R's system function.
However, my script involves source activate and other commands that are not available in /bin/sh shell. Is there a way I can use /bin/bash instead?
Thanks!
Invoke /bin/bash, and pass the commands via -c option in one of the following ways:
system(paste("/bin/bash -c", shQuote("Bash commands")))
system2("/bin/bash", args = c("-c", shQuote("Bash commands")))
If you only want to run a Bash file, supply it with a shebang, e.g.:
#!/bin/bash -
builtin printf %q "/tmp/a b c"
and call it by passing script's path to the system function:
system("/path/to/script.sh")
It is implied that the current user/group has sufficient permissions to execute the script.
Rationale
Previously I suggested to set the SHELL environment variable. But it probably won't work, since the implementation of the system function in R calls the C function with the same name (see src/main/sysutils.c):
int R_system(const char *command)
{
/*... */
res = system(command);
And
The system() library function uses fork(2) to create a child process that executes the shell command specified in command using execl(3) as follows:
execl("/bin/sh", "sh", "-c", command, (char *) 0);
(see man 3 system)
Thus, you should invoke /bin/bash, and pass the script body via the -c option.
Testing
Let's list the top-level directories in /tmp using the Bash-specific mapfile:
test.R
script <- '
mapfile -t dir < <(find /tmp -mindepth 1 -maxdepth 1 -type d)
for d in "${dir[#]}"
do
builtin printf "%s\n" "$d"
done > /tmp/out'
system2("/bin/bash", args = c("-c", shQuote(script)))
test.sh
Rscript test.R && cat /tmp/out
Sample Output
/tmp/RtmpjJpuzr
/tmp/fish.ruslan
...
Original Answer
Try to set the SHELL environment variable:
Sys.setenv(SHELL = "/bin/bash")
system("command")
Then the commands passed to system or system2 functions should be invoked using the specified shell.
I am trying to run subjobs (one for each chromosome) using R --vanilla. Since each chromosome is independent I want them to run parallel in the system. I have written the following script:
#!/bin/bash
for K in {20..21};
do
qsub -V -cwd -b y -q short.q R --vanilla --args arg1$K arg2$K arg3$K < RareMETALS.R > loggroup$K.txt; done
But somehow R opens interactively and not in command line as suppose... when trying the script itself
R --vanilla --args arg1 arg2 arg3 < RareMETALS.R > loggroup.txt; done
It runs perfectly calling the script.
Can someboby guide me, or point out which might be the problem.
My take on this would be to use echo instead of --args option to pass parameters to the script. I find separating the script and the Grid Engine code to be more straightforward:
for K in {20..21};
do
echo "Rscript RareMETALS.R arg1$K arg2$K arg3$K > loggroup$K.txt" | qsub -V -cwd -q short.q
done
As others have commented use Rscript.
Code seems cleaner to me, but there may be some limitations to using echo as opposed to --args I am unaware of.