I'm trying to run this command with multiple instances of a program, like this:
mpirun -oversubscribe -tag-output -np 1 /home/rd636/Posdoc_Posner/BioNetFit2_pull1/bin/BioNetFit2 -a load -c /home/rd636/Posdoc_Posner/BioNetFit2_pull1/parabolaA_1681692777.sconf : -oversubscribe -tag-output -np 24 /home/rd636/Posdoc_Posner/BioNetFit2_pull1/bin/BioNetFit2 -t particle -p 0 -a run -c /home/rd636/Posdoc_Posner/BioNetFit2_pull1/parabolaA_1681692777.sconf
In one node this mpirun command works, but when I request more than one node in my sbatch scriipt, the messages that the 24 slave processes (from the other node) receive are empty.
I asked to the HPC manager about this, and he said that I should use "srun" to run my jobs.
But how do I convert that mpirun command into an srun command?
Here is my job submission script:
#SBATCH --job-name=parabola2_cluster_2x10 # the name of your job
#SBATCH --output=/home/rd636/Posdoc_Posner/BioNetFit2_pull1/parabola_log.txt # this is the file your output and errors go to
#SBATCH --time=1:30:00 # 1 hour and 30 min
#SBATCH --workdir=/home/rd636/Posdoc_Posner/BioNetFit2_pull1/bin # your work directory
#SBATCH --mem=2000 # 2GB of memory
abbreviated by -N
#SBATCH --ntasks=25 # number of MPI tasts (total number of cores), or (Swarm_Size*nModels)+1
module purge
module load gcc/5.4.0
module load openmpi/1.10.2
module load glibc/2.23
module load boost/1.65.0-gcc-5.4.0
time mpirun -oversubscribe -tag-output -np 1 /home/rd636/Posdoc_Posner/BioNetFit2_pull1/bin/BioNetFit2 -a load -c /home/rd636/Posdoc_Posner/BioNetFit2_pull1/parabolaA_1681692777.sconf : -oversubscribe -tag-output -np 24 /home/rd636/Posdoc_Posner/BioNetFit2_pull1/bin/BioNetFit2 -t particle -p 0 -a run -c /home/rd636/Posdoc_Posner/BioNetFit2_pull1/parabolaA_1681692777.sconf
echo "Synchronous GA, 24 subparticles, 12 particles, 2 models, 100 generations"
I can't use the multiprogram file approach because my command is too long, and changing to relative paths is not an option, unfortunately.
Related
Here is my slurm job script. I requested 4 gpu and 1 computing node. My script is as follows:
#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --gres=gpu:4
#SBATCH --ntasks-per-gpu=12
#SBATCH --mem-per-gpu=40G
#SBATCH --time=0:15:00
module use /ifs/opt_cuda/modulefiles
module load python/gcc/3.10
module load cuda11.1/toolkit cuda11.1/blas cuda11.1/fft cudnn8.0-cuda11.1 tensorrt-cuda11.1/7.2.3.4
# activate TF venv
source /ifs/groups/rweberGrp/venvs/py310-tf210/bin/activate
python -c "import torch;print(torch.cuda.device_count())"
so the torch.cuda.device_count() should give me 4 but actually the output is 0
0
I have no idea why this is happening. Anyone has any idea? Thanks
I would like to have 4 different mpi executions of same program, with different number of nodes. The outputs should be n_out.txt depends on the nodes. I have tried the following .sh file,
#!/bin/bash
#SBATCH --partition=halley
#SBATCH --job-name=deniz
#SBATCH --output=out.txt
#SBATCH --nodes=16
mpicc parallelTournament.c -o parallelTournament -lm
mpiexec -n 16 --output-filename 16_out.txt ./parallelTournament 16
mpiexec -n 8 --output-filename 8_out.txt ./parallelTournament 16
mpiexec -n 4 --output-filename 4_out.txt ./parallelTournament 16
mpiexec -n 1 --output-filename 1_out.txt ./parallelTournament 16
But it gives an error:
A call to mkdir was unable to create the desired directory:
Directory: /home/16_out.txt/1
Error: Not a directory
Please check to ensure you have adequate permissions to perform
the desired operation.
I understand that I can not specify the output file name with --output-filename, but assign the directory that it should be located.
How can I have n_out.txt as input for all executions?
I'm trying to get a code to run in parallel using mpirun, in the PBS queue. The script I'm using is below:
#!/bin/bash
#PBS -S /bin/bash
#PBS -l nodes=1:ppn=4
#PBS -l walltime=4:10:00
#PBS -N job
#PBS -j oe
/usr/bin/mpirun -np 4 a.out
where a.out is the executable. The problem is that the mpirun I'm using is the one on the main node of the cluster. I want to use an mpirun installed on my own compute nodes, which is also in /usr/bin/mpirun.
I tried just copying mpirun in the folder and running with the line
./mpirun -np 4 hello.out
and that worked for some simple "hello world" program, but it doesn't work for the more complex code. It also works from my compute node if I just type
/usr/bin/mpirun -np 4 a.out
How do I use the mpirun of the compute node in the PBS script? I simply cannot use the stuff on the main node because it's an old linux version I'm not ready to change.
I have installed openmpi and slurm in two nodes. i want to use slurm to run mpi jobs. When i use srun to run non-mpi jobs, everything is ok. However, i got some errors when i use salloc to run mpi jobs. Environment and codes are as follows.
Env:
slurm 17.02.1-2
mpirun (Open MPI) 2.1.0
test.sh
#!/bin/bash
MACHINEFILE="nodes.$SLURM_JOB_ID"
# Generate Machinefile for mpich such that hosts are in the same
# order as if run via srun
#
srun -l /bin/hostname | sort -n | awk '{print $2}' > $MACHINEFILE
source /home/slurm/allreduce/tf/tf-allreduce/bin/activate
mpirun -np $SLURM_NTASKS -machinefile $MACHINEFILE test
rm $MACHINEFILE
command
salloc -N2 -n2 bash test.sh
ERROR
salloc: Granted job allocation 97
--------------------------------------------------------------------------
An ORTE daemon has unexpectedly failed after launch and before
communicating back to mpirun. This could be caused by a number
of factors, including an inability to create a connection back
to mpirun due to a lack of common network interfaces and/or no
route found between them. Please check network connectivity
(including firewalls and network routing requirements).
--------------------------------------------------------------------------
salloc: Relinquishing job allocation 97
Anyone can help? Thanks.
Since the beginning of November, I'm stuck in to run a parallel job in a Linux cluster. I already search A LOT on the internet searching for information but I simply can't progress. When I start to search for parallelism in R using cluster I discovered the Rmpi. It looked quite simple, but now I don't now more what to do. I have a script to send my job:
#PBS -S /bin/bash
#PBS -N ANN_residencial
#PBS -q linux.q
#PBS -l nodes=8:ppn=8
cd $PBS_O_WORKDIR
source /hpc/modulos/bash/R-3.3.0.sh
export LD_LIBRARY_PATH=/hpc/nlopt-2.4.2/lib:$LD_LIBRARY_PATH
export CPPFLAGS='-I/hpc/nlopt-2.4.2/include '$CPPFLAGS
export PKG_CONFIG_PATH=/hpc/nlopt-2.4.2/lib/pkgconfig:$PKG_CONFIG_PATH
# OPENMPI 1.10 + GCC 5.3
source /hpc/modulos/bash/openmpi-1.10-gcc53.sh
mpiexec --mca orte_base_help_aggregate 0 -np 1 -hostfile ${PBS_NODEFILE} /hpc/R-3.3.0/bin/R --slave -f sunhpc_mpi.r
And this is the beginning of my R program:
library(caret)
library(Rmpi)
library(doMPI)
cl <- startMPIcluster()
registerDoMPI(cl)
So here is my questions:
1- Is this way I should initialize the processes (i.e. using starMPIcluster whitout a parameter and using at the command line -np 1)?
2- Why when I use this commands the MPI complains with it's frase?
An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process....
OBS: He said that for all the 64 processes (because there are 8 nodes with 8 cpus and I'm creating 63 processes)
3- Why when I use this commands on a machine of 60 CPU's he just spawn two workers?
Finally, I got it!
To run a parallel program in R using the Rmpi in a cluster you need to configure the job script according to the system. Next on the command line:
mpiexec --mca orte_base_help_aggregate 0 -np 1 -hostfile ${PBS_NODEFILE} /hpc/R-3.3.0/bin/R --slave -f sunhpc_mpi.r
You have to modify to:
mpiexec -np NUM_PROC -hostfile ${PBS_NODEFILE} /hpc/R-3.3.0/bin/R --slave -f sunhpc_mpi.r
On the R code, you must not detail anything 'startMPIcluster()' So, the code will exactly as I wrote above.