slurm:all cpus in a node are allocated by a job which just need a subset of cpus - mpi

Question

I have every node configured as follow in slurm.conf
NodeName=node1 NodeAddr=xxx.xxx.xxx.xxx State=UNKNOWN Procs=32 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=128000 TmpDisk=65536
when I run the following command
srun -n 2 sleep 60
I found that all the core in a node would be allocated by this job. If another job want to run on this node, it would be bolcked until the previous job finishes.
scontrol show the job information as following
JobId=51 JobName=sleep
UserId=hadoop(1002) GroupId=hadoop(1002) MCS_label=N/A
Priority=4294901703 Nice=0 Account=hadoop QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
RunTime=00:00:12 TimeLimit=UNLIMITED TimeMin=N/A
SubmitTime=2018-07-16T21:46:56 EligibleTime=2018-07-16T21:46:56
StartTime=2018-07-16T21:46:56 EndTime=Unknown Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
LastSchedEval=2018-07-16T21:46:56
Partition=TOTAL AllocNode:Sid=node1:25124
ReqNodeList=(null) ExcNodeList=(null)
NodeList=xxx.xxx.xxx
BatchHost=xxx.xxx.xxx
NumNodes=1 NumCPUs=32 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=32,mem=125G,node=1,billing=32
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=125G MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
Gres=(null) Reservation=(null)
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=sleep
WorkDir=/home/hadoop
Power=
Use sacct to get the history jobs , I get the following output
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
51 sleep TOTAL hadoop 32 COMPLETED 0:0
51.0 sleep hadoop 2 COMPLETED 0:0
show the partition information:
PartitionName=TOTAL
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
AllocNodes=ALL Default=YES QoS=N/A
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0
Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO
MaxCPUsPerNode=UNLIMITED
Nodes=xxxxxxx
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=OFF
State=UP TotalCPUs=96 TotalNodes=3 SelectTypeParameters=NONE
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
It seems something wrong.

Answer 1

It's the problem casued by SelectType. I let it as the default value which I think is select/linear. As mentioned in Select Plugin Design Guide, select/linear is node-centric .
The select/linear and select/cons_res plugins have similar modes of operation. The obvious difference is that data structures in select/linear are node-centric, while those in select/cons_res contain information at a finer resolution (sockets, cores, threads, or CPUs depending upon the SelectTypeParameters configuration parameter).
I change SelectType to select/cons_res and restart the whole cluster, the problem is solved.