Home / server / Questions / 1166275
Accepted
Liquid
Asked: 2024-10-08 00:11:44 +0800 CST

Slurm not allocating GPUs properly

  • 772

I'm running slurm 21.08.5 installed through apt.

My cluster has 4 gpu machines (nd-gpu[001-005]) with 8 gpus each. I can run jobs as

srun --gres=gpu:8 nvidia-smi -L

And I see my gpus. I can also schedule real jobs with any from 0 to 8 gpus. However, the scheduling of resources isn't working properly. If I run:

 srun --gres=gpu:1 sleep 1000

The entire node will be allocated, and I won't be able to use the remaining 7 gpus on that node.

What follows is the gres.conf:

NodeName=nd-gpu001 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia7

the slurm.conf:

# See the slurm.conf man page for more information.  
#  
ClusterName=cluster  
SlurmctldHost=nd-cpu01  
SlurmctldHost=nd-cpu02  
#  

#GresTypes=  
#GroupUpdateForce=0  

ProctrackType=proctrack/cgroup  
ReturnToService=1  
SlurmctldPidFile=/var/run/slurmctld.pid  
SlurmctldPort=6817  
SlurmdPidFile=/var/run/slurmd.pid  
SlurmdPort=6818  
SlurmdSpoolDir=/var/spool/slurmd  
SlurmUser=slurm  
StateSaveLocation=/home/slurm/slurmctd  
TaskPlugin=task/affinity,task/cgroup  

# TIMERS  
InactiveLimit=0  
KillWait=30  

MinJobAge=300  

SlurmctldTimeout=120  
SlurmdTimeout=300  
Waittime=0  

# SCHEDULING  
SchedulerType=sched/backfill  
SelectType=select/cons_tres  

# LOGGING AND ACCOUNTING  
JobCompType=jobcomp/none  
JobAcctGatherFrequency=30  
SlurmctldDebug=info  
SlurmctldLogFile=/var/log/slurmctld.log  
SlurmdDebug=info  
SlurmdLogFile=/var/log/slurmd.log  

# COMPUTE NODES  
GresTypes=gpu
NodeName=nd-gpu[001-004] Sockets=2 CoresPerSocket=56 ThreadsPerCore=1 State=UNKNOWN Gres=gpu:8
PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP

and the cgroups.conf:

ConstrainDevices=yes
slurm

2 Answers

  1. I ran into the same problem when I set up my cluster.

    You already have SelectType=select/cons_tres, which is good.

    Additionally, you need:

    SelectTypeParameters=CR_Core_Memory

    and you need to set OverSubscribe on the partition.

    PartitionName=partitionName [other options ...] OverSubscribe=FORCE

    OverSubscribe defaults to NO, setting it to YES will allow Jobs to oversubscribe, but it has to be specified as an argument with each job. Setting it to FORCE will enable it for all jobs, which is usually what you want.

    (technically you only need OverSubscribe, but it has no effect until SelectTypeParameters is also set)

    • 1
  2. Best Answer

    The resolution for me was

    SchedulerType=sched/backfill  
SelectType=select/cons_tres 
SelectTypeParameters=CR_CPU

    Seems my local setup wasn't really able to assess how much memory each core could be allocated. Instead, the number of CPUs came in handy with the gres.conf and the nodes specification.

    • 0