Questions[slurm](server)

I'm running slurm 21.08.5 installed through apt.

My cluster has 4 gpu machines (nd-gpu[001-005]) with 8 gpus each. I can run jobs as

srun --gres=gpu:8 nvidia-smi -L

And I see my gpus. I can also schedule real jobs with any from 0 to 8 gpus. However, the scheduling of resources isn't working properly. If I run:

 srun --gres=gpu:1 sleep 1000

The entire node will be allocated, and I won't be able to use the remaining 7 gpus on that node.

What follows is the gres.conf:

NodeName=nd-gpu001 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu001 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu002 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu003 Name=gpu File=/dev/nvidia7
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia0
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia1
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia2
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia3
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia4
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia5
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia6
NodeName=nd-gpu004 Name=gpu File=/dev/nvidia7

the slurm.conf:

# See the slurm.conf man page for more information.  
#  
ClusterName=cluster  
SlurmctldHost=nd-cpu01  
SlurmctldHost=nd-cpu02  
#  

#GresTypes=  
#GroupUpdateForce=0  

ProctrackType=proctrack/cgroup  
ReturnToService=1  
SlurmctldPidFile=/var/run/slurmctld.pid  
SlurmctldPort=6817  
SlurmdPidFile=/var/run/slurmd.pid  
SlurmdPort=6818  
SlurmdSpoolDir=/var/spool/slurmd  
SlurmUser=slurm  
StateSaveLocation=/home/slurm/slurmctd  
TaskPlugin=task/affinity,task/cgroup  

# TIMERS  
InactiveLimit=0  
KillWait=30  

MinJobAge=300  

SlurmctldTimeout=120  
SlurmdTimeout=300  
Waittime=0  

# SCHEDULING  
SchedulerType=sched/backfill  
SelectType=select/cons_tres  

# LOGGING AND ACCOUNTING  
JobCompType=jobcomp/none  
JobAcctGatherFrequency=30  
SlurmctldDebug=info  
SlurmctldLogFile=/var/log/slurmctld.log  
SlurmdDebug=info  
SlurmdLogFile=/var/log/slurmd.log  

# COMPUTE NODES  
GresTypes=gpu
NodeName=nd-gpu[001-004] Sockets=2 CoresPerSocket=56 ThreadsPerCore=1 State=UNKNOWN Gres=gpu:8
PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP

and the cgroups.conf:

ConstrainDevices=yes

Is there any way to clear stale allocated GRES in Slurm?

I have one node where 4 GPUs are allocated while no jobs are running on the node. Rebooting the node does not release the GPUs.

user@control1:~$ scontrol show node node2
NodeName=node2 Arch=x86_64 CoresPerSocket=64
   CPUAlloc=0 CPUTot=256 CPULoad=0.05
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=gpu:tesla:8
   NodeAddr=node2 NodeHostName=node2 Version=21.08.5
   OS=Linux 5.15.0-83-generic #92-Ubuntu SMP Mon Aug 14 09:30:42 UTC 2023
   RealMemory=1025596 AllocMem=0 FreeMem=1025887 Sockets=2 Boards=1
   State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=DEFAULT
   BootTime=2023-09-19T12:58:23 SlurmdStartTime=2023-09-19T12:59:06
   LastBusyTime=2023-09-19T12:59:07
   CfgTRES=cpu=256,mem=1025596M,billing=256,gres/gpu=8,gres/gpu:tesla=8
   AllocTRES=gres/gpu=4
   CapWatts=n/a
   CurrentWatts=0 AveWatts=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

user@control1:~$ squeue -w node2 --state=all
   JOBID     NAME     USER    STATE         TIME        QOS PRIORITY   TIME_LIMIT NODELIST(REASON)

I've just set up slurm where one physical machine will be the only system in the cluster (so far). This is on Ubuntu 18.04.

I have slurmdbd running, but when I attempt to start up slurmd and slurmctld this times out. Why?

I'm issuing the following commands:

systemctl start slurmctld
systemctl start slurmd

I've also tried:

systemctl start slurmctld slurmd

and:

systemctl start slurmd slurmctld

This fails with the following, for slurmctld:

systemd[1]: slurmd.service: Can't open PID file /var/run/slurm-llnl/slurm-llnl/slurmd.pid (yet?) after start: No such file or directory
systemd[1]: slurmctld.service: Start operation timed out. Terminating.
systemd[1]: slurmctld.service: Failed with result 'timeout'.
systemd[1]: Failed to start Slurm controller daemon.

And for slurmd:

systemd[1]: slurmd.service: Start operation timed out. Terminating.
systemd[1]: slurmd.service: Failed with result 'timeout'.
systemd[1]: Failed to start Slurm node daemon.

However, when I start these manually (using two terminals) by issuing:

slurmctld -Dvvv
slurmd -Dvvv

Everything appears to work.

Why is this? How am I supposed to start slurm?

These are the service files (which should be standard, I didn't touch them except for adding verbose arguments, but then removing them again later):

# cat /lib/systemd/system/slurmd.service 
[Unit]
Description=Slurm node daemon
After=network.target munge.service
ConditionPathExists=/etc/slurm-llnl/slurm.conf
Documentation=man:slurmd(8)

[Service]
Type=forking
EnvironmentFile=-/etc/default/slurmd
ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/var/run/slurm-llnl/slurmd.pid
KillMode=process
LimitNOFILE=51200
LimitMEMLOCK=infinity
LimitSTACK=infinity

[Install]
WantedBy=multi-user.target
# cat /lib/systemd/system/slurmctld.service 
[Unit]
Description=Slurm controller daemon
After=network.target munge.service
ConditionPathExists=/etc/slurm-llnl/slurm.conf
Documentation=man:slurmctld(8)

[Service]
Type=forking
EnvironmentFile=-/etc/default/slurmctld
ExecStart=/usr/sbin/slurmctld $SLURMCTLD_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/var/run/slurm-llnl/slurmctld.pid

[Install]
WantedBy=multi-user.target

I followed the steps to troubleshoot here: https://slurm.schedmd.com/troubleshoot.html.

When running scontrol show slurmd, I get:

Active Steps             = NONE
Actual CPUs              = 1
Actual Boards            = 1
Actual sockets           = 1
Actual cores             = 1
Actual threads per core  = 1
Actual real memory       = 984 MB
Actual temp disk space   = 492 MB
Boot time                = 2019-03-27T17:53:56
Hostname                 = fedora2
Last slurmctld msg time  = NONE
Slurmd PID               = 1549
Slurmd Debug             = 4
Slurmd Logfile           = /var/log/slurmd.log
Version                  = 17.11.13-2

I don't know why slurmd on fedora2 can't communicate with the controller on fedora1. slurmctld daemon is running fine on fedora1.

The slurm.conf is as follows:

# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
#SlurmctldHost=fedora1
#
ControlMachine=fedora1
ControlAddr=192.168.1.4 
MailProg=/bin/mail 
MpiDefault=none
#MpiParams=ports=#-# 
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurm/slurmctld.pid
#SlurmctldPort=6817 
SlurmdPidFile=/var/run/slurm/slurmd.pid
#SlurmdPort=6818 
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
SlurmdUser=root 
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
TaskPlugin=task/affinity
# 
# 
# TIMERS 
#KillWait=30 
#MinJobAge=300 
#SlurmctldTimeout=120 
#SlurmdTimeout=300 
# 
# 
# SCHEDULING 
FastSchedule=1
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core
# 
# 
# LOGGING AND ACCOUNTING 
AccountingStorageType=accounting_storage/none
ClusterName=fedora
#JobAcctGatherFrequency=30 
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=verbose 
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=verbose 
SlurmdLogFile=/var/log/slurmd.log
# 
# 
# COMPUTE NODES 
NodeName=fedora1 NodeAddr=192.168.1.4 CPUs=1 State=UNKNOWN
NodeName=fedora2 NodeAddr=192.168.1.5 CPUs=1 State=UNKNOWN 
PartitionName=debug Nodes=fedora[1-2] Default=YES MaxTime=INFINITE State=UP

The output of tail /var/log/slurmd.log on fedora2, on multiple lines:

error: Unable to register: Unable to contact slurm controller (connect failure)

I run systemctl start slurmd.service, and it times out:

Job for slurmd.service failed because a timeout was exceeded.

The relevant lines from running systemctl status slurmd.service:

Mar 23 17:13:42 fedora1 systemd[1]: Starting Slurm node daemon...
Mar 23 17:13:43 fedora1 systemd[1]: slurmd.service: Can't open PID file /var/run/slurm/slurmd.pid (yet?) after start: No such file or directory
Mar 23 17:15:11 fedora1 systemd[1]: slurmd.service: Start operation timed out. Terminating.
Mar 23 17:15:11 fedora1 systemd[1]: slurmd.service: Failed with result 'timeout'.
Mar 23 17:15:11 fedora1 systemd[1]: Failed to start Slurm node daemon.

The contents of the slurm.conf file:

# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
#SlurmctldHost=fedora1
#
ControlMachine=fedora1
ControlAddr=192.168.1.4
#MailProg=/bin/mail 
MpiDefault=none
#MpiParams=ports=#-# 
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
#SlurmctldPort=6817 
SlurmdPidFile=/var/run/slurmd.pid
#SlurmdPort=6818 
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
#SlurmdUser=root 
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
TaskPlugin=task/affinity
# 
# 
# TIMERS 
#KillWait=30 
#MinJobAge=300 
#SlurmctldTimeout=120 
#SlurmdTimeout=300 
# 
# 
# SCHEDULING 
FastSchedule=1
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core
# 
# 
# LOGGING AND ACCOUNTING 
AccountingStorageType=accounting_storage/none
ClusterName=fedora
#JobAcctGatherFrequency=30 
JobAcctGatherType=jobacct_gather/none
#SlurmctldDebug=info 
SlurmctldLogFile=/var/log/slurmctld.log
#SlurmdDebug=info 
SlurmdLogFile=/var/log/slurmd.log
# 
# 
# COMPUTE NODES 
NodeName=fedora1 NodeAddr=192.168.1.4 CPUs=1 State=UNKNOWN
NodeName=fedora2 NodeAddr=192.168.1.5 CPUs=1 State=UNKNOWN
PartitionName=debug Nodes=fedora[1-2] Default=YES MaxTime=INFINITE State=UP