user3273814's questions

I followed the steps to troubleshoot here: https://slurm.schedmd.com/troubleshoot.html.

When running scontrol show slurmd, I get:

Active Steps             = NONE
Actual CPUs              = 1
Actual Boards            = 1
Actual sockets           = 1
Actual cores             = 1
Actual threads per core  = 1
Actual real memory       = 984 MB
Actual temp disk space   = 492 MB
Boot time                = 2019-03-27T17:53:56
Hostname                 = fedora2
Last slurmctld msg time  = NONE
Slurmd PID               = 1549
Slurmd Debug             = 4
Slurmd Logfile           = /var/log/slurmd.log
Version                  = 17.11.13-2

I don't know why slurmd on fedora2 can't communicate with the controller on fedora1. slurmctld daemon is running fine on fedora1.

The slurm.conf is as follows:

# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
#SlurmctldHost=fedora1
#
ControlMachine=fedora1
ControlAddr=192.168.1.4 
MailProg=/bin/mail 
MpiDefault=none
#MpiParams=ports=#-# 
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurm/slurmctld.pid
#SlurmctldPort=6817 
SlurmdPidFile=/var/run/slurm/slurmd.pid
#SlurmdPort=6818 
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
SlurmdUser=root 
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
TaskPlugin=task/affinity
# 
# 
# TIMERS 
#KillWait=30 
#MinJobAge=300 
#SlurmctldTimeout=120 
#SlurmdTimeout=300 
# 
# 
# SCHEDULING 
FastSchedule=1
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core
# 
# 
# LOGGING AND ACCOUNTING 
AccountingStorageType=accounting_storage/none
ClusterName=fedora
#JobAcctGatherFrequency=30 
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=verbose 
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=verbose 
SlurmdLogFile=/var/log/slurmd.log
# 
# 
# COMPUTE NODES 
NodeName=fedora1 NodeAddr=192.168.1.4 CPUs=1 State=UNKNOWN
NodeName=fedora2 NodeAddr=192.168.1.5 CPUs=1 State=UNKNOWN 
PartitionName=debug Nodes=fedora[1-2] Default=YES MaxTime=INFINITE State=UP

The output of tail /var/log/slurmd.log on fedora2, on multiple lines:

error: Unable to register: Unable to contact slurm controller (connect failure)

I run systemctl start slurmd.service, and it times out:

Job for slurmd.service failed because a timeout was exceeded.

The relevant lines from running systemctl status slurmd.service:

Mar 23 17:13:42 fedora1 systemd[1]: Starting Slurm node daemon...
Mar 23 17:13:43 fedora1 systemd[1]: slurmd.service: Can't open PID file /var/run/slurm/slurmd.pid (yet?) after start: No such file or directory
Mar 23 17:15:11 fedora1 systemd[1]: slurmd.service: Start operation timed out. Terminating.
Mar 23 17:15:11 fedora1 systemd[1]: slurmd.service: Failed with result 'timeout'.
Mar 23 17:15:11 fedora1 systemd[1]: Failed to start Slurm node daemon.

The contents of the slurm.conf file:

# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
#SlurmctldHost=fedora1
#
ControlMachine=fedora1
ControlAddr=192.168.1.4
#MailProg=/bin/mail 
MpiDefault=none
#MpiParams=ports=#-# 
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
#SlurmctldPort=6817 
SlurmdPidFile=/var/run/slurmd.pid
#SlurmdPort=6818 
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
#SlurmdUser=root 
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
TaskPlugin=task/affinity
# 
# 
# TIMERS 
#KillWait=30 
#MinJobAge=300 
#SlurmctldTimeout=120 
#SlurmdTimeout=300 
# 
# 
# SCHEDULING 
FastSchedule=1
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core
# 
# 
# LOGGING AND ACCOUNTING 
AccountingStorageType=accounting_storage/none
ClusterName=fedora
#JobAcctGatherFrequency=30 
JobAcctGatherType=jobacct_gather/none
#SlurmctldDebug=info 
SlurmctldLogFile=/var/log/slurmctld.log
#SlurmdDebug=info 
SlurmdLogFile=/var/log/slurmd.log
# 
# 
# COMPUTE NODES 
NodeName=fedora1 NodeAddr=192.168.1.4 CPUs=1 State=UNKNOWN
NodeName=fedora2 NodeAddr=192.168.1.5 CPUs=1 State=UNKNOWN
PartitionName=debug Nodes=fedora[1-2] Default=YES MaxTime=INFINITE State=UP

When I run slurmd, it gives a -bash: slurmd: command not found.

I ran sudo yum install slurm to install slurm. I don't know why it isn't working, or if I installed all the required packages for slurm.