Home / ubuntu / Questions / 1114612
In Process
Greg
Asked: 2019-02-01 20:36:33 +0800 CST

rcu_sched self-detected stall on CPU + watchdog: BUG: soft lockup - CPU#3 stuck for 22s

  • 772

Being unable to ssh into a machine I connected it to a monitor and found the following:

enter image description here

The machine is running Ubuntu Server 18.04 LTS and is a first generation 8 core Ryzen 1700. I've restarted the machine since and it works fine but am not sure what caused this in the first place and want to avoid it happening again.

enter image description here

enter image description here

kernel lockup watchdog scheduler

1 Answers

  1. From: What are all these "Bug: soft lockup" messages about?

    Situation

    In the system log (/var/log/messages or journalctl) a lot of the following messages is printed.

    May 25 07:23:59 XXXXXXX kernel: [13445315.881356] BUG: soft lockup - CPU#16 stuck for 23s! [yyyyyyy:81602]

    followed by various stack traces. This document tries to explain what the soft lockup messages mean.

    The error message itself doesn't tell you what is causing the problem.

    Cause

    A 'soft lockup' is defined as a bug that causes the kernel to loop in kernel mode for more than 20 seconds, without giving other tasks a chance to run. The watchdog daemon will send an non maskable interrupt (NMI) to all CPUs in the system who in turn print the stack traces of their currently running tasks.

    Reducing server load is normal solution:

    Resolution

    Under normal circumstances those messages may go away if the load decreased. This 'soft lockup' can happen if the kernel is busy, working on a huge amount of objects which need to be scanned, freed or allocated respectively. The stack traces of those tasks can give a first idea what the tasks were doing. However, to be able to examine the cause behind the messages, a kernel dump would be needed.

    You cannot disable those messages, however in some situations increasing the time when those soft lockups will be fired can relax the situation.

    Do do so just increase the following sysctl parameter: kernel.watchdog_thresh Default value for this parameter is 10 and to double the value might be a good start.

    • 4