Home / ubuntu / Questions / 1203022
In Process
Nyxynyx
Asked: 2020-01-15 14:20:03 +0800 CST

System has 128 GB RAM, but 80 GB is Used Immediately After Reboot

  • 772

My desktop system has 128 GB of RAM, which I expect most to be free after booting Ubuntu. However, free -m shows that 80 GB is in use, with only 40 GB free. Note that buff/cache is only under 1 GB, so this is not the cause of the high memory usage.

              total        used        free      shared  buff/cache   available
Mem:           125G         80G         44G         20M        825M         42G
Swap:            8G          0B          8G

Output of ps aux | awk '{sum+=$6} END {print sum / 1024}'

1368.03

Based on another SO question, this means that my processes only use 1GB of memory but the system as a whole (excluding cache) uses 79 GB.

Question: What is really eating almost 80 GB of my memory? Launching a process that takes up around 40 GB will cause the system to crash, the 80 GB used initially on startup does not get released!

Any insights greatly appreciated!

Output of ps -A --sort -rss -o comm,pmem | head -n 5

COMMAND         %MEM
mysqld           0.2
blueman-applet   0.0
caja             0.0
nm-applet        0.0

Output of cat /proc/meminfo

MemTotal:       131947364 kB
MemFree:        45513984 kB
MemAvailable:   45083132 kB
Buffers:           24900 kB
Cached:           299012 kB
SwapCached:            0 kB
Active:          1337380 kB
Inactive:         211000 kB
Active(anon):    1228248 kB
Inactive(anon):    20812 kB
Active(file):     109132 kB
Inactive(file):   190188 kB
Unevictable:        6864 kB
Mlocked:            6900 kB
SwapTotal:      34553848 kB
SwapFree:       34553848 kB
Dirty:                16 kB
Writeback:             0 kB
AnonPages:       1231920 kB
Mapped:           206100 kB
Shmem:             20896 kB
Slab:             521948 kB
SReclaimable:      77492 kB
SUnreclaim:       444456 kB
KernelStack:       20880 kB
PageTables:        23056 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    58584488 kB
Committed_AS:    4713068 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:   40960
HugePages_Free:    40960
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      390128 kB
DirectMap2M:     8921088 kB
DirectMap1G:    124780544 kB

Output of sudo lshw -C memory

  *-firmware              
       description: BIOS
       vendor: American Megatrends Inc.
       physical id: 0
       version: P3.50
       date: 12/24/2018
       size: 64KiB
       capacity: 15MiB
       capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer acpi usb biosbootspecification uefi
  *-memory
       description: System Memory
       physical id: 8
       slot: System board or motherboard
       size: 128GiB
     *-bank:0
          description: DIMM Synchronous 2134 MHz (0.5 ns)
          product: CMK128GX4M8X3800C19
          vendor: Unknown
          physical id: 0
          serial: 00000000
          slot: DIMM 0
          size: 16GiB
          width: 64 bits
          clock: 2134MHz (0.5ns)
     *-bank:1
          description: DIMM Synchronous 2134 MHz (0.5 ns)
          product: CMK128GX4M8X3800C19
          vendor: Unknown
          physical id: 1
          serial: 00000000
          slot: DIMM 1
          size: 16GiB
          width: 64 bits
          clock: 2134MHz (0.5ns)
     *-bank:2
          description: DIMM Synchronous 2134 MHz (0.5 ns)
          product: CMK128GX4M8X3800C19
          vendor: Unknown
          physical id: 2
          serial: 00000000
          slot: DIMM 0
          size: 16GiB
          width: 64 bits
          clock: 2134MHz (0.5ns)
     *-bank:3
          description: DIMM Synchronous 2134 MHz (0.5 ns)
          product: CMK128GX4M8X3800C19
          vendor: Unknown
          physical id: 3
          serial: 00000000
          slot: DIMM 1
          size: 16GiB
          width: 64 bits
          clock: 2134MHz (0.5ns)
     *-bank:4
          description: DIMM Synchronous 2134 MHz (0.5 ns)
          product: CMK128GX4M8X3800C19
          vendor: Unknown
          physical id: 4
          serial: 00000000
          slot: DIMM 0
          size: 16GiB
          width: 64 bits
          clock: 2134MHz (0.5ns)
     *-bank:5
          description: DIMM Synchronous 2134 MHz (0.5 ns)
          product: CMK128GX4M8X3800C19
          vendor: Unknown
          physical id: 5
          serial: 00000000
          slot: DIMM 1
          size: 16GiB
          width: 64 bits
          clock: 2134MHz (0.5ns)
     *-bank:6
          description: DIMM Synchronous 2134 MHz (0.5 ns)
          product: CMK128GX4M8X3800C19
          vendor: Unknown
          physical id: 6
          serial: 00000000
          slot: DIMM 0
          size: 16GiB
          width: 64 bits
          clock: 2134MHz (0.5ns)
     *-bank:7
          description: DIMM Synchronous 2134 MHz (0.5 ns)
          product: CMK128GX4M8X3800C19
          vendor: Unknown
          physical id: 7
          serial: 00000000
          slot: DIMM 1
          size: 16GiB
          width: 64 bits
          clock: 2134MHz (0.5ns)
  *-cache:0
       description: L1 cache
       physical id: b
       slot: L1 - Cache
       size: 3MiB
       capacity: 3MiB
       clock: 1GHz (1.0ns)
       capabilities: pipeline-burst internal write-back unified
       configuration: level=1
  *-cache:1
       description: L2 cache
       physical id: c
       slot: L2 - Cache
       size: 16MiB
       capacity: 16MiB
       clock: 1GHz (1.0ns)
       capabilities: pipeline-burst internal write-back unified
       configuration: level=2
  *-cache:2
       description: L3 cache
       physical id: d
       slot: L3 - Cache
       size: 64MiB
       capacity: 64MiB
       clock: 1GHz (1.0ns)
       capabilities: pipeline-burst internal write-back unified
       configuration: level=3

Output of sudo grep Memory /var/log/kern.log | grep reserved

Jan 14 16:53:53 server2 kernel: [    0.000000] Memory: 131807432K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283932K reserved, 0K cma-reserved)
Jan 14 17:47:08 server2 kernel: [    0.000000] Memory: 131807432K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283932K reserved, 0K cma-reserved)
Jan 14 17:52:04 server2 kernel: [    0.000000] Memory: 131807436K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283928K reserved, 0K cma-reserved)
Jan 14 18:09:33 server2 kernel: [    0.000000] Memory: 131807436K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283928K reserved, 0K cma-reserved)
Jan 14 18:25:01 server2 kernel: [    0.000000] Memory: 131807436K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283928K reserved, 0K cma-reserved)
Jan 14 18:43:40 server2 kernel: [    0.000000] Memory: 131820396K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2270968K reserved, 0K cma-reserved)
Jan 14 18:48:07 server2 kernel: [    0.000000] Memory: 131807428K/134091364K available (12300K kernel code, 2480K rwdata, 4208K rodata, 2428K init, 2704K bss, 2283936K reserved, 0K cma-reserved)
debian memory-usage ram memory-leak 16.04

1 Answers

  1. Partial answer...

    CPU: AMD Threadripper 2990WX
Motherboard: ASRock X399 Professional Gaming sTR4
128GB of RAM

    1. Update your motherboard BIOS

      • You're currently at BIOS version P3.50
      • BIOS 3.80, from 1/8/2020, can be downloaded here
      • Confirm that this is the correct web page for your model #
      • Backup important data before installing new BIOS
      • view the free -h command to confirm memory usage

    2. Go to https://www.memtest86.com/ and download/run their free memtest to test your memory. Get at least one complete pass of all the 4/4 tests to confirm good memory. This may take many hours to complete.

    3. Using proper ESD and power procedures, re-seat all RAM dimms into their sockets. (Wait with doing this until I tell you more about this.)

    • 2