Home / ubuntu / Questions / 1527871
Accepted
WKGuy
Asked: 2024-09-24 06:24:44 +0800 CST

Jdb2 constantly writes to HDD

  • 772

I need help with some hard drives that my system keeps constantly writing 4kb of data to them through a process called jbd2. The writing never ends and it is making my drives extremely hot due to the constant activity on them.

I'll give you the whole background of how I got here first:

I have an old laptop running ubuntu server that I have been using as a server for running things like Nextcloud and most recently Jellyfin.

Linux nextcloudlenovo 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

The storage for my Jellyfin media library has been an old 500GB hard drive formatted using EXT4 that was backed up with a crontab job into another 500GB hard drive. Both drives were connected using a 2-bay USB drive dock like this: Inland USB 2 Bay Docking Station

Everything had been working perfectly with my old drives, until I recently bought new 8TB hard drives to replace my 500GB ones which are now full.

As you may notice, the Docking Station is also a disk cloner, so before installing my new drives into the laptop, I disconnected the docking station from it and used it standalone to clone the data from the 500GB to the 8TB. After plugging the docking station to the server again with the new drives, the process seems to have been successful (all my data is in the new drives and I can access it no problem).

With all that background out of the way...

The problem I'm having now is that as soon as my new drives are mounted, jdb2 constantly doing 4kb writes to each of them. Here's a printout I made after letting the drive mounted for a few minutes (sda2-8 is the drive being affected):

Average:      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
Average:        0         1    749.83    209.51      0.08       0  systemd
Average:        0        90      5.53      0.00      0.00       0  kworker/u4:1-flush-8:0
Average:        0        99      0.22      0.00      0.00       0  kworker/u4:3-loop5
Average:        0       303      0.00     19.00      0.00       0  jbd2/sda2-8
Average:        0       375      0.00     14.76      0.00       0  systemd-journal
Average:      107       714      0.00      0.28      0.00       0  rsyslogd
Average:        0       717   1794.02   3103.34     34.04       0  snapd
Average:        0       722      1.12      0.00      0.00       0  udisksd
Average:        0       970      0.00      0.03      0.02       0  nmbd
Average:        0      3682      0.00      0.64      0.00       0  jbd2/sdc1-8
Average:        0      3684     58.33      0.00      0.00       0  ext4lazyinit
Average:        0      3967      1.62      0.00      0.00       0  kworker/u4:0-events_unbound
Average:        0      4793    984.14     71.50     49.24       0  run-httpd
Average:        0      4799      0.03      0.00      0.00       0  nextcloud-fixer
Average:        0      4810      0.36      0.00      0.00       0  start-php-fpm
Average:        0      4816    120.85      0.02      0.00       0  start_mysql
Average:        0      4827     29.23      0.00      0.00       0  nextcloud-cron
Average:        0      4851      0.49      0.00      0.00       0  start-redis-ser
Average:        0      4860    110.75      0.03      0.02       0  renew-certs
Average:        0      5040     36.26      0.05      0.00       0  redis-server
Average:        0      5349      4.76      0.00      0.00       0  mysqld_safe
Average:        0      5580     10.60    133.39      0.06       0  mysqld
Average:        0      5711     32.41      0.03      0.00       0  php-fpm
Average:        0      6365      0.01      0.00      0.00       0  httpd-wrapper
Average:        0      6420      4.42      0.03      0.00       0  httpd
Average:        0      6422      0.08      0.00      0.00       0  httpd
Average:        0      6570      0.73      0.00      0.00       0  php

At first I thought it was a duplicate of this question, so I added the noatime to fstab, and it didn't change anything. Then I thought it might be a duplicate of this other question, but I don't see a file under .local/share/gvfs-metadata. So I'm running out of options and ideas.

I tried plugging the USB docking station into a raspberry pi 4 which is the only other Linux computer I have readily available, to see if the drives would behave any differently (i.e. not perform the 4kb writes with jdb2), but in there, as soon as I mount the drives ext4lazyinit takes FOREVER to run. I left it running in the pi for more than 1 hour and it was still initializing the drives, while on the laptop ext4lazyinit finishes within 1 minute.

I'm kind of a noob as a sysadmin, so any help is greatly appreciated!

If you want even more details about the laptop in case it helps in any way, here is my lscpu dump:

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          36 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   2
  On-line CPU(s) list:    0,1
Vendor ID:                GenuineIntel
  Model name:             Pentium(R) Dual-Core CPU       T4400  @ 2.20GHz
    CPU family:           6
    Model:                23
    Thread(s) per core:   1
    Core(s) per socket:   2
    Socket(s):            1
    Stepping:             10
    CPU max MHz:          2200.0000
    CPU min MHz:          1200.0000
    BogoMIPS:             4389.61
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_pe
                          rfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm xsave lahf_lm pti dtherm
Caches (sum of all):      
  L1d:                    64 KiB (2 instances)
  L1i:                    64 KiB (2 instances)
  L2:                     1 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0,1
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          KVM: Mitigation: VMX unsupported
  L1tf:                   Mitigation; PTE Inversion
  Mds:                    Vulnerable: Clear CPU buffers attempted, no microcode; SMT disabled
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Unknown: No mitigations
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Vulnerable
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected
server

1 Answers

  1. Best Answer

    The JBD is the journaling block device that sits between the file system and the block device driver. The jbd2 version is for ext4. If you are seeing a lot of activity when nothing much is running, check if any of the logs are getting a lot of updates. Run the following to watch for log size changes.

    ls -lS /var/log/*log | head

    Run in a loop every 2s.

    watch ls -lS /var/log/*log | head

    If there are any that are larger than a few megabytes, use tail to look at the end of them and see if there are any errors that are occurring frequently.

    It is possible to change various file system flags and, indeed change file systems called tune2fs

    tune2fs command allows you to view and change various filesystem parameters on Linux ext2, ext3, or ext4 filesystems. Some parameters you can modify are filesystem volume name, maximum mount count, mount count, interval time between two filesystem checks, last checked time, etc.

    tune2fs can also convert an ext2 filesystem to an ext3 filesystem and an ext3 filesystem to an ext4 filesystem.

    NOTE: Back up your data before running these commands as they might cause corruption of the filesystem and data loss.

    For example to add the journel feature: -

    sudo tune2fs -O has_journal /dev/sdXY

    And to remove it you add the ^ before the flag.

    sudo tune2fs -O ^has_journal /dev/sdXY

    For more details of this feature see: -

    https://www.golinuxcloud.com/tune2fs-command-in-linux/

    • 2