Home / server / Questions / 981357
In Process
Gaia
Asked: 2019-08-31 09:25:37 +0800 CST

Extremely Low Random Read/Write Performance on (4096 Aligned) Advanced Format Disks

  • 772

On a SATA 3.0, 6.0 Gb/s

*-disk:0
       description: ATA Disk
       product: WDC WD20EFRX-68E
       vendor: Western Digital
       physical id: 0
       bus info: scsi@1:0.0.0

hw_sector_size: 512
logical_block_size: 512
physical_block_size: 4096

# parted /dev/sda print
Model: ATA WDC WD20EFRX-68E (scsi)
Disk /dev/sda: 2000GB
Sector size (logical/physical): 512B/4096B

I have the following fio results on an empty /dev/sda3 (ext4):

fio --name=seqread --rw=read --direct=1 --ioengine=libaio --bs=8k --numjobs=8 --size=1G --runtime=600  --group_reporting
seqread IOPS=8960, BW=70.0MiB/s (73.4MB/s)(8192MiB/117028msec)

fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=32k --numjobs=4 --size=2G --runtime=600 --group_reporting
seqwrite IOPS=1538, BW=48.1MiB/s (50.4MB/s)(8192MiB/170345msec); 0 zone resets

fio --name=randread --rw=randread --direct=1 --ioengine=libaio --bs=8k --numjobs=16 --size=1G --runtime=600 --group_reporting
randread IOPS=163, BW=1305KiB/s (1337kB/s)(765MiB/600078msec)

fio --name=randwrite --rw=randwrite --direct=1 --ioengine=libaio --bs=64k --numjobs=8 --size=512m --runtime=600 --group_reporting
randwrite IOPS=165, BW=10.3MiB/s (10.8MB/s)(4096MiB/395912msec); 0 zone resets

fio --name=randrw --rw=randrw --direct=1 --ioengine=libaio --bs=16k --numjobs=8 --rwmixread=90 --size=1G --runtime=600 --group_reporting
randrw read IOPS=141, BW=2258KiB/s (2313kB/s)(1324MiB/600086msec)
randrw write IOPS=15, BW=253KiB/s (259kB/s)(148MiB/600086msec); 0 zone resets

At first I thought I had incorrect sector alignment,

Number  Start          End             Size            File system  Name              Flags
 1      1048576B       511705087B      510656512B      fat32                          boot, esp
 2      511705088B     54198796287B    53687091200B                                   raid
 3      54198796288B   215260069887B   161061273600B   ext4         Linux filesystem
 4      215260069888B  2000398917119B  1785138847232B               Linux RAID        raid

but all start sectors are divisible by 4096, and parted tells me that 

(parted) align-check opt 1
1 aligned
(parted) align-check opt 2
2 aligned
(parted) align-check opt 3
3 aligned
(parted) align-check opt 4
4 aligned

SMART looks good: enter image description here

These fio results are similar with the system running on Debian 10 and an Arch Linux Live cd. These are not the fastest spinners, so I can live with the sequential results (though they should be higher) but the random R, W and RW results are unacceptable.

Kernel is 5.0.x (5.2.x in the live CD), and the scheduler is the default (and recommended)

# cat /sys/block/sda/queue/scheduler
[mq-deadline] none

Changing the scheduler to BFQ did not help.

AFAIK the sector alignment is fine, but either I am missing something or I have some defective drives (which are brand new and under warranty). <200 IOPS and <2.5MB/s renders the system unusable at times.

I am at loss here. How do I fix this? Or do I have defective drives (unlikely, since these are two identical drives with similar results) ?

performance

1 Answers

  1. You appear to be doing I/O within a filesystem with your fio jobs. If so, this adds a another confounding factor to your results. WARNING - doing I/O directly to a disk node can DESTROY the filesystem and data that is already there so be careful if you do choose to go directly to /dev/sda!

    You are going direct with only a maximum of 8 outstanding I/Os. This is likely too low to see your disk best performance (SATA disks can usually queue around 32 outstanding I/Os). You are also using multiple processes to submit I/O which will likely incur some overhead compared to submitting I/O asynchronously.

    Summary: can't tell if disks are defective because your fio jobs looks strange. Maybe not enough I/O is being submitted (fio mentions disk stats which you didn't include here - check what your disk utilisation looks like). Additionally things like caches on the disk itself may mean you eventually get back a distorted result due to the area you're working in being so small and multiple jobs working on the same area of the disk so be warned there too (I'd imagine this would be more likely in "sequential" cases).

    • 0