Home / server / Questions / 913519
In Process
Michael Martinez
Asked: 2018-05-24 14:30:16 +0800 CST

contradictory results dd and iostat, ext4 and xfs, on EBS volumes

  • 772

I'm seeing what seems to me to be contradictory results when looking at disk performance with dd versus iostat on two hosts (EC2 instances with an EBS drive). The hosts are identical except that one uses EXT4-formatted ebs and the other XFS-formatted ebs.

If I look at iostat, the EXT4 host seems to outperform the XFS host. TBoth are doing roughly the same write throughput (about 25MB/s) at about 100% utilization, but the EXT4 host on average has less await (lower disk latency). It's this smaller await that makes me say EXT4 is outperforming XFS:

EXT4:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdf              0.00    11.00    0.00 6331.00     0.00    26.96     8.72    71.00   11.32   0.16  99.60

XFS:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdf              0.00     2.00    0.00 6211.00     0.00    27.38     9.03   144.95   23.24   0.16 100.40

But, if using dd to measure performance, XFS is the clear winner, as it takes much less time to complete a fully synchronous write. The command is dd bs=1M count=256 if=/dev/zero of=./test conv=fdatasync:

EXT4:

2.8 MB/s

XFS:

24.0 MB/s

What would be the reason for EXT4 looking much better with iostat but looking much worse with dd?

UPDATE 5/25/2018: After running the hosts for a couple days, dd and sync now show equivalent response times for both ext4 and xfs. I suspect this has to do with a difference (if any?) in the way they handle something like sparse files. The first day the hosts were up they were both busy laying down a bunch of new files on the filesystem (this is a graphite carbon-cache application). This has settled down to where small updates are being written to these files, but new files are no longer being created and the total amount of used disk space is no longer increasing.

So, there must be something fundamentally different in the way XFS allocates new disk blocks versus EXT4. Any insight into what this could be, would be welcome.

linux

1 Answers

  1. Correct me if I'm wrong, but I believe you use EBS volumes.

    Try to format the ext4 partition without lazy initialization, as it can affect testing performance.

    mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/xvdf


    EBS Volumes
    Several factors will determine EBS volume performance. They are not entirely intuitive.

    1. Type of the volume. Provisioned IOPS io1, General Purpose gp2, Throughput Optimized HDD st1, Cold HDD sc1. characteristics.

    2. Size of the volume. Generally, bigger the volume, better the performance. Except for the EBS Provisioned IOPS (io1), volumes use a burst model [1] and I/O can fluctuate or drop significantly if all I/O credits are used. In short, each volume gets base minimum 100 IOPS, and for every +1 GB added (after ~33GB) performance will increase for 3 IOPS. Also the volume can burst up to 3000 IOPS if there are enough I/O credits available.

    Size vs IOPS

    1. EC2 Instance type. Larger the instance, faster the network performance. It's also important whether it's an EBS-Optimized instance or not.

    2. If you restored your volume from a snapshot, you'd have to pre-warm the volume to get the maximum performance.
      See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html

    More detauls:
    [1] https://aws.amazon.com/ebs/details/

    Update 1

    It's difficult to tell why you see those results as there's just not enough information in your post for me to deduce it.

    This CloudFormation template is my attempt to recreate your results.
    https://gist.github.com/an2io/c68b2119f18192d83a685651905623e9

    EXT4 
    wrqm/s  w/s     wMB/s   avgrq-sz    avgqu-sz    await   svctm    %util  
0.00    528.62  66.08   255.84      127.07      217.16  1.61     85.39    
2.69    287.21  35.86   255.71      142.20      504.73  3.52    101.01    
0.00    285.28  35.59   255.49      134.93      462.90  3.52    100.33    
3.02    277.52  34.54   254.89      115.54      460.62  3.51     97.32    
0.00      0.00   0.00     0.00        0.00        0.00  0.00      0.00    
512+0 records in  
512+0 records out  
536870912 bytes (537 MB) copied, 11.9003 s, 45.1 MB/s
    XFS 
    wrqm/s  w/s     wMB/s   avgrq-sz    avgqu-sz    await   svctm    %util
1.68    542.28  67.44   254.07      176.84      301.59  1.66     90.47
0.00    284.95  35.62   256.00      143.74      508.38  3.52    100.33
0.00    285.28  35.58   255.46      184.36      609.81  3.52    100.33
0.00    265.10  32.95   253.27      162.34      692.59  3.48     92.75
0.00      0.00   0.00     0.00        0.00        0.00  0.00      0.00

512+0 records in  
512+0 records out  
536870912 bytes (537 MB) copied, 11.7641 s, 45.6 MB/s
    df 
    Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.9G   72K  1.9G   1% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
/dev/xvda1      9.8G  1.2G  8.5G  12% /
/dev/xvdb       3.9G  8.1M  3.7G   1% /media/ephemeral0
/dev/xvdf1      3.7G  520M  3.0G  15% /mnt/ext4
/dev/xvdf2      3.9G  545M  3.3G  14% /mnt/xfs
    mount 
    /dev/xvdf1 on /mnt/ext4 type ext4 (rw,relatime,data=ordered)
/dev/xvdf2 on /mnt/xfs type xfs (rw,relatime,attr2,inode64,noquota)
    • 2