Home / server / Questions / 692381
In Process
MealstroM
Asked: 2015-05-16 11:50:11 +0800 CST

ubuntu kvm disk io slowdown after some tyme

  • 772

Host system:

Ubuntu Server x64 12.04 
mdadm raid 1 (/dev/sda /dev/sdb)
no lvm
dd bs=1M count=256 if=/dev/zero of=filename conv=fdatasync
avarage ~ 40 MB/s

NCQ on disks is disabled
WriteCache is disables

Guest system:

Ubuntu server i386 12.04
with lvm2 /10Gb /200Gb /200Gb disks all on lv-root (LV)
  --- Physical volume ---
  PV Name               /dev/vda5
  VG Name               root-vg
  PV Size               9.76 GiB / not usable 2.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              2498
  Free PE               0
  Allocated PE          2498

  --- Physical volume ---
  PV Name               /dev/vdb
  VG Name               root-vg
  PV Size               195.31 GiB / not usable 4.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              49999
  Free PE               0
  Allocated PE          49999

  --- Physical volume ---
  PV Name               /dev/vdc
  VG Name               root-vg
  PV Size               195.31 GiB / not usable 4.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              49999
  Free PE               0
  Allocated PE          49999

 dd bs=1M count=256 if=/dev/zero of=filename conv=fdatasync
    avarage ~ 30 MB/s
all disks in guest are RAWformat /VirtioBUS / No cache / IOmode=native

after some time write speed falls to 1 MB/s , but host system is not have loaded and dd test shows same 30-40 MB/s , cpu usage 10% . Guest reboot helps for a while. There is no errors/ faults / no mdadm rebuild or resync.

Have no idea where is a problem or where to dig.

Looks like this helps on guest: sync && echo 3 > /proc/sys/vm/drop_caches

Similar problem On a system with 64GB mem the Linux Buffer run full while copying with dd to dev null and io stops till manual drop_caches

linux

1 Answers

  1. I think what happens is that the initial performance of 30-40 MB/s is because of linux kernel's caching (and any other caching that may be going on on a hardware level). Once that caching has been "used up" actual disk access starts to kick in and performance drops.

    In addition in order for dd to have better performance set the bs= argument to a reasonably large size. Personally I like to set it to about 1/3-1/2 of available ram. Your setting of 1M is sub optimal and is the main reason for the low performance numbers. But even with optimal bs= setting you would see a performance drop at some point as explained above.

    • 0