El Yobo's questions

I'm having some serious disk performance problems while setting up a KVM guest. Using a simple dd test, the partition on the host that the qcow2 images reside on (a mirrored RAID array) writes at over 120MB/s, while my guest gets writes ranging from 0.5 to 3MB/s.

• The guest is configured with a couple of CPUs and 4G of RAM and isn't currently running anything else; it's a completely minimal install at the moment.
• Performance is tested using time dd if=/dev/zero of=/tmp/test oflag=direct bs=64k count=16000.
• The guest is configured to use virtio, but this doesn't appear to make a difference to the performance.
• The host partitions are 4kb aligned (and performance is fine on the host, anyway).
• Using writeback caching on the disks increases the reported performance massively, but I'd prefer not to use it; even without it performance should be far better than this.
• Host and guest are both running Ubuntu 12.04 LTS, which comes with qemu-kvm 1.0+noroms-0ubuntu13 and libvirt 0.9.8-2ubuntu17.1.
• Host has the deadline IO scheduler enabled and the guest has noop.

There seem to be plenty of guides out there tweaking kvm performance, and I'll get there eventually, but it seems like I should be getting vastly better performance than this at this point in time so it seems like something is already very wrong.

Update 1

And suddenly when I go back and test now, it's 26.6 MB/s; this is more like what I expected w/qcrow2. I'll leave the question up in case anyone has any ideas as to what might have been the problem (and in case it mysteriously returns again).

Update 2

I stopped worrying about qcow2 performance and just cut over to LVM on top of RAID1 with raw images, still using virtio but setting cache='none' and io='native' on the disk drive. Write performance is now appx. 135MB/s using the same basic test as above, so there doesn't seem to be much point in figuring out what the problem was when it can be so easily worked around entirely.

I'm looking for a way to replicate a all changes on a filesystem between two disks in a master->slave setup. One of the disks is much faster for reads, so I would like to direct all reads to this real disk for performance reasons; is it possible to do this with Linux software RAID? Is there a better approach?

I'm currently investigating migrating our infrastructure over to run on Amazon's EC2 and am trying to figure out the best way to set up a MySQL service. I'm leaning towards running our own MySQL instances, rather than going with Amazon's RDS, but am still considering the best approach for performance and cost on the instance itself.

In order to have persistent data, the MySQL data needs to be on an EBS volume (with some form of striped RAID, e.g. RAID0 or RAID10) to improve persistence. However, EBS IO is limited by the network interface (gigabit, so a theoretical maximum of 128 MB/s), while the ephemeral volumes have no such problem.

I did see a suggestion for running two MySQL servers on an instance, with a master running on the ephemeral disk (which we would also RAID) and a slave storing changes to an EBS volume, but this has some additional overhead and complexity (two servers).

What I was imagining is using some form of replicated file system such that I could have

• a filesystem on top of a RAID0 of ephemeral volumes to maximise performance
• all changes from the above immediately replicated to another RAID1 volume backed by multiple EBS volumes to ensure no data loss

The advantages of this would be

• best possible IO performance for the DB server; no network delay in IO
• decreased IO on EBS volumes (as all read IO will be done on the ephemeral volumes) so decreased cost
• good data security, as it's backed onto redundant EBS volumes

However, I haven't seen an appropriate system to replicate all changes from one volume to the other; is there a filesystem, or any other approach, which will do this? The distributed file systems, e.g. GlusterFS, DRBD etc seem to focus on replicating disks between servers, can they be set up to do what I'm interested in here?

I also haven't seen anything about other's taking this approach. Do I have a solution in need of a problem here (i.e. is performance good enough, so this whole idea is redundant)? Is there some flaw in the plan?

Note: admins, please vote for migration to ServerFault.

I'm using a LAMP setup with PHP running through mod_fcgid. For most requests this works well, but I've noticed that when I download a file but interrupt the download before it's complete, the php-cgi process that was serving the file blocks attempting to write more data until it the IPCCommTimeout is reached. Once the timeout is reached, the process gets interrupted and the process starts servicing other requests again.

Is there some available setting for fcgid that I can set to have it abort if nothing is capturing the output? Is there something I can do in PHP to handle it?

The problem doesn't occur if the download isn't interrupted; in fact, I only noticed it because I was trying to stream a FLV file using gddflvplayer, which seems to send a brief request to get the first few frames, then another to play it, and this causes the same problem.

FYI, this is the strace of the hanging cgi process; it sits like this until it's eventually interrupted, presumably by the process manager when the IPCCommTimeout is reached. My guess is that it's hanging trying to output the results of the readfile() call, but Apache is no longer listening (as the request has been cancelled by the user).

root@some-machine:~# strace -p 24837
Process 24837 attached - interrupt to quit
write(3, "\5|A\313%\35\337\376\275\237\230\266\242\371\37YjzD<\322\215\357\336:M\362P\335\242\214\341"..., 17432

The logs indicate that the request is eventually reaped due to the timeout

mod_fcgid: read data timeout in 240 seconds

The download code more or less just uses readfile to serve up the file, with a few headers involved as well (note: in this code, Header is more or less just a wrapper around header() to avoid problems in tests).

$filepath    = '/some/path/foo.flv';
$filename    = 'foo.flv';
$disposition = 'inline';

$h = Header::get();
$h->send('Pragma: public');
$h->send('Content-Transfer-Encoding: binary');
$h->send('Content-type: ' . FileSystem::get()->getMimeType($filepath));
$h->send('Content-Length: ' . FileSystem::get()->getFileSize($filepath));
$h->send('Content-Disposition: ' . $disposition . '; filename="' . $filename . '"');
$h->send('Content-transfer-encoding: 8bit');
$h->send('Expires: 0');
$h->send('Pragma: cache');
$h->send('Cache-Control: private');

flush();
readfile($filepath);

Note: this is the same as this SO post, but is possibly more appropriate here as I suspect the problem is server config related rather than code related.

I'm using a LAMP setup with PHP running through mod_fcgid. For most requests this works well, but I've noticed that when I download a file but interrupt the download before it's complete, the php-cgi process that was serving the file blocks attempting to write more data until it the IPCCommTimeout is reached. Once the timeout is reached, the process gets interrupted and the process starts servicing other requests again.

This problem does not occur if I use mod_php instead of mod_fcgid.

Is there some available setting for fcgid that I can set to have it abort if nothing is capturing the output? Is there something I can do in PHP to handle it?

The problem doesn't occur if the download isn't interrupted; in fact, I only noticed it because I was trying to stream a FLV file using gddflvplayer, which seems to send a brief request to get the first few frames (which it shows as a preview), then another to play it, and this causes the same problem.

FYI, this is the strace of the hanging cgi process; it sits like this until it's eventually interrupted, presumably by the process manager when the IPCCommTimeout is reached. My guess is that it's hanging trying to output the results of the readfile() call, but Apache is no longer listening (as the request has been cancelled by the user).

root@some-machine:~# strace -p 24837
Process 24837 attached - interrupt to quit
write(3, "\5|A\313%\35\337\376\275\237\230\266\242\371\37YjzD<\322\215\357\336:M\362P\335\242\214\341"..., 17432

The logs indicate that the request is eventually reaped due to the timeout

mod_fcgid: read data timeout in 240 seconds

The download code more or less just uses readfile to serve up the file, with a few headers involved as well (note: in this code, Header is more or less just a wrapper around header() to avoid problems in tests).

$filepath    = '/some/path/foo.flv';
$filename    = 'foo.flv';
$disposition = 'inline';

$h = Header::get();
$h->send('Pragma: public');
$h->send('Content-Transfer-Encoding: binary');
$h->send('Content-type: ' . FileSystem::get()->getMimeType($filepath));
$h->send('Content-Length: ' . FileSystem::get()->getFileSize($filepath));
$h->send('Content-Disposition: ' . $disposition . '; filename="' . $filename . '"');
$h->send('Content-transfer-encoding: 8bit');
$h->send('Expires: 0');
$h->send('Pragma: cache');
$h->send('Cache-Control: private');

flush();
readfile($filepath);

The server itself is running Debian Lenny with standard packages for php5-cgi, apache2, libapache2-mod-fcgid, but I also get the same results on a development box with Ubuntu 10.10.

Package information follows -

[foo@bar ~]$  dpkg -l | egrep '(apache2|php5)'
ii  apache2-mpm-worker                  2.2.9-10+lenny9            Apache HTTP Server - high speed threaded model
ii  apache2-utils                       2.2.9-10+lenny9            utility programs for webservers
ii  apache2.2-common                    2.2.9-10+lenny9            Apache HTTP Server common files
ii  libapache2-mod-fastcgi              2.4.6-1                    Apache 2 FastCGI module for long-running CGI scripts
ii  libapache2-mod-fcgid                1:2.2-1+lenny1             an alternative module compat with mod_fastcgi
ii  php5                                5.2.6.dfsg.1-1+lenny9      server-side, HTML-embedded scripting language (metapack
ii  php5-cgi                            5.2.6.dfsg.1-1+lenny9      server-side, HTML-embedded scripting language (CGI bina
ii  php5-cli                            5.2.6.dfsg.1-1+lenny9      command-line interpreter for the php5 scripting languag
ii  php5-common                         5.2.6.dfsg.1-1+lenny9      Common files for packages built from the php5 source
ii  php5-curl                           5.2.6.dfsg.1-1+lenny9      CURL module for php5
ii  php5-ffmpeg                         0.5.3.1-3                  ffmpeg support for php5
ii  php5-fileinfo                       1.0.4-1                    Fileinfo module for PHP 5
ii  php5-gd                             5.2.6.dfsg.1-1+lenny9      GD module for php5
ii  php5-imagick                        2.1.1RC1-1                 ImageMagick module for php5
ii  php5-mysql                          5.2.6.dfsg.1-1+lenny9      MySQL module for php5
ii  php5-suhosin                        0.9.27-1                   advanced protection module for php5