While troubleshooting a performance issue on a Debian Linux box running ext3 on software RAID 1 I redirected the output of a process writing useless temporary log files to /dev/null (ln -s /dev/null/ foo) which has reduced write operations by around 3MB/s. However the change seems to have also lead to a big increase in Write and Overall IOwait times which I'm at a loss to explain and would really appreciate some input on...
iostat: http://oi45.tinypic.com/5l2mx0.jpg
write latency (change applied at ~3am): http://oi46.tinypic.com/33be1kw.jpg
Update 18.04.13:
Write IOwait remains higher than before but I've applied the following all of which have been at least somewhat beneficial...
- Remounted the filesystem with noatime option.
- Changed to deadline ioscheduler.
- Made vm.swappiness 30 (host is home to ~1GB of database, a CMS and streaming audio).
- Turned readahead down to 256 down from 4096 for sda and sdb.
- Turned readahead up to 4096 from 256 for /dev/md2 (the main data partition).
Update 19.04.13:
- Converted the main filesystem from ext3 to ext4
Logically, the only thing that can increase response time (I/O wait) is increasing the number of operations that have to be carried out on the "same" thing.
Hypothesis: the I/O to /dev/null causes more operations at the driver level than the (possibly more buffered) I/O to a disk
Test: try switching it back to a disk file. If response time falls, there is something to do with /dev/null that takes longer than it should. If response time stays the same, then something happened at the same time as the change that affected response time.
--dave