Home / server / Questions / 233287
Accepted
growse
Asked: 2011-02-10 06:30:00 +0800 CST

COMSTAR and ZFS zvol giving bursty write pattern

  • 772

I've just built a small NAS/SAN box and am doing some testing with it. Configuration is:

  • 2GB RAM
  • 2x Opteron 265s
  • Iwill DK8N
  • 5x500GB SATA drives (1 spare, rest mirrored) = 1TB usable space
  • Emulex LPC10000DC
  • NexentaCore 3.0.2

The first thing I'm testing is COMSTAR performance over fibre-channel. I created a 700g zvol and exposed it over COMSTAR. Write-caching is enabled. My FC initiator is a Windows 7 box with a Q-Logic QLA2340 card in it.

Everything works great, except for write speeds. For some reason, I'm only seeing ~30MB/s, whereas I can peak at about 90MB/s read speeds, using IOMeter. Doing some hunting around, I ran a zpool iostat command and saw the following pattern:

$ zpool iostat tank 1
tank        88.2G   840G      0  5.47K      0  43.4M
tank        88.2G   840G      0  16.1K      0   128M
tank        88.4G   840G      1  6.69K  8.93K  52.6M
tank        88.4G   840G     52  4.89K   110K  38.3M
tank        88.4G   840G    151      0   280K      0
tank        88.4G   840G    112      0   253K      0
tank        88.4G   840G      1  9.89K  16.0K  78.6M
tank        88.4G   840G      0  14.3K      0   113M
tank        88.6G   839G    150  2.99K   318K  20.2M
tank        88.6G   839G     15      0  45.0K      0
tank        88.6G   839G      3      0  17.8K      0
tank        88.6G   839G      0      0      0      0
tank        88.6G   839G      0      0      0      0
tank        88.6G   839G      0  11.3K      0  89.9M
tank        88.6G   839G      0  13.8K      0   110M
tank        88.8G   839G     90  7.91K   184K  61.0M
tank        88.9G   839G    105     91   220K   140K
tank        88.9G   839G     42      0  84.6K      0
tank        88.9G   839G      0      0      0      0
tank        88.9G   839G      0  6.52K  7.99K  51.8M
tank        88.9G   839G      0  15.6K   1011   124M
tank        88.9G   839G     91  4.30K   193K  31.9M
tank        89.1G   839G      1  9.48K  15.8K  74.7M
tank        89.1G   839G     41      0  99.9K      0
tank        89.1G   839G    166      0   374K      0
tank        89.1G   839G     87      0   162K      0
tank        89.1G   839G      0      0  7.92K      0

It appears to be writing a lot, then stopping.

What's going on here?

performance solaris zfs fibre-channel zvol

1 Answers

  1. Best Answer

    That's normal for ZFS. What's happening is it's coalescing the writes in memory and then committing them to disk at the same time. This a great wiki about what it's doing and how you can tune it:

    http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide

    • 1