Karl Katzke's questions

I'm doing some benchmarking of nodes as we build out a distributed filesystem. Since files are going to be distributed and replicated across many nodes, we're using raid0 on the nodes themselves. However, I'm getting some odd performance numbers, and I'm curious if the StackOverflow community can help figure out why. I'm using fio as my benchmarking tool; note that version 1.38 did not work for me and I needed to use version 1.59 or 1.60.

Here is my fio configuration file:

[global]
directory=/mnt/gluster
lockfile=readwrite
ioengine=libaio
iodepth=8
rw=randrw
nrfiles=200
openfiles=30
runtime=900

[file]
filesize=16-8k/8M-512M
size=1GiB

For a raid0 using software raid, I got the following results (snipping to the essentials):

  read : io=285240KB, bw=324535 B/s, iops=79 , runt=900011msec
  write: io=283532KB, bw=322592 B/s, iops=78 , runt=900011msec

On a raid1 using software raid, I got the following results:

  read : io=683808KB, bw=778021 B/s, iops=189 , runt=900000msec
  write: io=488184KB, bw=628122 B/s, iops=153 , runt=795864msec

Single disk performance still beat the raid0 performance:

  read : io=546848KB, bw=622179 B/s, iops=151 , runt=900018msec
  write: io=486736KB, bw=591126 B/s, iops=144 , runt=843166msec

LVM Striped across the four disks with 4k extents:

  read : io=727036KB, bw=827198 B/s, iops=201 , runt=900007msec
  write: io=489424KB, bw=604693 B/s, iops=147 , runt=828800msec

Hardware RAID0 (HighPoint RocketRaid 2470)

  read : io=326884KB, bw=371918 B/s, iops=90 , runt=900008msec
  write: io=328824KB, bw=374125 B/s, iops=91 , runt=900008msec

Note that the first four results above are running just on the motherboard's SATA controllers. However, I reproduced the results with software RAID after moving to the RocketRaid card. These are 1tb SATA drives. Running multithreaded tests delivered about the same results. Is there any reason that RAID0 would be running this slow? I thought it would deliver performance in random I/O that was superior to a single drive or RAID1.

Follow-up: Based on some suggestions from Joe at Scalable Informatics (nice guy, buy his stuff!), I changed my test to use a deeper queue and a more random block size.

[global]
directory=/mnt/glusterfs
lockfile=readwrite
ioengine=libaio
iodepth=32
rw=randrw
numjobs=8

[file]
filesize=16-8k/8M-512M
blocksize_range=64k-1M
size=1GiB
nrfiles=50
openfiles=8
runtime=900

And the end result is that the HighPoint RocketRaid 2740 card sucks.

• Single Disk Performance (SuperMicro motherboard SATA): 43.2mb/s read, 42.6 mb/s write
• MD Raid 0 (RocketRAID used as HBA, with or without drivers loaded): 53.1 mb/s read, 54.1 mb/s write
• RocketRaid Raid 0: 29.4 mb/s read, 29.2 mb/s write
• MD RAID 0 (Motherboard SATA): 58.0 mb/s read, 58.3mb/s write

Performance connected to a RocketRaid card was slower across the board.

I'm going to leave this question open -- We're getting a new slate of RAID cards in within the next week or two in order to test, and I'm still looking for optimizing tips on getting more than single-disk performance, which doesn't seem to have been thoroughly answered.

I've inherited a bunch of hardware from a failed project. They're SuperMicro 2U servers with 24x SATA HDD or SSDs. I've been tasked with turning this rack of generic hardware into a IaaS cluster. I'm an old hand with Xen and the show-stopper bug for us with KVM has been fixed, so I'm good to go from most aspects.

Looking over the documentation, I'm not sure right now if I can run GlusterFS (or Nimbus/Cumulus or another cloud storage layer) on the same hardware as the VM Host. I'm trying to find out if anyone has any experience building out with this kind of configuration -- since I don't have separate storage nodes, will it be possible for me to make use of this hardware fully?

Any other suggested solutions, such as doing hardware passthrough to a dedicated storage virtual machine on each node?

We're seeing some storage errors from the esxi logs relating to our MD3200. I'm sort of a VMWare noob and am not sure where to go from here because I couldn't find a lot of documentation on the VMWare website, and the forums didn't seem to have any posts about it with actual answers. Everything is working, but I'm trying to proactively troubleshoot this.

sfcb-vmware_base|StoragePool Cannot get logical disk data from controller 0
sfcb-vmware_base|Volume Cannot get logical disk data from controller 0
sfcb-vmware_base|storelib-GetLDList-ProcessLibCommandCall failed; rval = 0x800E

The ESXi boxes are connected directly via SAS to the controller on the MD3200.

What do these errors actually mean, and what's a good path to start troubleshooting or solving them?

We've rebuilt a bunch of RPMs because we wanted to remove modules and change config options for items specific to our production environment. We're hosting them in a private repo, and have them signed with our own GPG key and have our company name appended to the package name.

TO save ourselves a step after installing the package, I'd like to replace the base config file with config files that are specific to our environment. I'm not very familiar with adding things to spec files. What's the easiest way to do this? Package file?

I am a Solaris newbie. I have a Solaris 10u8 machine that is running an attached J4400 and some internal drives. We're using multipathed SAS I/O (stmsboot; mpathadm), so the device mount points have been moved off from their normal c0t5d0 to long strings -- in the case of c0t5d0, it's now /dev/rdsk/c6t5000CCA00A274EDCd0. (I can see this with stmsboot -l0.

Normally, when replacing a disk on a Solaris system, I would run cfgadm -c unconfigure c0::dsk/c0t5d0. However, cfgadm -l does not list c6, nor does it list any disks. In fact, running cfgadm gets me the following:

bash# cfgadm -l /dev/rdsk/c0t5d0
Ap_Id                          Type         Receptacle   Occupant     Condition
/dev/rdsk/c0t5d0: No matching library found
bash# cfgadm -l /dev/rdsk/c6t5000CCA00A274EDCd0
cfgadm: Attachment point not found
bash# cfgadm -l c6t5000CCA00A274EDCd0
Ap_Id                          Type         Receptacle   Occupant     Condition
c6t5000CCA00A274EDCd0: No matching library found

So -- how do I administer and remove physical devices that are in multipath-managed controllers on Solaris 10u8?

I'm getting a bunch of apache errors that I'm having problems tracing down. They're on a RHEL system that runs a very high-volume Drupal website.

[Mon Sep 14 12:48:44 2009] [info] [client xx.xx.xxx.xx] (70007)The timeout specified has expired: core_output_filter: writing data to the network
[Mon Sep 14 12:50:19 2009] [info] [client xx.xxx.xx.xx] (104)Connection reset by peer: core_output_filter: writing data to the network
[Mon Sep 14 12:51:28 2009] [info] [client xx.xxx.xx.xx] (32)Broken pipe: core_output_filter: writing data to the network

Occasionally (every 24 to 36 hours) there will be a load spike and the site will become completely unresponsive. Load average climbs from a normal 1-1.5 to 200. Most of the httpd processes that are running will show as 'D' -- deadlocked -- and the only way to get the server to get back down to "interactive" is to three-finger-salute or wait until you get a prompt and killall -9 httpd.

Obviously, the site can't be taken down for me to do a bunch of strace work. I've checked the apache configuration and (again) as far as I can tell, EnableMMAP and EnableSendFile are disabled. The files are on an NFS v3 mount, but neither the NFS server, nor the mysql server, nor anything else, is reporting errors. Nothing appropriate in the system log or dmesg. The site is also too high of a load to reconcile individual requests with errors resulting from them.

At this point, I'm thinking network hardware error and I'd prefer to bring the site up on a second machine. Anyone have any thoughts before I do this?

How does an ethernet switch work on a port-by-port basis? Specifications are given for the speed of the switch "fabric", but is that universal bandwidth through a central processor or are there optimal places to plug things in where the traffic won't have to hit the main bus?

Example: If I have two ports that will be talking to one another a lot, will it be better to put them next to one another or will it be better to put them on different blocks of ports?

How smart are RAID controllers when they're under load?

Given a moderate-to-high-end controller and cabinet (i.e. current standard Dell or HP off-the-shelf kit...), RAID 5, 10gb fiber, at least 3/4 utilized space, and lots of small non-sequential with heavy mixed read/write access to files such as would be found on a file or email server...

Practical implementation question: Given the same amount of space, what would be faster, a small number of large, high speed drives (i.e. 4x 1TB 15,000 rpm) or a larger number of smaller, moderate or low speed drives (i.e. 9x 500GB 7200 or 10,000 rpm) drives?

Theoretical question: Do RAID cabinets/controllers know where the head of a drive currently is and the location they need to seek to so that they can assign reads to the drive with the least head-travel distance? Or does it matter?

What other variables come into play with minimizing response time and maximizing throughput with a large numbers of non-sequential small files on a shared storage array? Note that cache doesn't come into play that heavily because of the nature of the data.

I'm managing a server running NFS v4 with Pacemaker/OpenAIS. NFS is configured to use TCP. When I migrate the NFS server to another node in the Pacemaker cluster, even though the metadata is persisted, connections from the clients 'hang' and eventually time out after 90 seconds. After that 90 seconds, the old mountpoint becomes 'stale' and the mounted files can no longer be accessed.

The 90 second grace period seems to be part of the server configuration and not the client configuration. I see this message on the server:

kernel: NFSD: starting 90-second grace period

If I restart the NFS client on the client nodes after I migrate (unmounting and then remounting the share), then I don't experience the problem, but connections and file transfers still interrupted.

Three questions:

1. What is the 90 second grace period? What's it there for?
2. How can I keep the files from going stale on the clients without restarting them after I migrate the NFS server to another node?
3. Is it actually possible to migrate the NFS server without having large file uploads drop?