how to do filesystem testing?

My first answer would also be bonnie++, a quick search also reveals Iozone3. Both are available as Debian packages. If you do simpler tests (cp, mv, rm), you should consider clearing the cache kept by the kernel with echo 1 > /proc/sys/vm/drop_caches before each test otherwise it might cause differences that are not representative of the filesystem itself.

Here are two more options, I've found them really useful for filesystem benchmarking.

1) Postmark - it's a benchmark which emulates real-world usage of a filesystem accessed by a busy mail server.

Download: http://www.freshports.org/benchmarks/postmark/

When testing, make sure you've compared different options of ext3 logging - they may be quite different in terms of performance (writeback logging gave best result in my Postmark experiments).

2) Filebench - another great benchmark, giving you even more flexibility. If you have a specific reason for considering a move from ext3, you'll probably like filebench - it has many different workloads so that you can only concentrate on the performance bottleneck which interests you (file create/delete operations or sequential read/write vs random ones, to give you a few examples).

You have to google for it as I'm not allowed any more hyperlinks yet - the project itself is hosted on SourceForge, and a really nice quick start guide is found on OpenSolaris.org website.

One final note: be sure to spread the load across as many disks (spindles) as possible for the most accurate results. It also makes sense to allocate a few disks for benchmarking specifically, and newfs them before each run of your benchmark.

Don't just test for speed, also consider reliability. Try to, e.g., power down disks on a busy filesystem and see what's left.

The quality of the repair and recovery tools available is also important, and very hard to test discretely. Block structure may e.g. inhibit tools that try to salvage data in raw mode from a unsalvageable filesystem.

For more hints on testing a filesystem under very harsh beatings you may be interesting what the ZFS guys did: One Two

We use Rugg to test the file systems of our servers. It's Python based and easy to use. It works by creating a large file which it subdivides, then performs various operations on in sequence or parallel.

Edit: Here is an example script that will test /data (a partition) repeatedly until the script is killed:

#!/bin/bash

(
while true
do
  cd /data
  rm -rf *
  rugg 'zone 97%, subdivide 16 | blank | ensure blank | fill same text, ensure same'
done
) | tee /tmp/rugg.log

exit 0

Some more:

• fio which is similar to filebench mentioned above: "fio is an I/O tool meant to be used both for benchmark and stress/hardware verification. It has support for 13 different types of I/O engines (sync, mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio, and more), I/O priorities (for newer Linux kernels), rate I/O, forked or threaded jobs, and much more. It can work on block devices as well as files. fio accepts job descriptions in a simple-to-understand text format. Several example job files are included. fio displays all sorts of I/O performance information. It supports Linux, FreeBSD, and OpenSolaris."
• File System Exerciser (fsx) originally from Apple but nowadays there are variants for various OSes.

While bonnie++ is primarily focused on bench-marking performance it does work above the filesystem. It performs some tests that give different results per-filesystem. I suspect you are probably interested in the change in performance related to EFS as well, so you may want to try that at least as one of your tests.