We're currently running our application on Heroku and are seeing the limitations of its platform (mainly due to the cost of scaling).
We're looking to run our own setup on something like Amazon. There are a few competitors out there in the Cloud that offer pure SSD storage etc. that look attractive.
I wanted to basically find a standard test that I could run to spec out the performance of each of these platforms running a comparable instance.
In no way am I a performance expert so I'm not 100% sure what I'm looking for, I just wanted to see some side-by-sides to make sure we make an informed decision.
Can anyone suggest a test suite (or a few) that might give us a relative benchmark for things like disk, memory, cpu performance etc...?
You could try the Phoronix Test Suite or a subset of the tests to gather metrics about system performance. It's an easy way to see the CPU and disk subsystem capabilities, for instance.
The best test of all is your application, though. Benchmarks aren't really the same as your actual workloads. So perhaps you can try the various offerings in a limited basis to see if it works for your needs.