I've just moved two sites to a new server. They are both similar, Drupal based sites. One loads normally (1-3 seconds) but the other takes up to 12-15 seconds. After much logging and measuring I've found that most of the delay is waiting for a response from the server. That is, when I load the page in the browser, it takes up to 10 seconds for Apache on the server to get the request. I've double checked the DNS entries but cannot find anything wrong.
How can I narrow down what the cause of this may be?
You'll want to use something that can peer inside the request for clues. The Devel module allows you to see a breakdown of queries that are being run and how long they take. Other tools, like XHProf, give you even deeper insights.
Ultimately, "waiting for response" is a red herring. What it means doesn't line up with the intuitive leap you appear to have made. In this case, it's not that the server is delaying its response, it's that the server is preparing its response. Therefore, your job is to find out what's taking the server so long.
Common causes of this that I've seen in just the last few weeks:
skip_name_resolve)But again, these are all shots in the dark; you'll want to instrument and proceed deductively, rather than SWAGing your way through this.