Greg's questions

I wish I could be more specific about this issue and I am really asking for suggestions on where to look next.

We are running a PHP web app that is integrated with WordPress and that is serving a page every second during some parts of the day. Generally things run very well on a single dedicated server (quad core with 16GB RAM).

I am considering New Relic and their tools alerted me to the occasional page load where PHP seems to stall. It reaches a seemingly arbitrary point in the call trace and then spends many seconds (or tens of seconds) on something trivial. The most trivial example was a function that was a conditional followed by echo().

There are no other errors (in the Apache/PHP error log) or slow queries I can see that coincide with these events. They also don't seem to coincide with any particularly heavy CPU load, disk I/O or network I/O.

It happens a few times an hour, and one thing in common between all the functions that stall is that they are doing output to the page. Could something be blocking the output buffer? Or are there any other obvious culprits that might be causing this issue? What would you do next to troubleshoot?

Linux: CentOS 5.6

Apache: 2.2.3

MYSQL: 5.5

PHP: 5.3.9

APC: 3.1.9 (cache is healthy, hit rate close to 100%)

I am new to CDNs and experimenting with CloudFront. I have set everything up and all appears to be working fine. I can create a static image on a page and access it though my CloudFront distribution. I am using a custom origin (i.e. not an s3 bucket).

I'm worried that I might be worse off from a performance point of view though. I have a test page that is loading up the same 20 or so images with and without the CDN. Looking at the net panel in Firebug, the first time I load this page the images that are loaded directly from the origin server come in much faster. On subsequent page loads the benefits of the CDN become obvious -- after 3-5 refreshes the CDN is doing better than the origin server.

So I can see that on a popular page on our site that is being hit all the time, this will be a benefit. And I should expect a benefit because I'm in Seattle (around the corner from Amazon) and my server is in CA.

The thing is that if I leave the page for a few minutes and then reload, things are back to square one, with CloudFront being worse than the origin server. Is this expected? Do things drop out of the CDN "cache" so quickly?

Is it possible that something in my setup is hurting performance? Or is the reality that the CDN will only be a net positive for content that is currently being accessed every few seconds on average?

(cross posted from the AWS forum because I've been spoiled forever by SO's turnaround times)

UPDATE:

There are two good answers below that are worth looking at if you have questions about CloudFront performance. I recently found one explanation for my specific problem wasn't mentioned though. I had left TTL at 5 minutes as an oversight. Since I'm also using a custom origin there is an additional round trip to the authoritative nameserver to resolve that to the actual Amazon CloudFront domain. Now that the TTL setting is back to 12 hours it seems that the long loads happen more seldom.

I was looking at my Apache access log and came across some strange requests. A single IP address will access several resources (mostly css style sheets and images), but no actual pages.

Sometimes they are requesting a resource that no longer exists on the server, or one that is still under the web root but no longer used (e.g. a resource in an old WordPress theme).

Also:

• The requests list no referrer
• I get no useful information on the IP address by looking it up
• There doesn't seem to be any pattern among the IP addresses that are making these requests (e.g. different countries)

Are these just links from a stale cache somewhere? Could it be a sign of an attack of some sort?

Here is a typical example:

GET /wp-content/themes/my-theme/images/old-image.gif HTTP/1.1" 500 809 "-" "Mozilla/4.0 (compatible;)"

This was one of about 10 similar requests, some for existing resources, some for older resources. There is no other sign of this IP address in access_log.

Note the internal server error, which is a topic for a different thread. What I'm asking here is where would isolated requests like this come from?

I have a LAMP server (CentOS-based MediaTemple (DV) Extreme with 2GB RAM) running a customized Wordpress+bbPress combination .

At about 30k pageviews per day the server is starting to groan. It stumbled earlier today for about 5 minutes when there was an influx of traffic. Even under normal conditions I can see that the virtual server is sometimes at 90%+ CPU load. Using Top I can often see 5-7 httpd processes that are each using 15-30% (and sometimes even 50%) CPU.

Before we do a big optimization pass (our use of MySQL is probably the culprit) I would love to find the pages that are the main offenders and deal with them first. Is there a way that I can find out which specific requests were responsible for the most CPU-hungry httpd processes? I have found a lot of info on optimization in general, but nothing on this specific question.

Secondly, I know there are a million variables, but if you have any insight on whether we should be at the boundaries of performance with a single dedicated virtual server with a site of this size, then I would love to hear your opinion. Should we be thinking about moving to a more powerful server, or should we be focused on optimization on the current server?