rugbert's questions

I have my scaling triggers set to look for an un-healthy host count but it does not seem to be working.

Now to test this - I am SSHing into one of my instances and halting the HTTPD service. Then, when I navigate to the health overview, I will immediately see that the server I SSHd into now has the status of severe.

I would assume that at this point, after 1 minute has passed (as per my rules) a new server would be created, but that is not happening.

If I am understanding my rules correctly - there is now 1 (the above the upper threshold) unhealthy server, so we increment up 1. And then once the number of unhealthy servers is 0 (below the lower threshold) then remove 1 sever.

But yeah, I waited around 5 minutes and no new EC2 servers were provisioned.

I also have some settings for the health check:

Is this conflicting with my autoscaling rules somehow? I thought that the healthcheck file needs to return a 200 response to be considered healthy and if HTTPD is halted - they it would not return that response.

So what gives?

I have an Elastic Beanstalk server which I am using for my employer's main site, example.com and they want me to host one of their ancillary sites on it: go.example.com.

So I just created a new ebextension config to create a second vhost. The problem I found is that Apache (HTTPD) only wants to use the first vhost entry.

Here are my vhosts:

 # I have Apache listening on port 8080 because I have varnish in front of my sites.
    <VirtualHost *:8080>

        ServerName      example.com
        ServerAlias     www.example.com
        ServerAdmin     [email protected]

        RewriteEngine On
        RewriteCond %{HTTP_HOST} ^example.com
        RewriteRule ^(.*)$ http://www.example.com%{REQUEST_URI} [R=301,L]

        DocumentRoot "/var/app/current/httpdocs"

        <Directory "/var/app/current">
            AllowOverride All
            Require all granted
        </Directory>

        <Directory "/var/app/current/cgi-bin">
            AllowOverride All
            Options None
            Require all granted
        </Directory>

        <Directory "/var/app/current/httpdocs">
            Options FollowSymLinks
            AllowOverride All
            DirectoryIndex index.html index.php
            Require all granted
        </Directory>

    </VirtualHost>
    #go.example.com
    <VirtualHost *:8080>

      ServerName      go.example.com
      ServerAlias     staging.go.example.com
      ServerAdmin     [email protected]

      DocumentRoot    /var/www/go.example.com/httpdocs

      <Directory "/var/www/go.example.com">
          AllowOverride All
          Require all granted
      </Directory>

      <Directory "/var/www/go.example.com/httpdocs">
          Options FollowSymLinks
          AllowOverride All
          DirectoryIndex index.html index.php
          Require all granted
      </Directory>

So the server will always listen for example.com, and with the above vhost order example.com will serve /var/app/current/httpdocs while go.example.com is just a blank page.

If I swap the vhost order, so go.example.com is first, then example.com serves /var/www/go.example.com/httpdocs. And go.example.com is still a blank page.

Nothing is really jumping out at me, and I dont have this problem is I build a regular ol' EC2.

Our WordPress site has multiple views for every page and post depending on whether or not a specific query string is used.

For example, if visited example.com/blog-post-with-chart (a page which includes a chart as the body content) you would see the entire page. The navbar, content, footer..everything.

However, if you went to example.com/blog-post-with-chart/?view=embed then everything but the chart would be removed. So you would only see the chart. It's how other sites might embed our charts.

WordPress's get_query_var('view') doesn't return anything. I have also tried to just var_dump or print_r the $_GET superglobal which also dont show anything.

I've noticed that this only doesnt work if "view" is the first query string parameter. so If my URL is example.com/blog-post-with-chart/?fake=arg&view=embed then the chart view will work as expected.

It looks like Varnish is dropping the query arguments from being passed to the server. If I add the query parameter "nocache" then everything works expected. And that is because I have a Vanish rule:

            if (req.url ~ "nocache") {
                set req.backend = default;
                return(pass);
            }

As a workaround, I added a new rule to bypass the cache if the url contains view=

            if ( req.url ~ "view=" ) {
                set req.backend = default;
                return(pass);
            }

But how can I stop Varnish from "dropping" the query args? We definitely want that page cached, and we also want it cached with the query string so that embedded view is always cached.

EDIT - here is the VCL file I am using: https://gist.github.com/kyle-jennings/589e513808fa83b93a27d1cf3c10c642

The problem:

30-40 minutes after pointing the DNS from our old server towards our new server, all available memory gets used up and our (3) load balanced EC2 instances crash.

To make matters worse, it doesn't appear that Elastic Beanstalk is terminating instances which have crashed. I think that is because we can only select a single auto scaling trigger and memory usage is not one of the available triggers.

According to Chartbeat, our website seems to get 200-400 concurrent users (Google Analytics Real-Time reports shows 60-80 users).

I should also point out that I have "solved" the issue but installing Varnish on the EC2 instances. With Varnish in place, the servers do not crash and the frontend is very very fast. However, I wanted to know whether or not this was normal behavior for 200 users on 3 load balanced servers. I'm worried that there is something very wrong, or something that could be tweaked.

Spec overview:

On AWS we are using

• 3 to 6 load balanced and autoscaled t2.large EC2 servers (2 vCPUs and 8gigs memory)
• managed by Elastic Beanstalk (for the Github integration)
• a classic load balancer
• Cloudflare is used for the DNS and SSL termination.
• Apache 2.4
• PHP 5.6
• PHP FPM
• 64bit Amazon Linux/2.7.1 AMI

The configs for PHP and FPM are below:

• PHP info
• PHP and FPM settings

What I have found

I am switching the DNS around 10pm EST when traffic is low (200 users according to Chartbeat and 60 according to GA) to test things and gather info.

After about 30-40 minutes, all memory gets used up. Unfortunately I wasn't monitoring closely enough to notice whether the memory use steadily increased or if it just spiked. You can see from the image that latency times exploded as well.

At this point I checked the logs and saw that the server reached it's max_children setting:

[19-Sep-2018 22:50:40] NOTICE: fpm is running, pid 6842
[19-Sep-2018 22:50:40] NOTICE: ready to handle connections
[19-Sep-2018 23:03:21] NOTICE: Reloading in progress ...
[19-Sep-2018 23:03:21] NOTICE: reloading: execvp("php-fpm-5.6", {"php-fpm-5.6"})
[19-Sep-2018 23:03:21] NOTICE: using inherited socket fd=9, "/var/run/php-fpm/php5-fpm.sock"
[19-Sep-2018 23:03:21] NOTICE: using inherited socket fd=9, "/var/run/php-fpm/php5-fpm.sock"
[19-Sep-2018 23:03:21] NOTICE: fpm is running, pid 8293
[19-Sep-2018 23:03:21] NOTICE: ready to handle connections
[19-Sep-2018 23:33:01] WARNING: [pool www] server reached max_children setting (200), consider raising it

I should probably increase the max_children back up to 420 from 200. But I guess I wasn't realizing what max_children does (it handles each request right? And each page view could request multiple images, css, a php file, JS calls ect ect?).

But I was hoping that 3 EC2 servers would be able to handle this load. Especially considering that the current, older infrastructure (Rackspace) is basically 2 servers: 1 varnish cache and 1 server that serves the frontend of the site. Neither of those servers seem beefier than the new AWS servers, they only have 4 gigs memory. The PHPFPM configs are also much lower on that server:

pm = dynamic
pm.max_children      = 20
pm.start_servers     = 8
pm.min_spare_servers = 5
pm.max_spare_servers = 10

And thats whats crazy to me. How can the old server (plus varnish cache) with lower specs and lower fpm settings handle all this traffic, but my 3 to 6 load balanced EC2 servers cant?

Next Steps

1. Maybe EC2 servers just really suck compared to the old Rackspace server and I need to choose larger instances?
2. The RDS database is a big bottleneck, and until I adjusted it's settings wouldn't allow more than 40 connections. Maybe I need to use an EC2 server running mysql? (I have another, separate but related question open about this)
3. Memcache or Redis via elasticache might help so long as I can ensure it doesn't interfere with the admin section.
4. Opcache is enabled by default in php5.6 but is there anything else I need to do use it?
5. Add memory monitoring and additional autoscale triggers to elastic beanstalk

I recently migrated a WordPress website I inherited over to AWS from Rackspace and am experiencing large performance decreases. I'm pretty new to the whole DevOps thing so I'm not sure where to start looking.

The problem:

After I point the DNS (Cloudflare) from our old server to the new one, WordPress (specifically in the admin section) things are find for a short bit. But after an hour or so admin page loads timeout, or take up to 30 seconds to load. I haven’t tested all pages just yet, but this seems to happen on pages with editors, so “new post” or editing a page. When the page times out, I see a Cloudflare timeout message.

I should note that I am making this switch at around 11pm at night, no one but me should be logged in, and web traffic is at about 150 users.

I have also noticed that our RDS instance starts hitting 100% CPU usage, and the DB connections skyrocket up to about 1000 connections.

Because RDS has WELL surpassed it's max_connections limit, WordPress can no longer connect to the database and now the front end of the site shows the "unable to establish connection to database" message.

At this point, I could restart my ECV2 instances, but the RDS database is still sorting out the 1000 connections.

I have also noticed that it seems the (classic) Elastic Load Balancer stops distributing traffic evenly across both instances.

I'll have another go at it this weekend, but what should I be looking for in the logs? Before the EC2 instances shut down I tailed the logs and al I saw was:

pid 17186:tid 139743773734656] (70007)The timeout specified has expired: [client 127.0.0.1:58604] AH01075: Error dispatching request to : (polling), referer: http://m.facebook.com

Spec overview:

Server - On AWS we are using 2 to 6 load balanced/autoscaled m3.large EC2 servers, managed by Elastic Beanstalk (for the Github integration) and a classic load balancer, Cloudflare is used for the DNS and SSL termination.

We are using Apache 2.4 with PHP 5.6, and PHP-FPM all on a 64bit Amazon Linux/2.7.1 AMI

RDS - R3.Large running Aurora and part of a cluster with a read replica running. I tried using an R3 Large a couple days ago, and admin page loads were 15-30 seconds… Still very slow.

I should also mention that RDS was setup on it's, outside of Elastic Beanstalk. I don't think that should matter tho. There are however 2 other databases on that server, for a couple smaller sites that get basically no traffic and will be decommissioned soon.

• PHP info
• PHP and FPM settings
• Varnish Settings
• PHPfpm install script
• HTTPD configuration script

I have enabled object caching via W3TC, and I have added some Cloudflare rules to disable performance and apps for /wp-admin* as recommended here

A couple things I've read around the internet

• Maybe the ELB timeout limit is lower than the Apache limit and thats causing issues
• This article suggests changing my MPM from event to prefork or worker