James Simpson's questions

We are in the process of moving from AWS where we have a highly available system setup using EC2's auto scaling feature. However, we aren't using this to change the size of the pool based on resource usage, we are simply using it to spin up new instances when one of them fails or becomes unresponsive.

Without this auto scaling feature on other cloud providers (we are specifically looking at DigitalOcean, but it should apply anywhere), what are some options to achieve this setup? My first thought was to create an instance that monitors the others, but then that server becomes a single point of failure. Are there any services or established patterns to accomplish this whether automated or writing some scripts to the API without creating a single point of failure?

I installed MongoDB on Ubuntu 14.04 as per the documentation. It runs and works fine, except that it mysteriously stops every few hours and requires a manual restart with service mongod restart to get going again. There's nothing in the mongod.log when this happens.

In addition, when I restart and then check status with service mongod status, it gives me mongod stop/waiting event hough the application is able to connect to it. I've been searching around on how to debug this, but am stuck at this point.

I'm trying to setup keepalived + HAProxy as a redundant load balancer on an EC2 VPC (yes, I know that ELB is an option). I believe we have things configured correctly, but killing the master server doesn't seem to failover.

Server A Config:

vrrp_script chk_haproxy {
  script "pidof haproxy"
  interval 2
}

vrrp_instance VI_1 {
  interface eth0
  state BACKUP
  priority 100
  nopreempt

  virtual_router_id 33
  unicast_src_ip 172.30.1.100
  unicast_peer {
    172.30.1.101
  }

  authentication {
    auth_type PASS
    auth_pass PASSWORD
  }

  track_script {
    chk_haproxy
  }
  notify_master /etc/keepalived/master.sh
}

Server B Config:

vrrp_script chk_haproxy {
  script "pidof haproxy"
  interval 2
}

vrrp_instance VI_1 {
  interface eth0
  state BACKUP
  priority 100
  nopreempt

  virtual_router_id 33
  unicast_src_ip 172.30.1.101
  unicast_peer {
    172.30.1.100
  }

  authentication {
    auth_type PASS
    auth_pass PASSWORD
  }

  track_script {
    chk_haproxy
  }
  notify_master /etc/keepalived/master.sh
}

I've setup the security group rules to:

HTTP              TCP           80   0.0.0.0/0
Custom ICMP Rule  Echo Reply    N/A  0.0.0.0/0
SSH               TCP           22   0.0.0.0/0
Custom Protocol   VRRP (112)    All  0.0.0.0/0
Custom ICMP Rule  Echo Request  N/A  0.0.0.0/0

However, the following command always times out from the backup (and same with reverse on master):

nc -vz 172.30.1.100 112

Also, the following command never returns anything, making me think these are still not going through for some reason:

sudo tshark -f "vrrp"

I'm trying to setup an auto-healing EC2 instance using an Auto Scaling Group and a user-data startup script. If the current server has an issue where it is no longer reachable, the instance should terminate and a new one take its place. This is easy enough, but one requirement is proving difficult.

I need the replacement server to have the same private IP as the previous server. My thought is to have a secondary private IP (this is within a VPC) assigned to the original server, and then re-assign it to the new server.

1. I assume I can use the aws-cli installed during the user-data startup script to re-assign the private IP, but how do I know which server is being replaced and re-assign the IP from it (for example, if in the future the pool of servers is larger and 2 happen to go down at the same time).

2. If the original server is being terminated, am I going to be able to re-assign the private IP at all?

I've got a site with a LAMP stack and I'm looking at moving to a new hosting provider (currently on a dedicated server and looking to move to a cloud provider). What is the best/most reliable way to compare performance of my site on the different providers? Obviously I would spin up an instance and transfer of a copy of my site, but are there any tools or specific things I should be looking at to compare performance?

I've been using MongoDB on CentOS with ext4 for a few months now with no problems. I'm potentially going to be moving to Joyentcloud, and their SmartOS is based off of OpenSolaris and uses ZFS. I'm wondering if there could be some downsides to switching to this for MongoDB?

I currently use CentOS for on my server, and I've been trying to figure out the practical differences between Linux and OpernSolaris. I'm not a linux master, I merely know my way around the system and can generally install things if I need to (though I won't lie, I get tripped up on that sometimes).

If I switch to OpenSolaris, are there going to be major things that I am unable to do now or that at least won't work the same way? My stacks mainly just consist of PHP/MySQL or Node.js/MongoDB.

For the past few weeks I've been getting more and more reports about lag on one of my sites. I've finally been experiencing it first hand over the last week, but I haven't been able to pinpoint the problem.

The server load is never higher than about 0.5 out of 16 cores, and the memory usage tops out at around 12-13%. The issue isn't the database as the lag can happen on static resources. About 1 out of 10 page views gets a 502 error. About 1 in 5 pages takes 5-20 seconds to load. When looking at Chrome's network tab, it shows "waiting" for almost all of that time.

I rebooted the server last night, and it seemed okay for a few hours, but less than 12 hours later it was back to the normal lag issues. Anyone have any tips on where I can look to try and figure out the problem?

I've got a single dedicated server with a MongoDB database of around 10GB. I need to do daily backups, but I can't have downtime with the database. Is it possible to use a replica set on a single disk (with 2 instances of mongod running on different ports), and simply take the secondary one offline and backup the data files to an offsite storage such as S3 (journaling is turned on)? Or would using master/slave be better than a replica set?

Is this viable, and if so, what potential problems could I have? If not, how do I conceptualize this to work?

How do I setup MongoDB so that it will start back up automatically after a server reboot? This is on CentOS 5.5.

I start it with the following command, but if the server reboots, I have to login to SSH and run that again to get it going (not that big of a deal really, but would be nice to know that this isn't necessary if I don't have access to the server).

./mongod --config ./mongodb.config --fork --logpath /var/log/mongodb.log --logappend

I'm comparing different cloud hosting offerings and am trying to do detailed cost comparisons. A lot of the other providers have most everything bundled, like bandwidth, but Amazon's is all scattered. Since I am only used to using dedicated servers, I'm not 100% sure on everything I would be paying for with EC2 (for example if I had 2 small nodes).

I know that no bandwidth is included with an instance, so that is an extra $0.15/GB on top of the hourly instance cost. If I understand correctly, I need an elastic IP per instance ($0.01/hr). From what I gather it is also important to have EBS and possibly the elastic load balancing (however I'm not the exact needs for these so I'm not sure how to price them)? Any other costs involved?

I am working on a new project that will be hosted with Amazon EC2. Speed of requests is of utmost importance, and the low memory footprint of Nginx is very compelling. I've always used Apache, but if significant advantages can be had with Nginx then I would like to switch to that.

I see that a lot of people use Apache + Nginx, but are there any disadvantages as of 2011 to using Nginx with PHP-FPM to serve both static and dynamic content? I know that .htaccess is no longer an option with Nginx, but I also understand that there are ways around this.

Basically, will you get the best performance out of Apache + Nginx, or just Nginx?

I've got a large multi-year access_log and I need to view all of the entries for a certain IP address from SSH. The format is:

111.111.111.111 - ...

222.222.222.222 - ...

111.111.111.111 - ...

How would I only view the log entries for 111.111.111.111? I'm running CentOS.

This is a purely theoretical question, but what if I had a site that would normally only get a couple thousand hits a day, but for a few days each month that could shoot to several hundred thousand or even several million hits over the period of 1-3 days.

The site would be pretty bare-bones (as in, 2-3 total pages with 1-2 max MySQL queries on each page and some PHP), so bandwidth wouldn't be the issue, but sheer volume taking down the site would be the main concern. Cloud hosting seems like the best way to go, but would something like Amazon EC2, MediaTemple, or something else be the right choice in this case?

I've been using MyISAM exclusively for several years now and know the ins-and-outs pretty well of how to optimize it, but I've just recently started using InnoDB for some of my tables and don't know that much about it. What are some general tips to help improve the performance of these InnoDB tables (they were converted from MyISAM and have anywhere from 100k - 2M rows and most won't use transactions).

I just converted a MyISAM table to InnoDB with around 1.4 million rows. When I converted it to InnoDB, it now shows -1.4 million rows. The table still works as expected, but why does it shows negative in the rows column?

I was just looking around cPanel and noticed that the default mail account is taking up over a gigabyte of storage. I opened it up in webmail and saw there are over 200,000 spam e-mails sent to randomly generated e-mails like [email protected]. I am unable to delete them because the webmail installed on cPanel can't handle such a large number of e-mails and won't load the inbox. How do I clear out these e-mails?

If it matters, I am running CentOS 5.5 on a dedicated box, but I'd rather not have to delete from the command line if at all possible.

I'm testing out a new CDN service (http://cloudflare.com), and I'm trying to figure out if it is giving much of a gain. I have a very fast internet connection so I can't see any increase in speed (though I realize since I am fairly close to my server I shouldn't see a big gain anyway compared to international users), but I'd like to calculate how much gain there actually is. Is there anyway to do this?

I've been using MySQLTuner to optimize my database, and the only problem I am having is:

[!!] Maximum possible memory usage: 14.8G (126% of installed RAM)

I'm not sure what I should be changing to get this down under my 12G of RAM. I have a database with 110 tables, 10,000,000 rows (growing fairly quickly), and an average of 250 users online. My my.cnf has looks like:

query_cache_limit=1M

query_cache_size=128M

query_cache_type=1

max_user_connections=2000

max_connections=2000

interactive_timeout=100

wait_timeout=100

connect_timeout=100

thread_cache_size=128

key_buffer=16M

join_buffer=1M

max_allowed_packet=16M

table_cache=15360

record_buffer=1M

sort_buffer_size=4M

read_buffer_size=2M

max_connect_errors=10

thread_concurrency=8

myisam_sort_buffer_size=64M

server-id=1

I have a server with Dual Nehalem Quad Core Xeon 5520 and 12GB DDR3 RAM. The server's average memory usage is only around 10-15%, yet the Swap used shows 10%+. Is this normal, or could something be wrong that is causing this. I was under the impression that Swap was only used if there wasn't enough memory available.

I am running Apache/2.0.63 on CentOS 5.3.