Questions[opsworks](server)

I am using the EC2 plugin in Jenkins (https://wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+Plugin) for using AWS services.

As of now, a job which uses this cloud label spins up a single EC2 instance. Whereas, I need to spin multiple instances for this job.

A single EC2 instance will never be overloaded and hence the plugin will not spin any new EC2 instances; but the job needs multiple instances to distribute the tasks. There are ~100 tasks to be assigned to 20 machines one at a time (hence 5 per machine). A single instance can complete one task at a time.

How can I spin multiple EC2 instances (fixed number say 20) for a single job in Jenkins?

Thanks.

I'm running an OpsWorks stack with:

• 10 c3.2xlarge instances running across 4 availability zones.
• The CPU never surpasses 5% on each instance.
• Each instance is connecting (through a memcached layer) to a db.r3.8xlarge (32 vCPU, 244 GiB RAM) RDS instance.
• The DB's CPU is staying nicely between 15-20 percent.
• Each EC2 instance is set to 250 connections with the RDS (total 2,500 connections).
• Each instance's memory is gradually increasing but I think this is a separate issue.
• The load_1, load_5, load_10 metrics are all really high 0.9-1.0.
• The active concurrent users connecting (based on a 3 minute window) are getting stuck at between 2500-3000 connections.

We tried increasing RDS instance size and while it did stop our RDS from maxing out we're trying to figure out why the conncurrent connections seems to be pegging.

Is there some setting in the ELB that limits connections? The latency for calls is really high. I'm just trying to figure out something which might be throttling this since all the hardware seems to be running okay.

In short - OS Amazon Linux or Ubuntu. One EC2 instance (or OpsWorks instance?) is running. Once a day start second instance, update it with latest security updates, install my app. Shutdown first instance. Repeat every day...

AWS documentation states:

By default, AWS OpsWorks automatically installs the latest updates during setup... We recommend that you... Create and start new instances to replace your current online instances. Then delete the current instances. The new instances will have the latest set of security patches installed during setup.

How to automate this process?

I know, I can start one micro instance to manage this process via CLI, but I would prefer to use some AWS built-in tool, so I can get notifications on failure, to add some redundancy (if CLI instance fails), everything continues to work on new CLI / AWS API versions etc.

Am I missing some elegant approach?

Update: Ability to use existing Elastic IP is preferred.

So I've followed through this userguide for creating an AMI from an EBS backed Opsworks instance and it all seemed to go smoothly until I attempted to boot an instance in Opsworks from the new AMI.

Here's the setup log as a gist due to it exceeding the post size limit: https://gist.githubusercontent.com/hailwood/d064476a06dff214436378afea6b1930/raw/3024f52a3a93fdd47e3f72ddd4ba5960cb8ff9e2/error.log

But from what I can tell there is nothing interesting there. However when I ssh into the instance and check out /var/chef/runs/42bb....../chef.log I notice it;s only contents are the follwing (I've split this over two lines for easy reading):

/opt/aws/opsworks/current/bin/chef_command_wrapper.sh: line 96: 
/opt/chef/bin/chef-client: No such file or directory

Running a find / -name 'chef-client' tells me that the chef-client is actually installed in /usr/bin/chef-client but I'm not sure that's the issue as it does appear to be successfully running chef based on the error log above.

I have an Opsworks Rails Layer (nginx/unicorn) instance that has been running for a while, and I have successfully deployed my rails app to it many times directly from GitHub.

Yesterday, I started seeing a strange error where our latest code changes are not actually being deployed by Opsworks. i.e. I get a 500 error along with a stack trace that can only be from an older version of my github repo. The line where the error occurs no longer exists in the latest version of the code.

When I SSH into the instance and look at the /srv/www/<myapp>/current folder, I do see the latest code files, and I see a reference to the latest GitHub commit in the Opsworks Deploy log file. It looks like everything is being deployed properly (i.e. I see no errors in the log file and the Deploy finishes sucessfully) but obviously something is going wrong.

Does anyone have any ideas? Is there a code cache on the server that I don't know about? I finally tried stoping/restarting the instance and when it rebooted, I started getting this error: (111: Connection refused) while connecting to upstream - Opsworks Rails 4. I don't know if the two issues are related... maybe this is all about Unicorn not restarting properly... but right now, I'd really like to get to the bottom of why my latest GitHub repo code doesn't end up running on the server after what looks like a successful deployment.

Thanks in advance!