My employer has a need to add tags to AWS ec2 instances started in OpsWorks once they come online.
The instances are all MS Windows Server 2012 R2 Base in this stack. The goal is to add custom tags like "application":"foo" to the instances started in stack Foo and "application":"bar" in stack Bar.
I've found a github repo chef-aws-tag which I believe would do what I need if I ran Chef 11 on Linux. OpsWorks only does Windows Server 2012, and only with Chef 12.2.
When I add that chef-aws-tag recipe from the github repo to the Setup lifecycle event, the instance fails at the "running_setup" with "setup_failed".
The Chef failure log then reports this failure at the running_setup stage:
INFO: HTTP Request Returned 412 Precondition Failed : No such cookbook: aws
ERROR: Running exception handlers
FATAL: Net::HTTPServerException: 412 "Precondition Failed "
This matches a dependency in metadata.rb, the line: depends 'aws', '>= 0.2.4'
I understand the dependency is missing. It's not clear to me how to fulfil this dependency.
The recipe is in S3, Repository URL is https://s3-us-west-2.amazonaws.com/employer/ec2instance-tagging.zip
When I add the recipe to the Configure lifecycle event, the machine gets online without error but the tags are not added. The "stock" tags show up for the ec2 instance; the Keys "opsworks:instance" , "opsworks:layer:foo_layer" , "opsworks:stack" and "Name" all have Values I expect. The tags I want to add through the recipe ("application", "team", "environment") are not present.
The custom JSON is added at the Stack level:
{ "aws-tag": {
"tags": {
"team": "specialteam",
"application": "foo",
"environment": "development"
}
}
}
Is there a recipe or Cookbook specific to OpsWorks AWS I should be calling to set tags on instances after boot? How do I call that recipe?
I've read this AWS blog post on using OpsWorks to customize app deployment. Am I missing an identical "set aws tags on instance via chef" recipe or cookbook that is in the Amazon Web Services - Labs repository on GitHub?
The aws opsworks-cookbooks on github say "For Chef 12.2 Windows and Chef 12 Linux there are no built-in cookbooks" but I am hoping that is wrong :-) and that a cookbook has been added which I have not yet found. Alternatively, hopefully someone has done this already and documented it somewhere.
I am troubleshooting this following this AWS blog post Quickly Explore the Chef Environment in AWS OpsWorks (again written for Linux) and this SF question on debugging Chef on Opsworks
I was able to figure out some of this on my own. By opening a case with AWS Support I received enough pointers to resolve this.
Speed bumps to work around
Different cookbook directory structures for different platforms when using archive file.
If you provide your recipes through an archive file stored on AWS S3, AWS requires one directory structure for Linux and a second, different, directory structure for Windows.
Linux can put multiple cookbooks in one root directory. This is what
$ berks package
does by default.Windows does not accept that root directory. The
$ berks package
command does not have an option to build one archive with multiple cookbooks in the manner Windows accepts. I had to edit the tarfile manually.Ruby on Windows
There's an
"error: Seahorse::Client::NetworkingError: SSL_connect"
bug that may show up in the 'aws' cookbook. AWS wants https between services. This requires a working Certificate Authority bundle. Ruby on Windows cannot access the OS CA cert bundle. This can be fixed if:a) you install the aws-sdk gem, then
b) set in the recipe the path to
curl-ca-bundle.crt
Details on dependencies and community cookbooks.
The documentation on AWS for Chef 12 on Windows does not mention Berksfile nor community cookbooks. These are documented on AWS for Chef 11 on Linux.
Specify region when calling update.
Odd that running the recipe on the instance says the instance does not exist.
FATAL: Aws::EC2::Errors::InvalidInstanceIDNotFound: aws_resource_tag[i-5ccd24d4] (stemsoft::default line 27) had an error: Aws::EC2::Errors::InvalidInstanceIDNotFound: The instance ID 'i-5ccd24d4' does not exist
AWS error codes says InvalidInstanceID.NotFound can be caused by a) being out of the region, or b) the ID of a recently created instance has not propagated through the system.
Fix is to include
region stack['region']
in theaws_resource_tag
block of the recipe.Correct policies attached to correct roles
Ensure both 'aws-opsworks-ec2-role' and 'aws-opsworks-service-role' are set up with correct policy. 'aws-opsworks-service-role' can be left as-is, so OpsWorks Service can make API calls to other AWS services. 'aws-opsworks-ec2-role', the default IAM instance profile, needs permission on ec2 to create and describe tags.
Procedure
The goal of the steps below is to build an OpsWorks stack that runs Chef 12 on Microsoft Windows Server 2012, with a (custom) cookbook that contains a recipe to set a custom tag. The tag gets its info from custom json set in the stack in my example.
I have not found an existing cookbook that does this. I'd rather use an existing cookbook than build a custom cookbook.
At the end of this procedure, the recipe uses the chef community 'aws' cookbook to update a "team" tag from values set in custom json defined in the stack settings.
To do this, the recipe gets instance details using a search of existing data bag. It also gets region details using a search of a second existing data bag.
(It worked for me....)
git-bash.exe
chef generate cookbook cookbook_dir
$ chef generate cookbook cookbook_dir --berks --copyright 'EMPLOYER' --email '[email protected]'
depends 'aws'
$ ruby -c /path/to/cookbook_dir/recipes/default.rb
$ berks install
to initialize berksfile and track dependenciescookbook 'aws'
$ berks package
to build cookbook_dir.tar.gzINFO: AWS: Updating the following tags for resource i-12345678 (skipping AWS tags): { .... "team"=>"teamFooBar"}
Example recipe
Example recipe
example::default.rb