Questions[configuration-management](server)

I am investigating Ansible for server and application provisioning. My application is currently provisioned with shell scripts in Vagrant. Rather than rewriting my scripts, I took a sample and attempted to deploy it.

It appears to deploy fine, but I have seen a failure message after what looks like a series of successful steps:

» vagrant provision                                       ~/vm/blvagrant 1 ↵
[default] Running provisioner: ansible...

PLAY [web-servers] ************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.9.149]

TASK: [install python-software-properties] ************************************
ok: [192.168.9.149] => {"changed": false, "item": ""}

TASK: [add nginx ppa if it ubuntu 10.04 and up] *******************************
ok: [192.168.9.149] => {"changed": false, "item": "", "repo": "ppa:nginx/stable", "state": "present"}

TASK: [update apt repo] *******************************************************
ok: [192.168.9.149] => {"changed": false, "item": ""}

TASK: [install nginx] *********************************************************
ok: [192.168.9.149] => {"changed": false, "item": ""}

TASK: [copy fixed init for nginx] *********************************************
ok: [192.168.9.149] => {"changed": false, "gid": 0, "group": "root", "item": "", "mode": "0755", "owner": "root", "path": "/etc/init.d/nginx", "size": 2321, "state": "file", "uid": 0}

TASK: [service nginx] *********************************************************
ok: [192.168.9.149] => {"changed": false, "item": "", "name": "nginx", "state": "started"}

TASK: [write nginx.conf] ******************************************************
ok: [192.168.9.149] => {"changed": false, "gid": 0, "group": "root", "item": "", "mode": "0644", "owner": "root", "path": "/etc/nginx/nginx.conf", "size": 1067, "state": "file", "uid": 0}

PLAY RECAP ********************************************************************
192.168.9.149              : ok=8    changed=0    unreachable=0    failed=0

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

How do I go about getting additional debug information? I've already added ansible.verbose = true to my vagrant config which results in the dictionaries being displayed within the output above.

I have over 200 computers which can provide IPMI services. The servers are manufactured by several different companies (SuperMicro, Dell, etc.), and there are 6-7 BMC models from about 5 different vendors, and each model has it's own idiosyncrasies.

So far we have been configuring the BMCs by using a combination of DHCP and manually configuring each BMC. The manual configuration might be done using a bootable CD-ROM, configuration from the BIOS (If supported), from the host operating system with a utility like ipmitool, freeipmi, etc. or remotely using ipmitool if we can determine the network address of the device.

However, this manual configuration is rather tedious. In some cases we want to change a setting globally on all BMCs, which requires that an administrator run a command against dozens of boxes. Since the BMCs are provided by different vendors and each model of BMC might have it's own idiosyncrasies, the same command does not always work on all BMCs.

Are there any utilities which allow me to mass configure the BMCs on dozens of boxes? Say that I want to query a parameter on dozens of different BMCs, or change the password, disable HTTP access to the WebUI or disable the infamous cipher zero security hole.

Bonus points for any utility which would allow me to update the BMC firmware, which is necessary to mitigate several security vulnerabilities

I use and love Puppet. I moved to a new company and they are adopting Chef. So I'm trying to learn Chef but am having a hard time piecing it all together because I still think in Puppet =)

These are my questions:

• Is it better to setup roles in Ruby DSL, JSON, or from the management console? Why are there multiple ways to do the same thing?
• Can you organize cookbooks into subdirectories? eg: we have custom software that I'd like to write a cookbook for and stick that into: chef-repo/cookbooks/ourcompanystuff/customsoftwarecookbook would this be a good practice?
• Do I create a cookbook for each type of role that specifies what it does? Do I have these cookbooks include other cookbooks (i.e. the cookbook for my webserver role includes the apache cookbook). I'm not sure how cookbook inter-dependencies and inheritance are handled.
• Is there anything like Puppet's external node classifier so nodes automatically determine their roles?
• It seems like you can configure things with knife or within the management console, or editing JSON files? This is super confusing to me why there are so many ways to do things, it's paralyzing! Is there a reason to use one or the other? Coming from puppet it seems like it'd be easy to accidentally mis-configure something with these tools (i.e.- leaving something out)
• How can I automatically provision nodes with Chef in my dev cluster? With Puppet I fire up a VM that connects to the puppermaster and kicks off a puppet run and sets itself up (role is determined by external node classifier). How do I do this with Chef? Install chef with pem/rb files that tie it to a chef server, manually tell the node its roles with knife or editing this in the management interface, and then kicking off a chef-client run to set itself up?

I completed the getting started tutorials and I see they have EC2 tutorials, but I've never used EC2 so they're hard for me to follow. At this point I have hosted Chef running and I'm starting to play around with configuring a single node. Where do I go from here? Do I need to start looking at the public cookbooks?

The docs on Opscode are ok, but not nearly as great as Puppet's. Are there any other good Chef resources I might be missing in my searches?

So we've all probably had this situation: you debug some problem, only to realize it was caused by a config change you made six months ago, and you can't remember why you did it. So you undo it and fix the problem, and now some other problem comes back. Oh yeah, NOW I remember! Then you fix it properly.

It's because you didn't take proper notes, you fool! But what's a good way to do this?

In engineering we have loads of software meant to help us detect and track changes. Source control, code reviews, and so on. Every change is tracked, every change requires a comment as to what it is. And typical engineering departments require good comments so that in six months when you're figuring out why you broke it like that, you can use a historical 'blame' feature or binary search builds to pinpoint the problem. These tools are very effective communication tools and historical records.

But in serverland, we have 500 different services, all with different ways of configuring them. And they don't always have a text format (consider setting permissions on a folder or altering the pagefile location) though they may have a textual representation.

In our environment, we check in what config files that we can into Perforce, but there are very few of those. Can't exactly check in the Active Directory DB..though perhaps a dump that could be diff'd...

In the past I have tried keeping a manual change log in our wiki, but it's super hard to maintain the discipline to do this (I know, not a good excuse, but it really is tough).

MY QUESTION: What strategies and tools do you use to cope with this problem of tracking configuration changes to your servers?

-- Update --

Note: I'm not looking for shared-note taking tools (I'm familiar with OneNote, etc) so much as automated tools specifically meant to help with tracking server changes. There's no comprehensive tool for tracking server config changes, but perhaps there are some for specific applications like GPO's.

Also I am very interested in specific strategies that you've found useful. "We share notes in Sharepoint" is pretty vague. How do you maintain the discipline? What format do you use to track your changes? How do you organize your change data? I'd really like examples as well as ideas.

I manage several Linux servers for clients in several roles like email, caching, web serving, filtering, firewalling/routing, and so on.

Since I don't own these computers and just provide remote support, central management systems like Puppet don't seem like they are the correct tool. (Please correct me if you think I am wrong about this assumption)

What tools do you recommend to track changes of configuration files, package installs and so on?

I am thinking something like etckeeper may be near to what I need, but I want to know if there is something better.

Update

We will have backups of the systems, and I wouldn't expect this type of a tool to be an alternative to a backup. This is about keeping track of changes of configuration and having a system to know what changed when, by who, and hopefully why.