Your cluster sounds more like an HPC cluster than an OLTP one like mine, but I think the setup I'm using would work for you too. I call it the "mpdehaan trifecta" because thats the ircnick of the guy who wrote or manages the three tools involved.

1.) Cobbler for base-build provisioning. Cobbler is a project that aims to be the intersection of your kickstart, pxe, yum-repo, dhcp, dns, etc systems. Its by far the easiest way to get a kickstart setup up and running, and you can grow into the other features as needed.

2.) Puppet for configuration management. Ideally your cobbler built hosts are very barebones configs that know just enough to phone home to your puppet server on startup. Puppet will then apply your configuration settings and keep them consistent across your environment in perpetuity.

3.) Func for ad-hoc commands to multiple machines in parallel. For instance "deploy a new svn checkout of the code and restart apache". Its pretty easy to just use func to call the same bash command on a group of servers much like cluster-ssh. If you really want to get into it you can write your own modules for it with some really simple python.

All three of these tools have good wiki's and active irc channels for help on freenode.

Overview

In some ways, you have two questions here..

• How do I build and maintain standard servers?
• How do I maintain standard configuration and make changes later?

I've split my answer below addressing those two things separately but they are very closely related. I am addressing the technology solutions here and not any of the best practices that are related, such as change control.

If this does not cover the scope of your question, please clarify and I will be happy to elaborate. This is necessary foundation, which is critical for a well-run technology infrastructure.

Building Servers

I don't like images in the UNIX world; that is more of a Windows style approach. Even some Windows people seem to be refocusing on scripts for standard builds now.

Satellite seems to be getting somewhat popular in the RHEL world. Spacewalk is the Open Source counterpart. You definitely have to buy into the RHEL approach entirely to use this. This serves as both server building and configuration management.

Ideally, you would want to establish local mirrors and repositories on a fileserver for all necessary software.

First, take advantage of your distribution build automation, such as Kickstart in RHEL/CentOS. The Kickstart would be a baseline with variations, depending on your needs. The Kickstart builds can be initiated from a PXE server.

For the more advanced part of the build and anything that was not suitable for a Kickstart file, you could write your own custom scripts. However, you may find puppet or cfengine works well for you instead of custom scripts. I have found custom scripts to be the most flexible and are not limited to any single approach.

If you choose to write your own scripts, I recommend a core script for universal configuration. This would be security configuration, hardening, and anything that applies to all builds. Then a final script to finalize the server role. For example, a web server or a database server.

Maintaining Standards

What you describe also falls under maintaining configurations. Build standards, software updates, and other things are related to builds but in a lot of ways separate.

If you choose to rely on system packages as opposed to creating your own source based builds for your most important server roles, a lot of that can be maintained with native system utilities. This can be as simple a script to run a for loop against your server list and run a yum -y update package.

For configuration management, this is where puppet, cfengine, and other configuration management utilities come into play. These are very useful utilities and provide the necessary foundation without writing your own scripts from scratch.

When you update your configuration standards for your servers, it is important to backfill this into your standard server builds.

If you go down the pxe route, be sure to have a look at

http://etherboot.org/wiki/index.php

Gpxe will give you more flexibility with boot targets . It's quite easy to boot of an aoe blade and there's nothing like booting a kernel off a remote http server!!!!!!!!!! :-).

What server up times do you need?

It's impossible to create a perfect image, as your always going to need to apply security patches and software upgrades. If they are internet facing, you can't just ignore patching.

On the pxe side, you've got a few options, depending on your file I/O of you cluster. Either just boot a kernel and mount disk remotely over aoe or iscsi.

You can also do some very clever stuff with copy on write images. It's great for upgrades and rolling back any changes that might be problematic.

I've also had success using nfs root , using a clustered nfs solution. You can specify different files to be served depending on their client addresses.

Again, you must check if your application(s) likes running off nfs. It's not suited for every workload.

clustered nfs sever can contain 192.168.0.1:/etc/hostname 192.168.0.2:/etc/hostname

so, every client references the same file, but get served the file that's relevant for the client. It's pretty impressive stuff, however it's not simple!

All this will give you quicker rollout times if you centralise the file system on network storage. Imaging an operating system over a network to a remote disk takes time!!!!

If your using any of these solutions, make sure you have a well designed and fault tolerant network layer and that your nfs/SAN server is well designed + secure!

Losing the connection your NFS/SAN would be bad for server health. :-(

It's quite easy to craft some scripts for tftp/pxe to control the boot process.

If I knew more what your actually trying to cluster I could perhaps think of a solution which was more suited for you.

I recently finished a big project to roll out a centralized build/provisioning and configuration management system at $WORK. We're running CentOS.

My design, which I happen to really like, gives us a virtually one-click (well, one web GUI page) build process, using some custom PHP scripts to tie everything together through a simple but effective web UI.

The general theory is:

1. Do all installs from a single, unified, minimalist KickStart file (well, OK, one for x86 and one for x86-64, but still, virtually identical files with minimal package selection).
2. KickStat postinstall script bootstraps Puppet.
3. Puppet applies all node/host-specific configuration, package installation, etc.

I agree that images aren't a Unix-y way of doing things... they're really more suited to the Windows world where scripted/automated installation and configuration isn't as simple.

You can replace Puppet with any other configuration management system that fits the bill, but I happen to like Puppet's declarative nature and its concept of convergence.

This system yields a number of benefits:

• Puppet handles everything past a generic base install, so all required packages and configuration is in one place.
• The Puppet manifests serve as an additional source of low-level documentation.
• As long as you stick with Puppet (i.e. don't make local config changes, or merge them back into Puppet) it's trivial to build a duplicate of a machine.
• Since all hosts are built from a common base, you can pre-install hardware or VMs with the base (KickStart) package set and then turn them into functional nodes simply by adding classes as needed.
• Puppet allows "tagging" hosts for production or development, so it's incredibly easy to build a development/testing copy of a host, upgrade packages or make config changes as needed, test, and then merge back into production.