Imagine you're going to manage a number of servers with a number of different services that's used by a number of people. Now say you want to reconfigure or replace some software on one of those servers. Obviously you don't want to work on servers that are in production.
If this was a code change, as a developer, I would make the change on my local development machine, test it locally and commit the change to a version control system. The changes could then be deployed in a staging environment, tested further and finally deployed in a production environment. It would also be easy for me to roll back, if necessary.
Generally, or specifically, how do you achieve this in system administration?
(The first thing that comes to mind is to use virtual machines and put virtual machine images in version control, but I'm sure there is a lot of literature and clever solutions I'm not presently aware of.)
Disclaimer: I am one of the developers of Puppet.
The obvious way is to just apply the concepts: define a cycle of development/test/production, and push changes through them. Use version control to track systems.
In short order, starting down that path leads to discovering that you really want tools that automate those things - essentially, that you want to automate systems administration, so that you don't use those techniques on machines, you use them on the system that manages the machines.
Tools like Chef, Puppet, Salt, and CFEngine are all popular tools to address that second need. They work in the general direction of turning systems administration into a central solution that you can version control and test.
The DevOps movement is another source of good information on how to do this. While the precept is a better cooperation between Developers and Operations staff, it also tends down this same direction.
Short answer is "OS deployment management", "Configuration Management" and "Software Packaging". Long answer follows.
I'd like to add to Daniel Pittman's reply with a breakdown of what forms a "system" in systems administration.
A system or an environment is going to comprise of:
Enveloping these will be processes such as:
And you'd like to have these combine together to assist you with achieving non-functional goals such as:
This is a quick brain dump. I am sure more could be added to all lists.
Your question touches on a number of these without using the specific words. For example, you want to be able to deploy easily and revert back i.e. want maintainability; you want to do it in a test environment and test till it passes i.e. repeatability, testability and measurability; you're thinking of putting vm images in version control because you'd like repeatability of OS and configuration deployments.
There are plenty of tools to assist you with this, some of them mentioned by Daniel. Some others are:
Again, this isn't a comprehensive list, but something that I keep in my head to guide me and hopefully it'll help you out too.
In a Windows world, these issues around application lifecycle management are being addressed with System Center 2012.
In System Center Virtual Machine Manager (SCVMM), services are defined using 'Service Templates' (e.g. a classic three-tier service) and execution environments are defined as 'Clouds' (e.g. development, staging, production). Service Templates can be versioned and rolled out (in an automated fashion) to different Clouds. Under the covers, SCVMM does the work to provision, deploy and configure the virtualised hardware (VMs etc) and software (OS, app components etc).
System Center Service Manager is the piece that ties this together from a process point-of-view. For example, the issue management and change control.