Getting started with documentation

since 2003 I'm documenting everything in our inhouse wiki.

Servers

• hardware specs
• waranty information
• network information
• and of course installed software and configuration

Workflows

e.g. how-to add or delete a user and give him/her access to all relevant services

Important links

• link to all your web interfaces
• link to the monitoring URLs (nagios, munin, apc-monitoring...)
• link to the wiki (for the printed version!)

Emergency instructions

what to do if intranet server/internet/web server/etc are down

Important:

Choose a wiki engine with easy export to PDF!
Its not useful if you are in holiday, the server running your wiki is down and no one knows what to do because your documentation is offline

Have a look at twiki, docuwiki or mediawiki.

BTW:

there is a OpenOffice.org plugin to write directly to mediawiki - very convenient.

EDIT:

Its also nice to write down some infos to /home/adminuser/maintenance. This is done quick and can be very helpfull, if several admins work on a server. eg:

2009-06-27 -thorsten-
          running aptitude update && aptitude full-upgrade
          everything seems ok
2009-06-25 -andreas-
          cups-pdf wasn't reachable. restarted cups
2009-06-23 -thorsten-
          deleted old log under /var/log/squid
etc.

While you realize that while everyone wants (and needs) documentation, you also need to recognize that no-one has time to read and study the stuff.

So, don't write documentation that needs to be studied - instead, structure your documentation in a way that allows someone to quickly find the information they need, when they need it - which may well be while the system is down and the CTO is breathing down his/her neck.

With this in mind, some suggestions ...

• Avoid large blocks of text
• Bullet lists are your friend
• A clear diagram is golden
• Repetition is a good idea (1)
• Make it easy to update and extend

(1) Don't create one source of truth and force people to hunt it down. The more important the idea, the more you should repeat it.

Essential docs:

• Server documentation - specs/disk layouts/installed software/anything of note
• Common procedures - anything that is done that isn't 'trivial' should have a procedure documented, especially if it's something that hasn't been done before.

Keeping documentation in sync can be pretty much a 'fix it as you see mistakes' affair. Along with this needs to come the realization that documentation can and will be out of date, and that it should not be followed blindly without taking this into account. Documentation is there to assist an admin in tasks, not to be a step by step set of rules that replace critical thought.

Minimizing duplication - using something like wikis where you can link documentation together can help with this, instead of repeating information, you just link to it. The problem is that the person writing the documentation needs to know that the information they are about to duplicate already exists. This is usually a question of good organization.

I've found that creating a template is a big help. In my case it's a Word template but use whatever suites you. Create a skeleton file, complete with table of contents field and sections as desired. Once you've used it a couple of times, and made any fine tuning adjustments, you'll create new documents much faster. The consistency of format will be a great help, both for document creation and later use. The documentation needs to be stored in a logical place and in a logical directory structure.

I'm personally opposed to repetition for the simple fact that it makes upkeep unnecessarily difficult and time consuming. Rather than duplicate documents, or parts of documents, create references to other documents where appropriate. If something changes you should never have to change the relevant documentation more then once or in more than one place, otherwise you will have a collection of conflicting documents, which helps nobody.

While creating your documentation just keep in mind what it's for. Somebody at a later time will need to use it. Will it be usable to do the job without prior knowledge?

Not a direct answer to your question, but a pointer in the right direction:

I found The Practice of System and Network Administration, by Limoncelli and Hogan (aka the Sysadmin Bible) to be quite valuable because it is about "best practice" issues, such as documentation. If you don't already know about it, make sure you investigate it whenever you get the chance.

For me, the most important consideration is making it easy to use. If it's difficult to orchestrate then people will avoid it. I choose Trac's wiki as the medium for our documentation for these reasons:

• Centrally located.

  More than one active copy of any one document leads to confusion. If you're able to refer everybody to the same place, both contributors and audience, then you can simplify the process.

• Simple editing and formatting.

  So much time is wasted on pretty Word templates and conforming with the last author's styling. If you don't bog people down with this, then it's easier to edit on-the-go and contributors are more inclined to do so. Separate out items as much as you wish with TracLinks.

• Audit history.

  It's important to know who made what change, when and why. If you can tie it into change request tickets and configuration commit logs then even better. SVN commit hooks are great for this.