I've used tools like puppet to manage individual systems, with generally a high level of success. Where puppet falls down is that it isn't good at managing dependencies outside of an individual server.
For example, on a MySQL server I configure puppet to do the following:
- Configure authentication on the machine to hit my LDAP server
- Configure apt to use my local repository mirror
- Install MySQL packages
- Write my.cnf
- Start MySQL
- Create users in the database
In this set of steps there are a number of dependencies resolved - for example, I can't start the database service unless I've installed the packages, which I can't do unless the apt repo is configured correctly.
This MySQL server is one box in a master->master replication setup. In an ideal world, puppet (or another similar tool) would let me represent the fact that server B needs to wait until server A is available and then attempt to establish a replication relationship with it.
A lot of text here - basically what I'm asking is: are there any tools like puppet which can manage inter-machine dependencies like this?
If you're using Puppet, then exported resources (and thus stored configs) are the tool for the job. This allows you to share information between nodes. Including the actual replication settings, if need be.
Alternatively, and although I'm not a subscriber myself, some people treat their manifests/runs as non-deterministic. Meaning that a single run doesn't necessarily declare the complete outcome of a node. It may take two or more runs before the node reaches it's desired state.
Since configuring the MySQL slave before the master is availble wouldn't be entirely fatal, you may wish to go down this path. It's not quite as "smart" in my opinion though.
Not being a puppet user, I'm sure others will chime in with more appropriate answers, but it sounds like a wrapper script could be implemented with relative ease that checks before starting the service.
Puppet can do this just fine. Have each server export a resource to configure the other appropriate server(s) to do what needs to be done once they're up and running. I don't have an example for MySQL, but we configure a lot (like about 80, at current count) DRBD resources this way, as well as all our Nagios configs.
Control Tier may well be what you're after. You can use it in conjunction with puppet so the learning curve shouldn't be too steep.