What do you think are the best practices to maintain dozens (if not hundreds) of debian servers up-to-date ? Keeping in mind that :
- There are groups of servers (i.e identical webservers, DB Servers, ...)
- There can be several Debian issues (lenny, etch)
- Running a loop over all servers and doing apt-get update && upgrade is not acceptable (because it's what I'm doing at the moment :) ) It should be better than this !
Currently, when I finally finish all the upgrades, a new security update is posted, and I have to do it all over again.
Thanks in advance serverfault community !
I use apt-dater to manage upgrading all my Debian boxes. Seems to do the trick well enough. Haven't tried to scale it up to hundreds of hosts though.
Google solved this with debmarshal:
http://code.google.com/p/debmarshal/
Which lets you approve packages from an upstream repository for installation on your production hosts.
Then you can just run cron-apt in fully automatic mode.
Here's an intro video:
http://www.youtube.com/watch?v=L3hRToC23mQ
We were trialing using puppet to upgrade security fixes on non-essential packages. We would run apticron to email a list of updates for every server, then daily run a script that merged these updates into a puppet manifest file which gave the package and the version for each distribution. This would then update a bunch of files on the individual servers and kick off an upgrade script when a package needed upgrading. This worked reasonable well, but we haven't tested it quite as much as I'd like. This scheme did get around the limitation of Puppet of not having the same resource defined in multiple places.
I was also not comfortable with doing automatic upgrades of things like MySQL or PostgreSQL, where a random update would shut down a service, possibly in the middle of the day. These would still require manual updates.
Spacewalk and Debmarshall do look like suitable alternatives for our puppet scheme.
Apparently, Spacewalk now has preliminary support for Debian. That, together, maybe, with Puppet, would be my starting point. I'm pretty sure the guy developing the Debian support for Spacewalk will love you for working with him in taking Debian support to a higher level.
In the way of pull-based configuration systems like Puppet, there are also bcfg2 and cfengine. One or the other of those might suit your needs well. I'm rolling out bcfg2 in my lab right now.
A solution can be given by func
I'm not sure what type of solution you are expecting. You probably know about cron jobs, but I wouldn't update systems in the blind as there are human interventions needed (and that is why they pay you to do this, right?)
If you had completely identical systems you might consider using something like rsync to bring in the differences, but figuring out which files not to rsync could be difficult, and I wouldn't do this while services are running. At least the update scripts are set up to manage restarting the services and merging in configuration file differences.
Perhaps if you explain what the problem is with doing apt-get commands we could see what you want to avoid.
If the problem is bandwidth and time to download, perhaps you should set up one box to act as your local Debian repository. There are Debian guides on how to do that.
Here are some tips on how to minimize the number of things you need to update.
When you install Debian, don't install Desktop unless you really need to use X on that console. Most servers do not need X installed. This can decrease the number of packages on the system significantly, and then you don't need to update as many packages.
Check that the sources.list is including only the repositories you really need. If you had experimented with some repository and forgot about that, you might be bringing in updates you don't need or want.
If you have run into trouble with blindly doing updates on a production server, be careful to consult the Debian upgrade guides when there is a major update (4.0 to 5.0). These will go through very well if you follow the upgrade instructions. It isn't as easy as running apt-get dist-upgrade and walking away. Sometimes in the instructions there are even pointers on when to run aptitude rather than apt-get - there are small differences in them.
Do you now this tool "dancer's shell"? I like it and i use it. But i don't know if you can use it for so many hosts. Maybe you could try...
http://www.netfort.gr.jp/~dancer/software/dsh.html.en
And he is in the repository.
ClusterSSH. You logon to all servers and give them the exact same commands, so you can also react to the dialogs. If one server gets an extra question, just click on that one and it will be the only one that responds.
I've used it to upgrade 25 webservers from etch to lenny. Worked like a charm.
http://sourceforge.net/projects/clusterssh/
Cluster ssh is a good suggestion.
debmarshal isn't part of debian yet - I'm not even sure it will be a package - seems to be a completely different system with a specialized repository. As the speaker said, this is currently user hostile, not user friendly.
Spacewalk seems to be a clone of Redhat Network, at least in the web interface. I've had bad results from using Redhat Network to update systems. One time it hung, for no apparant reason, and caused service outage. I did a yum update immediately after and it handled that fine, so I can only assume the problem was from something that barfed on the RHN side. The other thing I don't like about RHN updates is you don't know when the update will happen, to watch for issues.