We have three main servers on ec2, web, database, and search. I logged in today to find:
77 packages can be updated. 45 updates are security updates.
which scares the crap out of me so I want to update these machines asap but I'm scared to just run the updates on a live running system. Is this safe to do, what's the best approach when it comes to doing security updates on production machines?
You should also understand what the updates are and if they are cause for concern. If a patch fixes a security flaw in a service you don't run, you don't need to worry about it.
You're being prompted to patch because you've got the package installed; if you just did a 'full' install without deciding which packages you actually need most of these updates could be redundant for you.
Andrew
If you use only Ubuntu repository package, then upgrade is very safe.
Also you can stop EC2 instance, make snapshot and then make update. If you have problems you can rollback.
It doesn't really matter that it's an EC2 machine in this case. You should apply the updates as you would to any other production machine- testing each one in a test environment before applying to your live environment!
Welcome to build and patch management. Most of it is 99% the same as physical servers, ec2 just let you get this far without hiring a sysadmin.
The answer is to know your distro's packaging tools and policies well and use that to make case by case evaluations of what packages are non-issue's you can do real quick, and which packages are maybe-issues you can handle case-by-case (and then how to handle ones that turn into real-issues).
It also fits into your overall cluster/availability design to be able to shift whatever live services you provide off a given server, do your patching, test/validate somehow, then shift workload back onto the server.
Sorry thats really high level, but you didn't tell us anything about your cluster or its workload.