The company I work for is currently investigating the deployment of an centralized automation system (like Salt or Puppet) for our servers (all Ubuntu/FreeBSD). We will probably go along with Salt, but I think it is irrelevant to my question.
My quesiton: Is there a good way for monitoring machines for local changes not included in the automation system?
For example: for a quick fix, someone started a service or modified a configuration file on a given machine. Is there a way to check for such things using Salt/Puppet/whatever? Or do I need to use external programs like AIDE for that?
You could use tripwire to monitor changes to all relevant files on the server. The flipside is that after each automated configuration change you'd have to reset the trips.
You could write a script with find(1) and md5(1) that takes the MD5sums of all relevant files and compares them against MD5sums stored offsite.
In the same vein as the Tripwire suggestion, I've been implementing AIDE on new systems. This is available on both of the platforms in your environment.
Aide will use aide.conf to figure out exactly what files/directories to monitor and how. It would be possible to write up a Puppet template that could set up aide.conf to not look at directores/files that are created (and can be verified) with rpm.
On the flip side, you could just list the file you want to monitor...
In order to re-initialize the Aide DB on package updates, aide --init needs to be run whenever yum is run via Puppet (then the newly created DB needs to be moved into place). Aide takes so very long to run this initialization that this is untenable (for us). It might be fine in your case.
Use nice to get aide to not be a resource hog. A nice factor of -19 was suggested by www.howtoforge.com.
Something I really like about salt is state testing:
Gives you a nice colourized report telling you what is in compliance with your defined states without actually applying the changes. That gives you what you want in that it shows you where the state of your systems deviates from what the automation system wants it to be. There are good reasons for that to be so, perhaps you are learning salt or have a delicate legacy system and don't want to go all out with an automation system. The more states you add to salt the better the better but it means you can grow organically as you learn.
http://docs.saltstack.com/ref/states/testing.html
Not sure if you already found a solution, but you could use Metafor Software to monitor changes to all relevant files and packages.
This would be a lot easier than AIDE, and way cheaper than Tripwire (Metafor is in free Beta).
In addition to monitoring changes not included in the automation system, you could use Metafor to verify the true state of your servers after Salt/Puppet runs.