Currently we have 4 bare metal servers and one small VM server (all running centos6) and I'm looking at setting up monitoring for the Percona MySQL db server and replication slave (bare metal), as well as the remaining 3 which are app servers.
Curious what the preferred way to setup a monitoring config would be. Maybe spin up a small dedicated VM that just runs the monitoring tools for watching the other 5 servers remotely? Would you make the VM on the same host as the servers to be monitored? Maybe use a different host like Digital Ocean to run the monitoring tools?
When setting up a new monitoring host (I use Nagios as well), I try to make sure that it has as little in common (as far as infrastructure goes) with the hosts to be monitored as possible.
So, if you're monitoring internal servers/services in a VMware environment, then maybe set up a standalone ESXi host for your monitoring VM. If you're monitoring services in AWS, then fire up a small DO/Linode VM to monitor from, etc. etc.
There are some monitoring metrics that require script execution on the monitored host, but for the vast majority of items, monitor from an external server as much as possible.