I'm consider running some performance profile tools like Cacti for my server, but I'm concerned that running RRDtool or Cacti slow down my servers performance? Is this a valid concern for me? Or rather, how can mitigate this? Can I run the profiling tools on a different server or something?
I suppose another tool to mention here would be Munin, and that leads me to slightly a large questions, is there any huge advantage of using Munin over Cacti? Information is a little sparse online.
It may cause some slow down on your server although not that much. Depending on server specs I doubt it will be noticeable. You can also run these programs on a different server to eliminate the load in theory but Cacti still has to go and grab the data constantly causing a small load increase.
I really don't think this would effect your performance massively.
The expensive part is the graph's pictures generation. Data collection itself is not that expensive.
In my experience, Munin can have some significant impact on performances. I was running it on an embeded platform (an Alix 2d13) that I used as a firewall. Adding Munin on that platform was making far too much load. The system was becoming completely unresponsive.
Your server is probably much more powerfull than my Alix board, but still, it could have some performance impact ...
Running the server components on your systems will generate some load, but the munin-node client is generally not resource heavy. It all depends on your "plugins" of course but in the default installation they are very lightweight.
I recommend using munin for server graphing and cacti for switch/network graphing (snmp).
Do you know the average load in your system ? If it's running of its limits you may have a bunch of problems. (Such as pid runout , oom-kill events etc). But if it's not too overloaded you won't have problems. Even 2% extra load it's not extremely tragic.
After all, you can reduce the "precision" of the measurements / plots and thus reduce the overhead. But, still is better to have the "server" part of the monitoring framework in a seperate server and have the clients distributed to any of your machines.