On our web server, we perform an incremental backup of our files every 4 hours using rsnapshot. To perform the actual copying of files, I've configured it to use a custom script, cp-nice that has the following contents:
nice -n19 ionice -c3 cp "$@"
As you can see, this calls cp via nice and ionice to keep it from overloading the system. As far as I can tell, this has the desired effect (backups are made without bringing the system to a halt).
However, our NewRelic monitoring system freaks out every time a backup is made, since it detects that system IO is at 100%. Is there a way to perform these backups without causing NewRelic to think there's an issue on the server? Perhaps there's some way to "whitelist" a process in NewRelic so that it doesn't count against IO?
This seems like a pretty common use case - certainly, someone must have encountered and solved a situation like this before!
niceandioniceonly set CPU/IO priority. To clarify, if there are two competing IO requests and one of which has a higher priority, that one will be fulfilled first.If there is no other competing IO/CPU, these processes will gladly consume all available resources.
The answer to this is to configure your monitoring checks to know when this type of activity happens and to either suppress alerts during that time frame or adjust alerting thresholds.