Using R for crunching system data?

We have looked at R as a common analysis and reporting back end for data from multiple performance test tools but unfortunately we have not had the time tom implement R for this task as we have simply been too busy performance testing servers...just not enough downtime.

Here's my take on R at least from a performance testing analytics perspective

• We need to correlate data which is inclusive of response time and system monitor data across dozens of hosts involved in the performance test. Using R gives us the ability to import data from multiple disparate data sources in multiple formats for a consolidated analysis set
For testing we need reproducibility, otherwise what we are engaging in is 'experimenting' and not 'testing.' R will help us to understand better the quality of our data set based upon the number of samples and also allow us to better understand how statistically close are our tests when we run tests back to back to check for consistency.
• The weak link in all of the open source performance test tools is analytics and the ability to correlate an increase in response time event to increases or drops in other system metrics. R should allow us to provide the same level of analytical capability on open source test tools that we can get today with the best of the commercial tools, and for the commercial tools it should allow us greater insight into the nature of the sample sets where we can calculate the area under the curve for the frequency of the samples and get a delta when compared to the optimal curve (one half of the bell curve with low standard deviation). This delta we will use to guide our feedback to development on where to spend the time and energy for a fix
• All of the tools on the market are poor when comparing to a robust performance requirement which is typically written as an SLA, i.e., a response time of 'x, y% of the time , under a load of Z.' R will allow us to better report on the requirements/SLA targets, especially where moving SLAs may be present based upon load.
• For technical benchmarking R will allow us to better compare the output of current and prior tests against different builds, finding statistically significant deviations in performance from one test to the next to report back to development

On the test server analytics side today we use a combination of monitored metrics collected during our tests, log analysis with Microsoft logparser and a reporting engine based upon LaTeX/PSTRICKS which outputs PDF. With R we expect to move to a more structured rules based analysis of results which we should be able to automate to a high degree where today we need a lot of manual intervention on the analytical side of the house for eyeballing results and formatting the output.

R is a programming language like any other, with strengths and weaknesses. The strengths are in the depth of statistical methods implemented - so, for example, if you want to fit your system loads to a generalized autoregressive conditional heteroscedastic (I'm not making that up) time series model you can. There might be an implementation of that in Python or Perl, but I doubt its as widely-used or tested.

For me its weakness is the programming language itself - fairly irregular and quirky in places. Full of traps. If you've never used a programming language before you might take to it, if you're a competent programmer in any current language (Python, Perl, C(++), VB?) you might hate it.

If the statistical techniques and graphics exist in your favourite language of choice then I'd go for that. If you could implement them quite easily in your favourite language of choice I'd think about doing that. If you want cutting-edge stats that don't exist in other programming languages, use R.