Home / server / Questions / 211996
Accepted
IZhen
Asked: 2010-12-14 04:10:08 +0800 CST

How to periodically collect hardware performance counters in linux?

  • 772

I'm trying to monitor various hardware performance counters of a running system, however it seems that current PMC tool e.g. oprofile or perf tool can only record counters after the program finished, e.g. ~>perf record <command>;

~>perf report

the perf record returns only after the target program terminates, I'd like to periodically collect these counters like what vmstat does.

Edit: I want to collect hardware performance counters, e.g cache-misses,instructions,branch-misses, these counters are available in modern process.

linux performance-monitoring

4 Answers

  1. Best Answer

    It seems that current perf don't support Sar style output. However a very close emulation is possible with the help of watch command.

    To collect counters periodically for the whole system, use the '-a' option. Below is an example of reporting the general hardware performance counters for every 60 seconds

    watch -n 60 sudo perf stat -a sleep 60

    It gives the output as

    Every 10.0s: sudo perf stat -a sleep 10                                                                                               Wed May 11 11:19:36 2011


 Performance counter stats for 'sleep 10':

       19956.091380  task-clock-msecs         #      1.995 CPUs
              26802  context-switches         #      0.001 M/sec
               1871  CPU-migrations           #      0.000 M/sec
                951  page-faults              #      0.000 M/sec
         4582605668  cycles                   #    229.634 M/sec  (scaled from 69.93%)
         1993617795  instructions             #      0.435 IPC    (scaled from 81.07%)
          374028141  branches                 #     18.743 M/sec  (scaled from 81.34%)
           19071123  branch-misses            #      5.099 %      (scaled from 79.77%)
          100724660  cache-references         #      5.047 M/sec  (scaled from 18.66%)
            5461106  cache-misses             #      0.274 M/sec  (scaled from 20.23%)

       10.001116547  seconds time elapsed

    The usage of sleep command is a bit counter intuitive at first glance. Here the perf stat will NOT collect counters specific to sleep command ,instead sleep command simply act as a way to tell the perf stat the duration of collection(I realize that from the example in the perf wiki ) Note to collect system wise counters, administrative privilege is required, so the sudo before perf stat is necessary.

    If you simply need collect counters specific to a running process(e.g., with pid 2785), then a sample command is

    watch -n 60 perf stat -p 2785 sleep 60

    Note this time, sudo is NOT required.

    • 2

  2. I find that sar is a simple way to my needs for monitoring a linux system. It periodically collects stats on many things and you can have cron send you emails periodically with a table of the results which you can then process.

    If you need to monitor more systems, or want fancy graphs or alerts, you might look at something like cacti or nagios.

    • 0

  3. don't forget collectl. collects a lot more data and its output is much earier to read than sar. if you do use sar, do youself a BIG favor and set the monitoring interval to 10 seconds rather than 10 minutes - only real gross problems will show up at a rate as coarse as 10 minutes. -mark

    • 0

  4. If you want to log really comprehensive data use Performance Co-Pilot. There's a nice GUI, too.

    • 0