IZhen's questions

I'd like to analyze how the processing time are distributed to individual CPU for a process ( KVM actually). In Linux a task can be migrated to other CPU during its life-cycle, the process may run in CPU0 for 1 sec then in CPU1 for 2 sec, but we can only get the overall processing time(3 sec) from /proc/pid/stat, but not 1sec and 2sec.

And we can get the per CPU aggregate processing time for all process in /proc/stat, but not for a particular process e.g. a KVM.

There are also some scheduling statics in /proc/pid/schedstat and /proc/schedstat, but I'm not clear how to derive it from statics like number of migrations.

This problem is also covered in the virt-top document, I think there is still no clear solution.

Do you have any ideas?

I'd like to replay the IIS or Tomcat access logs to do load testing. The access logs contain only Get request, so the session data/post data is not a problem here. However two timing problem is present at default IIS and Tomcat logs

1. The access log "time" entry is at the resolution of 1 seconds , say "06/Aug/2011:17:51:52 +0800". So when the request enter in the 1 seconds is unknown, they can comes in a bursty way or in a evenly way. The response time will be different, but how much different do not known.

2. The access log "time" entry records the request completion time not request arrival time. See Tomcat time,IIS log time. If the "time-taken" field is present, we can extract the "time taken" from the "time" to get the arrival time, but the real problem is that "time-taken" is not logged by default. It seems to me that the only we to solve this problem is try to persuade the client to turn "time-taken" on.

Both problem can be solved by modified the default logger, but it is a last resort. Are there any workaround or tricks to measure or calculate the correct "time" so that it can be used for load testing.

In a typical multi-tier web application, how to measure the response time from when the server receive the request to the time server send out the response? I want focus on the server side performance, so I do NOT want to include the time the response spent at the internet.

The trouble is that web server logs usually logs response time which include the network time. For example in IIS, the "time taken" field in the access log reports the time when the server receive the request to the time server receive the acknowledge of response from the end user, see this MSDN article for more detail.

Are there any monitoring tool or server configuration to enable such measurement?

I want to check the capacity and performance of my network. Besides checking the txkB/s and rxkB/s via Sar, I'd also like to see the average queue length of the network interface(so that the average queueing time in the interface can be calculated). It seems that netstat can give a per socket queue length, is it possible to get a per interface statics(a bit like Network Interface\Output Queue Length in Windows)?

A related and kind of reverse questions is How do I view the TCP Send and Receive Queue sizes on Windows?

Thanks

I'm trying to monitor various hardware performance counters of a running system, however it seems that current PMC tool e.g. oprofile or perf tool can only record counters after the program finished, e.g. ~>perf record <command>;

~>perf report

the perf record returns only after the target program terminates, I'd like to periodically collect these counters like what vmstat does.

Edit: I want to collect hardware performance counters, e.g cache-misses,instructions,branch-misses, these counters are available in modern process.

I'm trying to measure the average cpu frequency in an DVFS enabled cpu for specific interval , the obvious way of periodically sampling /proc/cpuinfo has very large variants. The cpufreq-stats driver gave me some hope ,e.g.

cat /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state 
2600000 118148
2000000 8562
1600000 11041
1200000 3428602    # lots of ticks is idle ticks

but unfortunately it is mixed with idle ticks(see patch "Do not account for idle time when tracking time_in_state"). During idle time it is probable that cpu will fall back to the lowest frequency ,thus skew the frequency distribution significantly .

The patch above utilize the account_idle_tick function to remove the idle ticks from statics, but it seems only works for xen kernel. Is there alternative way to measure average cpu frequency?