Home / server / Questions / 56094
Accepted
Lorin Hochstein
Asked: 2009-08-20 06:20:13 +0800 CST

How can I tell what processes are causing high loads if they are not high-cpu usage?

  • 772

Sometimes my servers will show a high load average in the "top" program (e.g. load is ~10 on a 4-core machine), but the actual CPU usage isn't particularly high.

I assume the issue is that there are many I/O-intensive jobs running. Is there any easy way to identify these jobs that are causing the load, if their "%CPU" values in top aren't that high?

linux

3 Answers

  1. Best Answer

    iostat can report statistics like that. Usually included in your distro in the package sysstat.

    dstat might also be worth a look, it's a modern replacement.

    • 7

  2. To find what's causing high load you can check few things.

    • vmstat -w will show you ovierwiem (processes, swap, mem, cpu, io, system)
    • pmstat -P ALL will provide you statistics (with %iowait) per cpu core
    • iostat -x look for high %util or long await or big average queue size
      • dig deeper with iotop
    • ps -ax look for state D which is uninterruptible sleep (usually IO), run it one more time check if they are still in D state
      • dig deeper with strace
      • check files and connections of those processes with lsof and netstat
    • sar/sysstat - with that tool you can explore not just "now" stats but also check what was happening yestarday at midnight
      • sar -b - overall io activities
      • sar -d - individual block device io activities
    • 5

  3. If you have IO accounting in your kernel, then you can use iotop to give information like that. Also, monitoring tools like collectd can record and report on the data.

    • 4