The load has been sitting at around 6 but there is very little running on the box. Any ideas what's causing this? It's a large AWS instance.
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 4 0 4815700 179440 548200 0 0 0 3 2 3 5 1 91 0 0 4 0 4821256 179444 548192 0 0 0 0 228 373 1 1 48 48 0 4 0 4821296 179444 548196 0 0 0 0 96 150 0 0 50 50 0 4 0 4822832 179444 548200 0 0 0 0 158 197 1 1 49 49 0 4 0 4823452 179444 548200 0 0 0 0 98 149 0 0 50 50 0 4 0 4823452 179444 548200 0 0 0 0 94 149 0 0 50 50
As noted by James C, it looks like you are having trouble with disk IO, not CPU time. If your instance is in the us-east data center they are having trouble there today and latency to EBS volumes is way up.
The right most column in the vmstat output indicates that there's a lot of i/o wait going on. 50% of CPU time. Could you please post the output of
iostat 10 6
as this should give us a good insight as to which disks are slowing stuff down.I'm not sure what EBS performance is like but if you're doing a lot of I/O then it might be having problems keeping up.