Home / server / Questions / 47308
Accepted
lazy1
Asked: 2009-07-29 15:01:34 +0800 CST

How to diagnose erratic disk behavior?

  • 772

I have a web site with users lighttpd and CGI scripts.

After upgrading to Fedora 11 (ext4) the disc access became erratic. The timing of python -c 'import cgi' varies between 0.1 to almost 10 seconds: graph

How can I diagnose the problem? (Tools, methods, best practices ...)

Update Jul 30, 2009:
Found out that several CGI process were hogging the drive. After killing them the graph is stable between 0.02 and 0.03. Still didn't get an answer on how to diagnose such problems.

cgi python lighttpd fedora ext4

3 Answers

  1. If it is fresh install then tools like makewhatis which are used by apropos, whatis might cause disk to be heavily used. Wait for few days for things to get stabilized (updatedb, prelink, makewhatis, etc.) then may be timings will be consistent.

    It would also depend on something else you are doing on server and what the cgi script is actually doing, where it is taking input from, size of input, etc.

    Also if disk is very old, use diagnostic tools (like seagate seatools) to look for controller / bad sector problems. The tools will also allow you to optionally repair the sector if drive is actually from seagate.

    • 1

  2. Do you really need/want ext4 on a production server? It's a still a mighty bit to green for my taste for a server.

    • 0
  3. Best Answer

    The only way to diagnose a problem like this is with lots and lots of data. Familiarize yourself with vmstat and iostat. A tool I recently learned about in this thread is dstat which effectively combines the two.

    For problems like the one you're describing, this command would likely be useful:

    $ dstat -M app -cdnygl

    It will report on CPU, IO (disk and net), interrupts, swap, and load average. As a nice little bonus, it will include the name of whatever process was "most expensive" a the time the snapshot was taken. Unfortunately that particular command produces output too wide to paste here, so here's a bit more conservative version:

    $ dstat -M app -cdn
--most-expensive-- ----total-cpu-usage---- -dsk/total- -net/total-
     process      |usr sys idl wai hiq siq| read  writ| recv  send
bacula-fd        0|  1   0  98   0   0   0| 426k  108k|   0     0 
bash             1|  2   2  96   0   0   0|   0    20k|1460B 1804B
apache2          8|  4   2  94   0   0   0|   0     0 |  76k   15k
                  |  1   3  96   0   0   0|   0     0 |1132B 1034B
apache2          1|  2   2  96   0   0   0|   0  8192B|  11k 3895B
                  |  2   1  96   0   0   0|   0    32k|3322B 1338B
kipmi0           1|  2   2  96   0   0   0|   0     0 |1309B 1146B
    • 0