Questions[io](server)

This is a software design question

I used to work on the following rule for speed

cache memory > memory > disk > network

With each step being 5-10 times the previous step (e.g. cache memory is 10 times faster than main memory).

Now, it seems that gigabit ethernet has latency less than local disk. So, maybe operations to read out of a large remote in-memory DB are faster than local disk reads. This feels like heresy to an old timer like me. (I just spent some time building a local cache on disk to avoid having to do network round trips - hence my question)

Does anybody have any experience / numbers / advice in this area?

And yes I know that the only real way to find out is to build and measure, but I was wondering about the general rule.

edit:

This is the interesting data from the top answer:

• Round trip within same datacenter 500,000 ns

• Disk seek 10,000,000 ns

This is a shock for me; my mental model is that a network round trip is inherently slow. And its not - its 10x faster than a disk 'round trip'.

Jeff attwood posted this v good blog on the topic http://blog.codinghorror.com/the-infinite-space-between-words/

We're quite interested in exploring the possibility of using SSD drives in a server environment. However, one thing that we need to establish is expected drive longevity. According to this article manufacturer's are reporting drive endurance in terms of 'total bytes written' (TBW). E.g. from that article a Crucial C400 SSD is rated at 72TB TBW. Do any scripts/tools exist under the Linux ecosystem to help us measure TBW? (and then make a more educated decision on the feasibility of using SSD drives)

I'm having a problem with a Linux system and I have found sysstat and sar to report huge peaks of disk I/O, average service time as well as average wait time.

How could I determine which process is causing these peaks the next time it happen?

Is it possible to do with sar? Can I find this info from the already recorded sar files?

Output of sar -d, system stall happened around 12.58-13.01pm.

12:40:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
12:40:01       dev8-0     11.57      0.11    710.08     61.36      0.01      0.97      0.37      0.43
12:45:01       dev8-0     13.36      0.00    972.93     72.82      0.01      1.00      0.32      0.43
12:50:01       dev8-0     13.55      0.03    616.56     45.49      0.01      0.70      0.35      0.47
12:55:01       dev8-0     13.99      0.08    917.00     65.55      0.01      0.86      0.37      0.52
13:01:02       dev8-0      6.28      0.00    400.53     63.81      0.89    141.87    141.12     88.59
13:05:01       dev8-0     22.75      0.03    932.13     40.97      0.01      0.65      0.27      0.62
13:10:01       dev8-0     13.11      0.00    634.55     48.42      0.01      0.71      0.38      0.50

I also have this follow-up question to another thread I started yesterday:

• Sudden peaks in load and disk block wait

I have a server that has a really high load. Nothing is jumping out at me in terms of CPU usage, and it's not swapping.

I think it's cause some processes are waiting for disk IO, and I want to see what's waiting.

Is there any programme that'll show me what processes are waiting for IO? I know about iotop but that shows what's currently doing IO.

Or is this a silly question? (If so explain how :) )

As much as I have read about iowait, it is still mystery to me.

I know it's the time spent by the CPU waiting for a IO operations to complete, but what kind of IO operations precisely? What I am also not sure, is why it so important? Can't the CPU just do something else while the IO operation completes, and then get back to processing data?

Also what are the right tools to diagnose what process(es) did exactly wait for IO.

And what are the ways to minimize IO wait time?