Questions[scheduler](server)

I have some highly floating point intensive processes doing very little I/O. One is called "xspec", which calculates a numerical model and returns a floating point result back to a master process every second (via stdout). It is niced at the 19 level. I have another simple process "cpufloattest" which just does numerical computations in a tight loop. It is not niced.

I have a 4-core i7 system with hyperthreading disabled. I have started 4 of each type of process. Why is the Linux scheduler (Linux 3.4.2) not properly limiting the CPU time taken up by the niced processes?

Cpu(s): 56.2%us,  1.0%sy, 41.8%ni,  0.0%id,  0.0%wa,  0.9%hi,  0.1%si,  0.0%st
Mem:  12297620k total, 12147472k used,   150148k free,   831564k buffers
Swap:  2104508k total,    71172k used,  2033336k free,  4753956k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                         
32399 jss       20   0 44728  32m  772 R 62.7  0.3   4:17.93 cpufloattest                                    
32400 jss       20   0 44728  32m  744 R 53.1  0.3   4:14.17 cpufloattest                                    
32402 jss       20   0 44728  32m  744 R 51.1  0.3   4:14.09 cpufloattest                                    
32398 jss       20   0 44728  32m  744 R 48.8  0.3   4:15.44 cpufloattest                                    
 3989 jss       39  19 1725m 690m 7744 R 44.1  5.8   1459:59 xspec                                           
 3981 jss       39  19 1725m 689m 7744 R 42.1  5.7   1459:34 xspec                                           
 3985 jss       39  19 1725m 689m 7744 R 42.1  5.7   1460:51 xspec                                           
 3993 jss       39  19 1725m 691m 7744 R 38.8  5.8   1458:24 xspec                                           

The scheduler does what I expect if I start 8 of the cpufloattest processes, with 4 of them niced (i.e. 4 with most of the CPU, and 4 with very little)

I'm trying to construct an experiment to measure the effect of ionice. What I'd like to do is to (per another answer on serverfault) cause frequent enough I/O that a sufficiently "niced" process is starved of any I/O.

Based on another answer on serverfault, I think I need to cause at least one actual I/O operation to a common cfq-scheduled device every 250ms. My thought was to write a trivial program that does has a loop that

• writes to a (configurable) file on the common device,
• does an fsync() (to force a definite I/O operation),
• uses usleep() to delay a configurable amount of time
• periodically uses lseek() to truncate the file (so that I don't fill the file system)

I then start up one instance of the program using ionice -c3 (idle scheduling class) against one file on the common device. I simultaneously run various instances with the default (best-effort) scheduling class, specifying a different file on the common device (varying the delay values).

My hypothesis was that for delay values of 250ms or more on the "best-effort" process, I would see progress made by the "idle" process; for values less than 250ms, I would see little to no progress made by the "idle" process.

My observation was that there was no difference in performance of the two processes; they both made similar progress. Just to be sure (in case the wall-clock indications that "best-effort" process was performing I/O much faster than every 250ms), I started multiple simultaneous instances of the "best-effort" process, specifying no (zero) delay. Still, I saw no difference in performance between the processes in the two scheduling classes.

Just to be sure, I re-checked the scheduler class:

$ cat /sys/block/xvda/queue/scheduler
noop anticipatory deadline [cfq] 

What is it that I'm missing about how the cfq scheduler works?

If it matters, this is on a 2.6.18 kernel.

When I renice a multithreaded process in Ubuntu 10.04, then view the results in top, it seems that only the main thread actually gets reniced. All the other threads retain their old nice value. What's the easiest way to renice all threads in a process instead of just the main thread?

I have an 8 drive RAID 10 setup connected to an Adaptec 5805Z, running Centos 5.5 and deadline scheduler.

A basic dd read test shows 400mb/sec, and a basic dd write test shows about the same.

When I run the two simultaneously, I see the read speed drop to ~5mb/sec while the write speed stays at more or less the same 400mb/sec. The output of iostat -x as you would expect, shows that very few read transactions are being executed while the disk is bombarded with writes.

If i turn the controller's writeback cache off, I dont see a 50:50 split but I do see a marked improvement, somewhere around 100mb/s reads and 300mb/s writes. I've also found if I lower the nr_requests setting on the drive's queue (somewhere around 8 seems optimal) I can end up with 150mb/sec reads and 150mb/sec writes; ie. a reduction in total throughput but certainly more suitable for my workload.

Is this a real phenomenon? Or is my synthetic test too simplistic?

The reason this could happen seems clear enough, when the scheduler switches from reads to writes, it can run heaps of write requests because they all just land in the controllers cache but must be carried out at some point. I would guess the actual disk writes are occurring when the scheduler starts trying to perform reads again, resulting in very few read requests being executed.

This seems a reasonable explanation, but it also seems like a massive drawback to using writeback cache on an system with non-trivial write loads. I've been searching for discussions around this all afternoon and found nothing. What am I missing?

Not sure this is the right place for this question, but here goes:

I'm trying to make some sense of the /proc/pid/sched and /proc/pid/task/tid/sched files for a highly threaded server process, however I was not able to find a good explanation of how to interpret this file ( just a few bits here: http://knol.google.com/k/linux-performance-tuning-and-measurement# ) . I assume this entry in procfs is related to newer versions of the kernel that run with the CFS scheduler?

CentOS distro running on a 2.6.24.7-149.el5rt kernel version with preempt rt patch.

Any thoughts?