Questions[context-switch](server)

I have server with ubuntu server operation system, i have some applications on it which works with network. With an increase in network traffic, CPU context swithching and interrupts are also increases 40 - 60 k per/second. What i must fix kernel optimizaion, NIC optimization maybe or what?

UPDATE

First of all thank you for you answers. I have 8 CPU. My cat /proc/interrupts

          CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  0:    6938741    6966303    6934714    6881839    6895772    6883046    6952545    6909960   IO-APIC-edge      timer
  1:          0          0          1          0          0          1          1          1   IO-APIC-edge      i8042
  8:          0          1          0          0          0          0          0          0   IO-APIC-edge      rtc0
  9:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          1          0          0          1          1          2          1   IO-APIC-edge      i8042
 16:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2
 17:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
 18:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
 19:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb5
 21:         21         23         22         21         23         21         23         21   IO-APIC-fasteoi   ipmi_si
 22:          3          5          6          6          2          5          3          6   IO-APIC-fasteoi   uhci_hcd:usb6, hpilo
 23:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   radeon
 41:     275729        555        587        549     275294        563        583        600   PCI-MSI-edge      cciss0
 42:          2          0          2          1          1          1          2          1   PCI-MSI-edge      cciss1
 46:   31600723   31636789   31668261   31721092   31643480   31719981   31650284   31692948   PCI-MSI-edge      eth0
NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC:   42250721   42318004   19164905   20751945   32012455   25335850   15889990   15935085   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0          0          0          0          0   Performance monitoring interrupts
IWI:          0          0          0          0          0          0          0          0   IRQ work interrupts
RES:  104005816   96594384   40149041   34906154   77175689   55787936   28455228   25633969   Rescheduling interrupts
CAL:     204860     543304    1318717    1176681     431344     876239    1046465    1257472   Function call interrupts
TLB:     308034     229917     230598     299353     362103     478994     256456     212019   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:       1263       1263       1263       1263       1263       1263       1263       1263   Machine check polls
ERR:          0
MIS:          0

I have very big:

 RES:  104150407   96747853   40291367   35052019   77327041   55940217   28595113   25775538   Rescheduling

from which it depends?

My network cards: bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.1.6 (Mar 7, 2011)

UPDATE 2

I made: ethtool -k eth0

And see: large-receive-offload: off

How can i make on it?

Thank you.

I have a machine running Debian GNU/Linux 5.0.8 (lenny) 8 cores and 12Gb of RAM. We have one core permanently around 40% ~ 60% wait time and trying to spot what is happening I realized that we have more interruptions than cpu context switches. I found that the normal ratio between context switch and interruptions is around 10x more context switching than interruptions but on my server the values are completely different.

backend1:~# vmstat -s
  12330788 K total memory
  12221676 K used memory
   3668624 K active memory
   6121724 K inactive memory
    109112 K free memory
   3929400 K buffer memory
   4095536 K swap cache
   4194296 K total swap
      7988 K used swap
   4186308 K free swap
  44547459 non-nice user cpu ticks
    702408 nice user cpu ticks
  13346333 system cpu ticks
1607583668 idle cpu ticks
 374043393 IO-wait cpu ticks
   4144149 IRQ cpu ticks
   3994255 softirq cpu ticks
         0 stolen cpu ticks
4445557114 pages paged in
2910596714 pages paged out
    128642 pages swapped in
    267400 pages swapped out
3519307319 interrupts
2464686911 CPU context switches
1306744317 boot time
  11555115 forks

Any ideas if that is an issue? And in that case, how can I spot the cause and fix it?

Update

Following the instructions of the comments and focusing on the core stuck in wait I checked the processes attached to that core and below you can find the list:

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND
    24 root      RT  -5     0    0    0 S    0  0.0   0:03.42 7 migration/7                                                                                                                              
    25 root      15  -5     0    0    0 S    0  0.0   0:04.78 7 ksoftirqd/7                                                                                                                              
    26 root      RT  -5     0    0    0 S    0  0.0   0:00.00 7 watchdog/7                                                                                                                               
    34 root      15  -5     0    0    0 S    0  0.0   1:18.90 7 events/7                                                                                                                                 
    83 root      15  -5     0    0    0 S    0  0.0   1:10.68 7 kblockd/7                                                                                                                                
   291 root      15  -5     0    0    0 S    0  0.0   0:00.00 7 aio/7                                                                                                                                    
   569 root      15  -5     0    0    0 S    0  0.0   0:00.00 7 ata/7                                                                                                                                    
  1545 root      15  -5     0    0    0 S    0  0.0   0:00.00 7 ksnapd                                                                                                                                   
  1644 root      15  -5     0    0    0 S    0  0.0   0:36.73 7 kjournald                                                                                                                                
  1725 root      16  -4 16940 1152  488 S    0  0.0   0:00.00 7 udevd                                                                                                                                    
  2342 root      20   0  8828 1140  956 S    0  0.0   0:00.00 7 sh                                                                                                                                       
  2375 root      20   0  8848 1220 1016 S    0  0.0   0:00.00 7 locate                                                                                                                                   
  2421 root      30  10  8896 1268 1016 S    0  0.0   0:00.00 7 updatedb.findut                                                                                                                          
  2430 root      30  10 58272  49m  616 S    0  0.4   0:17.44 7 sort                                                                                                                                     
  2431 root      30  10  3792  448  360 S    0  0.0   0:00.00 7 frcode                                                                                                                                   
  2682 root      15  -5     0    0    0 S    0  0.0   3:25.98 7 kjournald                                                                                                                                
  2683 root      15  -5     0    0    0 S    0  0.0   0:00.64 7 kjournald                                                                                                                                
  2687 root      15  -5     0    0    0 S    0  0.0   1:31.30 7 kjournald                                                                                                                                
  3261 root      15  -5     0    0    0 S    0  0.0   2:30.56 7 kondemand/7                                                                                                                              
  3364 root      20   0  3796  596  476 S    0  0.0   0:00.00 7 acpid                                                                                                                                    
  3575 root      20   0  8828 1140  956 S    0  0.0   0:00.00 7 sh                                                                                                                                       
  3597 root      20   0  8848 1216 1016 S    0  0.0   0:00.00 7 locate                                                                                                                                   
  3603 root      30  10  8896 1268 1016 S    0  0.0   0:00.00 7 updatedb.findut                                                                                                                          
  3612 root      30  10 58272  49m  616 S    0  0.4   0:27.04 7 sort                                                                                                                                     
  3655 root      20   0 11056 2852  516 S    0  0.0   5:36.46 7 redis-server                                                                                                                             
  3706 root      20   0 19832 1056  816 S    0  0.0   0:01.64 7 cron                                                                                                                                     
  3746 root      20   0  3796  580  484 S    0  0.0   0:00.00 7 getty                                                                                                                                    
  3748 root      20   0  3796  580  484 S    0  0.0   0:00.00 7 getty                                                                                                                                    
  7674 root      20   0 28376 1000  736 S    0  0.0   0:00.00 7 cron                                                                                                                                     
  7675 root      20   0  8828 1140  956 S    0  0.0   0:00.00 7 sh                                                                                                                                       
  7708 root      30  10 58272  49m  616 S    0  0.4   0:03.36 7 sort                                                                                                                                     
 22049 root      20   0  8828 1136  956 S    0  0.0   0:00.00 7 sh                                                                                                                                       
 22095 root      20   0  8848 1220 1016 S    0  0.0   0:00.00 7 locate                                                                                                                                   
 22099 root      30  10  8896 1264 1016 S    0  0.0   0:00.00 7 updatedb.findut                                                                                                                          
 22108 root      30  10 58272  49m  616 S    0  0.4   0:44.55 7 sort                                                                                                                                     
 22109 root      30  10  3792  452  360 S    0  0.0   0:00.00 7 frcode                                                                                                                                   
 26927 root      20   0  8828 1140  956 S    0  0.0   0:00.00 7 sh                                                                                                                                       
 26947 root      20   0  8848 1216 1016 S    0  0.0   0:00.00 7 locate                                                                                                                                   
 26951 root      30  10  8896 1268 1016 S    0  0.0   0:00.00 7 updatedb.findut                                                                                                                          
 26960 root      30  10 58272  49m  616 S    0  0.4   0:10.24 7 sort                                                                                                                                     
 26961 root      30  10  3792  452  360 S    0  0.0   0:00.00 7 frcode                                                                                                                                   
 27952 root      20   0 65948 3028 2400 S    0  0.0   0:00.00 7 sshd                                                                                                                                     
 30731 root      20   0     0    0    0 S    0  0.0   0:01.34 7 pdflush                                                                                                                                  
 31204 root      20   0     0    0    0 S    0  0.0   0:00.24 7 pdflush
 21857 deploy    20   0 1227m 2240  868 S    0  0.0   2:44.22 7 nginx                                                                                                                                    
 21858 deploy    20   0 1228m 2784  868 S    0  0.0   2:42.45 7 nginx                                                                                                                                    
 21862 deploy    20   0 1228m 2732  868 S    0  0.0   2:43.90 7 nginx                                                                                                                                    
 21869 deploy    20   0 1228m 2840  868 S    0  0.0   2:44.14 7 nginx                                                                                                                                    
 27994 deploy    20   0 19372 2216 1380 S    0  0.0   0:00.00 7 bash                                                                                                                                     
 28493 deploy    20   0  331m  32m  16m S    4  0.3   0:00.40 7 apache2                                                                                                                                  
 21856 deploy    20   0 1228m 2844  868 S    0  0.0   2:43.64 7 nginx
  3622 nobody    30  10 21156  10m  916 D    0  0.1   4:42.31 7 find                                                                                                                                     
  7716 nobody    30  10 12268 1280  888 D    0  0.0   0:43.50 7 find                                                                                                                                     
 22116 nobody    30  10 12612 1696  916 D    0  0.0   6:32.26 7 find                                                                                                                                     
 26968 nobody    30  10 12268 1284  888 D    0  0.0   1:56.92 7 find

Update

As suggested I take a look at /proc/interrupts and below the info there:

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
  0:         35          0          0 1469085485          0          0          0          0   IO-APIC-edge      timer
  1:          0          0          0          8          0          0          0          0   IO-APIC-edge      i8042
  8:          0          0          0          1          0          0          0          0   IO-APIC-edge      rtc0
  9:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          0          0        105          0          0          0          0   IO-APIC-edge      i8042
 16:          0          0          0          0          0          0          0  580212114   IO-APIC-fasteoi   3w-9xxx, uhci_hcd:usb1
 18:          0          0        142          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb6, ehci_hcd:usb7
 19:          9          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3, uhci_hcd:usb5
 21:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb2
 23:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4, ehci_hcd:usb8
1273:         0          0 1600400502          0          0          0          0          0   PCI-MSI-edge      eth0
1274:         0          0          0          0          0          0          0          0   PCI-MSI-edge      ahci
NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC:  214252181   69439018  317298553   21943690   72562482   56448835  137923978  407514738   Local timer interrupts
RES:   27516446   16935944   26430972   44957009   24935543   19881887   57746906   24298747   Rescheduling interrupts
CAL:      10655      10705      10685      10567      10689      10669      10667        396   function call interrupts
TLB:     529548     462587     801138     596193     922202     747313    2027966     946594   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
ERR:          0

All the values seems more or less the same for all the cores but this one IO-APIC-fasteoi 3w-9xxx, uhci_hcd:usb1 only affects to the core 7 (the same with the wait time of 40% ~ 60%) could be something attached to the usb port causing the issue?

Thanks in advanced

We have a server with 2 quad xeon 5550 processors. The only application installed on the server is SQL server. CPU usage rarely goes above 40%, and the transaction rate is 2,000 transactions per second.

Processor queue length is also about 2. Everything seems normal, except context switches which has an average of 88,000 and max of 232,000 during work hours.

I've read in SQL server documents that a context switch number larger than 15,000 is indicative of a CPU problem. Is this actually a problem, even though the CPU counters seem normal?

According to vmstat, my Linux server (2xCore2 Duo 2.5 GHz) is constantly doing around 20k context switches per second.

# vmstat 3
procs -----------memory----------  ---swap-- -----io----  -system-- ----cpu----
 r  b   swpd   free   buff  cache    si   so    bi    bo   in    cs us sy id wa
 2  0   7292 249472  82340 2291972    0    0     0     0    0     0  7 13 79  0
 0  0   7292 251808  82344 2291968    0    0     0   184   24 20090  1  1 99  0
 0  0   7292 251876  82344 2291968    0    0     0    83   17 20157  1  0 99  0
 0  0   7292 251876  82344 2291968    0    0     0    73   12 20116  1  0 99  0

... but uptime shows small load: load average: 0.01, 0.02, 0.01 and top doesn't show any process with high %CPU usage.

How do I find out what exactly is generating those context switches? Which process/thread?

I tried to analyze pidstat output:

# pidstat -w 10 1

12:39:13          PID   cswch/s nvcswch/s  Command
12:39:23            1      0.20      0.00  init
12:39:23            4      0.20      0.00  ksoftirqd/0
12:39:23            7      1.60      0.00  events/0
12:39:23            8      1.50      0.00  events/1
12:39:23           89      0.50      0.00  kblockd/0
12:39:23           90      0.30      0.00  kblockd/1
12:39:23          995      0.40      0.00  kirqd
12:39:23          997      0.60      0.00  kjournald
12:39:23         1146      0.20      0.00  svscan
12:39:23         2162      5.00      0.00  kjournald
12:39:23         2526      0.20      2.00  postgres
12:39:23         2530      1.00      0.30  postgres
12:39:23         2534      5.00      3.20  postgres
12:39:23         2536      1.40      1.70  postgres
12:39:23        12061     10.59      0.90  postgres
12:39:23        14442      1.50      2.20  postgres
12:39:23        15416      0.20      0.00  monitor
12:39:23        17289      0.10      0.00  syslogd
12:39:23        21776      0.40      0.30  postgres
12:39:23        23638      0.10      0.00  screen
12:39:23        25153      1.00      0.00  sshd
12:39:23        25185     86.61      0.00  daemon1
12:39:23        25190     12.19     35.86  postgres
12:39:23        25295      2.00      0.00  screen
12:39:23        25743      9.99      0.00  daemon2
12:39:23        25747      1.10      3.00  postgres
12:39:23        26968      5.09      0.80  postgres
12:39:23        26969      5.00      0.00  postgres
12:39:23        26970      1.10      0.20  postgres
12:39:23        26971     17.98      1.80  postgres
12:39:23        27607      0.90      0.40  postgres
12:39:23        29338      4.30      0.00  screen
12:39:23        31247      4.10     23.58  postgres
12:39:23        31249     82.92     34.77  postgres
12:39:23        31484      0.20      0.00  pdflush
12:39:23        32097      0.10      0.00  pidstat

Looks like some postgresql tasks are doing >10 context swiches per second, but it doesn't all sum up to 20k anyway.

Any idea how to dig a little deeper for an answer?

I've got munin monitoring virtual private server. And here're graphs for context switches:

By day: http://imgur.com/ljon5.gif

What could have caused this? I'm not worried about the pike, because it can be explained with heavy load which is confirmed by other graphs. But what about flatline? It looks like several days with zero switches, then it magically turns back on. And now it's off again. Any advice on where to look is greatly appreciated.