Home / server / Questions / 643986
In Process
Mike
Asked: 2014-11-13 20:09:41 +0800 CST

High network softirq requests for kernel 3.10 under a lot of traffic

  • 772

We have been experimenting rolling out 3.10 series kernel that's provided by centos via the centosplus repo for v6.

We have been noticing something interesting where under the same load the load for the server is 6-7x more on the 3.10 series then then stock 2.6 series. Using top we noticed this process using a lot of CPU

 [kworker/u66:2]

That lead us to install powertop and then saw this

            Usage       Events/s    Category       Description
         1110 ms/s     2045.2   Process        php-fpm: pool www
         36.0 ms/s     2165.4   Timer          tick_sched_timer
         57.7 ms/s     1285.0   Process        nginx: worker process
         13.3 ms/s     416.0        Timer          hrtimer_wakeup
         39.1 ms/s     350.7        Interrupt      [3] net_rx(softirq)

This is the same on a 2.6 series getting the same amount of traffic

            Usage       Events/s    Category       Description
         1795 ms/s     1654.0   Process        php-fpm: pool www
         45.3 ms/s     1110.4   Process        nginx: worker process
        562.8 µs/s     122.4        Process        /usr/bin/java -Xms200m -Xmx2000m -Xss256k -XX:MaxDirectMemorySize=516m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Dage
        497.1 µs/s      59.3        Process        /usr/sbin/gmond
         16.0 ms/s      30.2        Process        /usr/bin/redis-server 127.0.0.1:6379
          4.7 ms/s      32.8        Process        python /usr/bin/statsd-relay.py
         81.7 ms/s      0.00        Timer          tcp_delack_timer
         24.8 ms/s      0.00        Timer          tick_sched_timer
        549.4 µs/s       9.2        Process        java -Xmx6g -server -Dfile.encoding=utf-8 -XX:OnOutOfMemoryError=kill -9 %p -XX:+HeapDumpOnOutOfMemoryError -XX:HeapD
         15.2 ms/s      0.00        Interrupt      [3] net_rx(softirq)

As you can see the net_rx is 0 on 2.6 but we get as many as 4k/s on 3.10. The server specs are the same and removed all sysctl settings. I can replicate the issue just by installing 3.10 on a server.

the nics we have in are

06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
centos6

0 Answers