Home / server / Questions / 380935
In Process
TraderJoeChicago
Asked: 2012-04-19 16:37:10 +0800 CST

How to ban hardware interrupts with IRQBALANCE_BANNED_CPUS on Ubuntu?

  • 772

I would like to ban interrupts from certain CPUs. I heard about the IRQBALANCE_BANNED_CPUS option. I see irqbalance is running in the background of my machine. Where do I go to edit and how do I configure that option? For example, I want to exclude cpus 2,3,4,5 from interrupts. The argument descriptor is:

Provides a mask of cpus which irqbalance should ignore and never assign interrupts to

What does it mean by a mask? And where do I configure irqbalance with that option?

EDIT1: How to know that my configuration is in effect, in other words that my cpu is receiving NO interrupts? I am checking /proc/interrupts but some numbers are increasing there.

EDIT2: Now I booted my machine with IRQBALANCE_BANNED_CPUS=3e so only CPU 0 is NOT banned from interrupts. So I should expect to see cpo0 receiving a lot of interrupts and the other cpus not receiving interrupts, right? Here is my /proc/interrupts. The lines in bold are changing for ALL cpus. Lines 22, 24, 35 and LOC are changing.

            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       
   0:         26          0          0          0          0          0   IO-APIC-edge      timer
   1:          2          0          0          0          0          0   IO-APIC-edge      i8042
   6:          3          0          0          0          0          0   IO-APIC-edge      floppy
   8:          1          0          0          0          0          0   IO-APIC-edge      rtc0
   9:          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
  12:          4          0          0          0          0          0   IO-APIC-edge      i8042
  14:      13556          0          0          0          0          0   IO-APIC-edge      ata_piix
  15:          0          0          0          0          0          0   IO-APIC-edge      ata_piix
  18:          0          0          0          0          0          0   IO-APIC-fasteoi   ata_piix
  19:          2          0          0          0          0          0   IO-APIC-fasteoi   ohci1394
  20:          3          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb6
  21:        197        635         39          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4, uhci_hcd:usb7, HDA Intel
  22:        344       3506          0        702          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
  24:        162         48          0          0          0          0   IO-APIC-fasteoi   nvidia
  35:        174          0         47          0          0          0   IO-APIC-fasteoi   nvidia
  53:       3517          0          0          0          0          0   PCI-MSI-edge      eth0
 NMI:          0          0          0          0          0          0   Non-maskable interrupts
 LOC:      11007       8840       6480       5652       4272       3046   Local timer interrupts
 SPU:          0          0          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0          0          0   Performance monitoring interrupts
 PND:          0          0          0          0          0          0   Performance pending work
 RES:        292        169        217        125        122        126   Rescheduling interrupts
 CAL:         86        280        254        292        293        291   Function call interrupts
 TLB:       1147       1031       1348        616        177        322   TLB shootdowns
 TRM:          0          0          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0          0   Machine check exceptions
 MCP:          2          2          2          2          2          2   Machine check polls
 ERR:          5
 MIS:          0

EDIT3: It looks like IRQBALANCE_BANNED_CPUS option is completely IGNORED on Ubuntu. I tried rebooting my machine with 1, 3e and got interrupts all over. Just when I disable irqbalance by setting ENABLED=0 that I get a clean /proc/interrupts just on cpu0 and no other cpu.

linux ubuntu irq interrupts

4 Answers

  1. You set IRQBALANCE_BANNED_CPUS in /etc/default/irqbalance. I found this by looking in /etc/init.d/irqbalance. But what are the valid values for that setting? From Red Hat's man page:

    This is a hex mask without the leading '0x', on systems with large numbers of processors each group of eight hex digits is sepearated ba a comma ','. i.e. export IRQBALANCE_BANNED_CPUS=fc0 would prevent irqbalance from assigning irqs to the 7th-12th cpus (cpu6-cpu11) or export IRQBALANCE_BANNED_CPUS=ff000000,00000001 would prevent irqbalance from assigning irqs to the 1st (cpu0) and 57th-64th cpus (cpu56-cpu63).

    The concept of a mask is explained on wikipedia. Read that, then come back. Let's break down Red Hat's first example. The number that is written as fc0 in hexadecimal is written as 111111000000 in binary. Scanning right to left (i.e. from the least significant bit to the most significant bit), there are six zeroes. This means the 1st-5th cpus (cpu0-cpu5) can be assigned interrupts. Then, there are six ones. This means that the 7th-12th cpus (cpu6-cpu11) will not be assigned interrupts.

    It sounds like you want to allow cpu0 and cpu1 to receive interrupts but prevent cpu2, cpu3, cpu4, and cpu5 from being assigned interrupts. That means you need two zeroes and four ones, or 111100. This is 3C in hexadecimal. So, you'd create /etc/default/irqbalance with the contents

    ENABLED="1"
ONESHOT="0"
IRQBALANCE_BANNED_CPUS="3f"

    To see what is going on, try

    $ sudo service irqbalance stop
Stopping SMP IRQ Balancer: irqbalance.
$ source /etc/default/irqbalance 
$ sudo irqbalance --debug
    • 7

  2. an irqbalance bug prevents IRQBALANCE_BANNED_CPUS from working on NUMA machinges: http://code.google.com/p/irqbalance/issues/detail?id=43

    As of now, 1.0.5 is the latest irqbalance release and does not have the fix.

    • 2

  3. You have to use 3c as per above to make it work. Later versions do this automatically (at least by v 1.9 on debian/sid)

    • 0

  4. Adding to the suggestions above about troubleshooting irqbalance issues - A useful command for viewing the interrupt affinities resulting from irqbalance's run:

    find /proc/irq/ -name smp_affinity -print -exec cat {} ; | less

    • 0