I am trying to move all interrupts over to cores 0-3 to keep the rest of my cores free for high speed, low latency virtualization.

I wrote a quick script to set IRQ affinity to 0-3:

#!/bin/bash

while IFS= read -r LINE; do
    echo "0-3 -> \"$LINE\""
    sudo bash -c "echo 0-3 > \"$LINE\""
done <<< "$(find /proc/irq/ -name smp_affinity_list)"

This appears to work for USB devices and network devices, but not NVME devices. They all produce this error:

bash: line 1: echo: write error: Input/output error

And they stubbornly continue to produce interrupts evenly across almost all my cores.

If I check the current affinities of those devices:

$ cat /proc/irq/81/smp_affinity_list 
0-1,16-17
$ cat /proc/irq/82/smp_affinity_list
2-3,18-19
$ cat /proc/irq/83/smp_affinity_list
4-5,20-21
$ cat /proc/irq/84/smp_affinity_list
6-7,22-23
...

It appears "something" is taking full control of spreading IRQs across cores and not letting me change it.

It is completely critical that I move these to other cores, as I'm doing heavy IO in virtual machines on these cores and the NVME drives are producing a crap load of interrupts. This isn't Windows, I'm supposed to be able to decide what my machine does.

What is controlling IRQ affinity for these devices and how do I override it?

I am using a Ryzen 3950X CPU on a Gigabyte Auros X570 Master motherboard with 3 NVME drives connected to the M.2 ports on the motherboard.

(Update: I am now using a 5950X, still having the exact same issue)

Kernel: 5.12.2-arch1-1

Output of lspci -v related to NVME:

01:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01) (prog-if 02 [NVM Express])
    Subsystem: Phison Electronics Corporation E12 NVMe Controller
    Flags: bus master, fast devsel, latency 0, IRQ 45, NUMA node 0, IOMMU group 14
    Memory at fc100000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] Express Endpoint, MSI 00
    Capabilities: [d0] MSI-X: Enable+ Count=9 Masked-
    Capabilities: [e0] MSI: Enable- Count=1/8 Maskable- 64bit+
    Capabilities: [f8] Power Management version 3
    Capabilities: [100] Latency Tolerance Reporting
    Capabilities: [110] L1 PM Substates
    Capabilities: [128] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [200] Advanced Error Reporting
    Capabilities: [300] Secondary PCI Express
    Kernel driver in use: nvme

04:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01) (prog-if 02 [NVM Express])
    Subsystem: Phison Electronics Corporation E12 NVMe Controller
    Flags: bus master, fast devsel, latency 0, IRQ 24, NUMA node 0, IOMMU group 25
    Memory at fbd00000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] Express Endpoint, MSI 00
    Capabilities: [d0] MSI-X: Enable+ Count=9 Masked-
    Capabilities: [e0] MSI: Enable- Count=1/8 Maskable- 64bit+
    Capabilities: [f8] Power Management version 3
    Capabilities: [100] Latency Tolerance Reporting
    Capabilities: [110] L1 PM Substates
    Capabilities: [128] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [200] Advanced Error Reporting
    Capabilities: [300] Secondary PCI Express
    Kernel driver in use: nvme

05:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01) (prog-if 02 [NVM Express])
    Subsystem: Phison Electronics Corporation E12 NVMe Controller
    Flags: bus master, fast devsel, latency 0, IRQ 40, NUMA node 0, IOMMU group 26
    Memory at fbc00000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [80] Express Endpoint, MSI 00
    Capabilities: [d0] MSI-X: Enable+ Count=9 Masked-
    Capabilities: [e0] MSI: Enable- Count=1/8 Maskable- 64bit+
    Capabilities: [f8] Power Management version 3
    Capabilities: [100] Latency Tolerance Reporting
    Capabilities: [110] L1 PM Substates
    Capabilities: [128] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [200] Advanced Error Reporting
    Capabilities: [300] Secondary PCI Express
    Kernel driver in use: nvme

$ dmesg | grep -i nvme
[    2.042888] nvme nvme0: pci function 0000:01:00.0
[    2.042912] nvme nvme1: pci function 0000:04:00.0
[    2.042941] nvme nvme2: pci function 0000:05:00.0
[    2.048103] nvme nvme0: missing or invalid SUBNQN field.
[    2.048109] nvme nvme2: missing or invalid SUBNQN field.
[    2.048109] nvme nvme1: missing or invalid SUBNQN field.
[    2.048112] nvme nvme0: Shutdown timeout set to 10 seconds
[    2.048120] nvme nvme1: Shutdown timeout set to 10 seconds
[    2.048127] nvme nvme2: Shutdown timeout set to 10 seconds
[    2.049578] nvme nvme0: 8/0/0 default/read/poll queues
[    2.049668] nvme nvme1: 8/0/0 default/read/poll queues
[    2.049716] nvme nvme2: 8/0/0 default/read/poll queues
[    2.051211]  nvme1n1: p1
[    2.051260]  nvme2n1: p1
[    2.051577]  nvme0n1: p1 p2

I've purchased a VPS for dev work (at Linode) some time ago and am wishing to install NetBSD for the sake of experience, however its current support under the Xen hypervisor does not allow for SMP.

I would assume this would mean I could only use one core, am I correct? Or just one processor?

My content will contain a few not too intensive pages, although a lot of testing stuff and the odd script ran occasionally may be a bit intensive, I am wondering if a single Xeon core would be an alright solution for a small-medium dev website.

Don.

I am looking to upgrade one of our servers, an 8 core server running VMware Server 2. The VMs it runs are mostly web, file, and email servers; specifically, it runs three webservers, 2 email/file servers, and a few Jabber/XMPP Servers.

When we originally set this machine up, we had the two web servers configured with two vCPUs. We had some very serious performance issues. At the time we were only running 3 VMs with a total of 5 vCPUs. We were also running VMware Server 1 and a RAID1. We have now upgraded to a RAID10 and I would never consider anything less! :-)

The new server will have VMware ESXi on it instead of VMware Server. We are looking at either another 8-core machine with faster processors, or a 16- or 32-core server. I want to make sure that I avoid the issues we had before.

Because I had such issues with multiple vCPUs before, my plan was to replace the primary web server VM with multiple identical webserver VMs (2 - 4), each with a single vCPU, serving content from an NFS share, and place these in a load-balanced configuration. SO I would basically be replacing one VM with at least 4: 2 small webserver VMs, a file server VM, and a load balancing VM. My plan here being twofold: First, I can avoid any multiple vCPU isuses I saw before, and secondly, I can spawn new web server VMs to handle increased load.

Note that I understand I am not getting all the benefits of the load balancing because I still have a single point of failure.

Is this a good plan, or is it unnecessary? Can VMware ESXi handle multiple vCPUs better than VMware Server 1 could over three years ago? Which will perform better, a single large webserver VM with 2 - 4 vCPUs, or 4 VMs with a single vCPU, in a load-balanced configuration?

I have a server that is a quad-core i-7 with hyperthreading enabled. The kernel recognises this as 8 cpus, as it should. I want to create a couple virtual hosts running on this machine. virt-manager lets me select which cpus are used on which machine. I want to configure this so that I assign both cpus in a hyperthreaded pair to the same virtual machine. This way one machine under heavy load will not effect another virtual machine by sharing a single core due to the hyperthreading. But I do want hyperthreading available to each vm because I am running tasks that involve heavy parallelization.

So, how can I tell which cpus the kernel recognizes are a pair? I would suspect cpu 0 and cpu 1 make a pair, then cpu 2 and cpu 3, and so on, but I am not sure how to test/confirm.

I've noticed that the CPU usage on our 8-CPU database server, running SQL Server 2008, is not balanced at all.

Here are the 1 day averages for a random day a while back, which is typical and consistently asymmetric:

9, 15, 10, 21, 18, 21, 14, 9

(only thumbnail here, because the image is really tall, but click through for the full size image)

Compared with our 4-CPU web servers, which are all almost exactly and perfectly balanced all the time, that struck me as odd.

Now, this is a dedicated server, so the only thing running on it is SQL Server 2008 (and the built-in full text indexing, which we use pretty heavily), so I'm not sure why the CPU usage would be so asymmetric. Thoughts?