Questions[affinity](server)

I have shell scripts that I use to programmatically configure my Virtual Machines (VMs) on my vmware ESXi host. I use the "govc" command-line-interface (CLI) which provides a suite of commands to configure my VMs that run on my ESXI (i.e. powering on, setting ram, etc...)

I am trying now to set the CPU affinity for some of my virtual machines (i.e. VM #2 should use CPUs 9-16 where-as VM#1 should use CPU 2-8). However, I cannot find any govc command to do this programmatically, though I can very clearly see how to do this within the ESXI web-based gui:

I reference the govc API for commands here, but again, I could not find anything. The CPU affinity should be set for a specific VM, and there should be an argument of CPU numbers separated by comma or with hypen (-) for a range, I would imagine: https://github.com/vmware/govmomi/blob/master/govc/USAGE.md

Can anyone who is familiar with vmware ESXi and govc please help?

Thank you.

I have servers which deal with a large number of network packets. I've seen evidence that user space processes running on CPU0 have their performance impacted when network load is high. I'm fairly sure this relates to interrupt handling, specifically for network devices.

I'm thus experimenting with changing the affinity of network (eth) devices in order to test my hypothesis and see if I can improve performance.

I understand that in order to change IRQ affinity I must change the value in /proc/irq/XXX/smp_affinity.

When I change the value in that file I can only move the interrupt handling from one CPU to another. For example, in a 4 core system I can set that value to 1, 2, 4 or 8 and I see the interrupts move from cores 0,1,2 and 3 respectively (by monitoring /proc/interrupts). However, if I set the affinity value to any combination of cores then I don't see interrupts balanced across all cores. They always stick to one core. For example,

• Setting ff: CPU0 used
• Setting 12: CPU4 used

(Thus it seems only the lowest specified core is used)

I have stopped the irqbalance daemon in case this was affecting things (although I suspect not as my understanding would be that irqbalance would change the values of the smp_affinity file and I don't see that).

My questions thus are:

• How do I balance interrupts across cores?
• Is it indeed even possible to do so?

N.B. I have tried this on 2 machines: A ubuntu VM running in VBox with 2 cores and with the eth device using IO-APIC; A ubuntu real machine with 4 cores (8 hyperthreaded) and with eth device using PCI-MSI.

As a bonus question:

• Is it possible to balance PCI interrupts? From what I understand it should be definitely possible with IO-APIC interrupts.

OK, this is a really weird one and I'm not even sure how to properly describe it. We had a customer complain that a specific page on our website wasn't working, and one of our internal technicians was able to reproduce the issue as well. Most of the website is working fine. This is deployed on Azure App Service.

I checked running the exact same page as the technician, and it worked fine for me. The entire request is identical, except for authentication cookies. When I run the request, I get 200 OK, but the technician and the customer get 404 NOT FOUND.

The issue only started after we did a VIP swap this morning on Azure App Service (which I am new to). I deployed a service update this morning to the Staging Deployment Swap, then a few minutes later did the VIP Swap. I think that both the customer and the technician had their browser open and session active during the VIP Swap.

I've done some troubleshooting, and here is what I discovered. I can use Fiddler to capture the exact trace for the web page that works fine for me. Then I can copy only one value from the request for the technician that gets 404 Error, and suddenly I can reproduce the 404 error as well. The difference is one cookie:

Cookie: ARRAffinity=blahblahblahblah;

My basic understanding is that this is a key for identifying which server the user is connecting to so they get affinity to a specific instance in the load-balanced set (2 servers). We were able to fix the issue by having the technician and customer delete all cookies in their browser, but even logging out and back in wouldn't fix the issue.

Why would a "stale" affinity key cause a random 404 on one specific page? Is it possible that some of the user's requests are actually getting directed to the old staging deployment site, even though they are hitting the url that connects to the Production deployment site?

I have three processes running on Intel Core 2 Duo CPU. From watching output of 'top' and graphs of CPU load (drawn by MRTG, data collection via SNMP) I can see that CPU load is never more than 50%, and, most of the day, when those processes are busy CPU load has a ceiling at 50 %. I mean, CPU load grows up to 50% in the morning and stays there until late evening.

My first thought was that only one core was used at 100% thus giving 50% of both CPUs. But, as there are three processes running and from 'top' I see that both cores are being loaded, so this is not the case. schedtool shows that CPU affinity for those three processes is at default, 0x03, allowing them to use both cores. If I force one process to one core (schedtool -a 0x01), and two others to second (schedtool -a 0x02), cumulative usage grows beyond 50%.

Why three processes seem to consume only 50% of two cores? Why forcing them to different CPUs allows usage to grow higher? Any hints?

P.S. Processes in question are Counter-Strike servers.

I use a distributed user-space filesystem (GlusterFS) and I would like to be sure GlusterFS processes will always have the computing power they need.

Each execution node of my grid have 2 CPU, with 4 cores per CPU and 2 threads per core (16 "processors" are seen by Linux).

My goal is to guarantee that GlusterFS processes have enough processing power to be reliable, responsive and fast. (There is no marketing here, just the dreams of a sysadmin ;-)

I consider two main points :

• GlusterFS processes
• I/O for data access (on local disks, or remote disks)

I thought about binding GlusterFS instances on a specific "processor".

I would like to be sure that :

• No grid job will impact the kernel and the GlusterFS instances
• Researchers jobs won't be affected by system processes (I'd like to reserve a pool of cores to job execution and be sure that no system process will use these CPUs)

But what about I/O ? As we handle a huge amount of data (several terabytes), we'll have a lot of interuptions.

How can I distribute these operations on my processors ? What are the "best practices" ?

Thanks for your comments!