Questions[opteron](server)

According to numactl, this dual CPU Opteron box is UMA rather than the expected NUMA:

$ numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 65534 MB
node 0 free: 381 MB
node distances:
node   0 
  0:  10 

I think it should be NUMA because there are four 4-core CPUs*. As I understand it, each CPU has its own memory channel; if a CPU needs to access memory in a non-local node, it must use hypertransport, which costs more time than accessing memory in the local node. AMD explains it here.

The motherboard has 16 RAM slots, 8 for each CPU. All 16 are populated with 4GB each for a total of 64GB. Some more particulars:

• Processors: 2 x AMD 6128 ("Magny-Cours")
• Mobo: Supermicro H8DG6/i(-F)
• BIOS: AMI v02.68 -- mobo/BIOS manual (pdf)
• Linux: 2.6.32
• OS: Debian "wheezy"
• BIOS memory config:
  • bank interleaving: auto
  • node interleaving: auto
  • channel interleaving: auto
  • CS sparing: disabled
  • bank swizzle mode: enabled

Why is numactl reporting that this box is UMA?

*There are two CPU dies per package, so the motherboard only has two CPU sockets.

I'm in the process of attempting to benchmark a multithreaded application on my new HP Proliant Server which has an 2x Opteron 6272 and 64gb of ram.

When I run the application on a desktop machine (a range of i7s and a Xeon X5675 processors) the application will cause all cores to hit a near 100% utilization.

When I run the application on my server, no matter how many threads I run, the total cpu utilization of the application hovers around 20-25%. That is if I'm running with 32 threads, all 32 cores will hang at around 20%, if I run 16 threads they'll hang around 40%, and so on.

1. At first I suspected this had to do with the operating system, so I installed Windows 7 on the server so that the desktops and the server had the same OS.
2. Then I suspected it was the hardware, I changed the power management in the bios to High Performance. Even though this did increase the benchmark time, the same 20% utilization problem persists.
3. I can get all 32 cores at 100% using the y-cruncher benchmark. My custom benchmark is written in .NET, could this possibly have anything to do with it?

I'm perplexed by this problem. Anyone have an idea of what could cause this?

I am seeing latency on a Hyper-V host and its guests lately. When I run ping -f from a Linux prompt, I see the dots "pulsate" like a heartbeat indicating 2 successive latency spikes within 1 second followed by about 1-1.5 seconds of "normal" operation.

Details

This is what it looks like for ICMP echo request/reply packets send at an 50-ms interval (total length of the sequence is 5.5 seconds):

# time ping -c 100 -i 0.05 192.168.111.199
PING 192.168.111.199 (192.168.111.199) 56(84) bytes of data.
64 bytes from 192.168.111.199: icmp_seq=1 ttl=64 time=1.03 ms
64 bytes from 192.168.111.199: icmp_seq=2 ttl=64 time=0.480 ms
64 bytes from 192.168.111.199: icmp_seq=3 ttl=64 time=0.911 ms
64 bytes from 192.168.111.199: icmp_seq=4 ttl=64 time=3.01 ms
64 bytes from 192.168.111.199: icmp_seq=5 ttl=64 time=0.473 ms
64 bytes from 192.168.111.199: icmp_seq=6 ttl=64 time=0.439 ms
64 bytes from 192.168.111.199: icmp_seq=7 ttl=64 time=0.896 ms
64 bytes from 192.168.111.199: icmp_seq=8 ttl=64 time=0.425 ms
64 bytes from 192.168.111.199: icmp_seq=9 ttl=64 time=0.654 ms
64 bytes from 192.168.111.199: icmp_seq=10 ttl=64 time=2.58 ms
64 bytes from 192.168.111.199: icmp_seq=11 ttl=64 time=0.598 ms
64 bytes from 192.168.111.199: icmp_seq=12 ttl=64 time=0.511 ms
64 bytes from 192.168.111.199: icmp_seq=13 ttl=64 time=0.609 ms
64 bytes from 192.168.111.199: icmp_seq=14 ttl=64 time=0.621 ms
64 bytes from 192.168.111.199: icmp_seq=15 ttl=64 time=0.451 ms
64 bytes from 192.168.111.199: icmp_seq=16 ttl=64 time=1.05 ms
64 bytes from 192.168.111.199: icmp_seq=17 ttl=64 time=0.447 ms
64 bytes from 192.168.111.199: icmp_seq=18 ttl=64 time=0.462 ms
64 bytes from 192.168.111.199: icmp_seq=19 ttl=64 time=0.635 ms
64 bytes from 192.168.111.199: icmp_seq=20 ttl=64 time=1.90 ms
64 bytes from 192.168.111.199: icmp_seq=21 ttl=64 time=0.654 ms
64 bytes from 192.168.111.199: icmp_seq=22 ttl=64 time=0.634 ms
64 bytes from 192.168.111.199: icmp_seq=23 ttl=64 time=794 ms      <<<---- SPIKE
64 bytes from 192.168.111.199: icmp_seq=24 ttl=64 time=742 ms
64 bytes from 192.168.111.199: icmp_seq=25 ttl=64 time=690 ms
64 bytes from 192.168.111.199: icmp_seq=26 ttl=64 time=638 ms
64 bytes from 192.168.111.199: icmp_seq=27 ttl=64 time=586 ms
64 bytes from 192.168.111.199: icmp_seq=28 ttl=64 time=534 ms
64 bytes from 192.168.111.199: icmp_seq=29 ttl=64 time=482 ms
64 bytes from 192.168.111.199: icmp_seq=30 ttl=64 time=430 ms
64 bytes from 192.168.111.199: icmp_seq=31 ttl=64 time=379 ms
64 bytes from 192.168.111.199: icmp_seq=32 ttl=64 time=327 ms
64 bytes from 192.168.111.199: icmp_seq=33 ttl=64 time=275 ms
64 bytes from 192.168.111.199: icmp_seq=34 ttl=64 time=223 ms
64 bytes from 192.168.111.199: icmp_seq=35 ttl=64 time=171 ms
64 bytes from 192.168.111.199: icmp_seq=36 ttl=64 time=119 ms
64 bytes from 192.168.111.199: icmp_seq=37 ttl=64 time=66.8 ms
64 bytes from 192.168.111.199: icmp_seq=38 ttl=64 time=15.1 ms
64 bytes from 192.168.111.199: icmp_seq=39 ttl=64 time=0.657 ms
64 bytes from 192.168.111.199: icmp_seq=40 ttl=64 time=0.755 ms
64 bytes from 192.168.111.199: icmp_seq=41 ttl=64 time=0.744 ms
64 bytes from 192.168.111.199: icmp_seq=42 ttl=64 time=0.652 ms
64 bytes from 192.168.111.199: icmp_seq=43 ttl=64 time=0.507 ms
64 bytes from 192.168.111.199: icmp_seq=44 ttl=64 time=0.338 ms
64 bytes from 192.168.111.199: icmp_seq=45 ttl=64 time=0.455 ms
64 bytes from 192.168.111.199: icmp_seq=46 ttl=64 time=0.755 ms
64 bytes from 192.168.111.199: icmp_seq=47 ttl=64 time=0.530 ms
64 bytes from 192.168.111.199: icmp_seq=48 ttl=64 time=0.730 ms
64 bytes from 192.168.111.199: icmp_seq=49 ttl=64 time=0.385 ms
64 bytes from 192.168.111.199: icmp_seq=50 ttl=64 time=0.664 ms
64 bytes from 192.168.111.199: icmp_seq=51 ttl=64 time=0.440 ms
64 bytes from 192.168.111.199: icmp_seq=52 ttl=64 time=0.721 ms
64 bytes from 192.168.111.199: icmp_seq=53 ttl=64 time=0.963 ms
64 bytes from 192.168.111.199: icmp_seq=54 ttl=64 time=0.452 ms
64 bytes from 192.168.111.199: icmp_seq=55 ttl=64 time=0.535 ms
64 bytes from 192.168.111.199: icmp_seq=56 ttl=64 time=0.467 ms
64 bytes from 192.168.111.199: icmp_seq=57 ttl=64 time=0.591 ms
64 bytes from 192.168.111.199: icmp_seq=58 ttl=64 time=0.749 ms
64 bytes from 192.168.111.199: icmp_seq=59 ttl=64 time=0.440 ms
64 bytes from 192.168.111.199: icmp_seq=60 ttl=64 time=0.498 ms
64 bytes from 192.168.111.199: icmp_seq=61 ttl=64 time=816 ms      <<<---- SPIKE
64 bytes from 192.168.111.199: icmp_seq=62 ttl=64 time=764 ms
64 bytes from 192.168.111.199: icmp_seq=63 ttl=64 time=712 ms
64 bytes from 192.168.111.199: icmp_seq=64 ttl=64 time=660 ms
64 bytes from 192.168.111.199: icmp_seq=65 ttl=64 time=607 ms
64 bytes from 192.168.111.199: icmp_seq=66 ttl=64 time=556 ms
64 bytes from 192.168.111.199: icmp_seq=67 ttl=64 time=504 ms
64 bytes from 192.168.111.199: icmp_seq=68 ttl=64 time=452 ms
64 bytes from 192.168.111.199: icmp_seq=69 ttl=64 time=400 ms
64 bytes from 192.168.111.199: icmp_seq=70 ttl=64 time=348 ms
64 bytes from 192.168.111.199: icmp_seq=71 ttl=64 time=296 ms
64 bytes from 192.168.111.199: icmp_seq=72 ttl=64 time=244 ms
64 bytes from 192.168.111.199: icmp_seq=73 ttl=64 time=192 ms
64 bytes from 192.168.111.199: icmp_seq=74 ttl=64 time=140 ms
64 bytes from 192.168.111.199: icmp_seq=75 ttl=64 time=88.6 ms
64 bytes from 192.168.111.199: icmp_seq=76 ttl=64 time=36.6 ms
64 bytes from 192.168.111.199: icmp_seq=77 ttl=64 time=0.466 ms
64 bytes from 192.168.111.199: icmp_seq=78 ttl=64 time=0.559 ms
64 bytes from 192.168.111.199: icmp_seq=79 ttl=64 time=0.655 ms
64 bytes from 192.168.111.199: icmp_seq=80 ttl=64 time=0.555 ms
64 bytes from 192.168.111.199: icmp_seq=81 ttl=64 time=2.16 ms
64 bytes from 192.168.111.199: icmp_seq=82 ttl=64 time=0.660 ms
64 bytes from 192.168.111.199: icmp_seq=83 ttl=64 time=0.452 ms
64 bytes from 192.168.111.199: icmp_seq=84 ttl=64 time=0.454 ms
64 bytes from 192.168.111.199: icmp_seq=85 ttl=64 time=0.608 ms
64 bytes from 192.168.111.199: icmp_seq=86 ttl=64 time=0.471 ms
64 bytes from 192.168.111.199: icmp_seq=87 ttl=64 time=0.506 ms
64 bytes from 192.168.111.199: icmp_seq=88 ttl=64 time=0.435 ms
64 bytes from 192.168.111.199: icmp_seq=89 ttl=64 time=0.450 ms
64 bytes from 192.168.111.199: icmp_seq=90 ttl=64 time=0.582 ms
64 bytes from 192.168.111.199: icmp_seq=91 ttl=64 time=0.537 ms
64 bytes from 192.168.111.199: icmp_seq=92 ttl=64 time=0.452 ms
64 bytes from 192.168.111.199: icmp_seq=93 ttl=64 time=0.533 ms
64 bytes from 192.168.111.199: icmp_seq=94 ttl=64 time=0.638 ms
64 bytes from 192.168.111.199: icmp_seq=95 ttl=64 time=1.28 ms
64 bytes from 192.168.111.199: icmp_seq=96 ttl=64 time=1.86 ms
64 bytes from 192.168.111.199: icmp_seq=97 ttl=64 time=0.770 ms
64 bytes from 192.168.111.199: icmp_seq=98 ttl=64 time=0.822 ms
64 bytes from 192.168.111.199: icmp_seq=99 ttl=64 time=0.452 ms
64 bytes from 192.168.111.199: icmp_seq=100 ttl=64 time=385 ms      <<<---- SPIKE

--- 192.168.111.199 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 5112ms
rtt min/avg/max/mdev = 0.338/137.383/816.359/237.291 ms, pipe 16

real    0m5.502s
user    0m0.028s
sys     0m0.028s

From the ping result it looks like a full stop of data transmission for about 800 ms followed by a period of resumed operations for 23-24 request/reply sequences (around 1200 ms).

History and environment

This seems to be somewhat new and associated with a recent reboot 2 days ago - I have not seen it before on this host. Host is running Hyper-V 2008 R2 on a recent HP DL165 G7 (AMD Opteron 6238, 4x Intel 82576). The host's management interface as well as all guests (connected on a separate interface) are uniformly affected. The connection to the testing machine is either a local Ethernet LAN or a routed local IP Network (Ethernet all through).

Diagnostics so far:

The network itself is "clean":

# ping -f 192.168.112.187
PING 192.168.112.187 (192.168.112.187) 56(84) bytes of data.
....^C
--- 192.168.112.187 ping statistics ---
8633 packets transmitted, 8629 received, 0% packet loss, time 11208ms
rtt min/avg/max/mdev = 0.330/1.096/25.797/1.643 ms, pipe 2, ipg/ewma 1.298/1.499 ms

I also tried a different switch port / interface combination to rule out a hardware issue without any effect - the problem seems to persist on any interface.

Hyper-V guests of this host which on the same network (and get their traffic switched by the Hyper-V virtual switch) seem not to have any latency issues.

I've read about Hyper-V timing issues with AMD CPUs, but these appeared to only affect guest systems. Also, people occasionally see latencies for aggregated/bonded ethernet channels - we do not have those.

I'm doing some bioinformatics work where the limitation is amount of memory (i.e., the process is not parallelizable). I see servers like the Dell R715 that have two sockets for Opteron 6100 CPUs and 128GB of RAM possible with 16 x 8GB DIMMs.

Each CPU has a bank of DIMMs associated with it.

My question is, if I put Linux on a machine like that, is only half the RAM addressable by each CPU? In other words, if I run a single-threaded program under Linux will that process have access to 128GB of RAM or only 64GB of RAM?

I'm currently working with an AMD Opteron-based NUMA system. For the needs of my current project, I'd like to make Linux and all of the system processes to utilize only the CPU0 (and preferably, only one of its cores), leaving all other cores for my own egoistic needs. I know that it will really impact system performance, but it is OK now.

So, I'd like to know if it is possible to achieve at all without messing with the kernel source code. Any links to the up-to-date articles about NUMA and SMP implementation details in Linux will be appreciated, too.