Mohammad Hedayati's questions

I have two machines:

1) 2 x E5-2620 v3 @ 2.40GHz (microcode: 0x3c 2018-01-19), 8GB RAM (CentOS 7 w/ 4.11 kernel)

2) 2 x E5-2630 v3 @ 2.40GHz (microcode: 0x3c 2018-01-19), 64GB RAM (Fedora 22 w/ 4.11 kernel)

The kernels are custom compiled but otherwise without any modification. The .config for both builds are exactly the same (and neither have kpti since it was introduce on 4.14 iirc).

I wrote a code like:

beg = rdtsc_beg();
for (i = 0; i < 1000000; ++i)
{
    syscall(512); //ENOSYS
}
end = rdtsc_end();

printf("syscall: %lu cycles\n", (end - beg) / 1000000);

to measure the cost of a syscall. I get 99 cycles on machine 1 and 264 cycles on machine 2.

I can not understand why there is such a huge difference given almost anything that I think is relevant being the same.

Any ideas what might be causing the difference or any clue where I should be looking to find out the cause?

Edit:

I changed the code to:

volatile int ret = 0;

beg = rdtsc_beg();
for (i = 0; i < 1000000; ++i)
{
    ret++;
}
end = rdtsc_end();

printf("inc: %lu cycles\n", (end - beg) / 1000000);

And compiled statically:

400a3f:       0f a2                   cpuid
400a41:       0f 31                   rdtsc
400a43:       89 d6                   mov    %edx,%esi
400a45:       89 c7                   mov    %eax,%edi
400a47:       48 c1 e6 20             shl    $0x20,%rsi
400a4b:       89 ff                   mov    %edi,%edi
400a4d:       b8 80 96 98 00          mov    $0xf4240,%eax
400a52:       48 09 fe                or     %rdi,%rsi
400a55:       0f 1f 00                nopl   (%rax)
400a58:       8b 54 24 0c             mov    0xc(%rsp),%edx
400a5c:       83 c2 01                add    $0x1,%edx
400a5f:       48 83 e8 01             sub    $0x1,%rax
400a63:       89 54 24 0c             mov    %edx,0xc(%rsp)
400a67:       75 ef                   jne    400a58 <main+0x28>
400a69:       0f 01 f9                rdtscp
400a6c:       41 89 d0                mov    %edx,%r8d
400a6f:       89 c7                   mov    %eax,%edi
400a71:       0f a2                   cpuid

The result is 5 cycles on Machine 1 vs 14 cycles in Machine 2.

I have two nodes connected with dual-port Mellanox Connext-X3 VPI HCAs via an IB switch. The nodes are two socket machines with Hasswell CPUs and 2 16GB DIMMs per each socket (totaling 64GB). Everything seems to work perfectly, except for the performance numbers that don't seem right.

When I run ib_read_bw benchmark:

server# ib_read_bw --report_gbits
client# ib_read_bw server --report_gbits

---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      1000             37.76              37.76          0.072016
---------------------------------------------------------------------------------------

But when I run dual-port:

server# ib_read_bw --report_gbits -O
client# ib_read_bw server --report_gbits -O
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      2000             52.47              52.47          0.100073
---------------------------------------------------------------------------------------

I only get less than 40% improvement (am I wrong to expect ~2x the single port bandwidth)?

I don't know what could be the bottleneck here and how to find it.

Other configurations that may be helpful:

• Each socket has 8 cores, overall each machine has 32 HTs
• Each DIMM provides ~14GB/s bw (per socket mem-bw: ~28 GB/s, overall ~56 GB/s)
• I used Mellanox's Auto Tuning Utility tool to tune the interrupts.
• IB links are 4X 10.0 Gbps (FDR10) -- each 40 Gb/s
• I am using Mellanox OFED 4.3.

I want a UDP echo server to get packets, and reply exactly what it has received. How can I simply do this using netcat or socat? It should stay alive forever and handle packets coming from several hosts.

I'm using iozone version 3.373 to benchmark NFSv4 and NBD, but the results are a bit weird! It says that outputs are in KBytes/sec but I am getting values such as 3,411,938 for random read. I am using a 10/100 Ethernet network and best throughput using netperf is around 96Mbps. I am actually executing following command.

iozone -i 0 -i 1 -i 2 -i 8 -a -Rb nbd.xls -g 10M -q 1M