Evan's questions

I have several machines with ConnectX-7 Infiniband cards and they're plugged into an Nvidia QM9700 switch. I've confirmed 400 Gbit NDR at both ends (ibstat on the host and in the console on the switch). The machines are running Ubuntu 22.04 and the Mellanox 5.8-3.0.7.0 drivers. I've done a lot of testing with ib_write_bw and the most I can get is ~251 Gbit/s. The actual test commands are:

Server side (host_a):

numactl -N 0 -m 0 ib_write_bw -d mlx5_4 -F --report_gbits

Client side (host_b):

numactl -N 0 -m 0 ib_write_bw -d mlx5_4 -F --report_gbits --run_infinitely host_b

The cards are in the correct numa domains to match numactl, but I've tried other combinations of that with no luck. Output ends up looking something like this:

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_4
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x54 QPN 0x0058 PSN xxx RKey 0x1820e0 VAddr xxx
 remote address: LID 0x53 QPN 0x0058 PSN xxx RKey 0x1820e0 VAddr xxx
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      2353827          0.00               246.81             0.470754
 65536      2339084          0.00               245.27             0.467815
 65536      2338736          0.00               245.23             0.467746
 65536      2338574          0.00               245.22             0.467713
 65536      2338610          0.00               245.22             0.467720

I know this is probably a long shot, but wondering if anyone has actually achieved 400 gbit over infiniband with ib_write_bw that might know something we missed.

I'm buying servers lately and all of them have disks that support TCG Opal full-disk encryption (aka SED). What I'd like to do is:

• Store data encrypted-at-rest on the disks (NVMe & SAS).
• Not be required to enter a password/passphrase at server boot.
• Have encryption keys be stored in the server's TPM 2.0 module.

To simplify, the goal is to "lock" the data to the server, ensuring that if a disk is removed and inserted in another machine the data is inaccessible. I've seen sedutil but haven't yet played with it as it looks like it requires entering a passphrase at every boot. I have hundreds of servers I'd like to enable this on, so having to enter a passphrase (is it per-disk?) at reboot is not an option. TPM seems like a natural place to store this kind of thing, so I'm wondering if anyone's done anything like this. The only reference I can find to anything like this is in this Micron white paper. The servers are all running Ubuntu 16.04 (Xenial). There's more than one disk per server, if that matters.

I have a system with dual Intel E5-2620 v3 @ 2.40GHz, 64 GB ram, an LSI 3008 HBA, 8x 1.6 TB Intel S3510 SSDs. I've been playing benchmarking it in different configurations using fio and I've gotten some interesting results. Benchmarking each raw disk individually, I'm seeing ~450 MB/s random writes per device. In a RAID0 (stripe without parity) I'm seeing ~3500 MB/s random writes - basically 450 * 8 disks.

However, when I tried RAID 5, performance completely tanked. I realize raid5 should be significantly worse than raid0, but RAID5 performance using the same test was about 50 MB/s. I've used RAID 5 before and while the performance has never been great, I've never seen a 99% penalty for it. The fio test I'm running has 600 threads writing random data in 512k blocks. The filesystem on the device is xfs for all these tests.

Setting /sys/block/md0/md/stripe_cache_size to 32768 (the maximum possible value) increased the overall throughput to ~130 MB/s, leading me to assume that the problem is the lack of a writeback cache like those found in hardware RAID controllers. But I'm wondering if there's anything I can do to improve the mdadm raid 5 performance. Any idea what could be causing this, or how to improve raid 5 performance?

Interestingly, I also tried a 16-disk RAID-10 (same disks plus a second LSI HBA) and the performance was ~2400 MB/s - a 33% decrease from RAID 0. Given how RAID10 works, I would have expected the performance to be nearly identical to RAID 0.

For anyone interested, here's the fio config file:

[global]
rw=randwrite
direct=1
numjobs=600
group_reporting
bs=512k
runtime=120
ramp_time=5
size=10G


[raid]
new_group
directory=/raid/

I have a server and I want to ensure it's always running at maximum speed, never in energy-saving mode. I've tried a few strategies (aside from rebooting and changing the setting in the bios, which I would have to do for a few hundred servers) but I'm getting weird results from cpufreq-info. E.g.:

root@host:~# cpufreq-info -c 0
cpufrequtils 007: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to [email protected], please.
analyzing CPU 0:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 10.0 us.
  hardware limits: 1.20 GHz - 2.20 GHz
  available frequency steps: 2.20 GHz, 2.20 GHz, 2.10 GHz, 2.00 GHz, 1.90 GHz, 1.80 GHz, 1.70 GHz, 1.60 GHz, 1.50 GHz, 1.40 GHz, 1.30 GHz, 1.20 GHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance
  current policy: frequency should be within 2.00 GHz and 2.20 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 1.20 GHz (asserted by call to hardware).
  cpufreq stats: 2.20 GHz:6.50%, 2.20 GHz:0.24%, 2.10 GHz:0.34%, 2.00 GHz:0.05%, 1.90 GHz:0.04%, 1.80 GHz:0.03%, 1.70 GHz:0.03%, 1.60 GHz:0.03%, 1.50 GHz:0.03%, 1.40 GHz:0.03%, 1.30 GHz:0.03%, 1.20 GHz:92.66%  (8080636)

Note that the min speed shows "2.00 GHz", and the governor is set to "performance", yet the current CPU frequency is 1.20 GHz. However, if I just look in /proc/cpuinfo, the value there is consistently ~2.2 GHz:

root@host:~# egrep 'MHz|Xeon' /proc/cpuinfo ; cpufreq-info -c 0
model name  : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
cpu MHz     : 2200.000
model name  : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
cpu MHz     : 2201.000

Which of these is correct? Which do I believe? Aside from the BIOS changes, is there any way to guarantee maximum performance mode? For what it's worth, this is Ubuntu 12.04, with this kernel:

Linux host 3.2.0-76-generic #111-Ubuntu SMP Tue Jan 13 22:16:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I'm looking into using auto scaling groups for a tier of webservers that would be fronted by an ELB. One of the things I'm having a hard time with is how to give each new instance the proper DNS name. For example, I'd like webservers to have names like frontend-web-XXX.prod.example.com so their names would appear correct in logs and just ease of organization. I have two other tiers I'd ultimately like to make autoscaled and I'd like them to have names like api-web-XXX.prod.example.com as well. I have some experience with cloudformation templates and have spun up individual instances with associated Route53 records but I don't see any indication of how this can be done within an autoscaled group.

Assume I have an application with the following setup:

• 5 webservers running Apache
• 2 databases configured for failover

In Zabbix (or even another monitoring suite, for that matter), is it possible and how would I monitor "application status" and only alert when the following conditions are met:

• If 1 webserver stops responding, send an "info" alert
• If 3 webservers stop responding, send a warning
• If all webservers stop responding, send a critical
• If one DB is offline, send a warning
• If both DBs are offline, send a disaster

I know this looks like 5 questions, but the root question is: how do you create an alert that's abstracted away from a specific host? In every monitoring suite I've used, monitored items/alerts are bound to hosts, which leads to ugly workarounds like "check the database to see if all the servers checked in in the past 10 minutes," and if one hasn't, you get an alert email along the lines of "Host: database01.company.com - webserver05 hasn't checked in in 10 minutes". The email leads you to believe there's a problem with the database (since the check is tied to the database) but the problem is actually with the webserver. Ideally the alert would be tied to an abstract thing like "Corporate Application 1". Can this be done?

In my HAProxy load balancer, I have the following config chunks:

defaults
    mode                    http
    log                     global
    option                  httplog clf
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch

frontend  main_http *:80
    option forwardfor except 127.0.0.1
    option httpclose
    default_backend         backend_http

backend backend_http
    balance     roundrobin
    option httpchk
    server  node1 10.0.0.64:80 check port 80
    server  node2 10.0.0.65:80 check port 80
    server  node3 10.0.0.66:80 check port 80

On the nodes (Tomcat), I'm logging requests in this format (combined with x-forwarded-for in the first field and the real REMOTE_ADDR tacked onto the end):

pattern='%{X-Forwarded-For}i - %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i" %a'

This seems to work well for most requests, but in some cases, I assume in which the clients are behind proxies themselves, I'm seeing requests where the first field contains these values (each line represents one real request, I mangled the real IPs for privacy):

10.83.103.44, 10.83.103.44 
10.83.198.52, 10.83.198.52 
10.53.109.36, 10.53.109.36 
unknown, unknown 
192.168.1.43, 127.0.0.1 
192.168.11.189, 127.0.0.1 
10.1.6.3, 216.x.y.194, 10.37.52.202 
192.168.50.250, 38.x.y.5, 10.37.31.201 

According to the HAproxy docs, the last X-Forwarded-For should be the correct one that it appended, but this doesn't appear to be the case. My application uses the client IP for a geo lookup, so this isn't just a logging issue, it's actually screwing things up.

What I'd like to do is: instead of having HAproxy append the client's IP to the existing X-Forwarded-For header it receives, simply overwrite it. So if it receives X-Forwarded-For: 10.1.2.3 from IP addres 98.76.54.32, what it would send to the clients is just X-Forwarded-For: 98.76.54.32. Is there any way to do this? I'd like to know why such obvious junk is making it to the nodes - unknown, unknown is clearly junk info - but I'll settle for a workaround if one exists.

Thanks in advance.