Questions[rdma](server)

I am having a hard time finding any information at all about wether it's possible to use the Linux Kernel RXE driver (software RDMA over converged ethernet) with bonded ethernet interfaces (using dynamic link aggregation as in 802.3ad).

There seem to be some Mellanox HCAs which do support RDMA + LAG but in hardware and using the Mellanox specific drivers (https://docs.nvidia.com/networking/pages/viewpage.action?pageId=12013422 section "Link Aggregation).

So, does someone, perhaps with more insight into the inner workings of the linux soft-roce driver, know wether this is possible or even has experience with this themselves ?

Thanks :)

I've been reading about RoCE and it talks about transferring data zero copy between the memory in one server directly to the memory of an application in another server. Most articles make a point to say this is done without involvement from the CPU. I don't understand how this is accomplished though. How does the data get from the adapter to the RAM without any processing from the CPU? How does it know where the application's data is in RAM?

I have 10 servers with two CPUs each and one Mellanox 100G Infiniband NIC per CPU. Each NIC is connected to a single Mellanox 36 port 100G IB switch.

My RDMA application runs as one process per NUMA node and binds to the local NIC to avoid cross CPU traffic. Each node/process needs to connect to every other node using RC mode.

The problem I ran into is, it appears the default OpenSM routing forces me to use a certain NIC to reach a certain target node. So I would have to use both NICs from both NUMA nodes to reach all other nodes. That means I would need two PDs also, having to register all the memory twice.

Is there any way to allow a single NIC to be able to connect to any other NIC/port on the network?

Essentially I would like to make OpenSM think that each NIC is on it's own server, ie. pretending that no QPI traffic is possible.

See: https://docs.mellanox.com/display/MLNXOFEDv461000/OpenSM

Once MinHop matrices exist, each switch is visited and for each target LID a decision is made as to what port should be used to get to that LID.

Relevant code: https://github.com/linux-rdma/opensm/blob/844ab3b7edaad983449b5d3a4a773088b8daa299/opensm/osm_ucast_mgr.c#L201

I know loopback will go through the kernel network stack until reach IP layer, including syscall overhead and some memory copy overhead. DPDK and RDMA use different technology to avoid these.
So let's say I have two machine connected by dpdk/rdma, then I do net latency test, will that be faster than loopback on just one machine? I do a quick test of ping localhost on CPU E5-2630 v4 @ 2.20GHz, which on average is 0.010ms.

I come up with this question when I was testing my ceph cluster using vstart.sh, I want to minize network latency in order to carefully analyze how osd-related code affect latency.

In a Windows lab environment, I'm trying to test performance of SMB Direct using Mellanox RDMA adapters with global pause. I wanted to test first with Global Pause for simplicity before switching to PFC. I think I've set up everything correctly, but SMB Direct is not being used. I can tell because RDMA performance counters remain at zero and actual bandwidth is displayed in Task Manager.

I verified RDMA connectivity using nd_write_bw in both directions.

Server: Win2019 Datacenter with Mellanox ConnectX-5 100GbE
Client: Win10 Pro for Workstations with Mellanox ConnectX-4 Lx 25GbE
Both machines have WinOF2 2.40.50000 driver with Flow Control Rx/Tx enabled
The switch has global pause enabled and PFC disabled
I have enabled SMB Direct feature on the client.

On the client, Get-SmbMultichannelConnection is empty. What could be causing this?

Here is the server configuration:

PS C:\Users\Administrator> Get-NetOffloadGlobalSetting | Select NetworkDirect

NetworkDirect
-------------
      Enabled

PS C:\Users\Administrator> Get-NetAdapterRDMA fastnet

Name                      InterfaceDescription                     Enabled     PFC        ETS
----                      --------------------                     -------     ---        ---
fastnet                   Mellanox ConnectX-5 Adapter              True        False      False


PS C:\Users\Administrator> Get-NetAdapterHardwareInfo fastnet

Name                           Segment Bus Device Function Slot NumaNode PcieLinkSpeed PcieLinkWidth Version
----                           ------- --- ------ -------- ---- -------- ------------- ------------- -------
fastnet                              0 216      0        0    2        1      8.0 GT/s            16 1.1

PS C:\Users\Administrator> Get-SmbServerConfiguration | Select EnableMultichannel

EnableMultichannel
------------------
              True

PS C:\Users\Administrator> Get-SmbServerNetworkInterface

Scope Name Interface Index RSS Capable RDMA Capable Speed    IpAddress
---------- --------------- ----------- ------------ -----    ---------
*          6               True        False        1 Gbps   192.168.3.55
*          4               True        True         100 Gbps 192.168.10.55

PS C:\Users\Administrator> netstat.exe -xan | ? {$_ -match "445"}
  Kernel       4 Listener       192.168.10.55:445      NA                     0

And here is the client configuration:

PS C:\Windows\system32> Get-SmbClientConfiguration | Select EnableMultichannel

EnableMultichannel
------------------
              True

PS C:\Windows\system32> Get-SmbClientNetworkInterface

Interface Index RSS Capable RDMA Capable Speed   IpAddresses                                  Friendly Name
--------------- ----------- ------------ -----   -----------                                  -------------
6               True        True         40 Gbps {fe80::d1ac:ef30:6f70:a1bc, 169.254.161.188} chelsio40
4               True        True         25 Gbps {192.168.10.53}                              mlnx25
8               True        False        1 Gbps  {fe80::f912:79b2:e851:3cb4, 192.168.3.53}    housenet

PS C:\Windows\system32> Get-SmbConnection

ServerName    ShareName UserName              Credential            Dialect NumOpens
----------    --------- --------              ----------            ------- --------
192.168.10.55 NVMe      DESKTOP-K69SD7G\admin DESKTOP-K69SD7G\admin 3.1.1   5


PS C:\Windows\system32> Get-SmbMultichannelConnection
PS C:\Windows\system32> netstat.exe -xan | ? {$_ -match "445"}
PS C:\Windows\system32> netstat.exe -an | ? {$_ -match "445"}
  TCP    0.0.0.0:445            0.0.0.0:0              LISTENING
  TCP    192.168.10.53:58881    192.168.10.55:445      ESTABLISHED
  TCP    [::]:445               [::]:0                 LISTENING

Switch: Mellanox SN2100
PFC is enabled on the switch, so I disabled it for these specific ports and enabled Global Pause

int eth 1/10-1/12 flowcontrol send on force
int eth 1/10-1/12 flowcontrol receive on force
int eth 1/10-1/12 dcb priority mode off force