In a Windows lab environment, I'm trying to test performance of SMB Direct using Mellanox RDMA adapters with global pause. I wanted to test first with Global Pause for simplicity before switching to PFC. I think I've set up everything correctly, but SMB Direct is not being used. I can tell because RDMA performance counters remain at zero and actual bandwidth is displayed in Task Manager.
I verified RDMA connectivity using nd_write_bw in both directions.
Server: Win2019 Datacenter with Mellanox ConnectX-5 100GbE
Client: Win10 Pro for Workstations with Mellanox ConnectX-4 Lx 25GbE
Both machines have WinOF2 2.40.50000 driver with Flow Control Rx/Tx enabled
The switch has global pause enabled and PFC disabled
I have enabled SMB Direct feature on the client.
On the client, Get-SmbMultichannelConnection is empty. What could be causing this?
Here is the server configuration:
PS C:\Users\Administrator> Get-NetOffloadGlobalSetting | Select NetworkDirect
NetworkDirect
-------------
Enabled
PS C:\Users\Administrator> Get-NetAdapterRDMA fastnet
Name InterfaceDescription Enabled PFC ETS
---- -------------------- ------- --- ---
fastnet Mellanox ConnectX-5 Adapter True False False
PS C:\Users\Administrator> Get-NetAdapterHardwareInfo fastnet
Name Segment Bus Device Function Slot NumaNode PcieLinkSpeed PcieLinkWidth Version
---- ------- --- ------ -------- ---- -------- ------------- ------------- -------
fastnet 0 216 0 0 2 1 8.0 GT/s 16 1.1
PS C:\Users\Administrator> Get-SmbServerConfiguration | Select EnableMultichannel
EnableMultichannel
------------------
True
PS C:\Users\Administrator> Get-SmbServerNetworkInterface
Scope Name Interface Index RSS Capable RDMA Capable Speed IpAddress
---------- --------------- ----------- ------------ ----- ---------
* 6 True False 1 Gbps 192.168.3.55
* 4 True True 100 Gbps 192.168.10.55
PS C:\Users\Administrator> netstat.exe -xan | ? {$_ -match "445"}
Kernel 4 Listener 192.168.10.55:445 NA 0
And here is the client configuration:
PS C:\Windows\system32> Get-SmbClientConfiguration | Select EnableMultichannel
EnableMultichannel
------------------
True
PS C:\Windows\system32> Get-SmbClientNetworkInterface
Interface Index RSS Capable RDMA Capable Speed IpAddresses Friendly Name
--------------- ----------- ------------ ----- ----------- -------------
6 True True 40 Gbps {fe80::d1ac:ef30:6f70:a1bc, 169.254.161.188} chelsio40
4 True True 25 Gbps {192.168.10.53} mlnx25
8 True False 1 Gbps {fe80::f912:79b2:e851:3cb4, 192.168.3.53} housenet
PS C:\Windows\system32> Get-SmbConnection
ServerName ShareName UserName Credential Dialect NumOpens
---------- --------- -------- ---------- ------- --------
192.168.10.55 NVMe DESKTOP-K69SD7G\admin DESKTOP-K69SD7G\admin 3.1.1 5
PS C:\Windows\system32> Get-SmbMultichannelConnection
PS C:\Windows\system32> netstat.exe -xan | ? {$_ -match "445"}
PS C:\Windows\system32> netstat.exe -an | ? {$_ -match "445"}
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
TCP 192.168.10.53:58881 192.168.10.55:445 ESTABLISHED
TCP [::]:445 [::]:0 LISTENING
Switch: Mellanox SN2100
PFC is enabled on the switch, so I disabled it for these specific ports and enabled Global Pause
int eth 1/10-1/12 flowcontrol send on force
int eth 1/10-1/12 flowcontrol receive on force
int eth 1/10-1/12 dcb priority mode off force
