I have connected two rooms via a 10Gbit fiber connection. The connection has one MikroTik switch with four SFP+ on one end (room A) and an Ubiquiti Unifi 24 ports with two SFP+ (room B).
On Room A, I have a Mac mini 2018 with a 10Gbit ethernet connected to the MikroTik switch using an SFP+-RJ45 transceiver.
On Room B, I have a Supermicro machine with an SFP+ running ESXi with several VM's. One of those is a FreeNAS (TrueNAS) and another one is a Ubuntu 20.04 server.
All those machines are connected to a VLAN (id 45) that is used mostly for storage.
I wanted to test the speed of the network using iperf3. I am a bit disappointed, as I am getting a bit less than gigabit speeds from the Mac to the FreeNAS. I tested it against the Ubuntu server, and the speed was around 9Gbit, which seems more like it should be.
Upstream was a bit different: getting around 1.45Gbit from the FreeNAS to the Mac mini, and around 3.5Gbit from the Ubuntu to the Mac mini.
I also tested between then FreeNAS and the Ubuntu server, and it was around 9Gbit from the first to the last one, and around 2.3Gbit the other way around. Once I had jumbo frames enabled, and it was way faster both ways, but it didn't work well with the Mac, so I went back to MTU 1500.
Then I decided to boot the Mac mini with Ubuntu to see if this was an OS-related issue, and indeed, From Ubuntu client (Mac) to my Ubuntu server, I got up and down speeds of 9Gbit. But then I got around 2.Gbit from the FreeNAS, and 9Gbit speed the other way around:
| FreeNAS | Ubuntu Server | |
|---|---|---|
| From Mac (macOS) | 0.9 | 9 |
| To Mac (macOS) | 2.3 | 3.5 |
| From Mac (ubuntu) | 9 | 9 |
| To Mac (ubuntu) | 2.3 | 9 |
(speeds are in Gbits/s)
I realise now that the hardware is working as it should (except for the Mac that doesn't seem to play well with MTU 9000 with this MikroTik transceiver) - using it gets be 0Gbits from the other machines (though I haven't tested with Ubuntu).
My questions are: Is there a way to perform a test that is a bit more consistent between OS'es? And does it mean that my Mac will get slower transfers to/from the FreeNAS, or is it just iperf3 that's behaving inconsistently?
Ok, it turns out the Unifi switch didn't have jumbo frames enabled. Enabling it didn't fix it.
But setting the MTU of the vSwitch to 9000 did the trick, as well as setting the nics of the servers involved to 9000. Now I finally got 10Gbit speeds.