I have an Ubuntu 22.04 server, a MikroTik CRS328-24P-4S+RM switch, and Ethernet connected cameras. I wanted to increase the MTU on all systems so that jumbo frames are transferred, to reduce the CPU load on the server when receiving HD streams from many cameras, however I noticed that even with default MTU settings (1500 bytes) the cameras send jumbo frames already.
In more detail:
- on the server
ip l
shows for the network interface:2: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
; i.e. MTU 1500. - on the switch management web interface I checked the MTU for the interface towards the server and for the interfaces towards the cameras. They are set to MTU = 1500 and L2 MTU = 1592.
- the cameras I don't know how to check the MTU (they are a blackbox for me).
I checked size of transferred packets:
on the server I looked at the packets with tcpdump.
tcpdump -i eno2 -s 100 -e | less
shows packets like this:16:25:06.315612 <MAC redacted> (oui Unknown) > <MAC redacted> (oui Unknown), ethertype IPv4 (0x0800), length 9014: camera1.3957 > server.46179: UDP, length 8972
. The ethernet packets have a size of 9014 bytes.I disabled "offloading" on the server, with
ethtool --offload eno2 rx off tx off
as described at https://wiki.wireshark.org/CaptureSetup/Offloading.on the switch I can see that the interface to a camera has an RX rate of ~500 Mbps and an RX packet rate of ~7000 p/s. This comes out to
((500/8)*1024*1024)/7000
= ~9362 bytes/packet.
Questions:
did I make any mistake in my analysis, above?
is it possible for network devices to send UDP packets larger than the configured MTU of intermediate devices?
Appendix:
Results of ethtool --show-offload eno2
:
Features for eno2:
rx-checksumming: off
tx-checksumming: off
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: off
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off [requested on]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: off [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
There still was one interface on the switch that had a large MTU configured. After changing this MTU down to normal size (MTU=1500 and L2MTU to 1592), none of the cameras sends jumbo frames any more, but only normal-sized frames.
So my conclusion is: raising the L2 MTU on any interface on this switch (even an unused one) makes it possible to send jumbo frames between any interfaces.
Also, the MTU on the receiving server did not influence this behaviour. It is still configured at 1500, but the server successfully receives jumbo frames. This matches the suggestion by Tom Yan:
Since the cameras communicate over UDP, this idea sounds realistic.