Okay, I just finished resolving a jumbo frames issue between a few Xserves, a Netgear GSM7224, and a Drobo B800i. It turned out that the Xserves (Mac OS X 10.6.8 Server) and the Drobo B800i accept the MTU in bytes as normally expected (1500-9000), but the Netgear seems to have wanted it including the various ethernet headers/footers(trailers) and I ultimately ended up with the Xserves & Drobo configured with an MTU of 9000 and the Netgear ports set to an MTU of 9216.
I've used the following commands for testing & verifying the MTU between the two Xserves over the Netgear (Note: these are the Mac OS X commands, the Windows & Linux ones differ):
ping -D -s <mtu> <ip_address>
traceroute -F <ip_address> <mtu>
The former's usage is noted in the man page as, "Specify the number of data bytes to be sent. The default is 56, which translates into 64 ICMP data bytes when combined with the 8 bytes of ICMP header data." In testing, I found that to ping -D 1472 <ip_address> was the equivalent of MTU 1500 due to the 8 bytes of ICMP header data plus 20 bytes IP headers (see this and this). That all makes sense.
Now, why is the equivalent command for a 9000 MTU ping -D -s 8164 <ip_address>? I've verified this is the limit before I star getting "sendto: Message too long" errors, but also that 9000 MTU is working correctly as traceroute -F <ip_address> 9000 works and traceroute -F <ip_address> 9001 does not. So, why 8164? I would've expected 8972 (MTU - 28 bytes, just like for 1500 MTU).
Also, why 9216 MTU for the Netgear? I counted 42 bytes for the MAC & ethernet headers (inc. CRC), plus 20 for IP headers (which should eat into the MTU).
I'm really rusty on this math and know I'm just missing something.
Welcome to the strange and mysterious world of jumbo frames! It's normal that jumbo frame ethernet gear has MTU > 1518 and < 65K byte and you have to find a setting that is the lowest common denominator across your L2 domain to enable proper jumbo traffic.
My guess is that your ping/ICMP implementation only works for 8192 bytes payload, so 8164 + 28 (20 for the IP header and 8 for the ICMP header) gives you 8192 bytes.
MTU 9216 is also a standard 9K MTU on a lot of Cisco's gear, so my assumption is that Netgear wanted to be ''compatible'' to that.
Also note that MTU size specification should be taken seriously, a lot of vendors don't include 802.1Q (vLANs) or even L2 frame headers. Check with your switch vendor's documentation what they really specify when they're talking about MTU size.