I have a server instance on an OpenStack
that at a pretty high load starts loosing UDP
packets. I captured all outgoing packets using tcpdump
and some of them are missing, even though application logs imply that they should have been sent. Usual packet size is around 60-120 bytes.
Running netstat -s
gives:
[root@myServer] ~> netstat -s | grep Udp: -A 5
Udp:
3855490640 packets received
133199 packets to unknown port received.
89 packet receive errors
4116940753 packets sent
SndbufErrors: 1396176
When the server is under load, SndbufErrors
keeps increasing. I tried to figure out what might be causing it but with no luck even though it feels like this should be covered somewhere.
Q: What might be the reasons for this and how could I resolve this?
Investigation I've done:
Running
ifconfig -a
doesn't show any errors:UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:6361554048 errors:0 dropped:0 overruns:0 frame:0 TX packets:6902945025 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
I tried increasing
txqueuelen
to10'000
(by runningifconfig eth1 txqueuelen 10000
), but it didn't make a difference.Running several
sysctl
commands I get:net.core.rmem_max = 124928 net.core.wmem_max = 4194304 net.core.rmem_default = 124928 net.core.wmem_default = 124928
I tried increasing
net.core.rmem_max
andnet.core.wmem_max
to much bigger number16'777'216
, but still keep getting the same errors.Running
sar -n UDP 1 1
gives (approximated values, but no errors):05:47:31 PM idgm/s odgm/s noport/s idgmerr/s 05:48:46 PM 23000.00 24000.00 0.00 0.00
Running
ethtool
on theOpenstack
VM instance mostly results inOperation not supported
. Runningethtool
on theOpenstack
host server choosing its interface that is used to communicate with outside world, I get:[root@myServer] ~> ethtool em1 Speed: 1000Mb/s ... [root@myServer] ~> ethtool -g em1 Ring parameters for em1: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 256 RX Mini: 0 RX Jumbo: 0 TX: 256
I am not convinced that it can be related though as errors I see are inside the VM and not the
Openstack
host server. Update: I increased theRX
andTX
values, but to no success.
0 Answers