In December/2015 Amazon launched their VPC NAT Gateway service which is basically a managed alternative to creating your own (and it looks very good in comparison to NAT instances).
I was just about to deploy my own NAT instances and I decided to use the NAT Gateway instead to save me the hassle of configuring and maintaining my own NAT service.
The service works as advertised and I've been a happy chappy until I noticed a strange behaviour.
I use EC2 instances behind the NAT gateway to test a bunch of websites, running up to 4 parallel Firefox+selenium tests per EC2 instance. I noticed that tests that were previously running fine elsewhere had been failing on this environment.
So I spun-up 2x t2.large instances, one with a public elastic IP and direct access to the Internet; the other with a private IP, behind the NAT gateway.
Here are the results of running apache benchmark against some sites:
EIP
ab -kn 1000 -c 20 https://www.pinterest.com/
(...)
Time taken for tests: 17.331 seconds
ab -kn 1000 -c 20 http://speedtest.ftp.otenet.gr/files/test100k.db
(...)
Time taken for tests: 9.610 seconds
ab -kn 1000 -c 20 http://www.bbc.com/
(...)
Time taken for tests: 5.890 seconds
NAT
ab -kn 1000 -c 20 https://www.pinterest.com/
(...)
Time taken for tests: 25.980 seconds
ab -kn 1000 -c 20 http://speedtest.ftp.otenet.gr/files/test100k.db
(...)
Time taken for tests: 14.920 seconds
ab -kn 1000 -c 20 http://www.bbc.com/
(...)
Time taken for tests: 6.552 seconds
The server behind the NAT gateway is outperformed by the service with direct connection by a significant percentage in many cases.
AWS says that the service "Supports bursts of up to 10Gbps" which is more than what the individual instances I used can do.
I noticed that if I turn-off keep-alive (i.e. the -k) the performance becomes similar:
ab -n 1000 -c 20 https://www.pinterest.com/
EIP: 18.316 seconds
NAT: 17.763 seconds
What is this sorcery? What's going on? Are my benchmarks relevant?
What has been your experience?
