Home / server / Questions / 951860
Accepted
Vlad Nikiforov
Asked: 2019-02-02 02:22:05 +0800 CST

OpenVPN on AWS - no access to Internet

  • 772

I seem to have encountered a strange problem with configuring OpenVPN on an AWS EC2 instance (Ubuntu AMI). Connection gets established, two-factor authentication passes, but once connected, the client can't access the Internet.

For debugging, I started an endless ping on the client:

$ ping 87.250.250.242
PING 87.250.250.242 (87.250.250.242): 56 data bytes
64 bytes from 87.250.250.242: icmp_seq=0 ttl=54 time=33.589 ms
64 bytes from 87.250.250.242: icmp_seq=1 ttl=54 time=31.275 ms
64 bytes from 87.250.250.242: icmp_seq=2 ttl=54 time=42.907 ms
64 bytes from 87.250.250.242: icmp_seq=3 ttl=54 time=49.470 ms
64 bytes from 87.250.250.242: icmp_seq=4 ttl=54 time=29.772 ms

On the server I'm testing how far does the packet go. It does reach tun0 interface:

$ sudo tcpdump -i tun0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
08:52:10.261730 IP ip-10-8-1-6.ec2.internal > ya.ru: ICMP echo request, id 52451, seq 2684, length 64
08:52:11.264706 IP ip-10-8-1-6.ec2.internal > ya.ru: ICMP echo request, id 52451, seq 2685, length 64
08:52:12.268201 IP ip-10-8-1-6.ec2.internal > ya.ru: ICMP echo request, id 52451, seq 2686, length 64
08:52:13.272732 IP ip-10-8-1-6.ec2.internal > ya.ru: ICMP echo request, id 52451, seq 2687, length 64
08:52:14.275066 IP ip-10-8-1-6.ec2.internal > ya.ru: ICMP echo request, id 52451, seq 2688, length 64

However, it does not go any further - eth0 doesn't see it:

$ sudo tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

Internet from inside the EC2 instance is perfectly reachable:

$ curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

Looks like a typical NAT postrouting misconfiguration, but I triple-checked that everything is set up as needed.

IP forwarding is on:

$ cat /proc/sys/net/ipv4/ip_forward
1

Postrouting rule in iptables is present:

$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  ip-172-17-0-0.ec2.internal/16  anywhere
MASQUERADE  all  --  ip-10-8-1-0.ec2.internal/24  anywhere

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Explicit check also says it is there:

$ sudo iptables -t nat -C POSTROUTING -s 10.8.1.0/24 -o eth0 -j MASQUERADE && echo ok
ok

Ubuntu has ufw installed, but it is not active:

$ sudo ufw status
Status: inactive

Here is more information on the system:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:    16.04
Codename:   xenial

$ openvpn --version
OpenVPN 2.4.4 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Sep 26 2017
library versions: OpenSSL 1.0.2g  1 Mar 2016, LZO 2.08

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.2.50.1       0.0.0.0         UG    0      0        0 eth0
10.2.50.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.8.1.0        10.8.1.2        255.255.255.0   UG    0      0        0 tun0
10.8.1.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0

Server config:

$ cat /etc/openvpn/server.conf | egrep -v '^#' | egrep -v '^$'
port 1194
proto udp
dev tun
ca ca.crt
cert VPN.crt
key VPN.key  # This file should be kept secret
dh dh2048.pem
server 10.8.1.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 8.8.8.8"
keepalive 10 120
tls-auth ta.key 0 # This file is secret
compress lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 4
plugin openvpn-plugin-auth-pam.so openvpn
reneg-sec 43200
reneg-bytes 0
cipher AES-256-CBC
ncp-ciphers AES-256-GCM:AES-256-CBC:AES-128-GCM:AES-128-CBC:BF-CBC
crl-verify crl.pem

Client config:

client
dev tun
proto udp
remote <server IP> 1194
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
key-direction 1
compress lzo
verb 3
auth-user-pass
auth-nocache
reneg-sec 43200
cipher AES-256-CBC
<ca>
-----BEGIN CERTIFICATE-----
# redacted
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----
# redacted
-----END PRIVATE KEY-----
</key>
<tls-auth>
-----BEGIN OpenVPN Static key V1-----
# redacted
-----END OpenVPN Static key V1-----
</tls-auth>

The system information: EC2 instance built from ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180814 image.

Security groups allow all outbound traffic and the following inbound traffic:

  • UDP/1194 from anywhere
  • TCP/22 from my location (for debugging)

I read about disabling Source/Destination Check. Not sure it applies to my case, as I don't need to provide access to AWS VPC network, but tried that as well, and it didn't help.

What puzzles me the most, is that we already have a working VPN server set up in a similar way in another region, but for some reason I can't get a new instance to work. So I'm not even sure this is AWS-specific.

Can anyone suggest where to look for the problem?

amazon-ec2

1 Answers

  1. Best Answer

    Not sure what was the cause, but I just rebooted the instance and the problem magically disappeared.

    Before doing so, I tried to debug iptables (thanks to answers to this question) and it seemed like packets never reached iptables altogether (note the zeroes in the lines with rules for ip-10-8-1-0.ec2.internal/24):

    $ sudo iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 1249 packets, 93674 bytes)
 pkts bytes target     prot opt in     out     source               destination
   59  3938 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 2 packets, 140 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 923 packets, 58794 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all  --  any    any     anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 923 packets, 58794 bytes)
 pkts bytes target     prot opt in     out     source               destination
   29  1910 MASQUERADE  all  --  any    !docker0  ip-172-17-0-0.ec2.internal/16  anywhere
    0     0 LOG        all  --  any    eth0    ip-10-8-1-0.ec2.internal/24  anywhere             LOG level warning prefix "nat"
    0     0 MASQUERADE  all  --  any    eth0    ip-10-8-1-0.ec2.internal/24  anywhere

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 any     anywhere             anywhere

    After rebooting the packets go happily through:

    $ sudo iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 35 packets, 2438 bytes)
 pkts bytes target     prot opt in     out     source               destination
    1    70 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 1 packets, 70 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 281 packets, 17695 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all  --  any    any     anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 283 packets, 17975 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  any    !docker0  ip-172-17-0-0.ec2.internal/16  anywhere
   32  2088 MASQUERADE  all  --  any    eth0    ip-10-8-1-0.ec2.internal/24  anywhere

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 any     anywhere             anywhere

    I'm marking my question is answered, but I'm happy to transfer the correct answer flag to anyone who gives a hint on why this is happening.

    • 0