Home / server / Questions / 1052955
In Process
pgb
Asked: 2021-02-10 05:30:28 +0800 CST

pfSense OpenVPN on AWS VPC

  • 772

We are in the process of migrating our office servers to AWS to finally decommission. Since we are familiar with pfSense, we are using a mix of AWS services but want to use pfSense as a the OpenVPC server.

Our VPC has two subnets:

  • Public: 192.168.16.0/24
  • Private: 192.168.17.0/24

I created an EC2 instance running in pfSense with two interfaces: WAN -> 192.168.16.10, LAN -> 192.168.17.10.

Worth noting: we are not using pfSense as the NAT gateway for the private network, so the LAN interface in pfSense has a gateway configured at 192.168.17.1 (why so? because we only want to use pfSense as the VPN server, not as the NAT gateway at this point).

Then, we configured OpenVPN to listen on the WAN interface and with a tunnel network of 192.168.140.0/23 and IPv4 local networks of 192.168.17.0/24.

I ensured I have an outbound NAT rule on the LAN interface with 192.168.140.0/23 as source, and NAT set as either hybrid or manual (tried both).

On the EC2 instance, I stopped source destination check.

What works:

  • I can connect to OpenVPN
  • I can use pfSense as a DNS (not relevant now, but I want to use it later on)
  • From a console in pfSense, I can ssh another server in my private network (192.168.17.30).
  • From said server (192.168.17.30), I can ssh to pfSense.

What doesn't work:

  • I can't ssh to a server on the private network (192.168.17.30).
    • If I enable logs on my firewall rules, I can confirm that pfSense is letting the packets in.

What I tried already:

  • Wide open firewall rules.
  • Wide open AWS security groups.
  • Adding 192.168.140.0/23 to the route tables in my VPC (although since it's NATing, I understand I should not need it).

For diagnostics, I enabled packet capture. Here's the output when SSHing from the pfSense box:

18:59:06.034748 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.034969 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.035013 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.035538 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 38
18:59:06.035705 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.060000 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 41
18:59:06.060098 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.060752 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 1432
18:59:06.060905 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.062725 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 1056
18:59:06.063141 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.065800 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 48
18:59:06.065956 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.070607 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 508
18:59:06.070772 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.074492 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 16
18:59:06.074616 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.074640 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 44
18:59:06.074802 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.074818 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 44
18:59:06.074868 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.074920 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 68
18:59:06.081411 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 44
18:59:06.081477 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.081628 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.082931 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.082959 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0

and the same packet capture when SSHing from an OpenVPN client:

18:57:12.126970 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:13.131702 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:14.135982 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:15.145506 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:16.159995 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:17.152084 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:19.158465 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0

as you can see, NAT appears to be working, since the outgoing packet goes from 192.168.17.10, but no response is received from the server.

I posted a similar question on Netgate forums, but I included most of the information here.

openvpn amazon-web-services pfsense amazon-vpc

1 Answers

  1. I had "hardware checksum offload" enabled, and it needs to be disabled when you use pfSense on AWS (it really should be a default setting on the official image).

    This solved the issues we were experiencing.

    • 1