Home / server / Questions / 753774
Accepted
Bittrance
Asked: 2016-02-04 08:27:24 +0800 CST

Traffic not routed back to GKE cluster from remote network via VPN

  • 772

(Followup on GKE pod connecting via VPN?)

I am trying to connect a GKE cluster to a remote network using a GCE VPN to a Cisco ASA 5510. Ping from GKE pod 10.248.0.26 -> remote node 10.99.193.115 arrives at 10.99.193.115 and the ASA says that the echo reply goes back through the tunnel to GKE. However, tcpdump on 10.248.0.26 shows no replies coming in.

Firewall and routing as reported by Google Cloud Console:

Name    Source tag / IP range   Allowed protocols / ports   Target tags

default-allow-icmp  0.0.0.0/0   icmp    Apply to all targets
default-allow-internal  10.240.0.0/16   tcp:1-65535; udp:1-65535; icmp  Apply to all targets
default-allow-ssh   0.0.0.0/0   tcp:22  Apply to all targets
gke-zecluster-d6cc7a55-all  10.248.0.0/14   tcp; udp; icmp;     Apply to all targets
gke-zecluster-d6cc7a55-ssh  <public_ip>/32  tcp:22  gke-zecluster-d6cc7a55-node
gke-zecluster-d6cc7a55-vms  10.240.0.0/16   tcp:1-65535; udp:1-65535; icmp  gke-zecluster-d6cc7a55-node
k8s-fw-a1a92183fb18e11e5be3442010af0001     0.0.0.0/0   tcp:80,443  gke-zecluster-d6cc7a55-node
k8s-fw-a1aa3fe95b18e11e5be3442010af0001     0.0.0.0/0   tcp:2003    gke-zecluster-d6cc7a55-node

Name    Destination IP ranges   Priority    Instance tags   Next hop

default-route-3eed071cad0670e8  0.0.0.0/0   1000    None    Default internet gateway
default-route-7a9ddc4457c714a0  10.240.0.0/16   1000    None    Virtual network
gke-zecluster-d6cc7a55-7b61213c-b187-11e5-be34-42010af00015     10.248.0.0/24   1000    None    gke-zecluster-d6cc7a55-node-j4jx (Zone ze-zone-1)
gke-zecluster-d6cc7a55-7ec5f7a9-b187-11e5-be34-42010af00015     10.248.1.0/24   1000    None    gke-zecluster-d6cc7a55-node-rluf (Zone ze-zone-1)
vpn-1-tunnel-1-route-1  10.99.0.0/16    1000    None

Is there some logging I can turn on to see what goes on? As far as I can see, the VPN says nothing pertinent about this traffic, only:

15:24:51.058 sending DPD request
15:24:51.058 generating INFORMATIONAL_V1 request 3069408857 [ HASH N(DPD) ]
15:24:51.058 sending packet: from <gce-vpn-ip>[500] to <asa-ip>[500] (92 bytes)
15:24:51.092 received packet: from <asa-ip>[500] to <gce-vpn-ip>[500] (92 bytes)
15:24:51.092 parsed INFORMATIONAL_V1 request 146600869 [ HASH N(DPD_ACK) ]

If I modify the VPN tunnel (GCE VPN, ASA) to have the default net 10.240.0.0/16 at the GCE end traffic passes correctly in both directions.

I assume this is a routing issue, but what? Should not the route 10.248.0.0/24 send the traffic back to the GKE node? Or do I have to somehow declare the GKE network as a network?

vpn

2 Answers

  1. If IP address 10.248.0.26 belongs to a GKE node, then for doing ping between the GKE node and your remote node you will need to add a firewall rule on 10.248.0.26/24 network to allow incoming traffic to GKE node or all targets in that network from your remote source.

    • 1
  2. Best Answer

    In the end, I had to pick a different option. Setting the spec.hostNetwork option pushed the pod into the node address space, 10.240.0.0/16 for which the VPN worked fine.

    As far as I can tell, when you create a GKE cluster, there is some "magic" networking set up for the pod address space, which appears not to have correct routing as regards VPNs. It is possible that Karman is correct, but I can find no way to declare an explicit virtual network for the pods to stick the firewall rules on. Simply sticking them on the default network does not seem to help.

    Creating a new non-legacy network does not help as GKE refuses to create a cluster with pod address in an existing virtual network and GCE SDN refuses to create virtual subnetworks for an address space that GKE has already claimed.

    • 0