I have a VPN with a gateway, and an application server. The app server only knows about the gateway of the VPN. The gateway of the VPN knows about the gateway of the physical network the vms are hosted on.
Problem: I can't reach api.twitter.com from the application server. Observation : the cache is filled with entries showing the physical gateway IP address. Here is an excerpt :
ip -s route show cache 199.16.156.40
199.16.156.40 via 37.59.245.62 dev eth0 src 10.1.4.20
cache <redirected> used 170 age 22sec ipid 0x9e49
199.16.156.40 from 10.1.4.20 via 37.59.245.62 dev eth0
cache <redirected> age 25sec ipid 0x9e49
IPs beginning with 199 are twitter IPs. 37.59.245.62
is the IP of the physical gateway. 10.1.4.20
is the VPN IP of the VPN's gateway.
The IP of the physical gateway only appears for entries dealing with twitter's servers.
Why does this machine cache routes with IPs it does not have direct access to ?
Previous question showing that flushing the cache solves the problem temporarily
Why does it cache? The same reason caches are used in any other situation, to prevent the overhead of a route lookup. The reason you're seeing "external" gateways are more than likely ICMP messages coming back informing you that the that IP is unavailable.
I asked this somewhere else, and it turns out the solution was to turn off ICMP redirects.