I'm trying to understand what this output from traceroute means. I changed the IP addresses for privacy but retained the public/private IP range distinction.
traceroute.db -e -n 10.1.1.9
traceroute to (10.1.1.9), 30 hops max, 60 byte packets
1 10.0.0.1 0.596 ms 0.588 ms 0.577 ms
2 10.0.0.2 1.032 ms 1.029 ms 1.084 ms
3 10.0.0.3 3.360 ms 3.355 ms 3.338 ms
4 23.0.0.4 3.974 ms 4.592 ms 4.584 ms
5 23.0.0.5 13.442 ms 13.445 ms 13.434 ms
6 45.0.0.6 13.195 ms 12.924 ms 12.913 ms
7 67.0.0.7 52.088 ms 51.683 ms 52.040 ms
8 10.1.1.8 46.878 ms 44.575 ms 44.815 ms
9 10.1.1.9 45.932 ms 45.603 ms 45.593 ms
The first 10.0.* range is inside my organisation. The last 10.1.* range is another site of my organisation. The intermediate addresses belong to various ISPs. I expect that there is some kind of VPN between the two sites, but I don't know much about our network topology.
What I don't understand is how the route can go from a private address through public addresses back into private addresses. Searching led me to Public IPs on MPLS Traceroute, which gives a possible explanation: MPLS. Is MPLS the only possible or most likely explanation? Otherwise what does this tell me about our network infrastructure?
Bonus question for my edification: in this scenario, who is generating the ICMP TTL exceeded packets and if relevant mangling their source and destination addresses?
There are servers at both ends which are doing Network Address Translation (NAT). As the address passes though these servers the header address on the data packet gets rewritten to that servers Internet address. The server keeps track of which connections belong to which internal host.
Traceroute displays data from inside an ICMP packet indicating whether or not the host was reached in a given number of hops. The NAT routers do not alter this data. As a result you see the address that each host received the packet on.
Normally the servername on the far end in this case has been routed using DNAT (Destination NAT) to a host on the private network.
It is likely that the address is being passed over a VPN tunnel between two sites. The VPN be encapsulating the source and final addresses inside the packets being sent between hops 3 and 7. The effect is the same, although the mechanism is different. The routers at hops 3 and 7 would know the addresses ranges supported by the remote routers, and route the packets accordingly. Leaving hop 7 the IP destination would be 67.0.0.7 with a public address belonging to hop 3. This is invisible due to the way route tracing works. Depending on the VPN protocol some hops after hop 7 may not be traceable.
In some cases you may see an ISP routing over private addresses to a public address. This will appear as one ore more private addresses between two public addresses. If the intermediate routers with public addresses belong to the same organization, it is possible they have routing rules allowing end to end communication without translation.