Questions[mtr](server)

I have server A and server B, with A attempting to open TCP connections to server B. curl shows that approximately 5-10% of connection attempts time out.

mtr shows 0% loss to B, and at intermediate hops (except those hops that do not respond at all).
mtr --tcp shows ~5% loss to B, and 0% loss at intermediate hops (with the same caveat).

I've verified these results at multiple times.

Server A is an AWS EC2 instance which we've setup on a dedicated VPC to eliminate any issues on our end. We've been unable to reproduce this issue from Azure, GCP, a business ISP and a consumer ISP. We have been able to reproduce this via two different transit providers when originating from AWS, interestingly. However, I can't say for sure that they use the same hops leading into B, as we don't have mtr responses for a few hops prior to B.

Server B is a CDN edge server. (We're their customers as well but this doesn't seem relevant as we're now troubleshooting at the IP address level and it only affects a few of their edge servers.)

The issue is reproducible with both TCP and UDP, but we have been unable to reproduce over ICMP, regardless of packet size. In addition, the route taken appears to meaningfully differ between TCP/UDP and ICMP--ICMP packets seem to follow one route and one route only; TCP/UDP packets have multiple routes they follow but the ICMP route is not one of them.

Is there any further troubleshooting/investigation I can meaningfully do to narrow down why/where this is happening? AWS and the server B CDN provider are pointing the finger at each other unhelpfully, so any further evidence would be extremely helpful.

I was looking for a tool to display the continuous ping loss to multiple hosts, and I looked at mtr as it already has the display I was thinking about. I saw that it has this option which sounded promising:

  -F FILENAME, --filename FILENAME
          Reads the list of hostnames from the specified file.

Unfortunately, I cannot understand what mtr does with this list of hostnames. It seems to only pick up the first one from the list and report on it.

How is the "list of hostnames" used by mtr?

I have been running MTR to/from one of my servers and noticed something that looks odd to me. Because I am not really into this I will give you three outputs:

This is from the server to my home location:

My traceroute  [v0.75]
prag341.server4you.de (0.0.0.0)                                                                                                                         Sat Apr 16 12:31:36 2011
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                                        Packets               Pings
 Host                                                                                                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. v9-609a.s4y14.fra.routeserver.net                                                                                                  0.0%   143    6.6   2.9   0.7  15.6   2.4
 2. 217.118.16.161                                                                                                                     0.0%   143    0.7   5.7   0.4  67.3  13.2
 3. 217.118.16.25                                                                                                                      0.0%   143    3.3   5.3   3.3  63.5   8.6
 4. 194.25.211.53                                                                                                                      0.0%   143    3.4   5.5   3.2  61.1   9.1
 5. vie-sb2-i.VIE.AT.NET.DTAG.DE                                                                                                       0.7%   143   17.8  21.7  17.6 131.1  14.8
    vie-sb2-i.VIE.AT.NET.DTAG.DE
 6. at-vie05b-ri1-pos-5-0.aorta.net                                                                                                    0.7%   143   18.7  18.4  17.6  23.8   0.9
 7. at-vie05b-ri2-ge-2-1-9.aorta.net                                                                                                   0.0%   143   17.9  18.6  17.5  41.7   2.6
 8. at-vie01a-rd1-xe-1-0-0.aorta.net                                                                                                   0.0%   143   18.2  21.1  17.3 104.1  12.0
 9. at-vie-sk11-pe01-vl-20.upc.at                                                                                                      0.0%   143   18.2  20.6  17.7  55.7   7.0
10. at-vie-sk11-pe02-vl-1.upc.at                                                                                                       0.0%   143   17.8  19.6  17.3  55.2   6.6
11. ???

This is from my home location to the server:

My traceroute  

[v0.80]
joe-desktop (0.0.0.0)                                                                                                                                   Sat Apr 16 14:27:54 2011
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                                        Packets               Pings
 Host                                                                                                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.1.1                                                                                                                        0.0%    87    0.2   0.2   0.2   0.2   0.0
 2. ???
 3. 84.116.4.33                                                                                                                        0.0%    86    9.7   9.0   6.3  27.3   3.5
 4. at-vie-sk11-cia01-vl-2070.upc.at                                                                                                   0.0%    86   22.7  22.8  20.0  52.2   4.7
 5. at-vie-sk11-pe01-vl-2069.upc.at                                                                                                    0.0%    86   47.6  23.9  20.2  47.6   5.8
 6. at-vie01a-rd1-vl-2042.aorta.net                                                                                                    0.0%    86   21.7  25.0  20.1  61.7   8.5
 7. de-fra03a-rd1-xe-9-2-0.aorta.net                                                                                                   0.0%    86   21.3  22.8  19.6  44.0   5.0
 8. 84.116.132.154                                                                                                                     0.0%    86   20.2  22.8  19.3  41.0   4.1
 9. tge-5-1-0-353a.cr2.fra.routeserver.net                                                                                             0.0%    86   38.6  27.4  20.9 120.2  16.0
10. 217.118.16.130                                                                                                                     0.0%    86   23.7  26.9  20.8  73.0   9.8
11. 217.118.16.26                                                                                                                      0.0%    86   25.5  28.8  22.9  85.1  11.8
12. 217.118.16.165                                                                                                                    81.2%    86   68.2  37.5  25.0  68.2  10.3
13. prag341.server4you.de                                                                                                              0.0%    86   35.7  27.1  24.0  49.3   4.3

And this is from another server (amazon ec2) to the server:

My traceroute  [v0.75]
flimmit.com (0.0.0.0)                                                                                                                                   Sat Apr 16 12:32:50 2011
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                                        Packets               Pings
 Host                                                                                                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ip-10-48-192-3.eu-west-1.compute.internal                                                                                          0.0%   178    0.4   0.9   0.3  16.4   1.7
    ip-10-48-192-2.eu-west-1.compute.internal
 2. ec2-79-125-0-244.eu-west-1.compute.amazonaws.com                                                                                   0.0%   178    0.5   0.9   0.3  30.8   2.6
    ec2-79-125-0-242.eu-west-1.compute.amazonaws.com
 3. ???
 4. ???
 5. ???
 6. xe-4-1-0.dub10.ip4.tinet.net                                                                                                      36.5%   178    1.9   3.9   1.6  56.8   8.5
 7. xe-4-1-0.dub10.ip4.tinet.net                                                                                                       0.0%   178   12.1   9.7   1.6  92.5  10.5
    xe-0-1-0.lon14.ip4.tinet.net
    xe-2-1-0.lon14.ip4.tinet.net
 8. xe-0-1-0.lon14.ip4.tinet.net                                                                                                       0.0%   177   17.4  17.7  11.1 184.3  24.6
    xe-2-1-0.lon14.ip4.tinet.net
    213.200.77.234
 9. 213.200.77.234                                                                                                                     0.0%   177   25.2  23.7  12.0 162.5  16.0
    tge-4-2-0-0a.cr2.fra.routeserver.net
10. tge-4-2-0-0a.cr2.fra.routeserver.net                                                                                               0.6%   177  178.6  57.1  24.7 178.6  39.0
    217.118.16.26
11. 217.118.16.26                                                                                                                     47.2%   177   32.7  61.1  29.1 164.4  35.4
    217.118.16.165
12. 217.118.16.165                                                                                                                    28.2%   177   28.9  29.8  27.8  48.9   4.2
    prag341.server4you.de
13. prag341.server4you.de                                                                                                              1.1%   177   28.2  28.7  27.7  63.4   2.9

What looks weird to me is this very high loss >80% on the last hop from My home location to the server. The server is responding fine and services run smoothly.

It may be due to my lacking knowledge of networking but it would sound logical to me that loss rates should add up? But I often see MTR outputs where there are high loss rates on the way, but the final target loss is much lower.

So my questions are:

• In my particular case, is this an indicator of a possible problem I should pay attention to?

• In general, how do I interpret an output of mtr correctly? Can you recommend a good article / literature on that?