In the past few months I have been testing two ISP that are providing a 10 Mbps link service to the internet. I have been able to properly verify that they are providing that bandwidth up to the equipments in their premises. Problem is that in my geographical area there is no local peering between all providers. So basically the traffic from any of my clients that want to access my web servers have to go through their ISP, up to the continental US (wherever NAP they are connected to), to "bounce" to the ISP providing me the service to finally reach my site. Because of that it was requested from the selected ISPs to provide the 10 Mbps service symmetrical (upload is very important to me), up to any point in the continental USA. Provider A have routers collocated at a site in Miami and I was able to properly test the service with those routers. Provider B connects through three other local providers and does not have a server or device in the continental US that I can use for the test. For that reason I test their performances using M-LABS network test servers, Speedtest servers and Speakeasy speed test servers. All test where conducted on same conditions (servers, time, etc). Test from those sites shown that Provider A was effectively providing the requested symmetrical 10 Mbps service when tested from my servers. With Provider B only the download is showing 10 Mbps but NOT the upload which give only from 5 up to 7 Mbps depending of the selected test site. Provider B is now telling me that my tests are not fair and that they are indeed providing the requested service. Then they show me some data to a server at Texas using Iperf but performing the test only with UDP and NOT TCP. They said that Speedtest, M-Labs, etc. are not valid test for reasons that I cannot understand.
I will like to get your feedback about this. Am I wrong the way I’m testing them? Can somebody please explain in simple terms why Speedtest or M-Labs cannot be use as a valid way to test ISPs? In my opinion what is equal offers no advantage and at this point I have to remove Provider B but I want to be fair with both providers and if I’m doing something wrong I’m willing to amend my mistake to give provider B another chance. How do you test and measure your ISP service? Am I right with my assertions about providing an excellent service to my clients in terms of traffic/bandwidth when there is no local peering?
Please give me any advice.
Thanks.
In all honesty, I can't think of a reason why your tests for ISP B wouldn't work just fine. Short of owning a personal server in the US and uploading to that, I can't think of anything I would really do any differently. If you've got a client there that you can upload/download from, you can always
dda huge chunk from /dev/urandom, and SCP to and from them. No pretty graphs, but it's a decent real world test.All that being said, you did call ISP B's tech support line. It doesn't sound as though you're terribly happy with what they told you. In my opinion, that's as important if not more than your speeds testing. If your ISP support is going to give you the runaround, not troubleshoot thoroughly, and can't give explanations you can understand, it's really a bad sign. Even if they give you the bandwidth you want, it will make any minor issue or question you have become a massive PITA, because their support is so bad. With my company, our bandwidth is 'fast enough', but the support is excellent. It makes your job a lot faster when your ISP reacts quickly to DNS requests, questions, and incident investigations. Infinitely more useful than the extra Mbs or two they we don't use 90% of the time.
If you really want to be perfectly fair here, call ISP A. Ask them for their opinion on a good speed test. Or, if you're feeling a bit mischievous, claim that your speed tests resulted in lest than promised speeds. Phrase it all the exact same way you did with ISP B. If they provide a good or better explanation, or expend more effort into researching the problem, then they've won on two fronts. If they handle it the same as ISP B, they're still ahead. Note that there isn't any actual problem here, so you should expect them to eventually state "We can't see anything wrong here. Maybe it's at your end?" But how they get here, and how long it takes, can tell you quite a bit.
--Christopher Karel
One of the challenges with the test sites is that their results can be distorted by caches at the endpoint ISPs or anywhere in-between. They have made progress in detecting caching, but the relatively small test payloads are very "cache-able" and the caches are just doing the job that they are designed to do.
A more realistic test is to perform transfer(s) over time with data that will more than likely not be "cache-able" (e.g. few 100 megs) between two acceptable test points. The longer transfers will also assist in determining the overall "stability" and performance of the connection when under actual prolonged load.
Basically, when you use speed tests that are off-network for the ISP in question, the ISP will only guarantee any portion of the connection that is on their network or their peers network. The best way to gauge bandwidth is to use a speed test provided by the ISP (which is usually located regionally along with the primary DNS server (which is the best for pings to test latency).