Home / server / Questions / 377691
Accepted
Tom Marthenal
Asked: 2012-04-09 09:40:03 +0800 CST

Why is uploading to S3 so slow?

  • 772

I am using s3cmd to upload to S3:

# s3cmd put 1gb.bin s3://my-bucket/1gb.bin
1gb.bin -> s3://my-bucket/1gb.bin  [1 of 1]
  366706688 of 1073741824    34% in  371s   963.22 kB/s

I am uploading from Linode, which has an outgoing bandwidth cap of 50 Mb/s according to support (roughly 6 MB/s).

Why am I getting such slow upload speeds to S3, and how can I improve them?

Update:

Uploading the same file via SCP to an m1.medium EC2 instance (SCP from my Linode to the instance's EBS drive) gives about 44 Mb/s according to iftop (any compression done by the cipher is not a factor).

Traceroute: Here's a traceroute to the server it's uploading to (according to tcpdump).

# traceroute s3-1-w.amazonaws.com.
traceroute to s3-1-w.amazonaws.com. (72.21.194.32), 30 hops max, 60 byte packets
 1  207.99.1.13 (207.99.1.13)  0.635 ms  0.743 ms  0.723 ms
 2  207.99.53.41 (207.99.53.41)  0.683 ms  0.865 ms  0.915 ms
 3  vlan801.tbr1.mmu.nac.net (209.123.10.9)  0.397 ms  0.541 ms  0.527 ms
 4  0.e1-1.tbr1.tl9.nac.net (209.123.10.102)  1.400 ms  1.481 ms  1.508 ms
 5  0.gi-0-0-0.pr1.tl9.nac.net (209.123.11.62)  1.602 ms  1.677 ms  1.699 ms
 6  equinix02-iad2.amazon.com (206.223.115.35)  9.393 ms  8.925 ms  8.900 ms
 7  72.21.220.41 (72.21.220.41)  32.610 ms  9.812 ms  9.789 ms
 8  72.21.222.141 (72.21.222.141)  9.519 ms  9.439 ms  9.443 ms
 9  72.21.218.3 (72.21.218.3)  10.245 ms  10.202 ms  10.154 ms
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

The latency looks reasonable, at least until the server stopped responding to ping requests.

amazon-s3 amazon-web-services linode

3 Answers

  1. Just in case anyone stumbles upon this....

    I had an issue where it was really slow to upload from an ec2 instance to an s3 bucket, turned out to be really simple, the region of the bucket! I was using ec2 instances in north california, when the bucket was created as us standard it was really slow to transfer, with the bucket set to north california it was heaps faster.

    • 11
  2. Best Answer

    Just because you can go up to 50mbps does not mean you will always get 50mbps; The network path to S3 as well as the latency are also important.

    If you are able to use multi-part upload, you will be able to break a file down into multiple pieces and upload it using multiple threads, possibly increasing the upload speed.

    • 10

  3. I have seen many threads in a forum about slow upload to S3 using various clients, such as the free command line tools (written in Python, btw) and commercial ones.

    Although I have no hard evidence for you, switching the S3 client (e.g. Cloud Berry Explorer) might solve your problem. Try it! :)

    • 1