Home / server / Questions / 396100
Accepted
MB.
Asked: 2012-06-07 06:14:49 +0800 CST

S3fs performance improvements or alternative?

  • 772

I've been trying to use s3fs on an EC2 server but it's really slow. I spent the last 5 hours uploading 100MB of small files.

Is there anything I can do to improve the performance or is it something related to S3fs? If not, what's an alternative I could use?

amazon-ec2 s3fs

9 Answers

  1. Best Answer

    S3FS may not be the best choice for a large amount of smaller files. The overhead with S3FS is pretty high as well. I'd suggest using something like S3Curl

    You can even get parallel transfers going. Just remember it will never be fast like EBS / local storage.

    If you need to have it as a 'mountable' storage, the only alternatives to S3FS that I know of is S3Backer or s3ql

    • 11

  2. I just released v0.0.1 of https://github.com/kahing/goofys which was partly motivated by performance problems in s3fs. File creation speedup is 3-6x and time to first byte is 58x. Feedback welcome!

    • 9

  3. I just benchmarked riofs compared to s3fs. My test case was a relatively simple bash script that ran pngquant on each .png it found. On a testbucket with ~70 images of which ~20 where png (spread over a lot of subdirectories, which prob slows things down) the results where:

    s3fs: 3m54
    riofs: 15.9s

    So for this testcase riofs is ~15x faster! Setup was also very straightforward, although the docs are somewhat terse.

    Regarding the fact that the script still took 15.9s: it's not very efficient, also actually running pngquant on png's is a cpu intensive process.

    • 5

  4. https://github.com/kahing/goofys

    Goofys allows you to mount an S3 bucket as a filey system.

    It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have an on disk data cache (checkout catfs), and consistency model is close-to-open.

    • 3

  5. I used [1] and it's very good. It's lot of commands and makes accessing aws and s3 a lot easier.

    1. http://timkay.com/aws/
    • 1

  6. Just adding riofs to the mix. I could not see any benchmark for now but there is an issue/request that anybody can do.

    • 1

  7. I compared s3fs with riofs, and almost switched because of noticeably better performance. Problem is riofs is still not updated to v4 signatures, which means it won't work with Frankfurt or China s3 buckets. According to the owner, an update is on the way. If you don't use Frankfurt or China, riofs is a hands down better option.

    • 1

  8. This is kind old but may help who gets here.

    I'm using aws cli and its working very good (3 Mb/s)

    pip install awscli --upgrade --user
nano ~/.bash_profile
export PATH="~/.local/bin:$PATH"
aws s3 sync /sourcedir/ s3://bucketname/destdir/
    • 1

  9. Used rclone mount and is far much faster than s3fs

    • 0