Home / server / Questions / 554086
In Process
lev
Asked: 2013-11-12 01:12:07 +0800 CST

Move files from ftp server to s3

  • 772

I would like to set up an ftp server, where users will upload files, and for each file, put it on s3 storage, and delete it from the ftp server. (the server runs on ec2 ubuntu)

Here are the stuff I already tried, with no success..

  1. Mount s3 bucket using s3fs. I followed those instructions, but there is a bug in the latest version of s3fs, that prevents it from working. The bug was fixed on the develop branch, but I don't want to use unstable version on my production.

  2. Use vsftpd and using s3cmd sync via cron to sync the files periodically. The problem with that approach, is that s3cmd can start running in the middle of a file upload, and start synching the incomplete file. Also s3cmd doesn't give any feedback it the sync fails, so I have no way of knowing if I can delete the files after the sync command finished running.

  3. Use pure-ftpd's upload script feature (which allows to run a script after a file is finished uploading), but I noticed that if the file upload was failed in the middle, the script will run anyway, and I have no way of knowing if the upload was successful or not.

I've been at it for a few days now, and I'm at a loss here. Any suggestions will be welcomed.

ubuntu

2 Answers

  1. This might be worth looking into;

    https://github.com/seedifferently/boto_rsync

    Found here;

    Using RSYNC with Amazon S3

    • 1

  2. I have done it successfully using instruction in the following link https://github.com/s3fs-fuse/s3fs-fuse.git

    Have also created another user for ftp as using ubuntu user for this purpose is not good

    http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/managing-users.html

    However I have to still find out how I can delete the file from EC2 server after uploading it to S3 as both are mounted and once I delete file/folder from EC2 it deletes from S3

    • 0