Copy directory structure intact to AWS S3 bucket

I believe sync is the method you want. Try this instead:

aws s3 sync ./logdata s3://bucketname/

The following worked for me:

aws s3 cp ~/this_directory s3://bucketname/this_directory --recursive

AWS will then "make" this_directory and copy all of the local contents into it.

I had faced this error while using either of these commands.

$ aws s3 cp --recursive /local/dir s3://s3bucket/
OR
$ aws s3 sync /local/dir s3://s3bucket/

I even thought of mounting the S3 bucket locally and then run rsync, even that failed (or got hung for few hours) as I have thousands of file.

Finally, s3cmd worked like a charm.

s3cmd sync /local/dir/ --delete-removed s3://s3bucket/ --exclude="some_file" --exclude="*directory*"  --progress --no-preserve

This not only does the job well and shows quite a verbose output on the console, but also uploads big files in parts.

(Improving the solution of Shishir)

• Save the following script in a file (I named the file s3Copy.sh)

path=$1 # the path of the directory where the files and directories that need to be copied are located
s3Dir=$2 # the s3 bucket path

for entry in "$path"/*; do
    name=`echo $entry | sed 's/.*\///'`  # getting the name of the file or directory
    if [[ -d  $entry ]]; then  # if it is a directory
        aws s3 cp  --recursive "$name" "$s3Dir/$name/"
    else  # if it is a file
        aws s3 cp "$name" "$s3Dir/"
    fi
done

• Run it as follows:
  /PATH/TO/s3Copy.sh /PATH/TO/ROOT/DIR/OF/SOURCE/FILESandDIRS PATH/OF/S3/BUCKET
  For example if s3Copy.sh is stored in the home directory and I want to copy all the files and directories located in the current directory, then I run this:
  ~/s3Copy.sh . s3://XXX/myBucket

You can easily modify the script to allow for other arguments of s3 cp such as --include, --exclude, ...

Use the following script for copying folder structure:

s3Folder="s3://xyz.abc.com/asdf";

for entry in "$asset_directory"*
do
    echo "Processing - $entry"
    if [[ -d  $entry ]]; then
        echo "directory"
        aws s3 cp  --recursive "./$entry" "$s3Folder/$entry/"
    else
        echo "file"
        aws s3 cp "./$entry" "$s3Folder/"
    fi
done

I couldn't get s3 sync or s3 cp to work on a 55 GB folder with thousands of files and over 2 dozen subdirectories inside. Trying to sync the whole folder would just cause awscli to fail silently without uploading anything to the bucket.

Ended up doing this to first sync all subdirectories and their contents (folder structure is preserved):

nice find . -mindepth 1 -maxdepth 1 -type d | cut -c 3- | while read line; do aws s3 sync $"$line" "s3://bucketname/$line"; done

Then I did this to get the 30,000 files in the top level:

nice find . -mindepth 1 -maxdepth 1 -type f | cut -c 3- | while read line; do aws s3 cp "$line" "s3://bucketname/";

Make sure to watch the load on the server (protip you can use w to just show the load) and ctrl-z to suspend the command if load gets too high. (fg to continue it again).

Putting this here in case it helps anyone in a similar situation.

Notes:

-mindepth 1 excludes .

-maxdepth 1 prevents find from listing contents of sub-directories, since s3 sync handles those successfully.

cut -c 3- removes the "./" from the beginning of each result from find.

Alternatively you could also try minio client aka mc

$ mc cp Desktop/test/test/test.txt s3/miniocloud/Desktop/test/test/

Hope it help.

PS: I am one of the contributor to the project.

This works for me.. aws s3 sync mydir s3://rahuls-bucket/mydir