Am having some difficulties getting a few things working with cross-account data copies. Specifically, I'm trying to clone an S3 bucket from one AWS account (in eu-west-1
) to another (in eu-west-2
).
I've tried setting up bucket replication per this guide, adding a Replication Batch Operation to copy existing files, and have also tried a DataSync job per this guide.
Despite having followed the guides to the letter:
Replication metrics show no activity and the destination bucket is still empty 24 hours later.
The replication batch operation aborted with the reason
Job failure rate 100% is above 50%
.The CloudWatch log for the DataSync job provides only this detail:
finished with status Unable to connect to S3 endpoint
I wonder if I'm missing some prerequisite that the guides assume will have already been set up; eg. Is there a need to establish some kind of trust relationship between two AWS accounts before cross-account permissions will work?
ie, where the intended destination S3 bucket in account 987654321098 has a permissions policy that includes this snippet to identify the actioning role in the source account:
"Sid": "DataSyncCreateS3LocationAndTaskAccess",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::123456789012:role/source-account-datasync-s3-copy-role" }
..is that all that should be needed for the destination account to trust the IAM role from the other account?
The answer to the question seems to be no - cross-account actions work ok without any special trust at the account level (at least for S3).
I did some further research on the tutorials I was following and found various posts about how the instructions were also not working for others, so ended up following the instructions in this post to add my IAM user from the source account to the destination bucket's Bucket Policy, then as the source account user:
aws s3 sync s3://sourcebucket s3://destinationbucket
- which worked fine.Note 1: When trying to save changes to the bucket policy, having added a number of actions to the example given in the above link, I ran into an error along the lines of:
action doesn't apply to any resources
; The fix for this was to collapse the two statements into one as shown in this post. (I imagine that it's also possible to refine the actions lists for the individual..bucketname
and..bucketname/*
ARNs in the two-statement version.)Note 2: AWS CLI may not be the most cost-effective way of achieving the desired result?, but a deadline is looming and it got the job done.