Home / server / Questions / 571133
In Process
APZ
Asked: 2014-01-31 13:20:36 +0800 CST

Ensuring data consistency in s3 with multiple users writing/reading

  • 772

I need advice/ ideas about how can I ensure data consistency in my s3 bucket. I have 2-3 users pushing and reading resources from s3. As one can easily see this leads to data inconsistency( User A is downloading that resource from S3 while User B is still writing to it).

We are syncing saltstack code from s3. We have one bucket setup per environment(prod, staging) and are using AWS IAM roles to control access. Problem happens when I am trying to grab data from s3 and the other developer is pushing code to that bucket, we end up with downloaded resource in a inconsistent state.

I would really appreciate if you guys can share your thoughts, ideas, experiences in dealing with such kind of problem.

Thanks

amazon-s3

1 Answers

  1. That's why you have a production and a staging environment. The staging environment is supposed to be "Free for all" and then your production environment is for when your code is all nice and dandy.

    If you want to allow more granularity you should consider source code control for your SaltStack code/yaml. (git, mercurial, etc) Every user could have his/her own branch and then when push comes to shove, you have to merge everything into a 'staging' branch and then eventually deploy to 'production'

    The behavior that you are seeing with S3 is the default behavior for "eventual consistency" Basically, you will get the copy of the last updated files and won't get anything new until the 'other user' has finally pushed his/her change.

    This is what the docs say on the AWS site about S3 data consistency:

    Q: What data consistency model does Amazon S3 employ? Amazon S3 buckets in the US West (Oregon), US West (Northern California), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Tokyo), Asia Pacific (Sydney) and South America (Sao Paulo) Regions provide read-after-write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES. Amazon S3 buckets in the US Standard Region provide eventual consistency.

    And you can read about 'eventual consistency' here:

    http://en.wikipedia.org/wiki/Eventual_consistency

    • 1