Terraform: Choosing credentials for a remote state file

Your first question

if my provider setting is explicitly declaring the credentials to use inside the Terraform source code, why does my OS-level AWS configuration matter at all?

The error message "Failed to load backend: Error configuring the backend "s3"" is referring to your Backend S3 configuration.

Look in the file ./.terraform/terraform.tfstate and you will see the S3 Backend configuration.

The Terraform S3 Backend is different than the Terraform AWS Provider. The error message "No valid credential sources found for AWS Provider." is misleading. It implies that the AWS Provider configuration is used, which is false. S3 Backend credentials are configured separately and stored in the terraform.tfstate file.

Your OS-level AWS configuration matters because if no S3 Backend credentials are specified, as documented here https://www.terraform.io/docs/backends/types/s3.html, then Terraform defaults to using the following, in order:

1. Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
2. AWS Shared credentials file, default value is "~/.aws/credentials".

You didn't specify any credentials in your S3 Backend config so terraform is defaulting to the AWS Shared Credentials File.

Your S3 Backend configuration contains no credentials.

terraform {
  backend "s3" {
    bucket = "example_tf_states"
    key    = "global/vpc/us_east_1/example_state.tfstate"
    encrypt = true
    region     = "us-east-1"
  }
}

Your second question,

How can I make Terraform only use the creds defined in my Terraform configuration and ignore what's in my OS user profile?

First, Backends cannot contain interpolation, see https://www.terraform.io/docs/backends/config.html. So you cannot use any variables in the Backend config. e.g. this config is invalid

terraform {
  backend "s3" {
    bucket = "example_tf_states"
    key    = "global/vpc/us_east_1/example_state.tfstate"
    encrypt = true
    region     = "us-east-1"
    access_key = ${var.access_key}
    secret_key = ${var.secret_key}
  }
}

If you want to specify AWS credentials when running terraform init you specify backend configuration as options.

terraform init --backend-config="access_key=your_access_key" --backend-config="secret_key=your_secret_key"

This produces a S3 Backend config that looks like this, stored in the ./.terraform/terraform.tfstate file:

{
    "version": 3,
    "serial": 1,
    "lineage": "bd737d2d-1181-ed64-db57-467d14d2155a",
    "backend": {
        "type": "s3",
        "config": {
            "access_key": "your_access_key",
            "secret_key": "your_secret_key"
        },
        "hash": 9345827190033900985
    },

Again, the S3 Backend credentials are configured separately from your AWS Provider credentials.

Re-run terraform init and specify the credentials on the command line as --backend-config options to fix your error.

The error you're getting is specifically referring to configuring the S3 backend, which AFAIK does not inherit the settings from the AWS provider configuration; it too has access_key & secret_key configuration options which if you're not using ~/.aws/credentials you will need to explicitly configure.

You are better off setting up profiles in your ~/.aws/credentials files like

[profile1]
aws_access_key_id = xxxx
aws_secret_access_key = xxxxx
region = us-east-1

[profile2]
aws_access_key_id = xxxx
aws_secret_access_key = xxxx
region = us-west-2

Then in your provider you can tell it which profile to use

provider "aws" {
  profile = "profile2"
  region = "${var.region}"
}

It will keep the keys out of your terraform files which is a good thing if you ever want to put them into source control.

I also had the same issue, the one of easiest and the secure way is to fix this issue is that configure the AWS profile. Even if you are properly mentioned the AWS_PROFILE in your project, you have to mention it again in your backend.tf.

my problem was, I have already set up the AWS provider in the project as below and it is working properly.

provider "aws" {
region = "${var.AWS_REGION}"
profile = "${var.AWS_PROFILE}"
}

but end of the project I was trying to configure the S3 backend configuration file. therefore I have run the command terraform init and I also got the same error message.

Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.

Note that is not enough for the terraform backend configuration. you have to mention the AWS_PROFILE in the backend file as well.

• Full Solution

I'm using the terraform latest version at this moment. it's v0.13.5. Please note that you can't use variables in the backend file as well.

please see the provider.tf

provider "aws" {
region = "${var.AWS_REGION}"
profile = "${var.AWS_PROFILE}" # lets say profile is my-profile
}

for example your AWS_PROFILE is my-profile then your backend.tf should be as below.

terraform {
    backend "s3" {
    bucket = "my-terraform--bucket"
    encrypt = true
    key = "state.tfstate"
    region = "ap-southeast-2"
    profile = "my-profile" # you have to give the profile name here. not the variable("${var.AWS_PROFILE}")
  }
}

then run the terraform init