Unable to load AWS credentials from AWS provider

[snehal-kolte]

Team,
We are referring the https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/ and set up docker-compose.yml and docker-compose.override.yml like below

docker-compose.yml
version: "2"
services:
app:
build: my_image
ports:
- 8080:80
environment:
PORT: "80"

docker-compose.override.yml
version: "2"
networks:

credentials_network:
driver: bridge
ipam:
config:
- subnet: "169.254.170.0/24"
gateway: 169.254.170.1

services:

volumes:
  - %UserProfile%\.aws
environment:
  AWS_PROFILE: "default"
networks:
  credentials_network:
    # This special IP address is recognized by the AWS SDKs and AWS CLI
    ipv4_address: "169.254.170.2"

app:
depends_on:
- ecs-local-endpoints
networks:
credentials_network:
ipv4_address: "169.254.170.3"
environment:
AWS_DEFAULT_REGION: "us-east-1"
AWS_REGION: "us-east-1"
ABBREVIATED_REGION: "us01"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds"

Error:
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@29c5ee1d: profile file cannot be null, WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@59429fac: Failed to connect to service endpoint: ]

@hencrice Is there something missing in my configuration set up?
credentials file look like below
[default]
aws_access_key_id= xyz
aws_secret_access_key= xyz
aws_security_token="xyz"
aws_session_token="xyz"

[cbishop-elsevier]

@snehal-kolte

A couple of things I notice here that differ to my local development environment (which obtains credentials successfully):

First - I would suggest using a specific Named Profile, versus AWS_PROFILE: "default" . I suggest this for two reasons, its best practice, and I also seem to recall having issues using "default" myself when I first kicked out my local dev environment.

Second - try referencing an IAM Role (NOT your Production Task Exec Role - rather, a Dev Testing Role with equivalent privileges, as best practice) in your AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable so that the ecs-local-container-endpoints container can "vend" temporary credentials to requisite containers in your microservice via sts:AssumeRole rather than consistently providing the same set of auth creds. I've always had better luck with this approach, and again - it better aligns with best practice.

Below is an example snippet from my local dev environment docker-compose-override.yml:

version: "2"
networks:
    # This special network is configured so that the local metadata
    # service can bind to the specific IP address that ECS uses
    # in production
    credentials_network:
        driver: bridge
        ipam:
            config:
                - subnet: "169.254.170.0/24"
                  gateway: 169.254.170.1
services:
    # This container vends credentials to your containers
    ecs-local-endpoints:
        # The Amazon ECS Local Container Endpoints Docker Image
        image: amazon/amazon-ecs-local-container-endpoints
        volumes:
          # Mount /var/run so we can access docker.sock and talk to Docker
          - /var/run:/var/run
          # Mount the shared configuration directory, used by the AWS CLI and AWS SDKs
          # On Windows, this directory can be found at "%UserProfile%\.aws"
          - $HOME/.aws/:/home/.aws/
        environment:
          # define the home folder; credentials will be read from $HOME/.aws
          HOME: "/home"
          # You can change which AWS CLI Profile is used
          AWS_PROFILE: "aws-dev" # "default"
        networks:
            credentials_network:
                # This special IP address is recognized by the AWS SDKs and AWS CLI 
                ipv4_address: "169.254.170.2"
                
    # Here we reference the application container that we are testing
    # You can test multiple containers at a time, simply duplicate this section
    # and customize it for each container, and give it a unique IP in 'credentials_network'.
    app:
        depends_on:
            - ecs-local-endpoints
        networks:
            credentials_network:
                ipv4_address: "169.254.170.3"
        environment:
          AWS_DEFAULT_REGION: "us-east-1"
          AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/dev-ecs-service-fluentd-s3-es-role"

    # add aws-cli instance, to test it uses credentials vended by ecs-local-endpoints
    awscli:
        image: amazon/aws-cli
        depends_on:
            - ecs-local-endpoints
        networks:
            credentials_network:
                ipv4_address: "169.254.170.4"
        environment:
          AWS_DEFAULT_REGION: "us-east-1"
          AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/dev-ecs-service-fluentd-s3-es-role"
          # define the home folder; credentials will be read from $HOME/.aws
          HOME: "/home"
          # You can change which AWS CLI Profile is used
          # AWS_PROFILE: "aws-dev" # "dev-ecs-role" # "default"
        command: ["--debug", "s3", "ls", "s3://some-bucket-name-for-testing-access/"]

Hope this helps!

Cheers,

Chris Bishop