SDK not loading assumed role credentials unless run as sudo

[martinpllu]

Describe the bug

(Originally asked as question in #4252, but at this stage an issue feels more appropriate)

I want to be able to run SDK code with assumed role credentials. I have the following in ~/.aws/config

[profile base]
region = eu-west-1
output = json
cli_pager=

[profile mfa]
role_arn = arn:aws:iam::0123456789012:role/Developer
mfa_serial = arn:aws:iam::0123456789012:mfa/myuser
source_profile = base

With this config I can run CLI commands like aws s3 ls, and I periodically get prompted for an MFA code as expected.

Now I'd like to run a script with these credentials. For example:

import aws from "aws-sdk";

new aws.S3().listBuckets().promise().then(b => console.log(b));

My environment is set up as follows:

AWS_PROFILE=mfa
AWS_CONFIG_FILE=/Users/martin/.aws/config
AWS_SHARED_CREDENTIALS_FILE=/Users/martin/.aws/credentials
AWS_SDK_LOAD_CONFIG=1

When I run the script above I get the following error:

Error: connect EHOSTDOWN 169.254.169.254:80 - Local (192.168.68.110:60258)
    at internalConnect (net.js:934:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:452:18)
    at net.js:1022:9
    at processTicksAndRejections (internal/process/task_queues.js:77:11) {
  message: 'Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1',
  errno: -64,
  code: 'CredentialsError',
  syscall: 'connect',
  address: '169.254.169.254',
  port: 80,
  time: 2022-10-22T18:32:25.240Z,
  originalError: {
    message: 'Could not load credentials from any providers',
    errno: -64,
    code: 'CredentialsError',
    syscall: 'connect',
    address: '169.254.169.254',
    port: 80,
    time: 2022-10-22T18:32:25.240Z,
    originalError: {
      message: 'EC2 Metadata roleName request returned error',
      errno: -64,
      code: 'EHOSTDOWN',
      syscall: 'connect',
      address: '169.254.169.254',
      port: 80,
      time: 2022-10-22T18:32:25.240Z,
      originalError: [Object]
    }
  }
}

However, when I run the script using sudo, it works.

My first thought was that there was a permission issue with files in ~/.aws, perhaps in the files under ~/.aws/cli that are created when assuming a role. However even after running sudo chown -R $USER ~/.aws and sudo chmod -R 777 ~/.aws I still see the same error, unless I run the script as sudo.

Expected Behavior

see above

Current Behavior

see above

Reproduction Steps

see above

Possible Solution

No response

Additional Information/Context

No response

SDK version used

2.1238.0

Environment details (OS name and version, etc.)

MacOS Monterey 12.2.1, aws-sdk 2.1238.0, CLI version aws-cli/2.8.4 Python/3.10.8 Darwin/21.3.0 source/arm64 prompt/off

[ajredniwja]

Hey @martinpllu thanks for opening this issue, it does seem weird that you have to sudo run the script.
With that being said, I have my setup something like this:

~/.aws/config:

[profile test]
region = us-east-1
output = json

[profile mfa]

region = us-east-1
role_arn = arn:aws:iam::8787318798:role/aj
mfa_serial = arn:aws:iam::9393:mfa/aj
source_profile = test

~/.aws/credentials:

[test]
aws_access_key_id = id..
aws_secret_access_key = key..

It does seem to be some permissions issue looking at the original error EC2 Metadata roleName request returned error.
See source_profile - shared AWS config file setting section

Can you check having a setup like that works for you? thanks

[martinpllu]

Hi @ajredniwja, thanks for the response. I tried to replicate your config above as closely as possible:

~/.aws/config:

[profile test]
region = us-east-1
output = json

[profile mfa]
region = us-east-1
role_arn = arn:aws:iam::0123456789012:role/Developer
mfa_serial = arn:aws:iam::0123456789012:mfa/myuser
source_profile = test

~/.aws/credentials:

[test]
aws_access_key_id=...
aws_secret_access_key=...

My environment is as follows:

AWS_PROFILE=mfa
AWS_CONFIG_FILE=/Users/martin/.aws/config
AWS_SHARED_CREDENTIALS_FILE=/Users/martin/.aws/credentials
AWS_SDK_LOAD_CONFIG=1

I still see a similar error when running as non-sudo, i.e.

Error: connect EHOSTDOWN 169.254.169.254:80 - Local (192.168.68.110:61583)
    at internalConnect (net.js:934:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:452:18)
    at net.js:1022:9
    at processTicksAndRejections (internal/process/task_queues.js:77:11) {
  message: 'Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1',
  errno: -64,
  code: 'CredentialsError',
  syscall: 'connect',
  address: '169.254.169.254',
  port: 80,
  time: 2022-10-30T16:13:12.605Z,
  originalError: {
    message: 'Could not load credentials from any providers',
    errno: -64,
    code: 'CredentialsError',
    syscall: 'connect',
    address: '169.254.169.254',
    port: 80,
    time: 2022-10-30T16:13:12.605Z,
    originalError: {
      message: 'EC2 Metadata roleName request returned error',
      errno: -64,
      code: 'EHOSTDOWN',
      syscall: 'connect',
      address: '169.254.169.254',
      port: 80,
      time: 2022-10-30T16:13:12.605Z,
      originalError: [Object]
    }
  }
}

[ajredniwja]

It must have to be something with the config as I am not able to reproduce using the setup mentioned above.

[Zordrak]

This may be related?: #4406