EKS Pod Identity does not work with aws-sdk v2

[k4kratik]

Describe the bug

AWS Recently rollout out a new feature for EKS to authorize pods IAM access more seamlessly, it can be considered as the successor of IRSA (IAM Role for Service Account).

So even after following the steps mentioned in the docs, My pod is not able to get any IAM access. I am sure I am using the latest SDK version as mentioned here and also I setup my role and service account as mentioned here.

When I dug, found that code in the aws-sdk to utilize this new feature is missing. (ref)

Expected Behavior

This should work seamlessly and all IAM access should be authorized as expected.

Current Behavior

My pod is not able to get any IAM Access. I get error :

Error listing SQS queues: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1

Reproduction Steps

Setup Consists : EKS Cluster enabled pod identity add-on enabled + Setup Role and Service Account + Configure Pod to use this Service Account. [In case if it matters, we have set http_put_response_hop_limit to 1 , default is 2]

Now create a simple script and run :

aws.js

const AWS = require('aws-sdk');

// Set your AWS region to Mumbai (ap-south-1)
const region = 'ap-south-1';

AWS.config.credentials = new AWS.RemoteCredentials({
	  httpOptions: { timeout: 5000 }, // 5 second timeout
	  maxRetries: 10, // retry 10 times
	  retryDelayOptions: { base: 200 } // see AWS.Config for information
});

// Create an SQS service object
const sqs = new AWS.SQS();

// List all SQS queues
sqs.listQueues({}, (err, data) => {
  if (err) {
    console.error('Error listing SQS queues:', err.message);
  } else {
    console.log('SQS Queues:');
    data.QueueUrls.forEach((queueUrl, index) => {
      console.log(`${index + 1}. ${queueUrl}`);
    });
  }
});

Output:

Error listing SQS queues: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1

Possible Solution

• add and whitelist IP for pod-identity agent : 169.254.170.23
• add code to use the env var : AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
  I also raised a PR here: https://github.com/aws/aws-sdk-js/pull/4565/files

Additional Information/Context

No response

SDK version used

v2.1518.0

Environment details (OS name and version, etc.)

Amazon Linux 2

[aBurmeseDev]

Hi @k4kratik - thanks for reaching out.

I'm not able to reproduce this on my end. Upon further investigating, it's mentioned in the docs that:

If your workloads currently use credentials that are earlier in the chain of credentials, those credentials will continue to be used even if you configure an EKS Pod Identity association for the same workload.

Can you confirm if that's the case or not?

The error you're seeing generally indicates that credentials aren't being able to locate from either Config or Credential file. I would also confirm if the default profile is being used in your ~/.aws/credentials. Lastly, I would make sure temporary credentials aren't expired

const AWS = require('aws-sdk');
 
AWS.config.update({region: 'REGION'});

const sqs = new AWS.SQS({
  apiVersion: '2012-11-05',
  credentials: creds
});
console.log(sqs.config.credentials) // log check the credentials
sqs.listQueues({}, function(err, data) {
  if (err) {
    console.log("Error", err);
  } else {
    console.log("Success", data.QueueUrls);
  }
});

If the issue persists, I would try different SDK that supports assuming an IAM role from the EKS Pod Identity Agent from this list to narrow down the root cause.

Hope that helps,
John!

[k4kratik]

Hi @aBurmeseDev!
Thanks for checking.

Can you confirm if that's the case or not?
No, It's not the case. Only the one access method which I mentioned (pod identity) is available.

I am glad that you tried on your end to reproduce. Can you specify how was your setup ?

In my case it was -

1. EKS latest version
2. Nodes with IMDSv2 Enabled with Hop limit 1
3. Created mapping between service account and role using Pod Identity associations.
4. Using nodejs aws-sdk v2 (latest version)

@aBurmeseDev can you please try to reproduce under above mentioned circumstances?

Thanks!

[k4kratik]

@aBurmeseDev The issue persists, and I our team will need some time for migration plan from v2 to v3 and Currently we want v2 to support Pod Identity Associations as we have a lot of microservices running with aws-sdk v2.

I have also raised a PR here : #4565

would you like to have a look and give me some feedback if it makes sense?

and If aws-sdk v2 supports this new feature, why there is no mention of variable AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE anywhere?

if you this check changelog, they have added support to manage/create pod identity associations but I could not see anywhere where they have mentioned on how to use IAM access from this feature
https://github.com/aws/aws-sdk-js/blob/master/CHANGELOG.md?plain=1#L236-L237

Thanks!

[bryantbiggs]

@aBurmeseDev - theres a similar issue in the v3 SDK with a fix going in, can we add the same here?

• Issue: EKS Pod Identities does not work with default credential chain aws-sdk-js-v3#5709
• Fix PR: fix(credential-provider-node): add fromHttp credential provider to default chain aws-sdk-js-v3#5739

[kuhe]

I don't think this credential provider was implemented in AWS SDK for JavaScript (v2) (this repo).

But, credential providers are modular and can be mixed, as a workaround:

Docs: https://www.npmjs.com/package/@aws-sdk/credential-providers

import { fromHttp } from '@aws-sdk/credential-providers';
import * as AWS from 'aws-sdk';

const credentialProvider = fromHttp({ ... }); // use code or environment variables.
const credentials = await credentialProvider();

const client = new AWS.S3({ credentials });

[bryantbiggs]

@kuhe per the release notes on v2.1503.0, it reads that Pod Identity was added but its currently not working as expected c1ef7c7#diff-31729d20ef2ae5d600178d896e07b595d5635238aac19ec9d1f74c6bbbd48bc5R73-R76