IRSA with aws-sdk getting "InvalidToken: The provided token is malformed or otherwise invalid."

[0xknon]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
I am trying to implement IRSA in my EKS cluster. I used eksctl to create the service account. I deployed the docker to EKS cluster that has an API to upload file. When I try to upload the file, I get the error: "InvalidToken: The provided token is malformed or otherwise invalid.".

I then created a simple js script to test aws-sdk and I successfully reproduced the same error.

I had a doubt on whether I successfully granted the permission to the pod. So, I used aws-cli to check the permission. I ran kubectl exec to jump into the pod and ran aws s3 cp to upload a file to my target bucket. Since I have 2 buckets and I only granted the permission for one of them, I verified that I can only upload a file to the S3 bucket that I have granted the permission to the pod. Thus, I think the IRSA setting is correct. It may be just an issue occurs on aws-sdk.

Is the issue in the browser/Node.js?
Node.js

If on Node.js, are you running this on AWS Lambda?
No

Details of the browser/Node.js version
v15.13.0

SDK version number
2.767.0

To Reproduce (observed behavior)

const aws = require('aws-sdk');

aws.config.update({
  logger: console,
});

const s3 = new aws.S3({
  region: 'ap-east-1',
  apiVersion: '2006-03-01',
  params: {
    Bucket: 'bucket'
  },
});

(async function() {
  const response1 = await s3
    .putObject({ Key: 'res/putobject', Body: Buffer.from('test') })
    .promise();
  console.log(response1);
  console.log('done1');

  const response2 = await s3.upload({ Key: 'res/upload', Body: Buffer.from('test') }).promise();
  console.log(response2);
  console.log('done2');
})();

Here is the ClusterConfig

iam:
  withOIDC: true
  serviceAccounts:
  - metadata:
      name: project-name
      namespace: development
      labels: { aws-usage: "application" }
    roleName: role-assumed
    attachPolicyARNs:
    - "arn:aws:iam::xxxxx:policy/BucketWritePolicy"

Expected behavior
successfully upload file

[ajredniwja]

Hey @knyucross thanks for opening this issue, I believe this is related to #3700 can you please check if its still an issue with the latest version of the SDK?

[github-actions]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.