429 / too many requests API error reported as missing credentials

[michaelglass]

Describe the bug

When performing too many requests, the ruby error surfaced says

Aws::Sigv4::Errors::MissingCredentialsError: missing credentials, provide credentials with one of the following options:
  - :access_key_id and :secret_access_key
  - :credentials
  - :credentials_provider

but the underlying failed http request fails with a 429 error

Expected Behavior

The error raised would be something related to the underlying 429 error (instead of pointing to wrong credentials)

Current Behavior

The error raised is a Aws::Sigv4::Errors::MissingCredentialsError

Reproduction Steps

hammer the credentials endpoint with valid credentials

Possible Solution

bubble through the 429 error or wrap the 429 error in a 429-themed error.

Additional Information/Context

No response

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-s3

Environment details (Version of Ruby, OS environment)

ruby 3.1, debian bullseye, x86-64

[alextwoods]

What do you mean by "hammer the credentials endpoint with valid credentials"?

Can you provide any code to reproduce the issue? What service and operation are you trying to use and how are you constructing a client?

[michaelglass]

from rails, from an AWS instance, with credentials from the instance

1000.times.map { Thread.new { Aws::S3::Client.new(region: 'us-east-2').head_object(bucket: 'my_bucket', key: 'my_file.txt') } }.map(&:join)

reproduces the error for me.

sometimes I have to futz with the number (2000? 3000?) to consistently reproduce.

[alextwoods]

In general you should only be creating a single client - the AWS service clients are thread safe and can safely be shared and re-used across your application.

The InstanceProfileCredentials (EC2 credentials) is part of the Credential Provider Chain, which requires that credential providers that encounter errors or otherwise fail to load/fetch credentials don't raise errors and skip to the next provider in the chain - so I dont think that we can/should surface an error during that process. We also defer raising any missing credential related errors until an operation that requires credentials is called (since there are unsigned operations and it must be possible to use the client without credentials for those operations).

If you know the type of credentials your application needs ahead of time, you can explicitly check and configure them.

[michaelglass]

Yup -- the issue is not at all blocking. Just when we got the error it was misleading.

[alextwoods]

Yeah - thats fair - getting to the root cause issue there is difficult. I'll look into the possibility of adding some logging for such failures.

[solomon-maeng]

@alextwoods

What if I create 100 or more AWS S3 clients instead of a single client?

I'm asking because I want to know more about creating S3 clients.

[alextwoods]

There is some overhead in creating a client, so its generally recommended to create only a single client (per set of configuration options) and share it across your application.

When you create a service client, it will attempt to resolve all of the configuration options. For credentials (unless credentials are explicitly provided) it uses the credential provider chain, which will check each credential provider in order until one of them is able to resolve credentials. With the InstanceProfileCredentials it will make a request to the local EC2 credentials endpoint. If you're creating a large number of clients at one time, this could overwhelm that endpoint and cause it to start throttling requests.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.