Add a separate WARN log for SDK credentials refresh failure

[azlwwan]

The debug log level at here hides the error of fetch credentials failure. This blocks customers from monitoring this hidden failure which could result with system wide failure.

Describe the Feature

By adding a new WARN level log statement, customers can use MetricsFilter and Alarms in CloudWatch to detect this hidden failure to prevent the approaching system wide failure by replacing the alarming EC2 instances asap.

Is your Feature Request related to a problem?

Yes, when the credentials in BaseCredentialsFetcher expired after 60 mins, application logs in EC2 instance probably not be able to publish to CloudWatch, because awslog publish also uses the same credentials from EC2 instance role. This means there is no way for customers to monitor and defect this failure in CloudWatch in early manner.

Proposed Solution

By adding a new WARN level log statement similar like this, but using an explicit string, customer can use MetricsFilter and Alarms in CloudWatch to detect this hidden failure 15 mins before the credential expires. This 15 mins should be enough for customers to take action and replace the stale instances, but it will be better to have 30 mins.

Describe alternatives you've considered

Additional Context

Your Environment

• AWS Java SDK version used: 1.11
• JDK version used: OpenJDK 1.8.0_192
• Operating System and version: AL2012

[debora-ito]

It's a reasonable ask @azlwwan, marking as a feature request.

[debora-ito]

The log level in BaseCredentialsFetcher#handleError() was changed to WARN in SDK version 1.11.799.

Closing this, feel free to reopen if you have further questions.