ClientError without 'Error' key

[zalex5]

Describe the bug

Hi,
we have an issue with S3 boto3 client apis call.

Sometimes our code generates a KeyError while parsing ClientError exceptions and the missing key is 'Error'.

The code is:

    try:
        s3_client.copy(
            CopySource={"Bucket": object_bucket, "Key": object_key},
            Bucket=DEFAULT_CACHE_BUCKET,
            Key=object_key,
            ExtraArgs={
                "MetadataDirective": "COPY",
            },
        )
    except ClientError as err:
        if err.response['Error']['Code'] in ['ObjectNotInActiveTierError', 'InvalidObjectState']:
            err_msg = f"{err.response['Error']['Code']}: the object could have been already copied"
            logger.warning(err_msg)

In cloudwatch we see this failing with the following message:

[ERROR] KeyError: 'Error'
Traceback (most recent call last):
  File "/var/task/manage_storage_class.py", line 95, in lambda_handler
    create_temporary_copy(object_bucket, object_key, s3_client=s3_client)
  File "/var/task/manage_storage_class.py", line 32, in create_temporary_copy
    if err.response['Error']['Code'] in ['ObjectNotInActiveTierError', 'InvalidObjectState']:

Just to let you know, this issue is not easily replicable: it happened 20 times in the last four weeks (considering thousands/millions of calls).

We know we can bypass the error testing for the presence of the 'Error' key in the 'response' dictionary but we couldn't find any documents mentioning that the ClientError could miss that key.

Any information about this situation would be much appreciated,
Thank you

Expected Behavior

We expect to be able to access the 'Error' key every time a ClientError is raised.

Current Behavior

Sometimes the ClientError object has no 'Error' key inside the 'response' dictionary.

Reproduction Steps

The code is very basic

import logging
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError
from storage_management import DEFAULT_CACHE_BUCKET

S3_CONF = Config(retries={"max_attempts": 30, "mode": "adaptive"})
REGION = "eu-west-1"
RESTORE_DAYS = 15
RESTORE_TIER = "Bulk"

fmt = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
hnd = logging.StreamHandler()
hnd.setFormatter(fmt)

logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addHandler(hnd)

def create_temporary_copy(object_bucket, object_key, s3_client):
    try:
        s3_client.copy(
            CopySource={"Bucket": object_bucket, "Key": object_key},
            Bucket=DEFAULT_CACHE_BUCKET,
            Key=object_key,
            ExtraArgs={
                "MetadataDirective": "COPY",
            },
        )
    except ClientError as err:
        if err.response['Error']['Code'] in ['ObjectNotInActiveTierError', 'InvalidObjectState']:
            err_msg = f"{err.response['Error']['Code']}: the object could have been already copied"
            logger.warning(err_msg)

and also this (very similar):

import logging
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError
from storage_management import DEFAULT_CACHE_BUCKET

S3_CONF = Config(retries={"max_attempts": 30, "mode": "adaptive"})
REGION = "eu-west-1"
RESTORE_DAYS = 15
RESTORE_TIER = "Bulk"

fmt = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
hnd = logging.StreamHandler()
hnd.setFormatter(fmt)

logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addHandler(hnd)

def change_object_storage_class(
    object_bucket, object_key, dest_storage_class, s3_client
):
    try:
        s3_client.copy(
            CopySource={"Bucket": object_bucket, "Key": object_key},
            Bucket=object_bucket,
            Key=object_key,
            ExtraArgs={
                "StorageClass": dest_storage_class,
                "MetadataDirective": "COPY",
                "TaggingDirective": "COPY",
            },
        )
    except ClientError as err:
        if err.response['Error']['Code'] in ['ObjectNotInActiveTierError', 'InvalidObjectState']:
            err_msg = f"{err.response['Error']['Code']}: the storage class has already been changed"
            logger.warning(err_msg)

Possible Solution

No response

Additional Information/Context

Both the functions are executed in a lambda triggered by EventBridge (configured to react to a storage class transition).

SDK version used

The AWS lambda default for Python 3.9 runtime

Environment details (OS name and version, etc.)

Lambda runtime Python 3.9

[zalex5]

Just adding the cloudwatch logs:

log-events-viewer-result.csv