Botocore generates incorrect S3 URLs, leading to a 403 Access Denied

[shaskumar]

hello 👋, I am using boto3 (v1.34) client to list objects from s3. sometimes there is a random failure of type accessDenied 403 (~1 out of 10 times). Note- 403 is not a real issue but due to some different root cause which i have explained below.

Background(bototcore internals): botocore generates s3 http url based on bucket name and do http call to fetch objects. input for generating url contains bucket key along with several other keys like this - DEBUG:botocore.regions:Calling endpoint provider with parameters:{'Bucket': 'mybucket', 'Prefix': 'myobjects',.....}. it gets bucket name from the provided method param in s3_client.list_objects_v2(Bucket="mybucket", Prefix="myobject")), and the generated is url:'url': '[https://mybucket.s3.amazonaws.com/?list-type=..'

Issue: for the cases where random failure happens, input for generating url doesn't contain bucket key and hence generated url misses bucket name in it which leads to creation of wrong s3 url and throws 403 access denied. input param in this case (you can see bucket key is missing in this case and then generated url will also miss it) - DEBUG:botocore.regions:Calling endpoint provider with parameters:{'Prefix': 'myobjects',.....}, and generated url is - url': 'https://s3.amazonaws.com/?list-type=..

how often: this is very random, once in ~10 times.

error triggers from : s3_client.list_objects_v2(Bucket="mybucket", Prefix="myobject")

what could be possible cause of missing bucket key in input param during s3 url generation processing by botocore internals when failure case occurs. any ideas?

session = boto3.Session()
credentials = session.get_credentials()

s3_client = boto3.client(
    "s3",
    region_name="eu-central-1",
    aws_access_key_id=credentials.access_key,
    aws_secret_access_key=credentials.secret_key,
    aws_session_token=credentials.token,
)



s3_client.list_objects_v2(Bucket=mybucket,
        Prefix=myobject)

Note: no concurrency, only sequential execution.