Regression: Prefixes in S3 listing are unquoted and don't match the Key

[kokes]

Upgrading from moto 3.1.11, we found the following issue: Our prefixes foo%2Fbar now get interpreted as foo/bar, but only in prefixes (be it in plain list_objects_v2 or with Delimiter='/' directory listing). Here's a quick repro that works on 3.1.11 and breaks on 3.1.12 (same with all the other repros here):

from moto import mock_s3
import boto3

from pprint import pprint

with mock_s3():
    client = boto3.client("s3", region_name="us-east-1")
    client.create_bucket(Bucket="foo")
    client.put_object(Bucket="foo", Key="foo%2Fbar/data", Body=b"")

    data = client.list_objects_v2(Bucket="foo", Prefix="foo%2Fbar")
    # pprint(data)
    assert data["Contents"][0]["Key"].startswith(data["Prefix"]), data

The Prefix no longer matches the Key, here's my output:

{'Contents': [{'ETag': '"d41d8cd98f00b204e9800998ecf8427e"',
               'Key': 'foo%2Fbar/data',
               'LastModified': datetime.datetime(2022, 6, 14, 15, 22, 54, tzinfo=tzutc()),
               'Size': 0,
               'StorageClass': 'STANDARD'}],
 'EncodingType': 'url',
 'IsTruncated': False,
 'KeyCount': 1,
 'MaxKeys': 1000,
 'Name': 'foo',
 'Prefix': 'foo/bar',
 'ResponseMetadata': {'HTTPHeaders': {'x-amzn-requestid': '7KAA4EP6YH5ECI64G8ME57SEKT2S1P0BD2XCVC7SQRB6QMUL6T7S'},
                      'HTTPStatusCode': 200,
                      'RequestId': '7KAA4EP6YH5ECI64G8ME57SEKT2S1P0BD2XCVC7SQRB6QMUL6T7S',
                      'RetryAttempts': 0}}
Traceback (most recent call last):
  File "/Users/okokes/git/[...]/repro.py", line 13, in <module>
    assert data["Contents"][0]["Key"].startswith(data["Prefix"]), data
AssertionError: {'ResponseMetadata': {'RequestId': '7KAA4EP6YH5ECI64G8ME57SEKT2S1P0BD2XCVC7SQRB6QMUL6T7S', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '7KAA4EP6YH5ECI64G8ME57SEKT2S1P0BD2XCVC7SQRB6QMUL6T7S'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Contents': [{'Key': 'foo%2Fbar/data', 'LastModified': datetime.datetime(2022, 6, 14, 15, 22, 54, tzinfo=tzutc()), 'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'Size': 0, 'StorageClass': 'STANDARD'}], 'Name': 'foo', 'Prefix': 'foo/bar', 'MaxKeys': 1000, 'EncodingType': 'url', 'KeyCount': 1}

---

Here's a different repro with a directory listing.

from moto import mock_s3
import boto3

from pprint import pprint

with mock_s3():
    client = boto3.client("s3", region_name="us-east-1")
    client.create_bucket(Bucket="foo")
    client.put_object(Bucket="foo", Key="foo%2Fbar/data", Body=b"")

    data = client.list_objects_v2(Bucket="foo", Prefix="foo%2Fbar", Delimiter="/")
    pprint(data)
    assert data["CommonPrefixes"] == [{'Prefix': 'foo%2Fbar/'}], data["CommonPrefixes"]

I tried the same thing (manually, no repro here) with AWS S3 and found the behaviour to match 3.1.11.

---

One last repro: here's where it gets confusing - Delimiter='/'-style listing looks like it's listing a nested directory, but that goes against the principle of that functionality:

from moto import mock_s3
import boto3

from pprint import pprint

with mock_s3():
    client = boto3.client("s3", region_name="us-east-1")
    client.create_bucket(Bucket="foo")
    client.put_object(Bucket="foo", Key="foo%2Fbar/data", Body=b"")
    client.put_object(Bucket="foo", Key="foo/bar/data", Body=b"")

    data = client.list_objects_v2(Bucket="foo", Delimiter="/")
    pprint(data["CommonPrefixes"])

Which yields

$ python3 repro2.py
[{'Prefix': 'foo/bar/'}, {'Prefix': 'foo/'}]

---

Looking at the commits between 3.1.11 and 3.1.12, it would seem like this was the offending pull request - #5181 - but I haven't bisected this.

Let me know if you want more repros or details.

Thanks!

[kokes]

cc @pinzon who worked on the two commits related to this

[pinzon]

thanks @kokes for the detailed review and reproducer, I'll take a look.

[bblommers]

Thanks for raising this @kokes! This should now be fixed as of moto >= 3.1.14.dev15 - please let us know if you're running into any other issues.

[kokes]

@bblommers wonderful, thanks for the quick fix, I can confirm the dev version fixes my issue.