"Failed to parse headers" warning logged when reading S3 object

[timb07]

Since upgrading botocore to >= 1.11.0, the following warning is logged when reading S3 objects:

Failed to parse headers (url=https://my-bucket.s3.ap-southeast-2.amazonaws.com:443/myfile): Unknown, unparsed data: [<http.client.HTTPMessage object at 0x7fdae5f81d30>]
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 399, in _make_request
    assert_header_parsing(httplib_response.msg)
  File "/home/user/.local/lib/python3.5/site-packages/urllib3/util/response.py", line 66, in assert_header_parsing
    raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)
urllib3.exceptions.HeaderParsingError: Unknown, unparsed data: [<http.client.HTTPMessage object at 0x7fdae5f81d30>]

The code to demonstrate this uses django-storages, and is running on an EC2 instance that has role-based access to the S3 bucket in question:

from storages.backends.s3boto3 import S3Boto3Storage

storage = S3Boto3Storage(region_name="ap-southeast-2", bucket_name="my-bucket")
fh = storage.open("myfile", "rb")
b = fh.read()

It seems that this is caused by the botocore upgrade to 1.11.0, in particular #1495 which uses a newer (non-vendored) version of urllib3; that later version adds a header parsing check, and logs a warning (but continues) if the check fails.

I would think that the official Python client for S3 shouldn't be logging warnings. :p

[jamesls]

Thanks for digging into this, it looks like you have a PR for urllib3 (urllib3/urllib3#1439) that fixes this issue. Once this fix is in urllib3 we can update our version range to pull in this latest version.

[timb07]

I agree that all botocore needs to do is require an updated version of urllib3 (once the PR is accepted and released).

Just by way of information for anyone reading this issue, the problem only occurs when the content-type of the object requested from S3 is message/*; for example message/rfc822 was the type that specifically triggered the issue for me. There's more details in urllib3/urllib3#1438.

[timb07]

This is now fixed in release 1.12.28.