StringIO buffered ContentFile S3 upload fails due to encoding error in botocore

[radekwlsk]

With changes in boto/botocore#1985 save to FileField where content is StringIO based ContentFile (initialised with string instead of bytes) now raises TypeError: Unicode-objects must be encoded before hashing as it tries to calculate MD5 sum on string where it expects bytes.

Already mentioned the issue in botocore
boto/botocore#1985 (comment)

As a possible solution django-storages could ensure that content is bytes at

File "storages/backends/s3boto3.py", line 451, in _save
    obj.upload_fileobj(content, ExtraArgs=params)

[jschneier]

Similar to boto I don't know how to properly detect the encoding. Given force_bytes is already used which assumes utf-8 I think it's reasonable to do that but it isn't strictly correct.

[LincolnPuzey]

This is a duplicate of issue #708 , which has been happening since before that change in botocore.

The fundamental problem is django-storages passes a file object open in string mode to obj.upload_fileobj(), but the boto3 docs specify it must be open in binary mode (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Object.upload_fileobj)

See my PR #911 which fixes this issue.

[jschneier]

Fixed by #911