Fix Issue #708: Save files with string content #911

LincolnPuzey · 2020-07-27T05:10:14Z

Wraps file objects open in string mode so they produce bytes and can be passed to obj.upload_fileobj()

LincolnPuzey · 2020-07-27T08:35:35Z

This fixes #708, #843.

@jschneier Can I get a review on this?

You can see my new tests that reproduce the issue, fail in the build on commit c3d00d9.

The latest build is just flake8 failing which is also happening on master.

LincolnPuzey · 2020-07-27T08:41:38Z

storages/backends/s3boto3.py

@@ -440,6 +440,10 @@ def _save(self, name, content):
        name = self._normalize_name(cleaned_name)
        params = self._get_write_parameters(name, content)

+        # wrap content so read() always returns bytes. This is required for passing it
+        # to obj.upload_fileobj() or self._compress_content()
+        content = ReadBytesWrapper(content)


This is the core part of the fix. This approach has the benefit of not making any extra read() or seek() calls to the file object.

LincolnPuzey · 2020-09-19T05:46:58Z

@jschneier Can I get this reviewed? Most of the diff is in tests. The new tests in S3Boto3StorageTestsWithMoto reproduce the original error and show its fixed.

LincolnPuzey · 2020-10-05T08:47:05Z

@jschneier Still waiting for a review

LincolnPuzey · 2020-12-18T00:45:34Z

Also fixes #961

wiktr · 2021-01-17T13:53:33Z

@LincolnPuzey I see there are conflicts, could you please resolve them?
@jschneier please take a look at your earliest convenience. I think it's worth resolving this and related tickets.

I hit the described issue using versions:

boto3==1.16.56
botocore==1.19.56
django-storages==1.11.1

Would be lovely if django storages worked with s3 and boto3 out of the box, without workarounds.

LincolnPuzey · 2021-01-18T02:38:04Z

I'm happy to resolve the conflicts if @jschneier or another maintainer can indicate they're willing to review and merge the PR.

pauloxnet · 2021-01-21T17:03:58Z

Thanks for the PR , it fix a problem we have in our projects.
I can image all maintainers are busy, but I hope one of them can review this PR and then merge.

dvf · 2021-07-19T01:08:48Z

Any news here folks? The docs use an example that breaks too:

mertcangokgoz · 2023-05-24T09:30:14Z

Any update?

hugo-reveni · 2023-08-14T10:30:08Z

@LincolnPuzey here is another one interested in this topic, as we're having problems with text files uploaded from Admin using django-import-export v3.2.0

Any merge prevision?

This makes .read() always return bytes. If .read() returns a string, it will be encoded to bytes before being returned. The encoding to use can be specified, otherwise will use the .encoding property of the original file, otherwise will use utf-8.

…per. This correctly handles file-like-objects that are open in text/string mode by converting to strings returned by read() to bytes, which is what upload_fileobj() requires. This is done before gzip compressing, so also removed force_bytes() call in _compress_content().

Add these tests in a new test class that uses moto. Remove old test for saving ContentFile Move test for detecting content-type to this new class. Add some more tests around this. Fix tests that fail because settings.AWS_STORAGE_BUCKET_NAME is now defined. Fix tests that fail because content is always wrapped. Fix test for gzipped file since that now only takes bytes.

jschneier · 2023-09-04T19:29:06Z

Thanks, moto is a big improvement. Seems boto has grown several more features in the last few years that necessitated a few more tweaks.

hugo-reveni · 2023-09-05T08:03:09Z

Thanks a lot @jschneier, we will wait for the next release with this improvement 👏🏽

jschneier · 2023-09-05T09:43:51Z

No need to wait; I released it already.

…

On Tuesday, September 5, 2023, Hugo Camino ***@***.***> wrote: Thanks a lot @jschneier <https://github.com/jschneier>, we will wait for the next release with this improvement 👏🏽 — Reply to this email directly, view it on GitHub <#911 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAREDWFH7G5NSXIOOTOIAMTXY3MEPANCNFSM4PIMAX3Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

hugo-reveni · 2023-09-05T10:37:25Z

@jschneier sorry I didn't find new releases in Github tags section like past ones but now I see it was in here. Thanks!

jschneier · 2023-09-05T15:26:53Z

Ah I forgot to apply the tag, fixed. Thanks.

…

On Sep 5, 2023, at 06:37, Hugo Camino ***@***.***> wrote: @jschneier <https://github.com/jschneier> sorry I didn't find new releases in Github tags section like past ones but now I see it was in here <https://pypi.org/project/django-storages/>. Thanks! — Reply to this email directly, view it on GitHub <#911 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAREDWHFCPNSM33N4BVV3BLXY36HDANCNFSM4PIMAX3Q>. You are receiving this because you were mentioned.

dvf · 2023-09-05T16:15:54Z

It's people like @jschneier that make OSS awesome. Thanks Josh ❤️

LincolnPuzey force-pushed the fix_708_save_string_content branch 2 times, most recently from cdf3180 to 0c64612 Compare July 27, 2020 08:12

LincolnPuzey marked this pull request as ready for review July 27, 2020 08:24

LincolnPuzey commented Jul 27, 2020

View reviewed changes

LincolnPuzey force-pushed the fix_708_save_string_content branch from 61e733b to b24e9a9 Compare August 18, 2020 00:19

LincolnPuzey force-pushed the fix_708_save_string_content branch 2 times, most recently from 641cbfd to ca9c466 Compare September 19, 2020 05:37

LincolnPuzey mentioned this pull request Dec 18, 2020

StringIO buffered ContentFile S3 upload fails due to encoding error in botocore #961

Closed

pauloxnet approved these changes Jan 21, 2021

View reviewed changes

jmbowman mentioned this pull request Feb 16, 2023

feat!: Update from boto to boto3 storage backend. openedx/edx-platform#31759

Merged

LincolnPuzey and others added 6 commits September 4, 2023 12:26

Add alternate .venv to gitignore

587a42a

Add moto as test dependency.

667a354

tweaks

1f45042

jschneier force-pushed the fix_708_save_string_content branch from ca9c466 to 1f45042 Compare September 4, 2023 19:00

jschneier added 2 commits September 4, 2023 15:08

add close to ReadBytesWrapper

7e9f18a

add readable to ReadBytesWrapper

577bb21

jschneier merged commit 19a15c2 into jschneier:master Sep 4, 2023
15 checks passed

This was referenced Sep 4, 2023

ContentFile with str content throws TypeError #708

Closed

Unicode-objects must be encoded before hashing - update from 1.8 to 1.9.1 #843

Closed

Mocking s3 in tests properly with moto #937

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Issue #708: Save files with string content #911

Fix Issue #708: Save files with string content #911

LincolnPuzey commented Jul 27, 2020 •

edited

LincolnPuzey commented Jul 27, 2020

LincolnPuzey Jul 27, 2020 •

edited

LincolnPuzey commented Sep 19, 2020

LincolnPuzey commented Oct 5, 2020

LincolnPuzey commented Dec 18, 2020

wiktr commented Jan 17, 2021

LincolnPuzey commented Jan 18, 2021

pauloxnet commented Jan 21, 2021

dvf commented Jul 19, 2021

mertcangokgoz commented May 24, 2023

hugo-reveni commented Aug 14, 2023

jschneier commented Sep 4, 2023

hugo-reveni commented Sep 5, 2023

jschneier commented Sep 5, 2023 via email

hugo-reveni commented Sep 5, 2023

jschneier commented Sep 5, 2023 via email

dvf commented Sep 5, 2023

Fix Issue #708: Save files with string content #911

Fix Issue #708: Save files with string content #911

Conversation

LincolnPuzey commented Jul 27, 2020 • edited

LincolnPuzey commented Jul 27, 2020

LincolnPuzey Jul 27, 2020 • edited

Choose a reason for hiding this comment

LincolnPuzey commented Sep 19, 2020

LincolnPuzey commented Oct 5, 2020

LincolnPuzey commented Dec 18, 2020

wiktr commented Jan 17, 2021

LincolnPuzey commented Jan 18, 2021

pauloxnet commented Jan 21, 2021

dvf commented Jul 19, 2021

mertcangokgoz commented May 24, 2023

hugo-reveni commented Aug 14, 2023

jschneier commented Sep 4, 2023

hugo-reveni commented Sep 5, 2023

jschneier commented Sep 5, 2023 via email

hugo-reveni commented Sep 5, 2023

jschneier commented Sep 5, 2023 via email

dvf commented Sep 5, 2023

LincolnPuzey commented Jul 27, 2020 •

edited

LincolnPuzey Jul 27, 2020 •

edited