File is closed after s3 upload

[stumitchell]

hi I am seeing this issue with your library

jschneier/django-storages#382

[stumitchell]

class CustomS3Storage(S3Storage):

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to boto3 it wrongly closes
        the file upon upload where as the storage backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified size
        content_autoclose = self.new_temporary_file()

        # Write our original content into our copy that will be closed by boto3
        content_autoclose.write(content.read())
        content_autoclose.file = content.file

        # Upload the object which will auto close the content_autoclose instance
        result = super()._save(name, content_autoclose)

        # Cleanup if this is fixed upstream our duplicate should always close
        if not content_autoclose.closed:
            content_autoclose.close()

        return result

storage = CustomS3Storage()

Seems to help

[etianen]

I'll push out a patch release momentarily

[jakobkarlstrand]

I'll push out a patch release momentarily

Hi, is there a solution for this? I get ValueError I/O while uploading to S3

[etianen]

Can I have your full traceback?

…

On Tue, 19 Jul 2022 at 08:57, jaskop ***@***.***> wrote: I'll push out a patch release momentarily Hi, is there a solution for this? I get ValueError I/O while uploading to S3 — Reply to this email directly, view it on GitHub <#141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEKCHQPN2T664R3YFV6MLVUZNFNANCNFSM52ECLTPA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[jakobkarlstrand]

Actually, it worked now. However I now get an UnicodeDecodeError at /api/jour-reports

Traceback (most recent call last):
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 145, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 143, in _get_response
    response = response.render()
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/response.py", line 70, in rendered_content
    ret = renderer.render(self.data, accepted_media_type, context)
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/renderers.py", line 100, in render
    ret = json.dumps(
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/utils/json.py", line 25, in dumps
    return json.dumps(*args, **kwargs)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/utils/encoders.py", line 50, in default
    return obj.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 50: invalid start byte

[etianen]

I don't think this has anything to do with django-s3-storages. Your original problem was fixed in the latest release, as you've found. This seems to be some sort of unicode problem with data stored in a model somewhere.

…

On Tue, 19 Jul 2022 at 20:20, jaskop ***@***.***> wrote: Actually, it worked now. However I now get an UnicodeDecodeError at /api/jour-reports Traceback (most recent call last): File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 145, in _get_response response = self.process_exception_by_middleware(e, request) File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 143, in _get_response response = response.render() File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/template/response.py", line 105, in render self.content = self.rendered_content File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/response.py", line 70, in rendered_content ret = renderer.render(self.data, accepted_media_type, context) File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/renderers.py", line 100, in render ret = json.dumps( File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/utils/json.py", line 25, in dumps return json.dumps(*args, **kwargs) File ***@***.***/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 234, in dumps return cls( File ***@***.***/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File ***@***.***/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/utils/encoders.py", line 50, in default return obj.decode() UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 50: invalid start byte — Reply to this email directly, view it on GitHub <#141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEKCBWDJO4FSMA7U2OKO3VU35WXANCNFSM52ECLTPA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[stumitchell]

Hi I am still seeing this issue in version 0.13.7 I will continue to investigate

[etianen]

@stumitchell Cab you let me know if this is fixed for you in the latest commit to the master branch?

[jakobkarlstrand]

For me it was the following error:

I tried to return the file, or information of the file, in a response. Check if you are doing something similar