Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3Boto3Storage.open(name, "r") doesn't handle newlines properly #1351

Closed
craigds opened this issue Feb 13, 2024 · 2 comments · Fixed by #1381
Closed

S3Boto3Storage.open(name, "r") doesn't handle newlines properly #1351

craigds opened this issue Feb 13, 2024 · 2 comments · Fixed by #1381

Comments

@craigds
Copy link
Contributor

craigds commented Feb 13, 2024

With S3Boto3Storage, opening a file in text mode ("r") doesn't properly convert windows newlines ("\r\n") to "\n" as you'd expect. Neither does it handle \r as \n

@craigds craigds changed the title S3Boto3Storage.read() doesn't handle newlines properly S3Boto3Storage.open(name, "r") doesn't handle newlines properly Feb 13, 2024
@jschneier
Copy link
Owner

Thanks for opening an issue. When you say "as you expect" can you link me to docs where the expected behavior is to change the file contents in that way?

@craigds
Copy link
Contributor Author

craigds commented Feb 14, 2024

Thanks for your reply.

This is the default behaviour of text-mode files in Python: https://docs.python.org/3/library/functions.html#open

newline determines how to parse newline characters from the stream. It can be None, '', '\n', '\r', and '\r\n'. It works as follows:

  • When reading input from the stream, if newline is None, universal newlines mode is enabled. Lines in the input can end in '\n', '\r', or '\r\n', and these are translated into '\n' before being returned to the caller. If it is '', universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated.

  • When writing output to the stream, if newline is None, any '\n' characters written are translated to the system default line separator, os.linesep. If newline is '' or '\n', no translation takes place. If newline is any of the other legal values, any '\n' characters written are translated to the given string.

Django's builtin storage classes also adhere to this API, which is how we discovered this discrepancy - when changing from a django-builtin storage backend to a django-storages one, the newlines in one of our tests started playing up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants