Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set OutputStream.write_buffer's default buffer to binmode #439

Merged
merged 3 commits into from Mar 14, 2020

Conversation

henkesn
Copy link
Contributor

@henkesn henkesn commented Feb 19, 2020

Fixes #438

@coveralls
Copy link

coveralls commented Feb 19, 2020

Coverage Status

Coverage increased (+0.002%) to 95.769% when pulling 66324a7 on henkeinfo:binary-outstream-buffer into 8d91d00 on rubyzip:master.

Copy link
Member

@hainesr hainesr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Just one comment/suggestion from me.

@@ -57,7 +57,7 @@ def open(file_name, encrypter = nil)
end

# Same as #open but writes to a filestream instead
def write_buffer(io = ::StringIO.new(''), encrypter = nil)
def write_buffer(io = ::StringIO.new('').binmode, encrypter = nil)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we pass an IO in here that hasn't been set to binmode? Do we still see the behaviour you're trying to avoid? This change only sets a newly created StringIO to binmode. Is this a problem?

How about:

Suggested change
def write_buffer(io = ::StringIO.new('').binmode, encrypter = nil)
def write_buffer(io = ::StringIO.new(''), encrypter = nil)
io.binmode if io.respond_to?(:binmode)

Which would match behaviour in ::Zip::File.open_buffer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found another location where the binmode would be set if supported. I think this is a cleaner solution:
https://github.com/henkeinfo/rubyzip/blob/5ce4e13ddd33443f01fa33c8208423b76d99fce3/lib/zip/file.rb#L151
Yes, you're right with your comment. I rewrote the write_buffer to behave like the source above and also fixed the same thing in yet another method.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I spotted that when I was looking through File and wondered why it was inconsistent. Thanks for the catch.

This looks good to me.

Copy link
Member

@hainesr hainesr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a good patch to me.

@jdleesmiller I can't merge this myself, obvs, so does it look OK to you too?

Copy link
Member

@jdleesmiller jdleesmiller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch! A few comments inline.

In my case this is UTF-8, which produces encoding errors in encoding sensitive use cases.

I'd be interested to see what the actual use case was that caused the error --- if we can get a test that failed (pre-fix) in the way that motivated you to open the issue, I think that will be more likely to catch future regressions.

There have been binmode problems in the past, notably #119 / #201, which resulted in this line:

rubyzip/lib/zip/file.rb

Lines 150 to 151 in 8d91d00

# https://github.com/rubyzip/rubyzip/issues/119
io.binmode if io.respond_to?(:binmode)

However, all the tests still pass even if that line is deleted, so I feel like we have already have some gaps in this area.

lib/zip/file.rb Outdated Show resolved Hide resolved
test/output_stream_test.rb Outdated Show resolved Hide resolved
henkesn and others added 2 commits March 2, 2020 11:00
Co-Authored-By: John Lees-Miller <jdleesmiller@gmail.com>
@henkesn
Copy link
Contributor Author

henkesn commented Mar 2, 2020

Thank you for your feedback @jdleesmiller. My use case was an extension in my code where i filtered empty strings before storing to a database (String#presence). I already faced some issues with ActiveRecord in the past, when trying to save invalid UTF8 strings even for BLOB columns. But that's already some time ago and I neither know how to reproduce nor if that issues persist with newer versions.
Regarding the next line

rubyzip/lib/zip/file.rb

Lines 150 to 151 in 8d91d00

# https://github.com/rubyzip/rubyzip/issues/119
io.binmode if io.respond_to?(:binmode)

perhaps this line now is also superfluous because it internally calls OutputStream#write_buffer with my fix. But the yield call is done after this line and before the OutputStream#write_buffer, so removing this line potentially could break things.

Copy link
Member

@jdleesmiller jdleesmiller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, thanks for the updates. Looks good.

I think you're right that removing the existing binmode line would change what the user might get in the yield, so it's probably best to keep it.

@@ -6,6 +6,8 @@ class ZipOutputStreamTest < MiniTest::Test
TEST_ZIP = TestZipFile::TEST_ZIP2.clone
TEST_ZIP.zip_name = 'test/data/generated/output.zip'

ASCII8BIT = 'ASCII-8BIT'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This constant can now be removed, but I'll tidy it up post-merge.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx!

@jdleesmiller jdleesmiller merged commit b231b28 into rubyzip:master Mar 14, 2020
jdleesmiller added a commit that referenced this pull request Mar 14, 2020
jdleesmiller added a commit that referenced this pull request Mar 14, 2020
@henkesn henkesn deleted the binary-outstream-buffer branch March 16, 2020 10:46
@kitsunde
Copy link

This appears to have affected an upstream gem because of the encoding changing. weshatheleopard/rubyXL#377

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OutputStream.write_buffer returns IOString with default encoding
5 participants