Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Puma versions >= v5.0.3 throw Encoding::CompatibilityError: incompatible character encodings: US-ASCII and UTF-16LE #2583

Closed
jboler opened this issue Mar 24, 2021 · 8 comments · Fixed by #2585
Labels

Comments

@jboler
Copy link

jboler commented Mar 24, 2021

Describe the bug
We upgraded our Rails app from puma v3.12.6 to v5.2.2 and found that our CSV exports throw error:

Encoding::CompatibilityError: incompatible character encodings: US-ASCII and UTF-16LE

We progressively downgraded puma and found that v5.0.2 works but v5.0.3 exhibits the error.

Puma config:
Default config from rails new

To Reproduce
This is a clean Rails 6.1 app with only the controller action added in app/controllers/application_controller.rb and the puma version locked to v5.0.3. If you change puma to v5.0.2 the controller action works fine.

https://github.com/jboler/puma_bug

Desktop:

  • OS: MacOS 10.15.7 or Ubuntu 20.04
  • Puma Version >= 5.0.3
  • Ruby 2.7.2
@cjlarose cjlarose added the bug label Mar 24, 2021
@cjlarose
Copy link
Member

cjlarose commented Mar 24, 2021

Thanks for the detailed bug report @jboler!

I performed a git bisect to find the offending change and it points to b089768. @MSP-Greg any ideas?

My guess is that we're trying to concatenate a UTF-8 encoding string with a UTF-16LE encoding string in this case and that's what's failing.

@MSP-Greg
Copy link
Member

I can repo it, looking at it now. You mean the line that replaced several method calls? I should have seen that before, having already wrestled with another encoding issue...

str = part.bytesize.to_s(16) << line_ending << part << line_ending

@cjlarose
Copy link
Member

Yeah, my guess is that's the culprit, but I haven't played around with it.

@jboler
Copy link
Author

jboler commented Mar 24, 2021

Confirmed:

$ irb
2.7.2 :001 > 'x'.bytesize.to_s(16) << 'y'.force_encoding(Encoding::UTF_16LE)
Encoding::CompatibilityError (incompatible character encodings: US-ASCII and UTF-16LE)

@MSP-Greg
Copy link
Member

@jboler

I've got a patch, using Windows, so ran server in WSL2/Ubuntu, saved the file in Windows Edge (browser), the shell opened the file in Excel. Still looking at a couple of things...

@MSP-Greg
Copy link
Member

@jboler

Not sure if you can test it, but PR #2585 should fix the issue, and there's now a test for it.

I looked at several ways to fix the issue, this seemed the best. Ruby is lacking in methods to combine byte strings (and hence, ignore encoding)...

@jboler
Copy link
Author

jboler commented Apr 19, 2021

Thank you for fixing this! I've confirmed the fix works. Is there an ETA for the v5.2.3 release? Thanks again!

@MSP-Greg
Copy link
Member

@jboler

Thanks for testing it.

Myself and @wjordan are working on code to improve Puma's handling/assembly of response bodies.

I hate to ask, but PR #2595 should fix this issue, and also decrease the time Puma takes to assemble a much larger csv. If you have code that has a larger csv, I'd be interested to see whether there's a speed improvement.

But, a Puma speed improvement may be 'buried' by the time the app takes to respond.

Regardless, this can get fixed shortly...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants