fix: utf8 -> utf16 decoding bug on surrogate pairs #1486
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This fixes protobufjs#1473 The custom utf8 -> utf16 decoder appears to be subtly flawed. From my reading it appears the chunking mechanism doesn't account for surrogate pairs at the end of a chunk causing variable size chunks. A larger chunk followed by a smaller chunk leaves behind garbage that'll be included in the latter chunk. It looks like the chunking mechanism was added to prevent stack overflows when calling `formCharCode` with too many args. From some benchmarking it appears putting utf16 code units in an array and spreading that into `fromCharCode` wasn't helping performance much anyway. I simplified it significantly. Here's a repro of the existing encoding bug in a fuzzing suite https://repl.it/@turbio/oh-no-our-strings#decoder.js
turbio
commented
Sep 10, 2020
| chunk[i++] = t; | ||
| else if (t > 191 && t < 224) | ||
| chunk[i++] = (t & 31) << 6 | buffer[start++] & 63; | ||
| else if (t > 239 && t < 365) { |
There was a problem hiding this comment.
My understanding of this function is that the passed in buffer is always an array of uint8s, each byte being a utf8 code unit. The condition t < 365 is a bit odd under this assumption, from profiling it actually forces v8 to deopt when t is compared to a non uint8 value. Alternatively I may have grossly misunderstood how this is expected to work.
|
Hi @turbio, Would it be possible to have a test that shows the problem included in the PR? Thank you! |
|
@alexander-fenster yup! added a test which fails on the old |
alexander-fenster
approved these changes
Oct 9, 2020
alexander-fenster
changed the title
masad-frost
added a commit
to replit/crosis
that referenced
this pull request
Dec 29, 2020
protobufjs/protobuf.js#1486 released in protobufjs 6.10.2 and updated in @replit/protocol in replit/protocol#9 and released in 0.2.15
This was referenced May 20, 2022
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes #1473
The custom utf8 -> utf16 decoder appears to be subtly flawed. From my reading it appears the chunking mechanism doesn't account for surrogate pairs at the end of a chunk causing variable size chunks. A larger chunk followed by a smaller chunk leaves behind garbage that'll be included in the latter chunk.
The chunking mechanism was added to prevent stack overflows when calling
formCharCodewith too many args. From some simple benchmarking it doesn't seem like putting utf16 code units in an array and spreading that intofromCharCodeis helping performance much anyway. I've removed the chunking code which simplifies it significantly and fixes the particular issue we've encountered.Here's a repro of the existing encoding bug in a fuzzing suite
https://repl.it/@turbio/oh-no-our-strings#decoder.js