Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix handling of surrogates on decoding #550

Merged
merged 1 commit into from Jun 16, 2022

Conversation

JustAnotherArchivist
Copy link
Collaborator

This implements surrogate handling on decoding as it is in the standard library. Lone escaped surrogates and any raw surrogates in the input result in surrogates in the output, and escaped surrogate pairs get decoded into non-BMP characters. Note that raw surrogate pairs get treated differently on platforms/compilers with 16-bit wchar_t, e.g. Microsoft Windows.

Before this, well-formed JSON using surrogates only in encoded surrogate pairs was decoded correctly, but anything else containing surrogates would produce unexpected results or errors. This change makes ujson's decoding compatible with the standard library's.

Unfortunately, platforms with a 16-bit wchar_t cannot handle raw surrogate pairs correctly because PyUnicode_FromWideChar decodes those. That issue was always present (but untested) and is left unfixed here. I'm not sure it is possible to handle it correctly without completely changing the decoding approach, e.g. producing UTF-8 and using PyUnicode_FromStringAndSize (which, by the way, is what orjson does, though it rejects lone surrogates).

This implements surrogate handling on decoding as it is in the standard library. Lone escaped surrogates and any raw surrogates in the input result in surrogates in the output, and escaped surrogate pairs get decoded into non-BMP characters. Note that raw surrogate pairs get treated differently on platforms/compilers with 16-bit `wchar_t`, e.g. Microsoft Windows.
@JustAnotherArchivist
Copy link
Collaborator Author

Some quick benchmarks (main = 4ac30c9 vs e0e5db9):

  • Encoded surrogates: ujson.loads('"' + '\\uD83D\\uDCA9' * 1000 + '"') – slightly faster (16.8 μs with main, 16.4 μs with this)
  • Raw surrogates: ujson.loads(('"' + '\uD83D\uDCA9' * 1000 + '"').encode('utf-8', 'surrogatepass')) – slightly faster (6.55 μs with main, 6.30 μs with this)
  • ujson.loads('"' + 'a\\uD83D\\uDCA9' * 1000 + '"') (ASCII + encoded surrogates) – slightly faster (20.0 μs with main, 19.7 μs with this)
  • ASCII: ujson.loads('"' + 'a' * 10000 + '"') – same (20.7 μs)
  • UTF-8: ujson.loads('"' + '\U0001F4A9' * 1000 + '"') – same (6.4 μs)

The raw surrogate timing is a bit surprising to me since nothing in that code path changed. But it is reproducibly faster.
Obviously, since the lone surrogates were broken in various ways, it's not possible to compare that.

@hugovk hugovk added the changelog: Fixed For any bug fixes label Jun 16, 2022
@hugovk hugovk merged commit b47c3a7 into ultrajson:main Jun 16, 2022
adrianeboyd added a commit to adrianeboyd/srsly that referenced this pull request Jul 20, 2022
adrianeboyd added a commit to adrianeboyd/srsly that referenced this pull request Jul 20, 2022
adrianeboyd added a commit to adrianeboyd/srsly that referenced this pull request Jul 20, 2022
adrianeboyd added a commit to explosion/srsly that referenced this pull request Jul 20, 2022
adrianeboyd added a commit to adrianeboyd/srsly that referenced this pull request Jul 18, 2023
adrianeboyd added a commit to adrianeboyd/srsly that referenced this pull request Jul 18, 2023
adrianeboyd added a commit to adrianeboyd/srsly that referenced this pull request Jul 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
changelog: Fixed For any bug fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants