Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix "�� is not JSON serializable" on Python 3 #382

Merged
merged 3 commits into from Mar 14, 2020

Conversation

hugovk
Copy link
Member

@hugovk hugovk commented Mar 13, 2020

Fixes #258.

@hugovk hugovk added bug Something isn't working changelog: Fixed For any bug fixes labels Mar 13, 2020
@hugovk hugovk merged commit 657d6c7 into ultrajson:master Mar 14, 2020
@hugovk hugovk deleted the 258-��-is-not-JSON-serializable branch March 14, 2020 17:50
adrianeboyd added a commit to adrianeboyd/srsly that referenced this pull request Nov 4, 2020
Include upstream modifications:

* Removed generic serialization of objects/iterables: ultrajson/ultrajson@53f85b1
* Fix "�� is not JSON serializable" on Python 3: ultrajson/ultrajson#382
adrianeboyd added a commit to adrianeboyd/srsly that referenced this pull request Nov 4, 2020
Include upstream modifications:

* Removed generic serialization of objects/iterables: ultrajson/ultrajson@53f85b1
* Fix "�� is not JSON serializable" on Python 3: ultrajson/ultrajson#382
adrianeboyd added a commit to explosion/srsly that referenced this pull request Nov 10, 2020
Include upstream modifications:

* Removed generic serialization of objects/iterables: ultrajson/ultrajson@53f85b1
* Fix "�� is not JSON serializable" on Python 3: ultrajson/ultrajson#382
adrianeboyd added a commit to adrianeboyd/srsly that referenced this pull request Nov 10, 2020
Include upstream modifications:

* Removed generic serialization of objects/iterables: ultrajson/ultrajson@53f85b1
* Fix "�� is not JSON serializable" on Python 3: ultrajson/ultrajson#382
@JustAnotherArchivist
Copy link
Collaborator

I just came across this and am wondering what the error handler string "~E~" is. I think that would just throw a LookupError when any encoding error occurs (cf. '\ud800'.encode('utf-8', '~E~') and codecs.lookup_error('~E~')). Is that intentional? Why not just strict, which raises a UnicodeEncodeError instead? (Yeah, I realise this was two years ago.)

@hugovk
Copy link
Member Author

hugovk commented Apr 13, 2022

It was probably taken from https://stackoverflow.com/a/46202119/724176:

in this example, this will replace invalid characters/data with the string "~E~"

But I also see this PR didn't add any test case with that string, so we could certainly improve that and re-evaluate this!

@JustAnotherArchivist
Copy link
Collaborator

Yeah, about that...

import ujson

class A:
	def __repr__(self):
		return '\ud800'

ujson.dumps(A())

→ Segmentation fault

I'm not sure whether that replacement was ever a thing (couldn't find it anywhere in the Python docs, not even back in Python 3.0), but it definitely isn't now. I guess that belongs in the 'wrong Stack Overflow answers' cabinet.

This is similar to #522, and actually I realise now that there are more issues with that string conversion as well. Will follow up with another PR.

@hugovk
Copy link
Member Author

hugovk commented Apr 13, 2022

Thanks!

JustAnotherArchivist added a commit to JustAnotherArchivist/ultrajson that referenced this pull request Apr 17, 2022
Errors during `__repr__` itself as well as ones during the conversion to a bytes object were not handled, resulting in NULL pointer dereferencing.

Cf. ultrajson#382
JustAnotherArchivist added a commit to JustAnotherArchivist/ultrajson that referenced this pull request Apr 17, 2022
Errors during `__repr__` itself as well as ones during the conversion to a bytes object were not handled, resulting in NULL pointer dereferencing.

Cf. ultrajson#382
bwoodsend pushed a commit that referenced this pull request Apr 18, 2022
Errors during `__repr__` itself as well as ones during the conversion to a bytes object were not handled, resulting in NULL pointer dereferencing.

Cf. #382
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working changelog: Fixed For any bug fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

repr output problem: �� is not JSON serializable
2 participants