Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Fix too small JSON payload triggering md with high ratio #59

Merged
merged 1 commit into from Jul 14, 2021

Conversation

Ousret
Copy link
Owner

@Ousret Ousret commented Jul 14, 2021

Close #58~

To reproduce:

from charset_normalizer import from_bytes

results = from_bytes(b"{}").best()

Expected encoding = 'ASCII'; Got 'UTF-16 (Variant)'.

Why does this happen?

Because the first pass consists of assessing the "mess/chaos" ratio, and one of the detection plugins was unadjusted for a small JSON payload.

@Ousret Ousret added the bug Something isn't working label Jul 14, 2021
@codecov-commenter
Copy link

codecov-commenter commented Jul 14, 2021

Codecov Report

Merging #59 (bde9973) into master (53b2dab) will increase coverage by 0.09%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #59      +/-   ##
==========================================
+ Coverage   84.49%   84.59%   +0.09%     
==========================================
  Files          11       11              
  Lines        1058     1058              
==========================================
+ Hits          894      895       +1     
+ Misses        164      163       -1     
Impacted Files Coverage Δ
charset_normalizer/md.py 87.05% <100.00%> (ø)
charset_normalizer/models.py 86.97% <0.00%> (+0.52%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53b2dab...bde9973. Read the comment docs.

@Ousret Ousret merged commit 595514e into master Jul 14, 2021
@Ousret Ousret deleted the patch-json-detect branch July 14, 2021 19:16
@potiuk
Copy link

potiuk commented Jul 14, 2021

Great to get i addressed so quickly !

@Ousret
Copy link
Owner Author

Ousret commented Jul 14, 2021

With a change that big, comes to some responsibilities. Will make sure that this change is worth it.

@Ousret Ousret mentioned this pull request Jul 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Wrong encoding detected for empty JSON response
3 participants