Refactoring for potential performance improvements in loops

[adbar]

Experiments with ideas to potentially improve performance or code consistency without impacting readability (#111).

This PR:

1. defines caches and sets in cd.py
2. uses list comprehensions for language associations in cd.py
3. refactors duplicate code in md.py

Close #111

[codecov-commenter]

Codecov Report

Merging #113 (59d22e0) into master (59e48eb) will increase coverage by 0.04%.
The diff coverage is 91.30%.

@@            Coverage Diff             @@
##           master     #113      +/-   ##
==========================================
+ Coverage   86.19%   86.24%   +0.04%     
==========================================
  Files          11       11              
  Lines        1188     1185       -3     
==========================================
- Hits         1024     1022       -2     
+ Misses        164      163       -1     

Impacted Files	Coverage Δ
charset_normalizer/models.py	86.66% <83.33%> (+0.16%)	⬆️
charset_normalizer/md.py	87.82% <85.71%> (-0.04%)	⬇️
charset_normalizer/cd.py	95.91% <100.00%> (-0.03%)	⬇️
charset_normalizer/constant.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59e48eb...59d22e0. Read the comment docs.

[Ousret]

In quick tests, I did see a minor performance improvement. About 2 ms in average, that not bad.
There some lint errors to fix.

[Ousret]

Plus the tests show a warning.

=============================== warnings summary ===============================
tests/test_cli.py: 2767815 warnings
  /home/runner/work/charset_normalizer/charset_normalizer/charset_normalizer/models.py:231: DeprecationWarning: NotImplemented should not be used in a boolean context
    self._unicode_ranges = sorted(filter(None.__ne__, detected_ranges))

-- Docs: https://docs.pytest.org/en/stable/warnings.html

[adbar]

Hi @Ousret, I did the changes as requested, I'm not sure what the warning has to do with DeprecationWarning: NotImplemented.

[akx]

Did you measure that this makes things faster? For one, a common micro-optimization in Python is to alias global variables into locals within functions, since local variable access is faster than global variable access.

[Ousret]

Thanks for the adjustments @adbar , remain a few remarks, but I think this is about it.

[adbar]

@Ousret The linter throws an error since 8f46e15, I guess that List[str] was correct.

[Ousret]

More a limitation of mypy.

charset_normalizer/models.py:229: error: List comprehension has incompatible type List[Optional[str]]; expected List[str]

Don't get that you filtered out the None. type: ignore is more suited then. Or I missed something..?

[adbar]

I'm not so proficient with mypy but type: ignore could do the trick yes.

[adbar]

@Ousret @akx Thanks for the feedback, I think that's all for now.

@Ousret Could you please decide on the mypy issue?

[Ousret]

LGTM