Introduce default_encoding parameter to [set|autodetect] the encoding if the charset is missing from the headers

[deedy5]

PR summary:

1. Added default_encoding parameter
2. Updated github actions

---

Using the default encoding

When requests are made to a site without explicit character set information from the server, but the encoding is known, it's advisable to explicitly set the default encoding.

from curl_cffi import requests

resp = requests.get(..., default_encoding="shift-jis")
print(response.encoding)  # Shows charset given in the Content-Type charset, or else "shift-jis".
print(response.text)  # The text decoded with the Content-Type charset, or using "shift-jis".

Using character set auto-detection

When the server doesn't provide character set information reliably, and the encoding is unknown, enabling auto-detection allows for a best-effort guess when converting bytes to text. To activate auto-detection, set the default_encoding parameter to a function that takes input bytes and returns the appropriate character set for decoding.

Let's take a look at autodetection using charset-normalizer

from curl_cffi import requests
from charset_normalizer import detect

def autodetect(content):
    return detect(content).get("encoding")

# Using a session with character-set autodetection enabled.
s = requests.Session(default_encoding=autodetect)
resp = s.get(...)
print(resp.encoding)  # Shows charset given in the Content-Type charset, or else autodetect() result.
print(resp.text)  # The text decoded with the Content-Type charset, or encoding, detected via autodetect.

[T-256]

fyi httpx uses utf-8 with errors=replace as default fallback. it uses charset_normalizer as optional feature.

[deedy5]

requests - fallback to auto-detected encoding
https://github.com/psf/requests/blob/8dd3b26bf59808de24fd654699f592abf6de581e/src/requests/models.py#L924
https://github.com/psf/requests/blob/8dd3b26bf59808de24fd654699f592abf6de581e/src/requests/models.py#L790

httpx
https://github.com/encode/httpx/blob/392dbe45f086d0877bd288c5d68abf860653b680/httpx/_models.py#L587

[T-256]

requests - fallback to auto-detected encoding

Needs decision. (I prefer httpx way, to not force new dependency.)

[deedy5]

httpx PR with useful information
encode/httpx#2165

[yifeikong]

I also prefer to add this as an optional feature.

[deedy5]

@yifeikong Update the documentation where you see fit, I've added a description in the first post.

[T-256]

Nice, Thank you!

[deedy5]

The problem in guthub workflow check Build bdist wheels and test (macos-12) occurred when installing pipx.

curl_cffi/.github/workflows/build-and-test.yaml

Line 70 in 418e452

which pipx || brew install pipx && pipx ensurepath

But pipx already installed:
macos-12
macos-14
So I removed that action and all works.

[coletdjnz]

I recommend splitting out unrelated changes into another PR(s), as it helps with review (and for downstream users to audit)

[deedy5]

Split the PR into several, in this one I left only adding the default_encoder parameter and updating the github actions to remove the bug in the tests.

[yifeikong]

Generally, I think the encoding/charset properies should be kept, other wise it would be a breaking change, which should be introduced only when the major version bumps.

You can add new properties or change the underlying implementation, though.

Thanks for fixing the CI!

[deedy5]

@yifeikong 🏁