Migrate to the next version of Python requests when released

[potiuk]

Hello Maintainers,

I am a PMC member of Apache Airflow, and I wanted to give you a bit of heads-up with rather important migration to the upcoming version of requests library in your Python release.

Since you are using requests library in your project, you are affected. It's not as bad for you as you seem to use requests library optionally (when some extras are used) and LGPL dependencies can be used if they are not optional, however it would be safer if you use non-LGPL dependency also here).

As discussed at length in https://issues.apache.org/jira/browse/LEGAL-572 we found out that the 'chardetlibrary used byrequests` library was a mandatory dependency to requests and since it has LGPL licence, we should not release any Apache Software with it.

Since then (and since in Airflow we rely on requests heavily) we have been working with the requests maintainers and "charset-normalizer" maintainer to make it possible to replace chardet with MIT-licensed charset-normalizer instead so that requests library can be used in Python releases by Apache projects.

This was a bumpy road but finally the PR by @ashb has been merged: psf/requests#5797 and we hope soon a new version of requests library will be released.

This is just a heads-up. I will let you know when it is released, but I have a kind requests as well - I might ask the maintainers to release a release candidate of requests and maybe you could help to test it before it is released, that would be some re-assurance for the maintainers of requests who are very concerned about stability of their releases.

Let me know if you need any more information and whether you would like to help in testing the candidate when it is out.

[potiuk]

superset-github-bot bot added the Polidea label 33 minutes ago

I think the Bot needs some adjustment ;)

[potiuk]

Requests 2.26.0 released without the mandatory chardet dependency:

https://pypi.org/project/requests/2.26.0/

Instead of chardet, use the MIT-licensed charset_normalizer for Python3 to remove license ambiguity for projects bundling requests. If chardet is already installed on your machine it will be used instead of charset_normalizer to keep backwards compatibility. (#5797)

You can also install chardet while installing requests by specifying [use_chardet_on_py3] extra as follows:

shell pip install "requests[use_chardet_on_py3]"

Python2 still depends upon the chardet module.

[kivhub]

If chardet is already installed on your machine it will be used instead of charset_normalizer to keep backwards compatibility.

How to force using chardet in requests, please?

With request using charset-normalizer I am getting this error when calling Python via User Defined Transform in SAP BODS:

File "EXPRESSION", line 6, in <module>
File "c:\program files\python39\lib\site-packages\requests\__init__.py", line 48, in <module>
from charset_normalizer import __version__ as charset_normalizer_version
File "c:\program files\python39\lib\site-packages\charset_normalizer\__init__.py", line 11
SyntaxError: Non-ASCII character '\xd1' in file c:\program files\python39\lib\site-packages\charset_normalizer\__init__.py on
line 12, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details.

I am not able to define a source code encoding by placing magic comment into the source files (either as first or second line in the file) because the app probably modifies the script by itself.
After unistalling charset-normalizer (chardet installed) I am getting this error:

File "EXPRESSION", line 6, in <module>
File "c:\program files\python39\lib\site-packages\requests\__init__.py", line 134, in <module>
from . import packages
File "c:\program files\python39\lib\site-packages\requests\packages.py", line 15, in <module>
locals()[package] = __import__(package)
File "c:\program files\python39\lib\site-packages\idna\__init__.py", line 2, in <module>
from .core import (
File "c:\program files\python39\lib\site-packages\idna\core.py", line 5, in <module>
from typing import Union, Optional
ImportError: No module named typing.

[potiuk]

See the comment just above:

pip install "requests[use_chardet_on_py3]"

or just install chardet manually. Once charder is installed it is automatically used instead of charset-normalizer

[potiuk]

You can also open an issue for charset nomarlizer https://github.com/ousret/charset_normalizer - @Ousret is usually very responsive and releases new versions quickly and often. I saw several releases recently so it might well be the issue is already solved in latest versions

[potiuk]

However this error is not a problem of charset normalizer. The error indicates that you have problem with your default system encoding on Windows:

Please set the env variable as described here to get rid of the error:

https://dev.to/methane/python-use-utf-8-mode-on-windows-212i

Summary: Set the PYTHONUTF8=1 environment variable.

[kivhub]

See the comment just above:

pip install "requests[use_chardet_on_py3]"

or just install chardet manually. Once charder is installed it is automatically used instead of charset-normalizer

Thank you very much for your quick response. I uninstalled charset-normalizer, chardet and requests and installed requests with this command. The charset-normalizer was installed anyway back and I got the same error. Requests 2.26.0 use charset-normalizer (2.0.6) although there is chardet (4.0.0) in the site-packages folder.

[kivhub]

However this error is not a problem of charset normalizer. The error indicates that you have problem with your default system encoding on Windows:

Please set the env variable as described here to get rid of the error:

https://dev.to/methane/python-use-utf-8-mode-on-windows-212i

Summary: Set the PYTHONUTF8=1 environment variable.

I set this environment variable (without reboot on Windows 2016 Server), I can see it in the Windows command prompt with 'set' command but unfortunately I am still getting the same error (Non-ASCII character '\xd1' in file c:\program files\python39\lib\site-packages\charset_normalizer_init_.py on line 12, but no encoding declared).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.

[rusackas]

It looks like we're on requests 2.30.0 as of this PR, but were already on 2.26.0 before that.

However... chardet 5.1.0 seems to still exist in integration.txt (via tox) and in development.txt (via tabulator). I'm not sure if this usage represents a problem, but I'll try to ask around a bit more.

[rusackas]

Looks like we're on requests 2.31.0 since quite some time ago. The chardet situation has not changed. Assigning this to @john-bodley and @mistercrunch who might have solid opinions on versioning/licensing here.

[mistercrunch]

Mmmh, why is pip-compile still brining chardet then?

[mistercrunch]

Oh here it is, in integration.txt it's via tox and in development.txt, it's via tabulator -> tableschema -> apache-superset[hive]. It's like the glitter of python libraries, can't shake that one off...

I think it's ok to have LGPL on optional dependencies or in CI or dev-related deps, but still not great/desirable. There may be ASF material that says that more officially so you don't have to take my word for it.