Upper version limits in dependencies

[oberstet]

requests currently has a bunch of version restrictions:

• chardet>=3.0.2,<5
• idna>=2.5,<3
• urllib3>=1.21.1,<1.27

In particular with idna and urllib3, which are widely used themself, whenever another package triggers installation of an open-ended version of these libraries, a subsequent install of requests then will lead to warnings or errors the like

requests 2.25.1 requires idna<3,>=2.5, but you'll have idna 3.1 which is incompatible.

Are there actual reasons to limit the upper versions? Could these be lifted?

Putting version upper limits into widely used libraries leads to major headaches for downstream project. Even when following a dedicated-venv-per-app model, if that app has multiple dependencies that in turn depend on above, this is tricky to fix (order dependency, finding an intersecting version in the first place, ..). fwiw, I've spent countless hours on fixing fallout from such scenarios ...

[naorlivne]

Seems related to #5710, in that case the reasoning for version limits is that version 3.x of idna doesn't support python2.x anymore which leads to why it's proposed fix in #5711 has version markers with the upper limit being different for py3 & py2

[oberstet]

ah yes, rgd idna that seem like the same issue. your PR would "fix" idna .. it bumps the upper limit of idna on py 3 to <4 - which is practically ok. but why limit it at all (on py3)?

requests upper limits the version of idna - in an effort to still run on python 2? even if so, that should be left to python version markets in idna IMO, not requests. eg if requests v3 would run on python 2, why would idna want to limit to idna<2? I am assuming the python version being the only reason in idna for the upp limit.

similar holds true for urllib .. why the upper limit?

[naorlivne]

As far as I understad it it's so not to make too big a change untested, personally I'm with you that this should be left to unittesting to confirm but that's seems like the spirit of things in #5711 discussion which is a valid reasoning

[sigmavirus24]

Are there actual reasons to limit the upper versions?

Yes there are. I believe they're even documented - both in requests documentation as well as numerous issues complaining about this. There's only so much time the maintainers have these days. Version limits communicate to the users what we are able to support at the time of release as well as roughly what versions are known to work well. (This is the shortest answer.) Other projects play fast and loose with their dependencies because they have people who have more time on their hands and can react quickly to breaking releases in their dependencies. We don't

[oberstet]

thanks for explaining reasons and project resource limits! I understand.

guess we'll have to deal with it. for me, with our "main app" having a dependency DAG with 130 entries, and requests and idna on different paths in the DAG, that means a bit of pain, but hey, I should contribute not complain;)

[walchko]

Thanks for your all of your hard work and I really enjoy requests. But continuing to support Python 2 prolongs this problem of everyone transitioning off of it. The world has had more than enough time to migrate, wasting time being compatible with an unsupported language is a waste of your limited time.

[OlafvdSpek]

Can't these version limits be handled by package managers, such that incompatible versions won't be installed?
A runtime warning like this seems sub-optimal.

[karolyi]

Ahem, it's not just a runtime warning. Deployment won't even finish because of this and bail out with an error code, hindering deployments that use this package.

Is my best course of action really to fork this repo and remove the upper limits? Please say that there is another meaningful way that doesn't involve keeping a repo that pulls sane dependencies.

[karolyi]

For the time being, until #5711 will get merged, the solution is to use the pull request's code state:

That is, putting the following line into the requirements.txt:
git+https://github.com/psf/requests.git@refs/pull/5711/head

[sigmavirus24]

Can't these version limits be handled by package managers, such that incompatible versions won't be installed?

No. Most of our users don't use package managers and package managers have all too often shipped incompatible versions.

Is my best course of action really to fork this repo and remove the upper limits? Please say that there is another meaningful way that doesn't involve keeping a repo that pulls sane dependencies.

Sanity is the wrong term to use here. These dependencies represent what's supported and tested. Forking to use unsupported configuration is fine if you want, but don't open an issue with us when it breaks. By using a PR branch, you're using a fork - just one further out of your control.

[OlafvdSpek]

Can't these version limits be handled by package managers, such that incompatible versions won't be installed?

No. Most of our users don't use package managers and package managers have all too often shipped incompatible versions.

Just wondering, how do they install this library?

[oberstet]

Just wondering, how do they install this library?

pip? from wheels or source archives fetched from PyPI. or plain setuptools from source. fwiw, we're recommending to install our app (crossbar.io) into a separate, dedicated virtualenv. this minimizes version conflicts and allows the app to exactly control the installed versions, if possible even using hash pinned wheels. using distro package managers in contrast for python packages is "wrong" (IMO). sure, pip (and PyPI) is imperfect, but it's real, works and is commonly used.

[OlafvdSpek]

Just wondering, how do they install this library?

pip?

I thought pip was a package manager.

[sigmavirus24]

I thought you were talking about people who distribute requests for linux distributions. Those tend to be called package managers more than pip is called that. Sorry for the misunderstanding

[nateprewitt]

We've talked about the reasoning for our upper bound pins a fair amount in this thread, but I'll resummarize before closing. Requests is critical infrastructure for a large portion of the Python community. We've implemented upper bound pins as a reaction to previous large scale outages from new major versions of our dependencies being released. This is a defensive decision which, while preventing from upgrading to the latest and greatest, is not strictly incompatible and prevents these worst case scenarios.

As for the idna 3.0 issue, we addressed the Python 2.7 versioning problems in #5711 which was released in 2.26.0. We'll resolve this as complete for now.