Permit urllib3 >=2 for non-PyPy Python >=3.10 in order to help users of Poetry #830
Conversation
|
@hartwork this is my attempt to workaround the issue I think #826 tries to solve. Adding the extra marker made my Python 3.10 project get urllib3 2.x: And the change to pyproject.toml was to depend on my local copy of the vcrpy repository that contains this commit: --- a/pyproject.toml
+++ b/pyproject.toml
...
[tool.poetry.group.dev.dependencies]
+vcrpy = {path = "../../vcrpy"}
...
--- a/poetry.lock
+++ b/poetry.lock
...
[[package]]
name = "vcrpy"
version = "6.0.1"
description = "Automatically mock your HTTP interactions to simplify and speed
up testing"
optional = false
python-versions = ">=3.8"
-files = [
- {file = "vcrpy-6.0.1.tar.gz", hash = "sha256:9e023fee7f892baa0bbda2f7da7c8ac51165c1c6e38ff8688683a12a4bde9278"},
-]
+files = []
+develop = false
[package.dependencies]
PyYAML = "*"
-urllib3 = {version = "<2", markers = "platform_python_implementation == \"PyPy\""}
+urllib3 = [
+ {version = "<2", markers = "platform_python_implementation == \"PyPy\""},
+ {version = "*", markers = "platform_python_implementation != \"PyPy\" and python_version >= \"3.10\""},
+]
wrapt = "*"
yarl = "*" |
There was a problem hiding this comment.
Hi @pjonsson,
given that Poetry lock files need to work for all implementations including PyPy, the lock file should still pin urllib3 1.x even with this change so that PyPy is kept in the boat by the lock file. So I don't see how workaround would have a chance to change the situation. Please help me see it if it indeed does. Thank you.
|
On the logical level, the third condition for urllib3 version is mutually exclusive to the other two conditions that are already there. I don't really know Python though, but doesn't: +urllib3 = [
+ {version = "<2", markers = "platform_python_implementation == \"PyPy\""},
+ {version = "*", markers = "platform_python_implementation != \"PyPy\" and python_version >= \"3.10\""},
+]pin the version to |
|
@pjonsson my current understanding is this:
If this is a magic trick, I don't see yet why it would work. Please help me understand. |
|
@pjonsson PS: Comment python-poetry/poetry#8996 (comment) helped me understand Poetry's behavior better only three days ago. Maybe that thread is of interest to you also. |
|
@hartwork I don't know the inner workings of Poetry, but my understanding is that while the lock file is platform independent, Poetry provides a weaker guarantee than what you describe (otherwise the consequence should be that you are guaranteed to get the same package versions when running Your third point about requiring more turning into relaxing requirements is fair, there are other packages in my pyproject.toml that depend on urllib3, so for me, some version of urllib3 will be installed no matter what. Your third point is interesting in another way though, why are the current urllib3 constraints in the |
@pjonsson based on python-poetry/poetry#8996 (comment) my understanding is that Poetry's lock file needs to satisfy all marker rules at once. I consider that troublesome, to be clear, but that seems to be status quo.
I'm not fully sure how you ended up in that state and why Poetry was okay writing a lockfile like that (given that seems to go against the one-lock-for-all principle; we're probably missing something about it). Is there a chance you could provide a shell session like mine at python-poetry/poetry#8996 (comment) to reproduce your idea here in isolation on a local Bash Linux shell? (We can also do voice call on the topic if you'd be up for it but want to save some typing, just an idea.)
I can trigger them, but we have zero Poetry here so it would either be all green and not give new insights or be red from unrelated breakage and still not give new insights. Am I missing something?
They are in |
|
@hartwork sorry, was making dinner when your last reply was written. If there aren't any Poetry tests in this CI there's no need to trigger the tests since that won't make us any wiser. I agree with your perspective that things seem to be stuck at the status quo on the Poetry side, which is why I'm trying to get a workaround in vcrpy instead. My reading of the comments you link to is that the Poetry people are responding to exactly what you are asking ("Why can't I restrict urllib3 to >= 2?"), which is because that conflicts with the <2 requirement for PyPy as they say, but the more interesting question is which urllib3 should be installed for CPython 3.10 when vcrpy is required alongside some other dependency that depends on "urllib3 >1.0". See the second session below for more on that. Here's a reproduction of your linked session in the Python 3.10 docker image with a non-root user: so Poetry will force urllib3 <2 since the lock file is written the way it is. I find it silly that my CPython package installation is forced by constraints marked for PyPy, but that is apparently how Poetry is supposed to work. Here's the same session with this PR: So adding one more constraint for the remaining configurations forces an installation of some version of urllib3, but any version compatible with the other dependencies in the project is permitted. |
hartwork
changed the title
|
Hi @pjonsson, thanks for sharing the reproducer! I tried it out now and I confirm that it makes urllib3 >=2 available to users of CPython 3.10 in practice. I find this behavior of Poetry weird at best. I will let the CI run now so that it shows its green nature, and approve, and add @jairhenrique for a chance on a second opinion. PS: My personal quite-happy-with alternative to Poetry is this combo:
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## master #830 +/- ##
=======================================
Coverage 92.32% 92.32%
=======================================
Files 27 27
Lines 1811 1811
Branches 338 338
=======================================
Hits 1672 1672
Misses 92 92
Partials 47 47 ☔ View full report in Codecov by Sentry. |
Poetry makes platform indepdent lock files, so the PyPy marker is there even when using CPython >= 3.10. Add a third constraint that permits any urllib3 version when using Python >=3.10 and some other implementation than PyPy.
pjonsson
force-pushed
the
permit-urllib3
branch
from
6159410
to
52da776
Compare
|
I agree with your assessment of a surprising interpretation of the constraint "<2 for PyPy" when it turns out to mean "<2 for anything" in reality, and thanks for your help @hartwork! |
Poetry makes platform indepdent lock files,
so the PyPy marker is there even when using
CPython >= 3.10.
Add a third constraint that permits any urllib3
version when using Python >=3.10 and some
other implementation than PyPy.