poetry with "vcrpy==6.0.1" and "urllib=*" requirements does NOT install urllib3>2 because of vcrpy's platform-specific marker that should be ignored on CPython 3.10

[immerrr]

Description

I have tried upgrading urllib3 in a project of mine to a more recent version to fix a vulnerability, and ran into an issue, where poetry's dependency resolver didn't let me to upgrade past urllib3>2.

AFAICS from the logs (see below), poetry insists that "urllib3<2" is a mandatory requirement. If I check how vcrpy v6.0.1 specifies its urllib requirements it looks something like this:

    "urllib3 <2; python_version <'3.10'",
    "urllib3 <2; platform_python_implementation =='PyPy'",

I am running this on CPython 3.10, so, if I understand pep-508 for environment markers correctly, neither conditional applies to my system. Which in turn means, that both urllib3 <2 requirements should be ignored during dependency resolution.

poetry lock for vcrpy ends up excluding the python_version<'3.10' from the lockfile, which is perfectly fine:

[[package]]
name = "vcrpy"
version = "6.0.1"
description = "Automatically mock your HTTP interactions to simplify and speed up testing"
optional = false
python-versions = ">=3.8"
files = [
    {file = "vcrpy-6.0.1.tar.gz", hash = "sha256:9e023fee7f892baa0bbda2f7da7c8ac51165c1c6e38ff8688683a12a4bde9278"},
]

[package.dependencies]
PyYAML = "*"
urllib3 = {version = "<2", markers = "platform_python_implementation == \"PyPy\""}
wrapt = "*"
yarl = "*"

But my question is, since I'm not on pypy, why does it still apply the version requirement during installation?

Workarounds

I have seen #8996, and would have gladly left my comment there, but it is locked. Looks like the maintainers of the vcrpy package have come up with a workaround, but until that is released, I don't see a solution.

Poetry Installation Method

pipx

Operating System

Ubuntu 23.04

Poetry Version

Poetry (version 1.8.3)

Poetry Configuration

cache-dir = "/home/immerrr/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "/home/immerrr/.local/poetry-venvs"
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

https://gist.github.com/immerrr/b6cc83637ba914289e4e4ed205a2aec0#file-python-m-sysconfig-log

Example pyproject.toml

[tool.poetry]
name = "vcrpy-urllib-test"
version = "0.1.0"
description = ""
authors = ["immerrr <immerrr@gmail.com>"]
readme = "README.md"
package-mode = false

[tool.poetry.dependencies]
python = "^3.10"
vcrpy = ">6"
urllib3 = "*"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

https://gist.github.com/immerrr/b6cc83637ba914289e4e4ed205a2aec0#file-poetry-lock-and-install-log

[dimbleby]

I am running this on CPython 3.10 ... neither conditional applies to my system.

the system that you are currently running on is more or less irrelevant during locking, poetry builds cross platform solutions. As currently defined, there is no solution for PyPy.

so this is working as expected, you will need to add markers spelling out where you do and don't want new urllib3

please close

[immerrr]

@dimbleby please, bear with me for a bit.

the system that you are currently running on is more or less irrelevant during locking, poetry builds cross platform solutions.

makes sense

As currently defined, there is no solution for PyPy.

here, you may have much more context, but I'm not following, "there is no solution for pypy" w.r.t. what exactly?

so this is working as expected, you will need to add markers spelling out where you do and don't want new urllib3

I am fully onboard with PyPy-specific requirement ending up in the lockfile, it makes sense that if the package in the future is going to be installed on PyPy, poetry would need that requirement to do the resolution properly.

What I'm trying to get through is that there should be a consistent platform-independent lock state that says:

• on PyPy, use urllib3<2
• on CPython 3.10+, feel free to use whatever urllib3 you like

Because the package, vcrpy, to the best of my knowledge already has markers spelling out where urllib3 requirement is needed:

    "urllib3 <2; python_version <'3.10'",
    "urllib3 <2; platform_python_implementation =='PyPy'",

For all other platforms, there should be no requirement from vcpy side of things.

[dimbleby]

you say you want poetry to resolve to some version of urllib3 that satisfies urllib3<2 on PyPy. But your pyproject.toml unconditionally requires urllib3 = ">2".

Either: don't mention urllib3 at all - you'll get it anyway via vcrpy. Or add markers in your own requirements to spell out what version you want in which circumstances.

please close

[immerrr]

I appreciate your time in responding to me, and I think I'm getting somewhere.

So do I understand correctly that you are saying that since pyproject.toml cannot be resolved for one of the platforms (PyPy), it should not be resolved for any of the platforms? It would make some sense, but make poetry quite a bit less practical. I.e. any package that, say, is not available for darwin/windows should not be usable with poetry at all.

Ok, the urllib3>2 requirement from pyproject.toml was to just highlight the issue. Suppose, I release the restriction to urrlib3 = "*", now pyproject.toml can be satisfied for CPython and PyPy:

[tool.poetry.dependencies]
python = "^3.10"
vcrpy = ">6"
urllib3 = "*"

I have an env with CPython 3.10, where neither of the two vcrpy requirements apply:

    "urllib3 <2; python_version <'3.10'",
    "urllib3 <2; platform_python_implementation =='PyPy'",

Why is poetry still not trying to unstall urllib3 2+ as the latest available version for the current environment?

$ poetry lock && poetry install -vv
Updating dependencies
Resolving dependencies... (0.6s)
Using virtualenv: /home/immerrr/.local/poetry-venvs/vcrpy-urllib-test-B2DJqU4P-py3.10
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 0 updates, 0 removals, 7 skipped

  - Installing idna (3.7): Skipped for the following reason: Already installed
  - Installing multidict (6.0.5): Skipped for the following reason: Already installed
  - Installing pyyaml (6.0.1): Skipped for the following reason: Already installed
  - Installing urllib3 (1.26.18): Skipped for the following reason: Already installed
  - Installing vcrpy (6.0.1): Skipped for the following reason: Already installed
  - Installing wrapt (1.16.0): Skipped for the following reason: Already installed
  - Installing yarl (1.9.4): Skipped for the following reason: Already installed

[immerrr]

Actually, let me update the bugreport to avoid the situation with "unresolvable" pyproject.toml...

[dimbleby]

again, poetry builds a cross-platform solution. So

• if there is no cross-platform solution then poetry locking will fail
  • eg if you want some windows-only dependency then add a marker to your requirement saying that you want that dependency only on windows
• if you haven't manually split up your urllib3 requirement by using markers - I have suggested this twice already now - then the solution that poetry finds must work on all platforms eg in your case the only cross-platform solution is to choose a urllib3 < 2

you can always pip install . (or pip install -e .), pip will look at the requirements and only care about your current platform, that seems to be what you want.

this continues all to be working as expected, please close.