Ensure consistent extras formatting in output #2013

AndydeCleyre · 2023-11-02T04:53:53Z

This aims to resolve the discussion in #2004, ~~and is a WIP~~ (all contributions or alternatives are welcome).

~~_compat~~ utils now exports install_req_from_line ~~and canonicalize_ireq~~ for other modules to use. This uses our copy_install_requirement which is updated to canonicalize extra strings.

~~As of now, canonicalize_ireq only affects the extras attributes, as that's all we need. So it may be poorly named 🤷🏼~~

~~No tests yet.~~

To avoid circular imports, this relocates PIP_VERSION from _compat to utils, as IMO utils should be importable by as many other piptools modules as possible.

Contributor checklist

Included tests for the changes.
PR title is short, clear, and ready to be included in the user-facing changelog.

Maintainer checklist

Verified one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

piptools/_compat/pip_compat.py

webknjaz · 2023-11-03T03:31:48Z

piptools/utils.py

+        ireq, extras=set(map(canonicalize_name, ireq.extras))
+    )
+    if ireq.req:
+        ireq.req.extras = set(ireq.extras)


Did you mean

Suggested change

ireq.req.extras = set(ireq.extras)

ireq.req.extras = set(map(canonicalize_name, ireq.extras))

?

No, because they get normalized on line 101.

webknjaz · 2023-11-03T03:34:20Z

copy_install_requirement

No preference here.

AndydeCleyre · 2023-11-03T03:36:57Z

copy_install_requirement

No preference here.

@atugushev Do you think this logic belongs in or out of copy_install_requirement?

atugushev · 2023-11-04T13:17:55Z

piptools/utils.py

+    ireq = copy_install_requirement(
+        ireq, extras=set(map(canonicalize_name, ireq.extras))


Do you think this logic belongs in or out of copy_install_requirement?

That sounds reasonable. Let's move this normalization to copy_install_requirement and use this function in all our InstallRequirement constructors.

atugushev · 2023-11-04T13:24:23Z

piptools/utils.py

+
+
+def install_req_from_line(*args: Any, **kwargs: Any) -> InstallRequirement:
+    return canonicalize_ireq(_install_req_from_line(*args, **kwargs))


It might be worthwhile to normalize extras that could be passed to pip-compile --extra as well. What do you think @AndydeCleyre?

I'd like to first see the current display of the extras in that case, but I haven't been able to get those extras-names to show up at all in the output.

I'm not inclined to do it unless someone can demonstrate how or if it would make any difference.

AndydeCleyre · 2024-02-26T18:53:41Z

OK, here's a problem I could use help with:

reqs.in:

sentry_sdk[pure_eval]

$ pip-compile --no-header reqs.in

certifi==2024.2.2
    # via sentry-sdk
sentry-sdk[pure-eval]==1.40.5
    # via -r reqs.in
urllib3==2.2.1
    # via sentry-sdk

$ pip-sync /dev/null
$ pip install --dry-run --quiet --report=- -r reqs.in | jq -r '.install[].metadata.name'

urllib3
asttokens
certifi
executing
pure-eval
sentry-sdk
six

I'm demoting this to WIP again until this gets resolved.

In its current state, this branch has resolver_result missing items:

--- pip-tools-main.txt	2024-02-26 14:40:10.004827841 -0500
+++ pip-tools-2013.txt	2024-02-26 14:39:52.064453749 -0500
@@ -3,19 +3,12 @@
         'sentry-sdk[pure-eval]': ExtrasCandidate(base=LinkCandidate('https://files.pythonhosted.org/packages/f6/21/e41b8dd0da432a06b8bf10b7ba634d6e62e60b9a79aba15146a2581265cc/sentry_sdk-1.40.5-py2.py3-none-any.whl (from https://pypi.org/simple/sentry-sdk/)'),
         extras=frozenset({'pure-eval'})),
         'urllib3': LinkCandidate('https://files.pythonhosted.org/packages/a2/73/a68704750a7679d0b6d3ad7aa8d4da8e14e151ae82e6fee774e6e0d05ec8/urllib3-2.2.1-py3-none-any.whl (from https://pypi.org/simple/urllib3/) (requires-python:>=3.8)'),
-        '<Python from Requires-Python>': <pip._internal.resolution.resolvelib.candidates.RequiresPythonCandidate object at 0x75b8142a9af0>,
-        'asttokens': LinkCandidate('https://files.pythonhosted.org/packages/45/86/4736ac618d82a20d87d2f92ae19441ebc7ac9e7a581d7e58bbe79233b24a/asttokens-2.4.1-py2.py3-none-any.whl (from https://pypi.org/simple/asttokens/)'),
+        '<Python from Requires-Python>': <pip._internal.resolution.resolvelib.candidates.RequiresPythonCandidate object at 0x743e650b7b60>,
         'certifi': LinkCandidate('https://files.pythonhosted.org/packages/ba/06/a07f096c664aeb9f01624f858c3add0a4e913d6c96257acb4fce61e7de14/certifi-2024.2.2-py3-none-any.whl (from https://pypi.org/simple/certifi/) (requires-python:>=3.6)'),
-        'executing': LinkCandidate('https://files.pythonhosted.org/packages/80/03/6ea8b1b2a5ab40a7a60dc464d3daa7aa546e0a74d74a9f8ff551ea7905db/executing-2.0.1-py2.py3-none-any.whl (from https://pypi.org/simple/executing/) (requires-python:>=3.5)'),
-        'pure-eval': LinkCandidate('https://files.pythonhosted.org/packages/2b/27/77f9d5684e6bce929f5cfe18d6cfbe5133013c06cb2fbf5933670e60761d/pure_eval-0.2.2-py3-none-any.whl (from https://pypi.org/simple/pure-eval/)'),
-        'sentry-sdk': LinkCandidate('https://files.pythonhosted.org/packages/f6/21/e41b8dd0da432a06b8bf10b7ba634d6e62e60b9a79aba15146a2581265cc/sentry_sdk-1.40.5-py2.py3-none-any.whl (from https://pypi.org/simple/sentry-sdk/)'),
-        'six': LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/) (requires-python:>=2.7,
-        !=3.0.*,
-        !=3.1.*,
-        !=3.2.*)')
+        'sentry-sdk': LinkCandidate('https://files.pythonhosted.org/packages/f6/21/e41b8dd0da432a06b8bf10b7ba634d6e62e60b9a79aba15146a2581265cc/sentry_sdk-1.40.5-py2.py3-none-any.whl (from https://pypi.org/simple/sentry-sdk/)')
     },
-    graph=<pip._vendor.resolvelib.structs.DirectedGraph object at 0x75b8143a0a10>,
-    criteria={'sentry-sdk[pure-eval]': Criterion((SpecifierRequirement('sentry_sdk[pure_eval]'),
+    graph=<pip._vendor.resolvelib.structs.DirectedGraph object at 0x743e658c9af0>,
+    criteria={'sentry-sdk[pure-eval]': Criterion((SpecifierRequirement('sentry_sdk[pure-eval]'),
     via=None)),
     'sentry-sdk': Criterion((ExplicitRequirement(LinkCandidate('https://files.pythonhosted.org/packages/f6/21/e41b8dd0da432a06b8bf10b7ba634d6e62e60b9a79aba15146a2581265cc/sentry_sdk-1.40.5-py2.py3-none-any.whl (from https://pypi.org/simple/sentry-sdk/)')),
     via=ExtrasCandidate(base=LinkCandidate('https://files.pythonhosted.org/packages/f6/21/e41b8dd0da432a06b8bf10b7ba634d6e62e60b9a79aba15146a2581265cc/sentry_sdk-1.40.5-py2.py3-none-any.whl (from https://pypi.org/simple/sentry-sdk/)'),
@@ -30,26 +23,8 @@
     extras=frozenset({'pure-eval'}))),
     (SpecifierRequirement('urllib3>=1.26.11; python_version >= "3.6"'),
     via=LinkCandidate('https://files.pythonhosted.org/packages/f6/21/e41b8dd0da432a06b8bf10b7ba634d6e62e60b9a79aba15146a2581265cc/sentry_sdk-1.40.5-py2.py3-none-any.whl (from https://pypi.org/simple/sentry-sdk/)'))),
-    'pure-eval': Criterion((SpecifierRequirement('pure-eval; extra == "pure_eval"'),
-    via=ExtrasCandidate(base=LinkCandidate('https://files.pythonhosted.org/packages/f6/21/e41b8dd0da432a06b8bf10b7ba634d6e62e60b9a79aba15146a2581265cc/sentry_sdk-1.40.5-py2.py3-none-any.whl (from https://pypi.org/simple/sentry-sdk/)'),
-    extras=frozenset({'pure-eval'})))),
-    'executing': Criterion((SpecifierRequirement('executing; extra == "pure_eval"'),
-    via=ExtrasCandidate(base=LinkCandidate('https://files.pythonhosted.org/packages/f6/21/e41b8dd0da432a06b8bf10b7ba634d6e62e60b9a79aba15146a2581265cc/sentry_sdk-1.40.5-py2.py3-none-any.whl (from https://pypi.org/simple/sentry-sdk/)'),
-    extras=frozenset({'pure-eval'})))),
-    'asttokens': Criterion((SpecifierRequirement('asttokens; extra == "pure_eval"'),
-    via=ExtrasCandidate(base=LinkCandidate('https://files.pythonhosted.org/packages/f6/21/e41b8dd0da432a06b8bf10b7ba634d6e62e60b9a79aba15146a2581265cc/sentry_sdk-1.40.5-py2.py3-none-any.whl (from https://pypi.org/simple/sentry-sdk/)'),
-    extras=frozenset({'pure-eval'})))),
     '<Python from Requires-Python>': Criterion((RequiresPythonRequirement('>=3.8'),
     via=LinkCandidate('https://files.pythonhosted.org/packages/a2/73/a68704750a7679d0b6d3ad7aa8d4da8e14e151ae82e6fee774e6e0d05ec8/urllib3-2.2.1-py3-none-any.whl (from https://pypi.org/simple/urllib3/) (requires-python:>=3.8)')),
     (RequiresPythonRequirement('>=3.6'),
-    via=LinkCandidate('https://files.pythonhosted.org/packages/ba/06/a07f096c664aeb9f01624f858c3add0a4e913d6c96257acb4fce61e7de14/certifi-2024.2.2-py3-none-any.whl (from https://pypi.org/simple/certifi/) (requires-python:>=3.6)')),
-    (RequiresPythonRequirement('>=3.5'),
-    via=LinkCandidate('https://files.pythonhosted.org/packages/80/03/6ea8b1b2a5ab40a7a60dc464d3daa7aa546e0a74d74a9f8ff551ea7905db/executing-2.0.1-py2.py3-none-any.whl (from https://pypi.org/simple/executing/) (requires-python:>=3.5)')),
-    (RequiresPythonRequirement('!=3.0.*,!=3.1.*,!=3.2.*,>=2.7'),
-    via=LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/) (requires-python:>=2.7,
-    !=3.0.*,
-    !=3.1.*,
-    !=3.2.*)'))),
-    'six': Criterion((SpecifierRequirement('six>=1.12.0'),
-    via=LinkCandidate('https://files.pythonhosted.org/packages/45/86/4736ac618d82a20d87d2f92ae19441ebc7ac9e7a581d7e58bbe79233b24a/asttokens-2.4.1-py2.py3-none-any.whl (from https://pypi.org/simple/asttokens/)')))
+    via=LinkCandidate('https://files.pythonhosted.org/packages/ba/06/a07f096c664aeb9f01624f858c3add0a4e913d6c96257acb4fce61e7de14/certifi-2024.2.2-py3-none-any.whl (from https://pypi.org/simple/certifi/) (requires-python:>=3.6)')))
 })

AndydeCleyre · 2024-02-26T19:53:51Z

I thought that pip was handling "canonicalized" extras, but:

$ pip-sync /dev/null
$ pip install --dry-run --quiet --report=- 'sentry_sdk[pure_eval]' | jq -r '.install[].metadata.name'

urllib3
asttokens
certifi
executing
pure-eval
sentry-sdk
six

$ pip install --dry-run --quiet --report=- 'sentry_sdk[pure-eval]' | jq -r '.install[].metadata.name'

urllib3
certifi
sentry-sdk

So I need to step back and request a sanity check about what we should be doing here. @atugushev ?

webknjaz · 2024-02-27T04:49:28Z

Sorry, this fell off of my radar. Sounds like a bug in pip to fix as well?

AndydeCleyre · 2024-02-28T18:20:58Z

Sounds like a bug in pip to fix as well?

Maybe these:

AndydeCleyre · 2024-03-03T15:03:35Z

So I'll take this back out of draft but maybe we should consider it blocked until pip is fixed.

AndydeCleyre · 2024-05-04T16:15:37Z

Sounds like a bug in pip to fix as well?

Maybe these:

Implement PEP 685 pypa/pip#11445

Upgrade the vendored packaging to 22.0+ pypa/pip#11715

These were just marked as resolved! I haven't retested with pip from git yet but I'm optimistic.

EDIT: Yes, this seems to be working fine with pip's main branch.

In fact it might be proper to undo some of these changes, if we can require the next pip release as a minimum. I'll add a commit undoing changes as long as the test still passes and see what everyone thinks/tests.

_compat now exports install_req_from_line and canonicalize_ireq

canonicalize_ireq now returns a copy ireq instead of modifying in place

Based on the case in jazzband#2004

chrysle

This looks good to me. I think we should move it forward? We can still respond to any issues this might cause afterwards (though I'm optimistic).

AndydeCleyre added bug Something is not working good first issue A good item for first time contributors to work on extras Handling optional dependencies labels Nov 2, 2023

webknjaz linked an issue Nov 2, 2023 that may be closed by this pull request

pip==23.3 breaking "extras" behavior #2004

Open

webknjaz reviewed Nov 2, 2023

View reviewed changes

piptools/_compat/pip_compat.py Outdated Show resolved Hide resolved

AndydeCleyre force-pushed the bugfix/2004 branch 3 times, most recently from eb5215d to 4ffd903 Compare November 2, 2023 22:47

This comment was marked as resolved.

Sign in to view

webknjaz reviewed Nov 3, 2023

View reviewed changes

AndydeCleyre requested a review from atugushev November 4, 2023 02:37

atugushev reviewed Nov 4, 2023

View reviewed changes

AndydeCleyre force-pushed the bugfix/2004 branch from 09f3bee to aad06fb Compare November 5, 2023 04:14

AndydeCleyre changed the title ~~Ensure canonicalization of extras~~ Ensure consistent extras formatting in output Nov 5, 2023

AndydeCleyre marked this pull request as ready for review November 5, 2023 16:18

webknjaz self-requested a review December 3, 2023 03:26

AndydeCleyre force-pushed the bugfix/2004 branch from 07017c4 to 6fe8bb0 Compare December 3, 2023 19:09

AndydeCleyre mentioned this pull request Dec 3, 2023

pip==23.3 breaking "extras" behavior #2004

Open

AndydeCleyre force-pushed the bugfix/2004 branch from 6fe8bb0 to c810c78 Compare January 8, 2024 18:14

AndydeCleyre force-pushed the bugfix/2004 branch from c810c78 to 96243dc Compare January 22, 2024 04:54

AndydeCleyre force-pushed the bugfix/2004 branch 3 times, most recently from 4612ba7 to 7711ce6 Compare February 22, 2024 13:50

This comment was marked as resolved.

Sign in to view

AndydeCleyre marked this pull request as draft February 26, 2024 19:03

AndydeCleyre force-pushed the bugfix/2004 branch from 7711ce6 to 4c8c68c Compare March 3, 2024 18:17

AndydeCleyre marked this pull request as ready for review March 3, 2024 18:31

Viicos mentioned this pull request Mar 14, 2024

Prepare upgrade to Django 4.2 maykinmedia/objects-api#362

Merged

AndydeCleyre added 4 commits May 7, 2024 13:02

Ensure canonicalization of extras when creating ireqs

89624dc

_compat now exports install_req_from_line and canonicalize_ireq

Reorganize canonicalize_ireq and PIP_VERSION into utils

98070d0

canonicalize_ireq now returns a copy ireq instead of modifying in place

Add test: canonicalize_extras

8e18b45

Based on the case in jazzband#2004

Fold canonicalize-ireq-extras logic into copy_install_requirement

42dcce1

AndydeCleyre force-pushed the bugfix/2004 branch from b73365f to 42dcce1 Compare May 7, 2024 17:19

chrysle approved these changes May 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure consistent extras formatting in output #2013

Ensure consistent extras formatting in output #2013

AndydeCleyre commented Nov 2, 2023 •

edited

This comment was marked as resolved.

webknjaz Nov 3, 2023

AndydeCleyre Nov 3, 2023

webknjaz commented Nov 3, 2023

AndydeCleyre commented Nov 3, 2023

atugushev Nov 4, 2023 •

edited

atugushev Nov 4, 2023

AndydeCleyre Nov 5, 2023

AndydeCleyre Jan 9, 2024

This comment was marked as resolved.

This comment was marked as resolved.

AndydeCleyre commented Feb 26, 2024 •

edited

AndydeCleyre commented Feb 26, 2024

webknjaz commented Feb 27, 2024

AndydeCleyre commented Feb 28, 2024

AndydeCleyre commented Mar 3, 2024

AndydeCleyre commented May 4, 2024 •

edited

chrysle left a comment

	ireq.req.extras = set(ireq.extras)
	ireq.req.extras = set(map(canonicalize_name, ireq.extras))

		ireq = copy_install_requirement(
		ireq, extras=set(map(canonicalize_name, ireq.extras))



		def install_req_from_line(args: Any, *kwargs: Any) -> InstallRequirement:
		return canonicalize_ireq(_install_req_from_line(args, *kwargs))

Ensure consistent extras formatting in output #2013

Are you sure you want to change the base?

Ensure consistent extras formatting in output #2013

Conversation

AndydeCleyre commented Nov 2, 2023 • edited

Contributor checklist

Maintainer checklist

This comment was marked as resolved.

webknjaz Nov 3, 2023

Choose a reason for hiding this comment

AndydeCleyre Nov 3, 2023

Choose a reason for hiding this comment

webknjaz commented Nov 3, 2023

AndydeCleyre commented Nov 3, 2023

atugushev Nov 4, 2023 • edited

Choose a reason for hiding this comment

atugushev Nov 4, 2023

Choose a reason for hiding this comment

AndydeCleyre Nov 5, 2023

Choose a reason for hiding this comment

AndydeCleyre Jan 9, 2024

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

AndydeCleyre commented Feb 26, 2024 • edited

AndydeCleyre commented Feb 26, 2024

webknjaz commented Feb 27, 2024

AndydeCleyre commented Feb 28, 2024

AndydeCleyre commented Mar 3, 2024

AndydeCleyre commented May 4, 2024 • edited

chrysle left a comment

Choose a reason for hiding this comment

AndydeCleyre commented Nov 2, 2023 •

edited

atugushev Nov 4, 2023 •

edited

AndydeCleyre commented Feb 26, 2024 •

edited

AndydeCleyre commented May 4, 2024 •

edited