Add support for pip>=22.0, drop support for Python 3.6 #1567

di · 2022-02-02T13:50:51Z

This PR consolidates all the changes necessary to update pip-tools for pip>=22.0, namely:

Fixes an AssertionError due to the new pip console log handler
Fixes an issue due to pip's normalization changes in https://pip.pypa.io/en/stable/news/#id73
Drops support for Python 3.6

Fixes #1558, closes #1559.
Fixes #1566.
Fixes #1562, closes #1560, closes #1561, closes #1564.

Contributor checklist

Provided the tests for the changes.
Assure PR title is short, clear, and good to be included in the user-oriented changelog

Maintainer checklist

Assure one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

Since pip no longer guarantees that it will no longer normalize underscore (_) in distribution names to dash (-) in `pip freeze`, `pip list`, and `pip show` (https://pip.pypa.io/en/stable/news/#id73) , in the interest of minimizing churn, pip-compile should normalize requirements names instead. This will ensure there are no changes to requirements files compiled with earlier versions of pip-tools/pip.

This test doesn't actually modify the requirements file, so update it to start with a normalized requirement name so it can continue to expect a normalized requirement name.

This is where we diverge from pip's output: pip no longer guarantees that it will canonicalize requirement names in `pip freeze` but we will continue to output canonicalized names.

nicoa · 2022-02-02T13:54:36Z

RichPopStreamHandler didn't exist in pip<22 - feel free to take the commits from #1565 into this PR, taking this into account.

di · 2022-02-02T13:59:50Z

@nicoa Thanks, I missed that the tests were being run across multiple pip versions!

fsecada01

Thorough. Nicely done.

pauloxnet · 2022-02-02T15:05:17Z

LGTM

AndydeCleyre · 2022-02-02T15:25:45Z

Anyone else think we should take the opportunity to replace the assert with if+raise or similar, as asserts aren't guaranteed to run depending on interpreter optimizations?

IMO assert should only be in explicit tests.

piptools/utils.py

fsecada01 · 2022-02-02T15:39:19Z

Anyone else think we should take the opportunity to replace the assert with if+raise or similar, as asserts aren't guaranteed to run depending on interpreter optimizations?

IMO assert should only be in explicit tests.

I agree with you, which is what I did in my pull request.

wbolster

seems using the github suggestion ui is not ideal. this line should be removed now:

        line = str(ireq.req).lower()

piptools/utils.py

piptools/repositories/pypi.py

Co-authored-by: wouter bolsterlee <wouter@bolsterl.ee>

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

pradyunsg · 2022-02-03T21:30:22Z

This likely needs retitling, since the PR title goes into the changelog!

di · 2022-02-03T23:51:25Z

Good point @pradyunsg, updated the title.

pradyunsg · 2022-02-04T06:48:21Z

Thanks @di!

d0ugal · 2022-02-04T06:51:18Z

Thanks!

atugushev · 2022-02-04T10:54:44Z

This is now released as part of pip-tools 6.5.0! Thanks, @di, @hugovk, and all involved!

ssbarnea · 2022-02-04T12:29:04Z

I have the impression that this change may have have an interesting outcome in produced locked files:

di · 2022-02-04T15:03:52Z

Yes, this is expected, we're now fully canonicalizing every project name, so this is a one way change.

Without this, many dashes would have changed back to underscores (but not all) and this would have changed in the compiled file depending on which underlying pip version was used.

ssbarnea · 2022-02-04T15:39:38Z

I am happy to see canonicalizing! My only concern was that this aspect was not directly visible under commit title so most will not see it. Sadly that change would likely to keep me busy for at least one day, considering number of affected projects.

Until today I considered keeping pre-commit out of dependency lock as a good idea. Time to pay for own mistakes.

See jazzband/pip-tools#1567

The following comment is borrowed from this pull request, with changes: openedx/opaque-keys#218 This failure started happening in the automated requirements upgrade GitHub action: https://github.com/openedx/edx-submissions/runs/5704384647?check_suite_focus=true The bug was being caused by an interaction between pip and pip-tools, described here: jazzband/pip-tools#1558 The bug is fixed in pip-tools==6.5.0 - from this PR: jazzband/pip-tools#1567 This pull requests bumps the version of pip-tools manually to get the automated upgrades working again.

The following comment is borrowed from this pull request, with changes: openedx/opaque-keys#218 This failure started happening in the automated requirements upgrade GitHub action: https://github.com/openedx/edx-submissions/runs/5704384647?check_suite_focus=true The bug was being caused by an interaction between pip and pip-tools, described here: jazzband/pip-tools#1558 The bug is fixed in pip-tools==6.5.0 - from this pull request: jazzband/pip-tools#1567 This pull requests bumps the version of pip-tools manually to get the automated upgrades working again.

hugovk and others added 6 commits February 2, 2022 06:44

With the release of pip 22.0, bump pipprevious to 21.3

941e5c8

Drop support for EOL Python 3.6

f717176

Daily cron: Test pip main on 3.10 but not 3.6

eb88812

Update test that doesn't rewrite requirements file

4c25f28

This test doesn't actually modify the requirements file, so update it to start with a normalized requirement name so it can continue to expect a normalized requirement name.

Update test that compares our output to pip freeze

a38d6b6

This is where we diverge from pip's output: pip no longer guarantees that it will canonicalize requirement names in `pip freeze` but we will continue to output canonicalized names.

di force-pushed the get-ci-green branch 2 times, most recently from f440c47 to fd76111 Compare February 2, 2022 13:57

This was referenced Feb 2, 2022

Daily cron: Test pip main on 3.10 but not 3.6 #1560

Closed

With the release of pip 22.0, bump pipprevious to 21.3 #1561

Closed

Drop support for EOL Python 3.6 #1564

Closed

di force-pushed the get-ci-green branch from fd76111 to 6556c7f Compare February 2, 2022 14:08

This was referenced Feb 2, 2022

add RichPipStreamHandler as allowed handler in repositories/pypi.py #1565

Closed

Drop Python 3.6 support #1562

Closed

fsecada01 mentioned this pull request Feb 2, 2022

Update pypi.py #1559

Closed

4 tasks

fsecada01 approved these changes Feb 2, 2022

View reviewed changes

pauloxnet approved these changes Feb 2, 2022

View reviewed changes

This was referenced Feb 2, 2022

CI failing due to normalization of dashes to underscores in tests #1566

Closed

pip-compile fails with pip==22.0 #1558

Closed

wbolster reviewed Feb 2, 2022

View reviewed changes

piptools/utils.py Outdated Show resolved Hide resolved

wbolster requested changes Feb 2, 2022

View reviewed changes

rsalmaso reviewed Feb 2, 2022

View reviewed changes

piptools/utils.py Show resolved Hide resolved

di commented Feb 2, 2022

View reviewed changes

piptools/repositories/pypi.py Outdated Show resolved Hide resolved

wbolster approved these changes Feb 2, 2022

View reviewed changes

Copy ireq.req instead of mutating it

d1eba7f

Co-authored-by: wouter bolsterlee <wouter@bolsterl.ee>

Update README.rst

c06ab8f

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

pradyunsg added this to the 6.5.0 milestone Feb 3, 2022

pradyunsg added the enhancement Improvements to functionality label Feb 3, 2022

tomwojcik approved these changes Feb 3, 2022

View reviewed changes

di changed the title ~~Consolidated changes for pip>=22.0~~ Add support for pip>=22.0, drop support for Python 3.6 Feb 3, 2022

pradyunsg approved these changes Feb 4, 2022

View reviewed changes

pradyunsg merged commit 9fb2d42 into jazzband:master Feb 4, 2022

di deleted the get-ci-green branch February 4, 2022 07:14

antonymilne mentioned this pull request Feb 4, 2022

Kedro build-reqs breaks with pip>=22.0 kedro-org/kedro#1208

Closed

ssbarnea mentioned this pull request Feb 4, 2022

Fix for pip-tools 6.5.0 ansible/ansible-lint#1845

Merged

webknjaz mentioned this pull request Feb 4, 2022

[6.5.0] Packages needlessly uninstalled during sync due to non-canonicalized names (dots vs dashes) #1570

Closed

This was referenced Feb 4, 2022

Cannot install python-language-server[all] #1538

Closed

Add support for pip's 2020 dependency resolver #1539

Merged

This was referenced Feb 7, 2022

Manually upgrade pip-tools to 6.5.0 to fix pip/pip-tools bug. openedx/XBlock#573

Merged

Manually upgrade pip-tools to 6.5.0 to fix pip/pip-tools bug. openedx/opaque-keys#218

Merged

di mentioned this pull request Mar 17, 2022

pip-compile's logs omit diagnostic messages from pip's subprocess error logging, when e.g. psycopg2 processing fails due to missing system (non-pip-managed) tools #1583

Open

replaceafill added a commit to artefactual-labs/a3m that referenced this pull request Mar 28, 2022

Upgrade pip-tools to be compatible with pip>=22.0

b66593d

See jazzband/pip-tools#1567

MichaelRoytman mentioned this pull request Mar 29, 2022

fix: manually update pip-tools version to 6.5.1 to fix pip/pip-tools bug openedx/edx-submissions#160

Merged

MaferMazu mentioned this pull request Sep 2, 2022

chore: update requirements for maple version eduNEXT/eox-core#215

Merged

4 tasks

atugushev mentioned this pull request Dec 14, 2022

environment markers 'duplicated' (requirements + install_requires + pyproject.toml) #874

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for pip>=22.0, drop support for Python 3.6 #1567

Add support for pip>=22.0, drop support for Python 3.6 #1567

di commented Feb 2, 2022 •

edited by hugovk

nicoa commented Feb 2, 2022

di commented Feb 2, 2022

fsecada01 left a comment

pauloxnet commented Feb 2, 2022

AndydeCleyre commented Feb 2, 2022

fsecada01 commented Feb 2, 2022

wbolster left a comment

pradyunsg commented Feb 3, 2022

di commented Feb 3, 2022

pradyunsg commented Feb 4, 2022

d0ugal commented Feb 4, 2022

atugushev commented Feb 4, 2022

ssbarnea commented Feb 4, 2022

di commented Feb 4, 2022

ssbarnea commented Feb 4, 2022

Add support for pip>=22.0, drop support for Python 3.6 #1567

Add support for pip>=22.0, drop support for Python 3.6 #1567

Conversation

di commented Feb 2, 2022 • edited by hugovk

Contributor checklist

Maintainer checklist

nicoa commented Feb 2, 2022

di commented Feb 2, 2022

fsecada01 left a comment

Choose a reason for hiding this comment

pauloxnet commented Feb 2, 2022

AndydeCleyre commented Feb 2, 2022

fsecada01 commented Feb 2, 2022

wbolster left a comment

Choose a reason for hiding this comment

pradyunsg commented Feb 3, 2022

di commented Feb 3, 2022

pradyunsg commented Feb 4, 2022

d0ugal commented Feb 4, 2022

atugushev commented Feb 4, 2022

ssbarnea commented Feb 4, 2022

di commented Feb 4, 2022

ssbarnea commented Feb 4, 2022

di commented Feb 2, 2022 •

edited by hugovk