Release 24.1 #12613

sbidoul · 2024-04-06T15:41:16Z

It's April already.

Any @pypa/pip-committers up to take the RM hat this month?

pfmoore · 2024-04-06T16:10:02Z

Do we expect the packaging upgrade to be in 24.1? Because I don't have the bandwidth to deal with that change.

If the packaging upgrade isn't included, I could do it - but looking at the news fragments, it doesn't look like there's much in the release - do we even need a release? (I'm assuming we don't get a flurry of people merging changes to get them into the release - I don't have the bandwidth for that, either!)

pradyunsg · 2024-04-07T15:26:28Z

Do we expect the packaging upgrade to be in 24.1?

Yes.

If no one else has bandwidth, I'd be willing to be the RM for this one.

I expect we're going to have a beta period, to get some user feedback on the legacy version removal situation (thank you to @sbidoul for doing most of the work on #12300!) and to also make follow up fixes we'd likely need.

pfmoore · 2024-04-07T16:44:41Z

OK, if we're putting the packaging upgrade in, I'll leave it with you. I'm happy to help where I can, so feel free to ping me if you think I can help. I'll try to find some time to review @sbidoul's PR in the next day or two.

sbidoul · 2024-04-08T18:04:08Z

Regarding a beta release, I'm not entirely convinced it will help (my instinct says not enough people will notice/care), but that's up to the RM, of course ;)

notatallshaw · 2024-04-09T20:14:44Z

I would test a beta and I'm sure @potiuk would as well, I would also try to spread the word on social media as much as possible. However, I will say, the one time that pip recently had a beta it didn't get much exposure.

On release though, I will again mention that resolving is not sound #12317 and I have a fix here if anyone has time to provide any more feed back sarugaku/resolvelib#144

Also this PR seems fairly small and would be a great performance news item for Pip #12453 (and more complex but would be a big UX improvement news item, would be #12388 and #12404 if they are considered close to ready)

NeilBotelho · 2024-04-11T14:58:47Z

I'd say #12388 is mostly done, but I'm not sure if you would want to merge it into main by itself. It only adds support for parallel downloads without progress bars. #12404 adds support for parallel progress bars, but I'd like some more feedback on that before I'd be ready to call it done.

notatallshaw · 2024-04-28T15:15:41Z

A summary on macos runners which are currently blocking CI runs:

macos-latest has been updated to macos-14 running on ARM from macos-12 running on x86
The subversion package is not available by default and must be installed on macos-14
Until a couple of days ago Python 3.8 and 3.9 weren't available on macos-14
Old versions of virtualenv do not work on macos-14, and pip test suite currently pins Python 3.8 and 3.9 to virtualenv<20
Removing this restriction reveals 1 failure "Tests_UninstallUserSite::test_uninstall_editable_from_usersite" on macos-14 Python 3.8 and 3.9

So it needs to be decided whether to keep running x86 as well as ARM (#12593) or just stick with "latest" which is now ARM (#12617). And someone either needs to fix this test (I don't know what the issue is and have no way to reproduce locally) or whether to exclude Python 3.8 and 3.9 from MacOS ARM.

notatallshaw · 2024-04-28T15:52:30Z

I'd also like to nominate #12657 for 24.1, it's very simple, and I've measured a significant performance improvement on large number of packages to resolve, and even a slight perfornance improvement on small number of packages to resolve.

pradyunsg · 2024-04-29T19:46:11Z

Oh wow, this month has flown by. I'm catching up on my RM duties now. 😅

vx1920 · 2024-04-30T10:05:11Z

I did report small issue while ago: useless warnings in "pip list --outdated --verbose".
Possible fix with adding two lines to commands/list.py was described, it will allow suppress warnings optionally.
See #12540
Should I create pull request ?

colesbury · 2024-05-02T15:16:36Z

I know this was asked by Paul above and answered, but is the packaging upgrade still slotted to be in the 24.1 release? (From the outside, it's hard to tell how close or far away the PR is to being ready.)

Do you expect the 24.1 release before the Python 3.13 beta 1?

sbidoul · 2024-05-02T16:59:28Z

I posted a status update on the PR: #12300 (comment)

notatallshaw · 2024-05-03T18:49:58Z

As it seems there's very likely to be a beta that hopefully will get some wide coverage, I would like to make the case for pip maintainers to be a little bit more agressive merging approved open PRs that are in a good state:

https://github.com/pypa/pip/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved+draft%3Afalse+status%3Asuccess+sort%3Aupdated-desc+

I appreciate pip is under resourced and generally conservative in accepting PRs, but having a long open approved unmerged PR can be a little demoralizing to potential future contributors.

pfmoore · 2024-05-03T19:38:00Z

I've moved a few of those onto the 24.1 milestone, where I think they are clear-cut ones that should be merged. The others I think might need a little more consideration than I have time for right now. (I'm not arguing against them, though).

pradyunsg · 2024-05-06T11:26:15Z

OK, we're almost all set for 24.1b1 to be released today. I have one outstanding PR to merge and then will cut a release after grabbing lunch today.

pradyunsg · 2024-05-06T20:50:30Z

https://pypi.org/project/pip/24.1b1/ is up. I'm not updating get-pip.py or filing a PR to CPython with an updated wheel for this.

pfmoore · 2024-05-07T10:14:49Z

Are you going to post an announcement on discuss.python.org? It might be a good way to attract a few more beta testers.

pradyunsg · 2024-05-07T12:28:41Z

Yep, I plan to do that along with a blog post that serves as a call for beta testing.

That'll all have to be after my work day today though, since pip maintainance isn't a part of my day job anymore.

potiuk · 2024-05-07T12:34:48Z

FYI. I can confirm that the most complex case with installation and eager upgrade for latest airflow version with 24.1b1 works fine. Not super comprehensive, but passes as "smoke test" IMHO

pradyunsg · 2024-05-07T15:18:07Z

@potiuk Is it meaningfully quicker? :)

potiuk · 2024-05-07T17:20:53Z

Not too scientific but looks like ~ 15% (~550 seconds -> ~470 seconds) - on a very small sample.

notatallshaw · 2024-05-07T17:40:24Z

Not too scientific but looks like ~ 15% (~550 seconds -> ~470 seconds) - on a very small sample.

Roughly what is that running?

Anything that gets stuck doing a bit of backtracking should see more significant performance gains. But if the versions here are all tightly constrained that is about in line with what I would expect.

potiuk · 2024-05-07T18:34:30Z

Roughly what is that running?

Roughly this:

curl -fsSL https://github.com/apache/airflow/archive/main.tar.gz | tar xz -C /tmp/tmp.18llqGZdHv --strip 1
pip install --root-user-action ignore --editable '/tmp/tmp.18llqGZdHv[devel-ci]'

potiuk · 2024-05-07T18:59:38Z

Some more detailed stats on ARM as well (likely a bit more time needed to build some wheels):

PIP 24.0:

Total time: ~590 sec

collection + downloading: 320s
Building wheels: 220s
Installing: 46s

24.1b1:

Total time: ~550 sec

collection + downloading: 260s
Building wheels: 220s
Installing: 46s

So ~ 40s of collection/downloading shaved.

notatallshaw · 2024-05-07T19:18:33Z

That seems reasonable, pip hasn't got any faster at downloading, so the IO overhead will be the same, it's the collecting part that got faster.

There is an open PR for pip to start downloading in parallel. Interesting to see if it makes a noticeable difference in a future release.

potiuk · 2024-05-07T19:28:32Z

I will keep an eye. Same step with uv which we use in CI and dev env for speed:

#46 56.23 Resolved 631 packages in 48.60s
#46 142.9 Downloaded 617 packages in 1m 26s
#46 143.1 Installed 628 packages in 238ms

I am not sure when package building happens - but I assume it's both downloading and building that is heavily parallelised as well. and yeah, would be be cool to see those improvements in pip as well.

edgarrmondragon · 2024-05-08T15:36:16Z

I think I found an issue with 24.1b1 with a scenario that works fine on 24.0.

Consider these requirements:

# requirements.txt
sqlalchemy==1.4.*

Create a virtualenv that uses pip 24.1b1:

VIRTUALENV_PIP=24.1b1 virtualenv .venv --python python3.8
source .venv/bin/activate

Install the requirements:

pip install -r requirements.txt

and it fails

Collecting sqlalchemy==1.4.* (from -r requirements.txt (line 1))
  Using cached SQLAlchemy-1.4.52.tar.gz (8.5 MB)
  Preparing metadata (setup.py) ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2791, in requires
    deps.extend(dm[safe_extra(ext)])
KeyError: 'mariadb-connector'

Full output

Collecting sqlalchemy==1.4.* (from -r requirements.txt (line 1))
  Using cached SQLAlchemy-1.4.52.tar.gz (8.5 MB)
  Preparing metadata (setup.py) ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2791, in requires
    deps.extend(dm[safe_extra(ext)])
KeyError: 'mariadb-connector'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 179, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 174, in __bool__
    return any(self)
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 162, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 53, in _iter_built
    candidate = func()
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 185, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 231, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 303, in __init__
    super().__init__(
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 247, in _prepare
    self._check_metadata_consistency(dist)
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 229, in _check_metadata_consistency
    list(dist.iter_dependencies(list(dist.iter_provided_extras())))
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_internal/metadata/pkg_resources.py", line 249, in iter_dependencies
    return self._dist.requires(extras)
  File "/Users/me/repro/pip-24.1-sqlalchemy/.venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2793, in requires
    raise UnknownExtra(
pip._vendor.pkg_resources.UnknownExtra: SQLAlchemy 1.4.52 has no such extra feature 'mariadb-connector'

pradyunsg · 2024-05-08T15:42:05Z

@edgarrmondragon could you file an issue for this (and run the pip install with --debug)?

edgarrmondragon · 2024-05-08T15:53:26Z

@edgarrmondragon could you file an issue for this (and run the pip install with --debug)?

Will do! (at some point later today 😅 )

hmc-cs-mdrissi · 2024-05-08T18:26:59Z

I strongly suspect that error is related to hyphen/underscore normalization in extra names (mariadb-connector vs mariadb_connector) and pkg_resources vs packaging/pip having inconsistent normalization logic. I'd guess packaging upgrade may have touched extra normalization logic and I've seen inconsistencies across pkg_resources normalization/pip in past elsewhere as well with extras having hyphens.

edit: The last time I looked into this error my initial thought was fix seems best placed in pkg_resources itself and it doing lookups in normalized aware manner.

notatallshaw · 2024-05-18T14:20:52Z

So ~ 40s of collection/downloading shaved.

Running your steps of editable installing apache-airflow[devel-ci], but using --dry-run to take out the install time, and running twice so all the wheels are cached to take out the I/O time, as neither of these have been improved between 24.0 and 24.1 this is the improvement I see:

24.0: ~208 s
24.1b1: ~154 s

So an absolute time not too far out from your own testing, and a ~25% improvement in resolving and building metadata from sdists for this set of requirements (note the building itself won't have improved). Interestingly I see a much bigger performance improvement with apache-airflow[all] (~60%), so it seems all hits some pathological case that devel-ci does not, but such is the nature of dependency resolving.

I'm going to make a follow up issue to look at the final install step, as that is taking a surprising amount of time, but that's for future versions of pip.

sbidoul added this to the 24.1 milestone Apr 6, 2024

sbidoul pinned this issue Apr 6, 2024

pradyunsg self-assigned this Apr 8, 2024

ichard26 added the type: maintenance Related to Development and Maintenance Processes label Apr 19, 2024

ichard26 mentioned this issue Apr 26, 2024

[fix]: Move error message formatting recommendation into pip docs #12626 #12642

Open

hugovk mentioned this issue May 1, 2024

[TODO][CI][pip upstream changes] Fix failing nightlies running against pip's main branch jazzband/pip-tools#2078

Closed

pradyunsg mentioned this issue May 6, 2024

Release 24.1.beta1 #12673

Merged

mayeut mentioned this issue May 8, 2024

feat: add CPython 3.13 --disable-gil build pypa/manylinux#1564

Merged

edgarrmondragon mentioned this issue May 8, 2024

Installation error for package with un-normalized extra names in pip 24.1b1 #12688

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release 24.1 #12613

Release 24.1 #12613

sbidoul commented Apr 6, 2024

pfmoore commented Apr 6, 2024

pradyunsg commented Apr 7, 2024

pfmoore commented Apr 7, 2024

sbidoul commented Apr 8, 2024

notatallshaw commented Apr 9, 2024 •

edited

NeilBotelho commented Apr 11, 2024

notatallshaw commented Apr 28, 2024 •

edited

notatallshaw commented Apr 28, 2024

pradyunsg commented Apr 29, 2024

vx1920 commented Apr 30, 2024

colesbury commented May 2, 2024 •

edited

sbidoul commented May 2, 2024

notatallshaw commented May 3, 2024 •

edited

pfmoore commented May 3, 2024

pradyunsg commented May 6, 2024

pradyunsg commented May 6, 2024

pfmoore commented May 7, 2024

pradyunsg commented May 7, 2024

potiuk commented May 7, 2024

pradyunsg commented May 7, 2024

potiuk commented May 7, 2024

notatallshaw commented May 7, 2024

potiuk commented May 7, 2024

potiuk commented May 7, 2024

notatallshaw commented May 7, 2024

potiuk commented May 7, 2024

edgarrmondragon commented May 8, 2024

pradyunsg commented May 8, 2024 •

edited

edgarrmondragon commented May 8, 2024

hmc-cs-mdrissi commented May 8, 2024 •

edited

notatallshaw commented May 18, 2024

Release 24.1 #12613

Release 24.1 #12613

Comments

sbidoul commented Apr 6, 2024

pfmoore commented Apr 6, 2024

pradyunsg commented Apr 7, 2024

pfmoore commented Apr 7, 2024

sbidoul commented Apr 8, 2024

notatallshaw commented Apr 9, 2024 • edited

NeilBotelho commented Apr 11, 2024

notatallshaw commented Apr 28, 2024 • edited

notatallshaw commented Apr 28, 2024

pradyunsg commented Apr 29, 2024

vx1920 commented Apr 30, 2024

colesbury commented May 2, 2024 • edited

sbidoul commented May 2, 2024

notatallshaw commented May 3, 2024 • edited

pfmoore commented May 3, 2024

pradyunsg commented May 6, 2024

pradyunsg commented May 6, 2024

pfmoore commented May 7, 2024

pradyunsg commented May 7, 2024

potiuk commented May 7, 2024

pradyunsg commented May 7, 2024

potiuk commented May 7, 2024

notatallshaw commented May 7, 2024

potiuk commented May 7, 2024

potiuk commented May 7, 2024

notatallshaw commented May 7, 2024

potiuk commented May 7, 2024

edgarrmondragon commented May 8, 2024

pradyunsg commented May 8, 2024 • edited

edgarrmondragon commented May 8, 2024

hmc-cs-mdrissi commented May 8, 2024 • edited

notatallshaw commented May 18, 2024

notatallshaw commented Apr 9, 2024 •

edited

notatallshaw commented Apr 28, 2024 •

edited

colesbury commented May 2, 2024 •

edited

notatallshaw commented May 3, 2024 •

edited

pradyunsg commented May 8, 2024 •

edited

hmc-cs-mdrissi commented May 8, 2024 •

edited