Regression in pip 22.3 when using wheel directly with Python 3.9 on Windows

[radoering]

Description

In python-poetry/poetry#6950, we noticed a regression considering the embedded pip wheel in virtualenv. The issue can only be reproduced with Python 3.9 on Windows with pip 22.3 and 22.3.1. Since the bug can be reproduced without virtualenv, I take my chance here. We've already figured out that the bug comes from the update of the vendored pep517, strictly speaking from pypa/pyproject-hooks#144 so maybe the fix has to be done there.

The issue can be reproduced by running python pip-22.3.1-py3-none-any.whl\pip install tomli-2.0.1.tar.gz which fails with an OSError.

Expected behavior

Command succeeds.

pip version

22.3.1

Python version

3.9

OS

Windows

How to Reproduce

1. Download the pip 22.3.1 wheel from pypi.org
2. Download any tar.gz from pypi.org (e.g. tomli)
3. Run python pip-22.3.1-py3-none-any.whl\pip install tomli-2.0.1.tar.gz

Output

Processing c:\temp\tomli-2.0.1.tar.gz
  Installing build dependencies ... done
ERROR: Could not install packages due to an OSError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Me\\AppData\\Local\\Temp\\tmp54jjkhzq_in_process.py'
Check the permissions.

Output with -v

Using pip 22.3.1 from C:\temp\pip-22.3.1-py3-none-any.whl\pip (python 3.9)
Processing c:\temp\tomli-2.0.1.tar.gz
  Running command pip subprocess to install build dependencies
  Collecting flit_core<4,>=3.2.0
    Using cached flit_core-3.8.0-py3-none-any.whl (62 kB)
  Installing collected packages: flit_core
  Successfully installed flit_core-3.8.0
  Installing build dependencies ... done
ERROR: Could not install packages due to an OSError.
Check the permissions.
Traceback (most recent call last):
  File "C:\Python39\lib\importlib\_common.py", line 35, in _tempfile
    os.write(fd, reader())
  File "C:\Python39\lib\zipfile.py", line 2331, in read_bytes
    with self.open('rb') as strm:
  File "C:\Python39\lib\zipfile.py", line 2315, in open
    stream = self.root.open(self.at, zip_mode, pwd=pwd)
  File "C:\Python39\lib\zipfile.py", line 1511, in open
    zinfo = self.getinfo(name)
  File "C:\Python39\lib\zipfile.py", line 1438, in getinfo
    raise KeyError(
KeyError: "There is no item named 'pip\\\\_vendor\\\\pep517\\\\in_process/_in_process.py' in the archive"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\commands\install.py", line 400, in run
    requirement_set = resolver.resolve(
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\resolution\resolvelib\resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\resolution\resolvelib\factory.py", line 491, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\resolution\resolvelib\factory.py", line 453, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\resolution\resolvelib\factory.py", line 206, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\resolution\resolvelib\candidates.py", line 297, in __init__
    super().__init__(
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\resolution\resolvelib\candidates.py", line 162, in __init__
    self.dist = self._prepare()
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\resolution\resolvelib\candidates.py", line 231, in _prepare
    dist = self._prepare_distribution()
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\resolution\resolvelib\candidates.py", line 308, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\operations\prepare.py", line 491, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\operations\prepare.py", line 577, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\operations\prepare.py", line 69, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\distributions\sdist.py", line 48, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\distributions\sdist.py", line 118, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\distributions\sdist.py", line 95, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_internal\utils\misc.py", line 685, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_vendor\pep517\wrappers.py", line 173, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "C:\temp\pip-22.3.1-py3-none-any.whl\pip\_vendor\pep517\wrappers.py", line 307, in _call_hook
    with _in_proc_script_path() as script:
  File "C:\Python39\lib\contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "C:\Python39\lib\importlib\_common.py", line 52, in as_file
    with _tempfile(path.read_bytes, suffix=path.name) as local:
  File "C:\Python39\lib\contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "C:\Python39\lib\importlib\_common.py", line 40, in _tempfile
    os.remove(raw_path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Me\\AppData\\Local\\Temp\\tmpo9i8on3h_in_process.py'

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

python pip-22.3.1-py3-none-any.whl\pip install tomli-2.0.1.tar.gz

This is not a supported way of using pip.

See https://peps.python.org/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file

Either install pip in the environment, or use pip --python ... install to manage a different interpreter's packages.

[pfmoore]

The issue can be reproduced by running python pip-22.3.1-py3-none-any.whl\pip install tomli-2.0.1.tar.gz which fails with an OSError.

Running pip like that isn't a supported way of running pip. Can you give an example of the failure in a supported command? I assume that will probably involve some sort of virtualenv invocation, as you mentioned that was where you first found the issue.

To be clear, virtualenv might be running pip from the wheel - it's usually OK as long as you know what you're doing, but it's the "as long as you know what you're doing" bit that's critical. In particular, I wouldn't normally recommend installing a sdist like this, so I'm curious to know how that command came to be invoked.

[radoering]

This is not a supported way of using pip.

See https://peps.python.org/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file

Good to know. Thanks for the link.

Can you give an example of the failure in a supported command?

I don't think so.

so I'm curious to know how that command came to be invoked.

The original (full) command is something like

C:\temp\testvenv\Scripts\python.exe C:\temp\pip-22.3-py3-none-any.whl\pip install --use-pep517 --disable-pip-version-check --isolated --no-input --prefix C:\temp\testvenv --upgrade --no-deps tomli-2.0.1.tar.gz

It's used as a fallback to install packages in a virtual environment if there is no pip inside the virtual environment.

[pfmoore]

Yeah, that's just not supported. You could try using the new zipapp distribution of pip.