ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: /tmp/pip-install

[allen-munsch]

I was wondering if anyone else has seen this type of error?

-----> Building on the Heroku-20 stack
-----> Using buildpacks:
       1. https://github.com/heroku/heroku-buildpack-cli
       2. https://github.com/heroku/heroku-buildpack-python
       -----> heroku-cli app detected
=== Fetching and vendoring Heroku CLI into slug
=== Installing profile.d script
 ›   Warning: Our terms of service have changed: 
 ›   https://dashboard.heroku.com/terms-of-service
 heroku/7.59.4 linux-x64 node-v12.21.0
=== Heroku CLI installation done
-----> Python app detected
-----> Using Python version specified in runtime.txt
 !     Python has released a security update! Please consider upgrading to python-3.9.12
       Learn More: https://devcenter.heroku.com/articles/python-runtimes
       -----> No change in requirements detected, installing from cache
-----> Using cached install of python-3.9.10
-----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
-----> Installing SQLite3
-----> Installing requirements with pip
       Looking in indexes: https://<authtoken>@pypi.fury.io/somespam/
       Processing /tmp/pip-install-xxx/dependencies_ba123xxx/wheels/httplib2-0.20.2-py3-none-any.whl
       ERROR: Could not install packages due to an OSError: [Errno 2] 
No such file or directory: '/tmp/pip-install-xxx/dependencies_ba123xxx/wheels/httplib2-0.20.2-py3-none-any.whl'
 !     Push rejected, failed to compile Python app.
 !     Push failed

a hacky setup.py

#!/usr/bin/env python
import os
from setuptools import setup, find_packages
build_dir = '/home/jm/pycharm_projects/python-wheels'
with open('requirements.txt') as f:
    requirements = f.read().replace(build_dir, os.getcwd()).split('\n')
setup(name='dependencies',
      version='1.1',
      install_requires=requirements,
)

example from the build.txt

aiven-client @ file:///home/jm/pycharm_projects/python-wheels/wheels/aiven_client-2.14.8-py3-none-any.whl
amqp @ file:///home/jm/pycharm_projects/python-wheels/wheels/amqp-5.0.9-py3-none-any.whl
aniso8601 @ file:///home/jm/pycharm_projects/python-wheels/wheels/aniso8601-7.0.0-py2.py3-none-any.whl
appdirs @ file:///home/jm/pycharm_projects/python-wheels/wheels/appdirs-1.4.4-py2.py3-none-any.whl

pip install wheel
python setup.py bdist_wheel
pip wheel -r requirements.txt
zip --exclude "./venv_3_9_10/*" --exclude ".git/*" -r ../dependencies.zip ./
cd ../
curl --progress-bar --verbose -F package=@dependencies.zip https://${FURY_PUSH_TOKEN}@push.fury.io/somespam/

requirements.txt

--index-url https://<auth_token>:@pypi.fury.io/somespam/
dependencies==1.1

possibly related?:

• Install on Heroku fails with pip error python-poetry/poetry#4297
• Concurrent poetry install fails due to cache issue python-poetry/poetry#5295
• Potentially a concurrency issue while installing packages python-poetry/poetry#4568
• POETRY_CACHE_DIR or cache-dir config are not used at all. python-poetry/poetry#2445
• Could not install packages due to an OSError: [Errno 2] No such file or directory: '' pypa/pip#9685

Works on the first time, but not twice:

pip -vvv install -r "$(pwd)/requirements.txt" \
--exists-action=w \
--src=$(pwd)/v1/src \
--disable-pip-version-check \
--no-cache-dir

Output, reproduction:

Using pip 21.2.4 from /home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-mnj10j5n
Created temporary directory: /tmp/pip-req-tracker-b5le8ezx
Initialized build tracking at /tmp/pip-req-tracker-b5le8ezx
Created build tracker: /tmp/pip-req-tracker-b5le8ezx
Entered build tracker: /tmp/pip-req-tracker-b5le8ezx
Created temporary directory: /tmp/pip-install-y3vd0371
Looking in indexes: https://<auth_token>@pypi.fury.io/somespam/
Requirement already satisfied: dependencies==1.1 in ./v3/lib/python3.9/site-packages (from -r /home/jm/pycharm_projects/somespam/requirements.txt (line 2)) (1.1)
Processing /tmp/pip-install-enrqy_za/dependencies_b08b9414cf1b484c997d8ceeef5018a1/wheels/aiven_client-2.14.8-py3-none-any.whl
  Added aiven-client@ file:///tmp/pip-install-enrqy_za/dependencies_b08b9414cf1b484c997d8ceeef5018a1/wheels/aiven_client-2.14.8-py3-none-any.whl from file:///tmp/pip-install-enrqy_za/dependencies_b08b9414cf1b484c997d8ceeef5018a1/wheels/aiven_client-2.14.8-py3-none-any.whl (from dependencies==1.1->-r /home/jm/pycharm_projects/somespam/requirements.txt (line 2)) to build tracker '/tmp/pip-req-tracker-b5le8ezx'
  Removed aiven-client@ file:///tmp/pip-install-enrqy_za/dependencies_b08b9414cf1b484c997d8ceeef5018a1/wheels/aiven_client-2.14.8-py3-none-any.whl from file:///tmp/pip-install-enrqy_za/dependencies_b08b9414cf1b484c997d8ceeef5018a1/wheels/aiven_client-2.14.8-py3-none-any.whl (from dependencies==1.1->-r /home/jm/pycharm_projects/somespam/requirements.txt (line 2)) from build tracker '/tmp/pip-req-tracker-b5le8ezx'
> /home/jm/.pyenv/versions/3.9.10/lib/python3.9/zipfile.py(1239)__init__()
-> try:
(Pdb) c
ERROR: Could not install packages due to an OSError.
Traceback (most recent call last):
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 315, in run
    requirement_set = resolver.resolve(
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 366, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 212, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 202, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 197, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 197, in <listcomp>
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 395, in iter_dependencies
    yield self._factory.make_requirement_from_spec(str(r), self._ireq)
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 494, in make_requirement_from_spec
    return self._make_requirement_from_install_req(ireq, requested_extras)
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 433, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 204, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 295, in __init__
    super().__init__(
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
    dist = self._prepare_distribution()
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 305, in _prepare_distribution
    return self._factory.preparer.prepare_linked_requirement(
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 570, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 61, in _get_prepared_distribution
    return abstract_dist.get_pkg_resources_distribution()
  File "/home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/distributions/wheel.py", line 26, in get_pkg_resources_distribution
    with ZipFile(self.req.local_file_path, allowZip64=True) as z:
  File "/home/jm/.pyenv/versions/3.9.10/lib/python3.9/zipfile.py", line 1239, in __init__
    try:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-enrqy_za/dependencies_b08b9414cf1b484c997d8ceeef5018a1/wheels/aiven_client-2.14.8-py3-none-any.whl'

After doing some debugging, it looks like the tmp directories are being written into the eggs-info:

01:10:37 jm@jm somespam ±|fix-teardown-reviewapp-dbs ✗|→ cat /home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/dependencies-1.1-py3.9.egg-info/*

PKG-INFO
SOURCES.txt
dependency_links.txt
requires.txt
top_level.txt
Metadata-Version: 2.1
Name: dependencies
Version: 1.1
Summary: UNKNOWN
Home-page: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN

UNKNOWN

aiven-client@ file:///tmp/pip-install-enrqy_za/dependencies_b08b9414cf1b484c997d8ceeef5018a1/wheels/aiven_client-2.14.8-py3-none-any.whl
# ... many other packages, in the same format
psycopg2@ file:///tmp/pip-install-enrqy_za/dependencies_b08b9414cf1b484c997d8ceeef5018a1/wheels/psycopg2-2.9.3-cp39-cp39-linux_x86_64.whl
# ... truncated

README.md
setup.py
dependencies.egg-info/PKG-INFO
dependencies.egg-info/SOURCES.txt
dependencies.egg-info/dependency_links.txt
dependencies.egg-info/requires.txt
dependencies.egg-info/top_level.txt

> /home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py(495)make_requirement_from_spec()
-> return self._make_requirement_from_install_req(ireq, requested_extras)

(Pdb) l
490           specifier: str,
491           comes_from: InstallRequirement,
492           requested_extras: Iterable[str] = (),
493       ) -> Optional[Requirement]:
494           ireq = self._make_install_req_from_spec(specifier, comes_from)
495  ->         return self._make_requirement_from_install_req(ireq, requested_extras)
496   
497       def make_requires_python_requirement(
498           self, specifier: Optional[SpecifierSet]
499       ) -> Optional[Requirement]:
500           if self._ignore_requires_python or specifier is None:

(Pdb) specifier
'aiven-client@ file:///tmp/pip-install-enrqy_za/dependencies_b08b9414cf1b484c997d8ceeef5018a1/wheels/aiven_client-2.14.8-py3-none-any.whl'

(Pdb) comes_from
<InstallRequirement object: dependencies==1.1 in ./v3/lib/python3.9/site-packages (from -r /home/jm/pycharm_projects/somespam/requirements.txt (line 2)) editable=False>

[edmorley]

Hi! Let's move this into the existing support ticket, since we'll be able to discuss app specifics, which may be private :-)