You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Originally reported privately to avoid early disclosure.
Report
In the test suite of https://pypi.org/project/setuptools/,
test_virtualenv unintentionally downloads unrelated projects from PyPI
(some of which don't exist):
It only happens in tests, so it's not a high severity issue. I'm
reporting it privately here so that y'all hopefully don't need to race
against possible bad actors :)
The happens because setuptools passes filenames to pip's --find-links,
for example:
--find-links
/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/foobar-2.4.tar.gz
But, --find-links only accepts directories (and remote URLs), so the
option has no effect, and pip goes on to install foobar-2.4 from PyPI.
I'm attaching a log of the tests, run without connectivity to the
Internet (which makes them fail).
When online, the tests download packages from PyPI, and pass.
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-6.0.2, py-1.9.0, pluggy-0.13.1
rootdir: /builddir/build/BUILD/setuptools-50.3.2, configfile: pytest.ini
plugins: virtualenv-1.7.0, shutil-1.7.0
collected 684 items / 2 skipped / 682 selected
pkg_resources/__init__.py ... [ 0%]
pkg_resources/api_tests.txt . [ 0%]
pkg_resources/tests/test_find_distributions.py ... [ 1%]
pkg_resources/tests/test_markers.py . [ 1%]
pkg_resources/tests/test_pkg_resources.py ................ssss [ 4%]
pkg_resources/tests/test_resources.py .................................. [ 9%]
................................................ [ 16%]
pkg_resources/tests/test_working_set.py ................................ [ 20%]
.. [ 21%]
setuptools/namespaces.py . [ 21%]
setuptools/package_index.py .. [ 21%]
setuptools/sandbox.py . [ 21%]
setuptools/command/develop.py . [ 21%]
setuptools/command/install_lib.py . [ 21%]
setuptools/tests/test_archive_util.py X [ 22%]
setuptools/tests/test_bdist_deprecations.py . [ 22%]
setuptools/tests/test_bdist_egg.py .x. [ 22%]
setuptools/tests/test_build_clib.py . [ 22%]
setuptools/tests/test_build_ext.py ... [ 23%]
setuptools/tests/test_build_meta.py .................................... [ 28%]
........................................................................ [ 39%]
.......... [ 40%]
setuptools/tests/test_build_py.py ... [ 40%]
setuptools/tests/test_config.py ....................................... [ 46%]
setuptools/tests/test_dep_util.py . [ 46%]
setuptools/tests/test_depends.py . [ 46%]
setuptools/tests/test_develop.py .s.... [ 47%]
setuptools/tests/test_dist.py ........x.x.................... [ 52%]
setuptools/tests/test_dist_info.py .. [ 52%]
setuptools/tests/test_distutils_adoption.py ... [ 53%]
setuptools/tests/test_easy_install.py .................................. [ 58%]
................ [ 60%]
setuptools/tests/test_egg_info.py .........................x............ [ 65%]
........................ [ 69%]
setuptools/tests/test_extern.py ... [ 69%]
setuptools/tests/test_find_packages.py ............ [ 71%]
setuptools/tests/test_glob.py .... [ 72%]
setuptools/tests/test_install_scripts.py .s.s [ 72%]
setuptools/tests/test_manifest.py ...................................... [ 78%]
............................. [ 82%]
setuptools/tests/test_msvc14.py ssss [ 83%]
setuptools/tests/test_namespaces.py .... [ 83%]
setuptools/tests/test_packageindex.py .................... [ 86%]
setuptools/tests/test_register.py . [ 86%]
setuptools/tests/test_sandbox.py .......... [ 88%]
setuptools/tests/test_sdist.py ............... [ 90%]
setuptools/tests/test_setopt.py . [ 90%]
setuptools/tests/test_setuptools.py ..................... [ 93%]
setuptools/tests/test_test.py .... [ 94%]
setuptools/tests/test_unicode_utils.py . [ 94%]
setuptools/tests/test_upload.py . [ 94%]
setuptools/tests/test_upload_docs.py .. [ 94%]
setuptools/tests/test_virtualenv.py ..sssssF.. [ 96%]
setuptools/tests/test_wheel.py ...................... [ 99%]
setuptools/tests/test_windows_wrappers.py sss [100%]
=================================== FAILURES ===================================
____________________ test_test_command_install_requirements ____________________
virtualenv = <pytest_virtualenv.VirtualEnv object at 0x7fd1e4d080d0>
tmpdir = local('/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0')
def test_test_command_install_requirements(virtualenv, tmpdir):
# Ensure pip/wheel packages are installed.
virtualenv.run(
"python -c \"__import__('pkg_resources').require(['pip', 'wheel'])\"")
> _check_test_command_install_requirements(virtualenv, tmpdir)
setuptools/tests/test_virtualenv.py:180:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
setuptools/tests/test_virtualenv.py:171: in _check_test_command_install_requirements
virtualenv.run(
/usr/lib/python3.9/site-packages/pytest_virtualenv.py:151: in run
return super(VirtualEnv, self).run(args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pytest_virtualenv.VirtualEnv object at 0x7fd1e4d080d0>
cmd = ['python', 'setup.py', 'test', '-s', 'test'], capture = False
check_rc = True
cd = '/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0'
shell = False
kwargs = {'env': {'CFLAGS': '-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-...fstack-clash-protection -fcf-protection', 'CONFIG_SITE': 'NONE', 'HISTCONTROL': 'ignoredups', 'HISTSIZE': '1000', ...}}
p = <Popen: returncode: 1 args: ['python', 'setup.py', 'test', '-s', 'test']>
out = None, _ = None
err = CalledProcessError(1, ['python', 'setup.py', 'test', '-s', 'test'])
> ???
E subprocess.CalledProcessError: Command '['python', 'setup.py', 'test', '-s', 'test']' returned non-zero exit status 1.
/builddir/build/BUILDROOT/python-pytest-shutil-1.7.0-10.fc34.noarch/usr/lib/python3.9/site-packages/pytest_shutil/workspace.py:132: CalledProcessError
---------------------------- Captured stdout setup -----------------------------
created virtual environment CPython3.9.1.final.0-64 in 232ms
creator CPython3Posix(dest=/tmp/tmp63y4m4cd/.env, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(extra_search_dir=/usr/share/python-wheels,download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/builddir/.local/share/virtualenv)
added seed packages: pip==20.3.3, setuptools==50.3.2, wheel==0.35.1
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
----------------------------- Captured stdout call -----------------------------
running develop
running egg_info
writing setuptools.egg-info/PKG-INFO
writing dependency_links to setuptools.egg-info/dependency_links.txt
writing entry points to setuptools.egg-info/entry_points.txt
writing requirements to setuptools.egg-info/requires.txt
writing top-level names to setuptools.egg-info/top_level.txt
reading manifest file 'setuptools.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'setuptools.egg-info/SOURCES.txt'
running build_ext
Creating /tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/setuptools.egg-link (link to .)
Adding setuptools 50.3.2 to easy-install.pth file
Installing easy_install script to /tmp/tmp63y4m4cd/.env/bin
Installing easy_install-3.9 script to /tmp/tmp63y4m4cd/.env/bin
Installed /builddir/build/BUILD/setuptools-50.3.2
Processing dependencies for setuptools==50.3.2
Finished processing dependencies for setuptools==50.3.2
----------------------------- Captured stderr call -----------------------------
warning: no files found matching '*.py' under directory 'tests'
warning: no files found matching '*.py' under directory 'docs'
warning: no files found matching '*.conf' under directory 'docs'
warning: no files found matching '*.css' under directory 'docs'
warning: no files found matching '*.css_t' under directory 'docs'
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
warning: no previously-included files found matching 'pyproject.toml'
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f731bbe12b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/bits/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f731bbe1e80>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/bits/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f731bbe1ca0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/bits/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f731bbe1310>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/bits/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f731bbe1f10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/bits/
ERROR: Exception:
Traceback (most recent call last):
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 224, in _main
status = self.run(options, args)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
return func(self, options, args)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/commands/wheel.py", line 154, in run
requirement_set = resolver.resolve(
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 121, in resolve
self._result = resolver.resolve(
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 445, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 339, in resolve
failure_causes = self._attempt_to_pin_criterion(name, criterion)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 207, in _attempt_to_pin_criterion
criteria = self._get_criteria_to_update(candidate)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 198, in _get_criteria_to_update
for r in self._p.get_dependencies(candidate):
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 170, in get_dependencies
return [
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 170, in <listcomp>
return [
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 260, in iter_dependencies
yield self._get_requires_python_dependency()
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 244, in _get_requires_python_dependency
requires_python = get_requires_python(self.dist)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 239, in dist
self._prepare()
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 226, in _prepare
dist = self._prepare_distribution()
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 318, in _prepare_distribution
return self._factory.preparer.prepare_linked_requirement(
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 480, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 503, in _prepare_linked_requirement
local_file = unpack_url(
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 253, in unpack_url
file = get_http_url(
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 130, in get_http_url
from_path, content_type = download(link, temp_dir.path)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/network/download.py", line 150, in __call__
resp = _http_get_download(self._session, link)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/network/download.py", line 131, in _http_get_download
resp = session.get(target_url, headers=HEADERS, stream=True)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_internal/network/session.py", line 428, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 528, in request
prep = self.prepare_request(req)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 456, in prepare_request
p.prepare(
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_vendor/requests/models.py", line 316, in prepare
self.prepare_url(url, params)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pip/_vendor/requests/models.py", line 390, in prepare_url
raise MissingSchema(error)
pip._vendor.requests.exceptions.MissingSchema: Invalid URL '/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/bits-4.2.tar.gz': No schema supplied. Perhaps you meant http:///tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/bits-4.2.tar.gz?
Traceback (most recent call last):
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/setuptools/installer.py", line 126, in fetch_build_egg
subprocess.check_call(cmd)
File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/tmp/tmp63y4m4cd/.env/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp6whudypf', '--quiet', '--find-links', '/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/foobar-2.4.tar.gz', '--find-links', '/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/bits-4.2.tar.gz', '--find-links', '/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/bobs-6.0.tar.gz', '--find-links', '/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/pieces-0.6.tar.gz', 'bits==4.2']' returned non-zero exit status 2.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/setup.py", line 3, in <module>
setup(
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 152, in setup
_install_setup_requires(attrs)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 147, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/setuptools/dist.py", line 673, in fetch_build_eggs
resolved_dists = pkg_resources.working_set.resolve(
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pkg_resources/__init__.py", line 764, in resolve
dist = best[req.key] = env.best_match(
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1049, in best_match
return self.obtain(req, installer)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1061, in obtain
return installer(requirement)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/setuptools/dist.py", line 732, in fetch_build_egg
return fetch_build_egg(self, req)
File "/tmp/tmp63y4m4cd/.env/lib/python3.9/site-packages/setuptools/installer.py", line 128, in fetch_build_egg
raise DistutilsError(str(e)) from e
distutils.errors.DistutilsError: Command '['/tmp/tmp63y4m4cd/.env/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp6whudypf', '--quiet', '--find-links', '/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/foobar-2.4.tar.gz', '--find-links', '/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/bits-4.2.tar.gz', '--find-links', '/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/bobs-6.0.tar.gz', '--find-links', '/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/pieces-0.6.tar.gz', 'bits==4.2']' returned non-zero exit status 2.
=========================== short test summary info ============================
Reviewing what the test does, it seems unlikely that the tests could pass as described:
--find-links only accepts directories
If that were the case, the tests would surely fail, as the tests explicitly import modules from the various requirements.
Still, it's obvious from the "MissingSchema" error (in the logs) that something was causing the file:// schema to be removed from each of the find_links, as as_uri() is used to generate the dependency_links/find_links.
Troubleshooting
This is the command that the test runs when it defers to pip:
Then by referencing that sdist in a pip wheel command:
draft $ pip wheel --find-links file:///Users/jaraco/draft/bits/dist/bits-4.2.tar.gz --index-url https://127.0.0.1 bits==4.2
Looking in indexes: https://127.0.0.1
Looking in links: file:///Users/jaraco/draft/bits/dist/bits-4.2.tar.gz
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdfe2973250>: Failed to establish a new connection: [Errno 61] Connection refused')': /bits/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdfe2973a30>: Failed to establish a new connection: [Errno 61] Connection refused')': /bits/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdfe2973be0>: Failed to establish a new connection: [Errno 61] Connection refused')': /bits/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdfe2973d90>: Failed to establish a new connection: [Errno 61] Connection refused')': /bits/
Processing ./bits/dist/bits-4.2.tar.gz
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdfe2973f40>: Failed to establish a new connection: [Errno 61] Connection refused')': /bits/
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-wheel-dahiy4y2/bits_ea589c753024466e9bebae681c1484f9/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-wheel-dahiy4y2/bits_ea589c753024466e9bebae681c1484f9/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-pip-egg-info-k0qdmrye
cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-wheel-dahiy4y2/bits_ea589c753024466e9bebae681c1484f9/
Complete output (3 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
The problem is not that it fails to build but that it attempts to download the project from the index URL.
I've found with this diff, I can observe the output:
diff --git a/setuptools/tests/test_virtualenv.py b/setuptools/tests/test_virtualenv.py
index 950c74c5d..da5e39f53 100644
--- a/setuptools/tests/test_virtualenv.py
+++ b/setuptools/tests/test_virtualenv.py
@@ -181,6 +181,9 @@ def test_test_command_install_requirements(virtualenv, tmpdir):
"python -c \"__import__('pkg_resources').require(['pip', 'wheel'])\"")
# uninstall setuptools so that 'setup.py develop' works
virtualenv.run("python -m pip uninstall -y setuptools")
+ # disable index URL so bits and bobs aren't requested from PyPI
+ virtualenv.env['PIP_NO_INDEX'] = '1'
+ virtualenv.env['PIP_VERBOSE'] = '1'
_check_test_command_install_requirements(virtualenv, tmpdir)
And in particular, when running tox -- -s -k 'test_test_command_install and not easy_install' -p no:cov, I see:
Subsequently, reporter indicates the issue exists in other tests:
Apparently there are more (sorry for not including all at once!): test_easy_install.py downloads "foobar" and "foo-bar".
It does appear as if many tests in test_easy_install indicate PIP_NO_INDEX, but apparently not all.
After patching to ensure verbose output from pip, I've found running this command can elicit the failures:
diff --git a/setuptools/tests/test_easy_install.py b/setuptools/tests/test_easy_install.py
index dc00e697d..c0271c4e9 100644
--- a/setuptools/tests/test_easy_install.py+++ b/setuptools/tests/test_easy_install.py@@ -38,6 +38,11 @@ from .files import build_files
from .textwrap import DALS
+@pytest.fixture(autouse=True)+def pip_always_verbose(monkeypatch):+ monkeypatch.setenv('PIP_VERBOSE', 'true')++
class FakeDist:
def get_entry_map(self, group):
if group != 'console_scripts':
tox -- -k test_easy_install -v -s -p no:cov | grep pypi.org
* https://pypi.org/simple/foobar/
Fetching project page and analyzing links: https://pypi.org/simple/foobar/
Getting page https://pypi.org/simple/foobar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foobar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foobar/ HTTP/1.1" 304 0
* https://pypi.org/simple/foobar/
Fetching project page and analyzing links: https://pypi.org/simple/foobar/
Getting page https://pypi.org/simple/foobar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foobar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foobar/ HTTP/1.1" 304 0
* https://pypi.org/simple/foobar/
Fetching project page and analyzing links: https://pypi.org/simple/foobar/
Getting page https://pypi.org/simple/foobar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foobar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foobar/ HTTP/1.1" 304 0
* https://pypi.org/simple/foobar/
Fetching project page and analyzing links: https://pypi.org/simple/foobar/
Getting page https://pypi.org/simple/foobar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foobar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foobar/ HTTP/1.1" 304 0
* https://pypi.org/simple/foo-bar/
Fetching project page and analyzing links: https://pypi.org/simple/foo-bar/
Getting page https://pypi.org/simple/foo-bar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foo-bar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foo-bar/ HTTP/1.1" 404 13
Could not fetch URL https://pypi.org/simple/foo-bar/: 404 Client Error: Not Found for url: https://pypi.org/simple/foo-bar/ - skipping
* https://pypi.org/simple/foo-bar/
Fetching project page and analyzing links: https://pypi.org/simple/foo-bar/
Getting page https://pypi.org/simple/foo-bar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foo-bar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foo-bar/ HTTP/1.1" 404 13
Could not fetch URL https://pypi.org/simple/foo-bar/: 404 Client Error: Not Found for url: https://pypi.org/simple/foo-bar/ - skipping
* https://pypi.org/simple/foo-bar/
Fetching project page and analyzing links: https://pypi.org/simple/foo-bar/
Getting page https://pypi.org/simple/foo-bar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foo-bar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foo-bar/ HTTP/1.1" 404 13
Could not fetch URL https://pypi.org/simple/foo-bar/: 404 Client Error: Not Found for url: https://pypi.org/simple/foo-bar/ - skipping
* https://pypi.org/simple/foo-bar/
Fetching project page and analyzing links: https://pypi.org/simple/foo-bar/
Getting page https://pypi.org/simple/foo-bar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foo-bar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foo-bar/ HTTP/1.1" 404 13
Could not fetch URL https://pypi.org/simple/foo-bar/: 404 Client Error: Not Found for url: https://pypi.org/simple/foo-bar/ - skipping
* https://pypi.org/simple/foobar/
Fetching project page and analyzing links: https://pypi.org/simple/foobar/
Getting page https://pypi.org/simple/foobar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foobar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foobar/ HTTP/1.1" 304 0
* https://pypi.org/simple/foobar/
Fetching project page and analyzing links: https://pypi.org/simple/foobar/
Getting page https://pypi.org/simple/foobar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foobar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foobar/ HTTP/1.1" 304 0
* https://pypi.org/simple/foobar/
Fetching project page and analyzing links: https://pypi.org/simple/foobar/
Getting page https://pypi.org/simple/foobar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foobar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foobar/ HTTP/1.1" 304 0
* https://pypi.org/simple/foobar/
Fetching project page and analyzing links: https://pypi.org/simple/foobar/
Getting page https://pypi.org/simple/foobar/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/foobar/" in the cache
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/foobar/ HTTP/1.1" 304 0
setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_python_requires Ignoring indexes: https://pypi.org/simple
Ignoring indexes: https://pypi.org/simple
Applying this diff:
setuptools main $ git diff
diff --git a/setuptools/tests/test_easy_install.py b/setuptools/tests/test_easy_install.py
index dc00e697d..9eb6d5612 100644
--- a/setuptools/tests/test_easy_install.py
+++ b/setuptools/tests/test_easy_install.py
@@ -38,6 +38,17 @@ from .files import build_files
from .textwrap import DALS
+@pytest.fixture(autouse=True)
+def pip_disable_index(monkeypatch):
+ """
+ Important: Disable the default index for pip to avoid
+ querying packages in the index and potentially resolving
+ and installing packages there.
+ """
+ monkeypatch.setenv('PIP_NO_INDEX', 'true')
+ monkeypatch.setenv('PIP_VERBOSE', 'true')
+
+
class FakeDist:
def get_entry_map(self, group):
if group != 'console_scripts':
Setuptools Tests Search PyPI
Originally reported privately to avoid early disclosure.
Report
In the test suite of https://pypi.org/project/setuptools/,
test_virtualenv unintentionally downloads unrelated projects from PyPI
(some of which don't exist):
It only happens in tests, so it's not a high severity issue. I'm
reporting it privately here so that y'all hopefully don't need to race
against possible bad actors :)
The happens because setuptools passes filenames to pip's --find-links,
for example:
--find-links
/tmp/pytest-of-mockbuild/pytest-0/test_test_command_install_requ0/foobar-2.4.tar.gz
But, --find-links only accepts directories (and remote URLs), so the
option has no effect, and pip goes on to install foobar-2.4 from PyPI.
I'm attaching a log of the tests, run without connectivity to the
Internet (which makes them fail).
When online, the tests download packages from PyPI, and pass.
Analysis
The affected test appears to be test_test_command_install_requirements.
Reviewing what the test does, it seems unlikely that the tests could pass as described:
If that were the case, the tests would surely fail, as the tests explicitly import modules from the various requirements.
Still, it's obvious from the "MissingSchema" error (in the logs) that something was causing the
file://schema to be removed from each of the find_links, as as_uri() is used to generate the dependency_links/find_links.Troubleshooting
This is the command that the test runs when it defers to pip:
['/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp6ptlgv1_/.env/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp94_0fhx4', '--quiet', '--find-links', 'file:///private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pytest-of-jaraco/pytest-69/test_test_command_install_requ0/foobar-2.4.tar.gz', '--find-links', 'file:///private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pytest-of-jaraco/pytest-69/test_test_command_install_requ0/bits-4.2.tar.gz', '--find-links', 'file:///private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pytest-of-jaraco/pytest-69/test_test_command_install_requ0/bobs-6.0.tar.gz', '--find-links', 'file:///private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pytest-of-jaraco/pytest-69/test_test_command_install_requ0/pieces-0.6.tar.gz', 'bits==4.2']I was able to replicate the issue, first by creating a 'bits' sdist:
Then by referencing that sdist in a
pip wheelcommand:The problem is not that it fails to build but that it attempts to download the project from the index URL.
I've found with this diff, I can observe the output:
And in particular, when running
tox -- -s -k 'test_test_command_install and not easy_install' -p no:cov, I see:So I'll commit just the NO_INDEX setting.
Patch committed at #2515 and released as 51.1.1.
Follow-up
Subsequently, reporter indicates the issue exists in other tests:
It does appear as if many tests in
test_easy_installindicatePIP_NO_INDEX, but apparently not all.After patching to ensure verbose output from pip, I've found running this command can elicit the failures:
Applying this diff:
demonstrates the fix:
Patch applied in a499dee, pending release with 51.2.0.
The text was updated successfully, but these errors were encountered: