Use of (Deprecated) pep517 Package Hides setup.py Syntax Errors

[Aaron-Beal-CardinalPeak]

Running python -m piptools compile when the setup.py file has invalid Python in it leads to this unhelpful error message from the pep517 package:

Traceback (most recent call last):
  File "C:\Users\ab\.pyenv\pyenv-win\versions\3.8.10\Lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\ab\.pyenv\pyenv-win\versions\3.8.10\Lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\ab\git\test\venv\lib\site-packages\piptools\__main__.py", line 17, in <module>
    cli()
  File "C:\Users\ab\git\test\venv\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\ab\git\test\venv\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\ab\git\test\venv\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\ab\git\test\venv\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\ab\git\test\venv\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\ab\git\test\venv\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\ab\git\test\venv\lib\site-packages\piptools\scripts\compile.py", line 408, in cli
    dist = meta.load(os.path.dirname(os.path.abspath(src_file)))
  File "C:\Users\ab\git\test\venv\lib\site-packages\pep517\meta.py", line 71, in load
    path = Path(build_as_zip(builder))
  File "C:\Users\ab\git\test\venv\lib\site-packages\pep517\meta.py", line 58, in build_as_zip
    builder(dest=out_dir)
  File "C:\Users\ab\git\test\venv\lib\site-packages\pep517\meta.py", line 53, in build
    _prep_meta(hooks, env, dest)
  File "C:\Users\ab\git\test\venv\lib\site-packages\pep517\meta.py", line 28, in _prep_meta
    reqs = hooks.get_requires_for_build_wheel({})
  File "C:\Users\ab\git\test\venv\lib\site-packages\pep517\wrappers.py", line 172, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "C:\Users\ab\git\test\venv\lib\site-packages\pep517\wrappers.py", line 322, in _call_hook
    self._subprocess_runner(
  File "C:\Users\ab\git\test\venv\lib\site-packages\pep517\wrappers.py", line 75, in quiet_subprocess_runner
    check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
  File "C:\Users\ab\.pyenv\pyenv-win\versions\3.8.10\Lib\subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Users\ab\.pyenv\pyenv-win\versions\3.8.10\Lib\subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\Users\\ab\\git\\test\\venv\\Scripts\\python.exe', 'C:\\Users\\ab\\git\\test\\venv\\lib\\site-packages\\pep517\\in_process\\_in_process.py', 'get_requires_for_build_wheel', 'C:\\Users\\ab\\AppData\\Local\\Temp\\tmp8_bd4noo']' returned non-zero exit status 1.

This issue has been brought up in the issue tracker for that project (pypa/pyproject-hooks#142). The response was that the API being used was a proof-of-concept and is now deprecated. As such pip-tools should look into using an alternative.

Environment Versions

1. Windows 10
2. Python version: 3.8.10
3. pip version: 22.1.1
4. pip-tools version: 6.6.1

Steps to replicate

1. In an empty directory create a setup.py file that contains invalid python. Mine has a single line: setup(
2. Create a virtual environment and activate it. Then pip install pip-tools.
3. Run python -m piptools compile.

Expected result

Print something mentioning that thesetup.py file failed to be parsed/loaded.

Actual result

Get the above error message.

[AndydeCleyre]

Something as simple as this seems pretty close, but seems to fail on setup.cfg-only builds:

diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
index 0a2e828..224bfdc 100755
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -6,8 +6,8 @@ import tempfile
 from typing import IO, Any, BinaryIO, List, Optional, Tuple, Union, cast
 
 import click
+from build.util import project_wheel_metadata
 from click.utils import LazyFile, safecall
-from pep517 import meta
 from pip._internal.commands import create_command
 from pip._internal.req import InstallRequirement
 from pip._internal.req.constructors import install_req_from_line
@@ -405,12 +405,14 @@ def cli(
             constraints.extend(reqs)
         elif is_setup_file:
             setup_file_found = True
-            dist = meta.load(os.path.dirname(os.path.abspath(src_file)))
-            comes_from = f"{dist.metadata.get_all('Name')[0]} ({src_file})"
+            metadata = project_wheel_metadata(
+                os.path.dirname(os.path.abspath(src_file))
+            )
+            comes_from = f"{metadata.get_all('Name')[0]} ({src_file})"
             constraints.extend(
                 [
                     install_req_from_line(req, comes_from=comes_from)
-                    for req in dist.requires or []
+                    for req in metadata.get_all("Requires-Dist") or []
                 ]
             )
         else:
diff --git a/setup.cfg b/setup.cfg
index 9e9215e..5d84058 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -30,8 +30,8 @@ packages = find:
 zip_safe = false
 install_requires =
     # direct dependencies
+    build
     click >= 7
-    pep517
     pip >= 21.2
     # indirect dependencies
     setuptools  # typically needed when pip-tools invokes setup.py

[AndydeCleyre]

Although the error output when using build instead of pep517 directly isn't exactly clear either:

/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/build/__init__.py", line 482, in _handle_backend
    yield
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/build/__init__.py", line 369, in get_requires_for_build
    return set(get_requires(config_settings))
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/pep517/wrappers.py", line 172, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/pep517/wrappers.py", line 322, in _call_hook
    self._subprocess_runner(
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/build/__init__.py", line 320, in _runner
    self._hook_runner(cmd, cwd, extra_environ)
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/pep517/wrappers.py", line 75, in quiet_subprocess_runner
    check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/build-env-syt2th92/bin/python', '/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmp_1mc294c']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/bin/pip-compile", line 33, in <module>
    sys.exit(load_entry_point('pip-tools', 'console_scripts', 'pip-compile')())
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/andy/Code/pip-tools/piptools/scripts/compile.py", line 408, in cli
    metadata = project_wheel_metadata(
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/build/util.py", line 54, in project_wheel_metadata
    env.install(builder.get_requires_for_build('wheel'))
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/build/__init__.py", line 368, in get_requires_for_build
    with self._handle_backend(hook_name):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/build/__init__.py", line 490, in _handle_backend
    raise BuildBackendException(exception, f'Backend subprocess exited when trying to invoke {hook}')
build.BuildBackendException: Backend subprocess exited when trying to invoke get_requires_for_build_wheel

[AndydeCleyre]

I've started #1629 for this; feel free to suggest some good error language, here or there.

[AndydeCleyre]

OK, current output for the error in that branch looks like:

/home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse setup.py

I'm going to promote the PR from draft to ready-for-review, and please try it out and let me know what you think.

Thanks!

[AndydeCleyre]

Just updating that the message's setup.py bit is now the full absolute path.

[Aaron-Beal-CardinalPeak]

Fixed my last issue (was on an ancient 16.4.1 version of virtualenv; updating to latest/20.14.1) and I'm now able to see your patch work. That's exactly what I was hoping for, thanks!

[AndydeCleyre]

Thanks for the issue, it was a push that helped me get that tooling change done that I'd been looking forward to/dreading, and the new concise error is much more communicative.

Closing as fixed since it's merged and will be in the next release.