Poetry cannot resolve scikit-image dependencies from a private PyPi server

[orf]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: MacOS 11
• Poetry version: 1.1.4

Issue

Given a project that depends on scikit-image, and using an external (private) repository that does not implement PyPi's JSON api, running poetry debug resolve scikit-image resolves scikit-image to have no dependencies (output at the end). However this is incorrect, and the console output clearly shows there is an exception while trying to invoke the PEP517 module.

Unfortunately this manifests itself as a poetry lock working just fine, but incorrectly specifying that there are no dependencies and thus scikit-image fails to import.

I've tried various steps to debug this:

1. Creating different venvs with different python versions
2. Explicitly setting the scikit-image dependency with source="pypi"
3. Trying to download the source package and invoking pep517 manually to see what happens
4. Upgrading pip, wheel and setuptools everywhere I can. Side note: I cannot see where WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available. is coming from 😱

I'm willing to help create a fix for this, but I need some pointers as I'm in the dark about what could be going wrong here.

The output:

❯ poetry debug resolve scikit-image -vvv
Resolving dependencies...
   1: fact: document-model-gateway is 1.0.0
   1: derived: document-model-gateway
   1: fact: document-model-gateway depends on scikit-image (^0.17.2)
   1: selecting document-model-gateway (1.0.0)
   1: derived: scikit-image (^0.17.2)
onfido: 1 packages found for scikit-image >=0.17.2,<0.18.0
PyPI: No release information found for scikit-image-0.12.dev0, skipping
PyPI: No release information found for scikit-image-0.9.2, skipping
PyPI: 1 packages found for scikit-image >=0.17.2,<0.18.0
onfido: Downloading sdist: scikit-image-0.17.2.tar.gz
PackageInfo: PEP517 build failed: Command ['/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/bin/python', '-'] errored with the following return code 1, and output:
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the '/private/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/bin/python -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "<stdin>", line 6, in <module>
  File "/private/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/lib/python3.9/site-packages/pep517/meta.py", line 53, in build
    _prep_meta(hooks, env, dest)
  File "/private/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/lib/python3.9/site-packages/pep517/meta.py", line 28, in _prep_meta
    reqs = hooks.get_requires_for_build_wheel({})
  File "/private/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/lib/python3.9/site-packages/pep517/wrappers.py", line 160, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "/private/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/lib/python3.9/site-packages/pep517/wrappers.py", line 255, in _call_hook
    self._subprocess_runner(
  File "/private/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/lib/python3.9/site-packages/pep517/wrappers.py", line 75, in quiet_subprocess_runner
    check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
  File "/usr/local/Cellar/python@3.9/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/bin/python', '/private/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/lib/python3.9/site-packages/pep517/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpptu68ng0']' returned non-zero exit status 1.
input was : import pep517.build
import pep517.meta

path='/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpl54mmc0o/scikit-image-0.17.2'
system=pep517.build.compat_system(path)
pep517.meta.build(source_dir=path, dest='/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/dist', system=system)

   1: selecting scikit-image (0.17.2)
   1: Version solving took 17.438 seconds.
   1: Tried 1 solutions.

Resolution results:

scikit-image 0.17.2

The output from poetry debug resolve against PyPi: https://gist.github.com/orf/b91ba6ac62583523cb8f6999ef69fe96

[sinoroc]

I cannot see where WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available. is coming from

Try: virtualenv --upgrade-embed-wheels

[sinoroc]

Something, I have not quite understood: does the fact that you are using a private index server plays a role or not?

[orf]

I can only assume so - resolution works fine with pypi. As I understand it Poetry uses a PyPi JSON api to resolve dependencies on PyPi, which isn't available on private repositories. This means that the PEP517 build stage isn't triggered and doesn't fail.

After running virtualenv --upgrade-embed-wheel the pip warning is gone but it still fails to resolve in the same way described above.

The problem is that there is no information about why the pep517 process failed, only that it errored. Displaying the stdout would really help here, but I'm not sure that's possible.

In the gist I linked at the end, this is the resolution from pypi:

Resolution results:
six             1.15.0
cycler          0.10.0
decorator       4.4.2
kiwisolver      1.3.1
numpy           1.19.4
pillow          8.0.1
pyparsing       2.4.7
python-dateutil 2.8.1
imageio         2.9.0
matplotlib      3.3.3
networkx        2.5
pywavelets      1.1.1
scipy           1.5.4
tifffile        2020.12.4
scikit-image    0.17.2

[orf]

I guess there are two bugs here: the first is that the PEP517 process fails to run on scikit-image, and the second is that Poetry seems to ignore PEP517 failures during resolution.

[sinoroc]

I guess there are two bugs here: the first is that the PEP517 process fails to run on scikit-image, and the second is that Poetry seems to ignore PEP517 failures during resolution.

I would think so as well. Some fix is probably required in poetry, so that such build failures are caught.

On the other hand, your issue should be reported on scikit-image's side, the real blocker could be on their side I would say. Their setup.py seems incredibly complicated. Also I can not really see clearly what Python versions they claim to support. The trove classifiers for version 0.17.2 seem to say only Python 3.6 and 3.7 are supported (see here).

[orf]

They have python_requires='>=3.6',, and they build wheels for Python 3.8. The latest RC has wheels for 3.9.

I'll definitely report it to scikit-image if needed, but I'm not sure how to reproduce the pep517 failure. I naievely tried to untar the source release and run something like the script in the output message:

import pep517.build
import pep517.meta

path='/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpl54mmc0o/scikit-image-0.17.2'
system=pep517.build.compat_system(path)
pep517.meta.build(source_dir=path, dest='/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/dist', system=system)

however this complained that there was a missing json file at one of the paths, which implies I'm missing a step. Is there anything you can do to point me in the right direction here?

[sinoroc]

I'm not sure how to reproduce the pep517 failure

What about creating a fresh virtual environment, and trying to pip-install in that environment? Have you already tried that?

With pep517 you can do python -m pep517.build path/to/scikit-image-0.17.2.

[Korijn]

I went in and debugged what happens inside that subprocess call mentioned at the end of the stacktrace:

subprocess.CalledProcessError: Command '['/private/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/bin/python', '/private/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpegwzhoyc/.venv/lib/python3.9/site-packages/pep517/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/k3/splrymbj4wn9fgzcfgkzy93h0000gn/T/tmpptu68ng0']' returned non-zero exit status 1.

The pep517 get_requires_for_build_wheel hook attempts to run scikit-image's setup.py, but fails on missing dependencies that are imported in setup.py, in this case it is numpy. This is caused by the complete omission of a pyproject.toml so there is no [build-system].requires to parse or anything, so numpy is not installed.

The confusing thing is though, that Azure Artifacts Feed (which we use) does appear to support a metadata API that could be called, so I'm wondering why does Poetry by default treat any non-pypi repository as a legacy repository?

[neersighted]

This is now obsolete for several reasons:

Poetry should now successfully report PEP 517 build errors -- I cannot reproduce a silent PEP 517 failure when resolving using a (bad) sdist from a custom source on 1.2.x.

Poetry now uses wheels wheels whenever possible to gather metadata: #6547.

In addition, PEP 691 and PEP 658 have gained acceptance as the standard ecosystem-wide way to support serving metadata in a PEP 503 repository. This will significantly improve this process moving forward.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.