Extra requirements aren't properly pulled in by pip-compile

[liberato-whisper]

Pip-compile doesn't properly resolve extra requirements into the requirements.txt output. I can get pip-compile to pass while passing in two packages that have conflicting requirements, which shouldn't be possible. Seems to be a reproduction of #768 with real packages.

Environment Versions

1. OS Type: macOS 11.6.5
2. Python version: 3.9.1
3. pip version: 22.0.4
4. pip-tools version: 6.5.1

Steps to replicate

1. Create a new, clean python venv with python 3.9.1
2. pip install pip-tools==6.5.1
3. Create a requirements.in with the following:

gitlint-core[trusted-deps]==0.17.0
click==8.0.4

3. Run pip-compile
4. Run pip-sync

Expected result

pip-compile at step 3 should fail. gitlint-core[trusted-deps]==0.17.0 requires click==8.0.3 (via the extra dependencies trusted-deps), but the click==8.0.4 is included in the outputted requirements.txt via the requirements.in.

Actual result

pip-compile passes, outputting:

#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile
#
arrow==1.2.2
    # via gitlint-core
click==8.0.4
    # via
    #   -r requirements.in
    #   gitlint-core
gitlint-core[trusted-deps]==0.17.0
    # via -r requirements.in
python-dateutil==2.8.2
    # via arrow
sh==1.14.2
    # via gitlint-core
six==1.16.0
    # via python-dateutil

The conflict is then found in step 4, pip-sync, with the following error:

The conflict is caused by:
    The user requested click==8.0.4
    gitlint-core[trusted-deps] 0.17.0 depends on Click>=8
    gitlint-core[trusted-deps] 0.17.0 depends on Click==8.0.3; extra == "trusted-deps"

[AndydeCleyre]

Thanks!

I just want to add for now that this is also reproducible with #1519.

[AndydeCleyre]

Good news: this (properly) fails with ResolutionImpossible when using #1539 with the new resolver option.

e.g. with pipx:

$ pipx run --spec 'pip-tools @ git+https://github.com/atugushev/pip-tools@new-resolver' pip-compile --resolver backtracking 

or with zpy:

% pipz runpkg 'pip-tools @ git+https://github.com/atugushev/pip-tools@new-resolver' pip-compile --resolver backtracking 

[richafrank]

It looks like the legacy resolver returns the existing Click>=8 when it encounters the second (stricter) dependency on Click==8.0.3 from gitlint-core[trusted-deps].

[atugushev]

With pip-compile --resolver backtracking it fails now into ResolutionImpossible.

Details

$ pip-compile --resolver backtracking --verbose
Using indexes:
  https://pypi.org/simple

                          ROUND 1
  Collecting gitlint-core[trusted-deps]==0.17.0
    Downloading gitlint_core-0.17.0-py2.py3-none-any.whl (64 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.3/64.3 kB 836.2 kB/s eta 0:00:00
  Collecting click==8.0.4
    Using cached click-8.0.4-py3-none-any.whl (97 kB)
  Collecting sh>=1.13.0
    Downloading sh-1.14.2-py2.py3-none-any.whl (40 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 kB 1.2 MB/s eta 0:00:00
  Collecting arrow>=1
    Using cached arrow-1.2.2-py3-none-any.whl (64 kB)
  ERROR: Cannot install click==8.0.4 and gitlint-core[trusted-deps]==0.17.0 because these package versions have conflicting dependencies.

  The conflict is caused by:
      The user requested click==8.0.4
      gitlint-core[trusted-deps] 0.17.0 depends on Click>=8
      gitlint-core[trusted-deps] 0.17.0 depends on Click==8.0.3; extra == "trusted-deps"

  To fix this you could try to:
  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

Traceback (most recent call last):
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 385, in resolve
    raise ResolutionImpossible(self.state.backtrack_causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('click==8.0.4'), parent=None), RequirementInformation(requirement=SpecifierRequirement('Click>=8'), parent=ExtrasCandidate(base=LinkCandidate('https://files.pythonhosted.org/packages/64/dd/d8f1226885b107bda6b97b63566381a37d61187562f32eae77e71e8fa261/gitlint_core-0.17.0-py2.py3-none-any.whl#sha256=cb99ccd736a698b910385211203bda94bf4ce29086d0c08f8f58a18c40a98377 (from https://pypi.org/simple/gitlint-core/) (requires-python:>=3.6)'), extras=frozenset({'trusted-deps'}))), RequirementInformation(requirement=SpecifierRequirement('Click==8.0.3; extra == "trusted-deps"'), parent=ExtrasCandidate(base=LinkCandidate('https://files.pythonhosted.org/packages/64/dd/d8f1226885b107bda6b97b63566381a37d61187562f32eae77e71e8fa261/gitlint_core-0.17.0-py2.py3-none-any.whl#sha256=cb99ccd736a698b910385211203bda94bf4ce29086d0c08f8f58a18c40a98377 (from https://pypi.org/simple/gitlint-core/) (requires-python:>=3.6)'), extras=frozenset({'trusted-deps'})))]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/albert/Projects/pip-tools/.venv/bin/pip-compile", line 33, in <module>
    sys.exit(load_entry_point('pip-tools', 'console_scripts', 'pip-compile')())
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/albert/Projects/pip-tools/piptools/scripts/compile.py", line 487, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/Users/albert/Projects/pip-tools/piptools/resolver.py", line 614, in resolve
    is_resolved = self._do_resolve(
  File "/Users/albert/Projects/pip-tools/piptools/resolver.py", line 652, in _do_resolve
    resolver.resolve(
  File "/Users/albert/Projects/pip-tools/.venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

I'll close this based on the above, but please let us know if it doesn't resolve your issue. Thanks!