--extra-index-url not propagating to indirect dependencies

[jamesbraza]

1. The package accelerate indirectly depends on torch
2. Currently (Jan 17, 2024), torch only supports Python 3.12 in its nightly dev builds, which requires https://download.pytorch.org/whl/nightly extra
3. Running pip-compile --extra-index-url=https://download.pytorch.org/whl/nightly will not use the extra for accelerate

Environment Versions

1. OS Type macOS
2. Python version: $ python -V Python 3.12.1 (main, Jan 6 2024, 20:42:19) [Clang 14.0.3 (clang-1403.0.22.14.1)]
3. pip version: $ pip --version 23.3.2
4. pip-tools version: $ pip-compile --version 7.3.0

Steps to replicate

# requirements.in
# NOTE: when torch is a direct requirement, resolution works
torch

# requirements.in
# NOTE: when torch is an indirect requirement, resolution does not work
accelerate>=0.26

Run: pip-compile --extra-index-url=https://download.pytorch.org/whl/nightly

Expected result

Successful compilation of both the above requirements.in

Actual result

Second requirements.in above will fail to compile with the below error:

...
  File "/Users/user/code/repo/venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 439, in resolve
    raise ResolutionImpossible(self.state.backtrack_causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('torch>=1.10.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/a6/b9/44623bdb05595481107153182e7f4b9f2ef9d3b674938ad13842054dcbd8/accelerate-0.26.1-py3-none-any.whl (from https://pypi.org/simple/accelerate/) (requires-python:>=3.8.0)')), RequirementInformation(requirement=SpecifierRequirement('torch>=1.10.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/63/9c/c10fc10df1d4968406b3f3cffe5a7d9988a8583e3423fc4156d6c91ab62d/accelerate-0.26.0-py3-none-any.whl (from https://pypi.org/simple/accelerate/) (requires-python:>=3.8.0)'))]
...
pip._internal.exceptions.DistributionNotFound: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[chrysle]

Thanks for the analysis! This sounds like a bug to me.

[webknjaz]

@jamesbraza does pip install do the right thing?

[jamesbraza]

Yeah it does if you just pip install -r requirements.in --extra-index-url=https://download.pytorch.org/whl/nightly, thanks for asking

[AndydeCleyre]

With Python 3.12.2, pip 24.0, and pip-tools from main, I successfully compiled a reqs.in with just accelerate>=0.26, without using any extra index. It locked torch==2.2.1. Is that surprising?

[jamesbraza]

@AndydeCleyre it's unsurprising because PyTorch 2.2 started supporting 3.12. The error only happens with torch<2.2. I am not sure that this issue can be repro'd based on my OP any more now that PyTorch 2.2 is released