Add support for pip's 2020 dependency resolver #1539

atugushev · 2021-11-22T08:30:28Z

What's new?

Added new option --resolver [backtracking|legacy] to pip-compile (default is legacy).

How to use?

To enable 2020 dependency resolver run pip-compile --resolver=backtracking.

Backtracking resolver example

$ echo "oslo.utils==1.4.0" | pip-compile - --resolver=backtracking --allow-unsafe --annotation-style=line -qo-
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --allow-unsafe --annotation-style=line --output-file=- --resolver=backtracking -
#
babel==2.10.3             # via oslo-i18n, oslo-utils
iso8601==1.0.2            # via oslo-utils
netaddr==0.8.0            # via oslo-utils
netifaces==0.11.0         # via oslo-utils
oslo-i18n==2.1.0          # via oslo-utils
oslo-utils==1.4.0         # via -r -
pbr==0.11.1               # via oslo-i18n, oslo-utils
pytz==2022.1              # via babel
six==1.16.0               # via oslo-i18n, oslo-utils

# The following packages are considered to be unsafe in a requirements file:
pip==22.1.2               # via pbr

Legacy resolver example

$ echo "oslo.utils==1.4.0" | pip-compile - --resolver=legacy --allow-unsafe -qo-
Could not find a version that matches pbr!=0.7,!=2.1.0,<1.0,>=0.6,>=2.0.0 (from oslo.utils==1.4.0->-r -)
Tried: 0.5.2.5.g5b3e942, 0.5.0, 0.5.1, 0.5.2, 0.5.4, 0.5.5, 0.5.6, 0.5.7, 0.5.8, 0.5.10, 0.5.11, 0.5.12, 0.5.13, 0.5.14, 0.5.15, 0.5.16, 0.5.17, 0.5.18, 0.5.19, 0.5.20, 0.5.21, 0.5.22, 0.5.23, 0.6, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.9.0, 0.10.0, 0.10.0, 0.10.1, 0.10.1, 0.10.2, 0.10.2, 0.10.3, 0.10.3, 0.10.4, 0.10.4, 0.10.5, 0.10.5, 0.10.6, 0.10.6, 0.10.7, 0.10.7, 0.10.8, 0.10.8, 0.11.0, 0.11.0, 0.11.1, 0.11.1, 1.0.0, 1.0.0, 1.0.1, 1.0.1, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.9.0, 1.9.0, 1.9.1, 1.9.1, 1.10.0, 1.10.0, 2.0.0, 2.0.0, 2.1.0, 2.1.0, 3.0.0, 3.0.0, 3.0.1, 3.0.1, 3.1.0, 3.1.0, 3.1.1, 3.1.1, 4.0.0, 4.0.0, 4.0.1, 4.0.1, 4.0.2, 4.0.2, 4.0.3, 4.0.3, 4.0.4, 4.0.4, 4.1.0, 4.1.0, 4.1.1, 4.1.1, 4.2.0, 4.2.0, 4.3.0, 4.3.0, 5.0.0, 5.0.0, 5.1.0, 5.1.0, 5.1.1, 5.1.1, 5.1.2, 5.1.2, 5.1.3, 5.1.3, 5.2.0, 5.2.0, 5.2.1, 5.2.1, 5.3.0, 5.3.0, 5.3.1, 5.3.1, 5.4.0, 5.4.0, 5.4.1, 5.4.1, 5.4.2, 5.4.2, 5.4.3, 5.4.3, 5.4.4, 5.4.4, 5.4.5, 5.4.5, 5.5.0, 5.5.0, 5.5.1, 5.5.1, 5.6.0, 5.6.0, 5.7.0, 5.7.0, 5.8.0, 5.8.0
There are incompatible versions in the resolved dependencies:
  pbr!=0.7,<1.0,>=0.6 (from oslo.utils==1.4.0->-r -)
  pbr!=2.1.0,>=2.0.0 (from oslo.i18n==5.1.0->oslo.utils==1.4.0->-r -)

Contributor checklist

Provided the tests for the changes.
Assure PR title is short, clear, and good to be included in the user-oriented changelog

Maintainer checklist

Assure one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

astrojuanlu · 2021-11-22T11:48:35Z

I confirm that this fixes my original use case in #1277 🚀

(.venv) juanlu at arion2 in /tmp/reqs
$ echo 'jupyterlab==3.0.0rc13' > requirements.in
(.venv) juanlu at arion2 in /tmp/reqs
$ pip-compile
Could not find a version that matches jupyter-server~=1.0.1,~=1.1,~=1.5 (from jupyterlab==3.0.0rc13->-r requirements.in (line 1))
Tried: 0.0.1, 0.0.1, 0.0.2, 0.0.2, 0.0.3, 0.0.3, 0.0.4, 0.0.4, 0.0.5, 0.0.5, 0.1.0, 0.1.0, 0.1.1, 0.1.1, 0.2.0, 0.2.0, 0.2.1, 0.2.1, 0.3.0, 0.3.0, 1.0.0, 1.0.0, 1.0.1, 1.0.1, 1.0.2, 1.0.2, 1.0.3, 1.0.3, 1.0.4, 1.0.4, 1.0.5, 1.0.5, 1.0.6, 1.0.6, 1.0.7, 1.0.7, 1.0.8, 1.0.8, 1.0.9, 1.0.9, 1.0.10, 1.0.10, 1.0.11, 1.0.11, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.1.2, 1.1.2, 1.1.3, 1.1.3, 1.1.4, 1.1.4, 1.2.0, 1.2.0, 1.2.1, 1.2.1, 1.2.2, 1.2.2, 1.2.3, 1.2.3, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.4.1, 1.4.1, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.6.2, 1.6.2, 1.6.3, 1.6.3, 1.6.4, 1.6.4, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.10.1, 1.10.1, 1.10.2, 1.10.2, 1.11.0, 1.11.0, 1.11.1, 1.11.1, 1.11.2, 1.11.2
Skipped pre-versions: 1.0.0rc0, 1.0.0rc0, 1.0.0rc1, 1.0.0rc1, 1.0.0rc2, 1.0.0rc2, 1.0.0rc3, 1.0.0rc3, 1.0.0rc4, 1.0.0rc4, 1.0.0rc5, 1.0.0rc5, 1.0.0rc6, 1.0.0rc6, 1.0.0rc7, 1.0.0rc7, 1.0.0rc8, 1.0.0rc8, 1.0.0rc9, 1.0.0rc9, 1.0.0rc10, 1.0.0rc10, 1.0.0rc11, 1.0.0rc11, 1.0.0rc12, 1.0.0rc12, 1.0.0rc13, 1.0.0rc13, 1.0.0rc14, 1.0.0rc14, 1.0.0rc15, 1.0.0rc15, 1.0.0rc16, 1.0.0rc16, 1.7.0a1, 1.7.0a1, 1.7.0a2, 1.7.0a2
There are incompatible versions in the resolved dependencies:
  jupyter-server~=1.0.1 (from jupyterlab==3.0.0rc13->-r requirements.in (line 1))
  jupyter-server~=1.1 (from jupyterlab-server==2.0.0->jupyterlab==3.0.0rc13->-r requirements.in (line 1))
  jupyter-server~=1.5 (from nbclassic==0.2.8->jupyterlab==3.0.0rc13->-r requirements.in (line 1))
(.venv) juanlu at arion2 in /tmp/reqs
$ pip-compile --resolver=2020
#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile --resolver=2020
#
argon2-cffi==21.1.0
    # via notebook
attrs==21.2.0
    # via jsonschema
babel==2.9.1
    # via jupyterlab-server
backcall==0.2.0
    # via ipython
bleach==4.1.0
    # via nbconvert
certifi==2021.10.8
    # via requests
cffi==1.15.0
    # via argon2-cffi
charset-normalizer==2.0.7
    # via requests
debugpy==1.5.1
    # via ipykernel
decorator==5.1.0
    # via ipython
defusedxml==0.7.1
    # via nbconvert
entrypoints==0.3
    # via
    #   jupyter-client
    #   nbconvert
idna==3.3
    # via requests
ipykernel==6.5.1
    # via notebook
ipython==7.29.0
    # via
    #   ipykernel
    #   jupyterlab
ipython-genutils==0.2.0
    # via
    #   jupyter-server
    #   nbformat
    #   notebook
jedi==0.18.1
    # via ipython
jinja2==3.0.3
    # via
    #   jupyter-server
    #   jupyterlab
    #   jupyterlab-server
    #   nbconvert
    #   notebook
json5==0.9.6
    # via jupyterlab-server
jsonschema==4.2.1
    # via
    #   jupyterlab-server
    #   nbformat
jupyter-client==7.0.6
    # via
    #   ipykernel
    #   jupyter-server
    #   nbclient
    #   notebook
jupyter-core==4.9.1
    # via
    #   jupyter-client
    #   jupyter-server
    #   jupyterlab
    #   nbconvert
    #   nbformat
    #   notebook
jupyter-server==1.0.11
    # via
    #   jupyterlab
    #   jupyterlab-server
    #   nbclassic
jupyterlab==3.0.0rc13
    # via -r requirements.in
jupyterlab-pygments==0.1.2
    # via nbconvert
jupyterlab-server==2.0.0rc7
    # via jupyterlab
markupsafe==2.0.1
    # via jinja2
matplotlib-inline==0.1.3
    # via
    #   ipykernel
    #   ipython
mistune==0.8.4
    # via nbconvert
nbclassic==0.2.4
    # via jupyterlab
nbclient==0.5.9
    # via nbconvert
nbconvert==6.3.0
    # via
    #   jupyter-server
    #   notebook
nbformat==5.1.3
    # via
    #   jupyter-server
    #   nbclient
    #   nbconvert
    #   notebook
nest-asyncio==1.5.1
    # via
    #   jupyter-client
    #   nbclient
    #   notebook
notebook==6.4.6
    # via nbclassic
packaging==21.3
    # via
    #   bleach
    #   jupyterlab
    #   jupyterlab-server
pandocfilters==1.5.0
    # via nbconvert
parso==0.8.2
    # via jedi
pexpect==4.8.0
    # via ipython
pickleshare==0.7.5
    # via ipython
prometheus-client==0.12.0
    # via
    #   jupyter-server
    #   notebook
prompt-toolkit==3.0.22
    # via ipython
ptyprocess==0.7.0
    # via
    #   pexpect
    #   terminado
pycparser==2.21
    # via cffi
pygments==2.10.0
    # via
    #   ipython
    #   jupyterlab-pygments
    #   nbconvert
pyparsing==3.0.6
    # via packaging
pyrsistent==0.18.0
    # via jsonschema
python-dateutil==2.8.2
    # via jupyter-client
pytz==2021.3
    # via babel
pyzmq==22.3.0
    # via
    #   jupyter-client
    #   jupyter-server
    #   notebook
requests==2.26.0
    # via jupyterlab-server
send2trash==1.8.0
    # via
    #   jupyter-server
    #   notebook
six==1.16.0
    # via
    #   bleach
    #   python-dateutil
terminado==0.12.1
    # via
    #   jupyter-server
    #   notebook
testpath==0.5.0
    # via nbconvert
tornado==6.1
    # via
    #   ipykernel
    #   jupyter-client
    #   jupyter-server
    #   jupyterlab
    #   notebook
    #   terminado
traitlets==5.1.1
    # via
    #   ipykernel
    #   ipython
    #   jupyter-client
    #   jupyter-core
    #   jupyter-server
    #   matplotlib-inline
    #   nbclient
    #   nbconvert
    #   nbformat
    #   notebook
urllib3==1.26.7
    # via requests
wcwidth==0.2.5
    # via prompt-toolkit
webencodings==0.5.1
    # via bleach

# The following packages are considered to be unsafe in a requirements file:
# setuptools

FlorentJeannot · 2021-11-22T13:48:28Z

Hello @atugushev

Thank you for pinging me to test your branch, and thank you for taking the time to work on this feature!

I tried it and I noted some remarks:

ℹ️ From now on, -c requirements.txt does not work if the requirements.txt contains for example a direct reference. It requires a package name with a constraint. This is not related to your PR, but to how pip works. I thought it was interesting to mention it.
❓ The listing of dependencies in the generated .txt file is a bit weird, for example:

certifi==2021.10.8
    # via
    #   requests
    #   requests[socks]

I don't know if we want to output the same package name twice?

⚠️ The generated .txt file does not output the same dependencies depending on the order in the .in file.

For example:

crochet==2.0.0
Scrapy==2.5.1

Will output:

[...]
twisted==21.7.0
    # via
    #   crochet
    #   twisted[http2]
[...]

While:

Scrapy==2.5.1
crochet==2.0.0

Will output:

[...]
 twisted[http2]==21.7.0
    # via
    #   crochet
    #   scrapy
    #   twisted[http2]
[...]

atugushev · 2021-11-22T16:16:41Z

@FlorentJeannot thanks for the feedback and good catch! Looks like we need more tests for packages with extras. I'll address the bugs shortly.

FlorentJeannot · 2021-11-22T16:23:32Z

Thank you @atugushev. I found another one, not sure if it's related to your changes though.

When I declare a local file which contains an egg (for example file:./vendored/package.zip[requests]), the output of the .txt is the absolute path of the file on my machine.

Let me know if you need an exemple and thank you.
Also let me know when you have made some fixes, so that I can test it again and let you know if I spot other things.

webknjaz · 2021-11-22T18:01:52Z

@FlorentJeannot if you could send a PR with a failing test, it'd be easier to come up with a fix separately, once it's committed in Git. Per https://pganssle-talks.github.io/xfail-lightning/.

piptools/repositories/base.py

piptools/repositories/local.py

piptools/repositories/pypi.py

AndydeCleyre · 2021-11-22T18:06:38Z

When I declare a local file which contains an egg (for example file:./vendored/package.zip[requests]), the output of the .txt is the absolute path of the file on my machine.

@FlorentJeannot I don't know if @atugushev wants to merge this before or after #1329

piptools/resolver.py

webknjaz · 2021-11-22T18:20:13Z

piptools/resolver.py

+            PIP_EXISTS_ACTION="i"
+        ):
+            # Mark direct/primary/user_supplied packages
+            for ireq in self.constraints:


I don't understand why not apply this conversion to the constraints list during the initialization. Also, I don't really understand why this needs to be mutable. Is that expectation that it'll be modified by external callers post-initialization?

I don't understand why not apply this conversion to the constraints list during the initialization.

Do you mean inside Resolver.__init__()?

piptools/scripts/compile.py

piptools/utils.py

piptools/writer.py

AndydeCleyre · 2021-11-22T18:43:13Z

I wonder if the varying extras behavior noted by @FlorentJeannot is resolved by @richafrank's sitting PRs.

@atugushev Do you think we'll be finalizing this one before any other PRs, or are there some you plan to review and merge first?

tests/test_cli_compile.py

Co-Authored-By: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>

It shouldn't raise an error when a preexisting output file's pins conflict with the input files' constraints. jazzband#1539 (comment)

jazzband#1539 (comment)

Co-authored-by: Richard Frank <rich@quantopian.com>

atugushev · 2022-06-29T22:36:12Z

I'm so happy this finally merged. Many thanks to my heroes @AndydeCleyre, @FlorentJeannot, @richafrank, @ssbarnea, @thomas-riccardi, @webknjaz, and all others involved for your support.

deifactor · 2022-08-04T02:39:15Z

One thing I noticed is that this is significantly slower than the legacy resolver, presumably because it's not using the pip-compile cache. What's worse is that for things that don't have pypi-built wheels for the target host, it has to rebuild the wheels every time, which is especially painful when doing things involving cython.

ssbarnea · 2022-08-04T07:18:35Z

@deifactor It is not a bug, just an opportunity for you to make a pull-request and address this performance regression! ;)

astrojuanlu · 2022-08-05T10:10:35Z

Worth opening an issue for wider visibility?

It shouldn't raise an error when a preexisting output file's pins conflict with the input files' constraints. jazzband/pip-tools#1539 (comment)

jazzband/pip-tools#1539 (comment)

taeefnajib · 2023-04-27T22:09:51Z

Hi @atugushev I have a question! I need both flytekit and bentoml installed in the same project. protobuf = ">=4.21.1,<5.0.0" is required by flytekit and protobuf<4.0 is required by bentoml. When I try pip-compile requirements.in I get an error. But after using your solution, i.e. pip-compile requirements.in --resolver=backtracking pip-compile generates requirements.txt file. But the version of protobuf is 3.20.3 How can I have both versions of protobuf in my requirements.txt file?

ryanhiebert · 2023-04-27T22:11:23Z

@taeefnajib Python doesn't allow you to install two different versions of the same dependency in the same environment, so we can't support two versions of a package in a requirements.txt file. You'll need to find a set of dependencies that all work with the same version of protobuf correctly. This isn't always easy, but it is necessary.

taeefnajib · 2023-04-27T22:23:06Z

@ryanhiebert thanks. let me see why bentoml locked the version of protobuf to <4.0

taeefnajib · 2023-04-27T23:17:02Z

@ryanhiebert Just a quick question! When I put bentoml and flytekit in my requirements.txt file manually (without mentioning versions), and use pip install -r requirements.txt it installs multiple versions of protobuf It's pip-compile that cannot generate a requirements.txt file with multiple versions. Do you know why is that happening and if I can find a workaround?

AndydeCleyre · 2023-04-27T23:20:48Z

@taeefnajib it's probably installing one version, then uninstalling that to install a different version. Run pip list to see the final state.

taeefnajib · 2023-04-27T23:28:28Z

@AndydeCleyre you are right. I can only see 3.20.3 Thanks for the clarification

luhn · 2023-04-27T23:28:48Z

It looks like pip will install flytekit 1.2.11 (latest release is 1.5.0), which requires protobuf>=3.6.1,<4, in order to satisfy all dependencies.

>> echo "bentoml" >> req.txt
>> echo "flytekit" >> req.txt
>> pip install -qr req.txt
>> pip list | grep protobuf
protobuf                                     3.20.3
>> pip list | grep flytekit
flytekit                                     1.2.11

taeefnajib · 2023-04-27T23:32:03Z

@luhn thanks. Yes, it's installing 1.2.11 even though I was hoping to install 1.5.0

atugushev added enhancement Improvements to functionality resolver Related to dependency resolver labels Nov 22, 2021