pip-compile claims conflict between boto3 and dvc (whereas pip installs smoothly)

[AmitAronovitch]

Trying to specify boto3 and dvc[s3] (both unrestricted) in requirements.in causes "incompatible versions" (while pip install can easily install them together without warnings).
See example below.

Conflict seems to be caused because the version that piptools had itself selected at stage1 conflicts with secondary dependencies.

Same results are generated if I add botocore<1.20.50 instead of dvc[s3]

Environment Versions

1. Linux (Ubuntu groovy)
2. Python version: 3.8.6
3. pip version: 21.1.2
4. pip-tools version: 6.1.0

Steps to replicate

$ cat requirements.in
dvc[s3]
boto3
$ pip-compile -v requirements.in

Expected result

requirements.txt containing a valid solution:
At the time of writing, the following is consistent:

dvc[s3]==2.3.0
boto3==1.17.49

Actual result

Error including the following

There are incompatible versions in the resolved dependencies:
  botocore<1.21.0,>=1.20.93 (from boto3==1.17.93->-r requirements.in (line 2))
  botocore<1.20.50,>=1.20.49 (from aiobotocore[boto3]==1.3.0->dvc[s3]==2.3.0->-r requirements.in (line 1))
  botocore<2.0a.0,>=1.12.36 (from s3transfer==0.4.2->boto3==1.17.93->-r requirements.in (line 2))

However, note that line 2 did not specify any constraint. In the verbose outpyt, I can see that the version of boto3 was selected in an earlier round, despite being incompatible with the restrictions on botocore.

[hugo-ricateau]

Hi @atugushev, any idea of when such resolver related issues could be tackled? I think it should be top priority as it almost kills all the purpose of pip-compile in such situations.

[hugo-ricateau]

There are more and more problems arising from this one; solving it should now be considered an emergency (we are unfortunately seriously thinking to drop out pip-tools if this issue is not solved rapidly as it regularly impacts our deployment pipeline and creates a lot of debt in requirements management).

@atugushev any info on a potential patch?

[jli]

In case it's useful for others, an alternative approach I've experimented with is to just use the pip resolver, by installing the requirements.in spec file in a Docker image and then running pip freeze to generate the lock file.

Here's a longer write-up of how I use this approach to freeze dependencies when using conda and pip together: https://gist.github.com/jli/b2d2d62ad44b7fcb5101502c08dca1ae
I've also tried this approach with just pip: just do pip install -r /requirements.in && pip freeze > /requirements.lock instead of the conda env create and conda env export commands.

This is missing a bunch of the pip-compile features (eg the resulting lock file doesn't have comments indicating what installed all the transitive dependencies), but it replicates the core functionality of going from a high-level spec file to a fully versioned/pinned lock file.

[hugo-ricateau]

Sure, thanks for the suggestion; there are other tools having more or less the same features. However, moving to an alternate approach will be a no return move for us (so I hope my previous message will highlights the criticality of this issue).

[atugushev]

I'm working on a new resolver, it's pretty close to the finish line and should fix this and the other issues. Could you try this atugushev#10?

UPDATE: try #1539 instead

[atugushev]

I've tried with the new resolver - it works:

see details

❯ pip-compile --resolver backtracking
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --resolver=backtracking
#
aiobotocore[boto3]==1.3.0
    # via
    #   dvc
    #   s3fs
aiohttp==3.8.1
    # via aiobotocore
aioitertools==0.9.0
    # via aiobotocore
aiosignal==1.2.0
    # via aiohttp
appdirs==1.4.4
    # via dvc
async-timeout==4.0.2
    # via aiohttp
atpublic==3.0.1
    # via flufl-lock
attrs==21.4.0
    # via aiohttp
boto3==1.17.49
    # via
    #   -r requirements.in
    #   aiobotocore
botocore==1.20.49
    # via
    #   aiobotocore
    #   boto3
    #   s3transfer
certifi==2021.10.8
    # via
    #   dulwich
    #   requests
cffi==1.15.0
    # via pygit2
charset-normalizer==2.0.12
    # via
    #   aiohttp
    #   requests
colorama==0.4.4
    # via
    #   dvc
    #   rich
commonmark==0.9.1
    # via rich
configobj==5.0.6
    # via dvc
decorator==5.1.1
    # via jsonpath-ng
dictdiffer==0.9.0
    # via dvc
diskcache==5.4.0
    # via dvc
distro==1.7.0
    # via dvc
dpath==2.0.6
    # via dvc
dulwich==0.20.32
    # via dvc
dvc[s3]==2.3.0
    # via -r requirements.in
flatten-dict==0.4.2
    # via dvc
flufl-lock==3.2
    # via dvc
frozenlist==1.3.0
    # via
    #   aiohttp
    #   aiosignal
fsspec==2021.5.0
    # via
    #   dvc
    #   s3fs
ftfy==6.1.1
    # via python-benedict
funcy==1.17
    # via dvc
future==0.18.2
    # via grandalf
gitdb==4.0.9
    # via gitpython
gitpython==3.1.27
    # via dvc
grandalf==0.6
    # via dvc
idna==3.3
    # via
    #   requests
    #   yarl
jmespath==0.10.0
    # via
    #   boto3
    #   botocore
jsonpath-ng==1.5.3
    # via dvc
mailchecker==4.1.12
    # via python-benedict
multidict==6.0.2
    # via
    #   aiohttp
    #   yarl
nanotime==0.5.2
    # via dvc
networkx==2.6.3
    # via dvc
packaging==21.3
    # via dvc
pathspec==0.9.0
    # via dvc
phonenumbers==8.12.43
    # via python-benedict
ply==3.11
    # via
    #   dvc
    #   jsonpath-ng
psutil==5.9.0
    # via dvc
pyasn1==0.4.8
    # via dvc
pycparser==2.21
    # via cffi
pydot==1.4.2
    # via dvc
pygit2==1.9.0
    # via dvc
pygments==2.11.2
    # via rich
pygtrie==2.3.2
    # via dvc
pyparsing==2.4.7
    # via
    #   dvc
    #   grandalf
    #   packaging
    #   pydot
python-benedict==0.25.0
    # via dvc
python-dateutil==2.8.2
    # via
    #   botocore
    #   python-benedict
python-fsutil==0.6.0
    # via python-benedict
python-slugify==6.1.0
    # via python-benedict
pyyaml==6.0
    # via python-benedict
requests==2.27.1
    # via
    #   dvc
    #   python-benedict
rich==11.2.0
    # via dvc
ruamel-yaml==0.17.21
    # via dvc
ruamel-yaml-clib==0.2.6
    # via ruamel-yaml
s3fs==2021.5.0
    # via dvc
s3transfer==0.3.7
    # via boto3
shortuuid==1.0.8
    # via dvc
shtab==1.5.3
    # via dvc
six==1.16.0
    # via
    #   configobj
    #   flatten-dict
    #   jsonpath-ng
    #   python-dateutil
smmap==5.0.0
    # via gitdb
tabulate==0.8.9
    # via dvc
text-unidecode==1.3
    # via python-slugify
toml==0.10.2
    # via
    #   dvc
    #   python-benedict
tqdm==4.62.3
    # via dvc
typing-extensions==4.1.1
    # via
    #   aioitertools
    #   dvc
urllib3==1.26.8
    # via
    #   botocore
    #   dulwich
    #   requests
voluptuous==0.12.2
    # via dvc
wcwidth==0.2.5
    # via ftfy
wrapt==1.13.3
    # via aiobotocore
xmltodict==0.12.0
    # via python-benedict
yarl==1.7.2
    # via aiohttp
zc-lockfile==2.0
    # via dvc

# The following packages are considered to be unsafe in a requirements file:
# setuptools

[jondo]

@AmitAronovitch, is this solved for you as the issue author?

[hugo-ricateau]

@jondo, yes it solves the issue (I have the exact same one); however, the PR is still awaiting to be merged...

[atugushev]

This has been fixed in #1539 with the backtracking resolver, try pip-compile --resolver backtracking. The resolver is released as part of pip-tools v6.8.0. Please let us know if it doesn't resolve your issue. Thanks!