Make it possible to figure out why packages conflict

[gnilrets]

What's the problem this feature will solve?

Using pip-tools 7.3.0

I'm trying to generate a requirements.txt file from a requirements.in file that has constraints based on another file. There's a dependency conflict, but pip-compile no longer provides enough useful information to troubleshoot the issue. I feel like it was easier to interpret in earlier versions of pip-compile (maybe 1-2 years ago?), but now it's practically uninterpretable.

Here's my dbt.in file:

-c dagster.txt

dagster-dbt~=0.22.9

dbt-snowflake~=1.7.0
snowflake-connector-python[secure-local-storage]~=3.4
sqlfluff~=2.3.5
sqlfluff-templater-dbt~=2.3.5

When I try to pip-compile it, I get an error:

pip-compile -v --strip-extras requirements/dbt.in > requirements/dbt.txt
Using pip-tools configuration defaults found in 'pyproject.toml'.
Using indexes:
  https://pypi.org/simple

                          ROUND 1
  ERROR: Cannot install dagster and dbt-core because these package versions have conflicting dependencies.
Traceback (most recent call last):
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 316, in _backjump
    name, candidate = broken_state.mapping.popitem()
KeyError: 'dictionary is empty'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 434, in resolve
    success = self._backjump(causes)
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in _backjump
    raise ResolutionImpossible(causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('importlib-metadata; python_version < "3.9"'), parent=LinkCandidate('https://files.pythonhosted.org/packages/7f/50/9fb3a5c80df6eb6516693270621676980acd6d5a9a7efdbfa273f8d616c7/alembic-1.13.1-py3-none-any.whl (from https://pypi.org/simple/alembic/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('importlib-metadata~=6.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/9d/a8/718950dfc5975f49f5b05cd3604d266a5d68c37e2e2e95d353363d33a01c/dbt_semantic_interfaces-0.4.4-py3-none-any.whl (from https://pypi.org/simple/dbt-semantic-interfaces/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('importlib-metadata; python_version < "3.9"'), parent=LinkCandidate('https://files.pythonhosted.org/packages/7f/50/9fb3a5c80df6eb6516693270621676980acd6d5a9a7efdbfa273f8d616c7/alembic-1.13.1-py3-none-any.whl (from https://pypi.org/simple/alembic/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('importlib-metadata~=6.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/eb/97/d7c34a8f0b23b9483f9adcd3183221198654b0ad883ba9b659ecdb1ca816/dbt_semantic_interfaces-0.4.3-py3-none-any.whl (from https://pypi.org/simple/dbt-semantic-interfaces/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('importlib-metadata; python_version < "3.9"'), parent=LinkCandidate('https://files.pythonhosted.org/packages/7f/50/9fb3a5c80df6eb6516693270621676980acd6d5a9a7efdbfa273f8d616c7/alembic-1.13.1-py3-none-any.whl (from https://pypi.org/simple/alembic/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('importlib-metadata~=6.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/a1/50/a76646f7031fdb63834a2b70226752b1d4d09c02d70b58273474ef3a9017/dbt_semantic_interfaces-0.4.2-py3-none-any.whl (from https://pypi.org/simple/dbt-semantic-interfaces/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('importlib-metadata; python_version < "3.9"'), parent=LinkCandidate('https://files.pythonhosted.org/packages/7f/50/9fb3a5c80df6eb6516693270621676980acd6d5a9a7efdbfa273f8d616c7/alembic-1.13.1-py3-none-any.whl (from https://pypi.org/simple/alembic/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('importlib-metadata~=6.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/74/7e/522b489d8735c7d44e2184b16a0ee7ffa8ec89c9c58b39f195e87659cc6b/dbt_semantic_interfaces-0.4.1-py3-none-any.whl (from https://pypi.org/simple/dbt-semantic-interfaces/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('importlib-metadata; python_version < "3.9"'), parent=LinkCandidate('https://files.pythonhosted.org/packages/7f/50/9fb3a5c80df6eb6516693270621676980acd6d5a9a7efdbfa273f8d616c7/alembic-1.13.1-py3-none-any.whl (from https://pypi.org/simple/alembic/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('importlib-metadata~=6.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/67/a5/9e503109f2228fcbf5ee3b5b6546248b55e53b06141cce75140a6b645288/dbt_semantic_interfaces-0.4.0-py3-none-any.whl (from https://pypi.org/simple/dbt-semantic-interfaces/) (requires-python:>=3.8)'))]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/piptools/scripts/compile.py", line 659, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 104, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

The only thing I can tell clearly is:

ERROR: Cannot install dagster and dbt-core because these package versions have conflicting dependencies.

The dagster.txt file does not have any dbt-core packages, but presumably dagster and dbt-core both have some shared dependency that is in conflict. How can I use pip-compile to figure out what that is?

Describe the solution you'd like

A human-readable version of the garbage following pip._vendor.resolvelib.resolvers.ResolutionImpossible in the error message above.

[webknjaz]

Seems like a bug in the depresolver (resolvelib that pip relies on) or our exception handling.
Try different pip versions, because we don't test each pip-tools release with every pip version. At least try using the pip version that was latest at the time of release.

This is because we use pip internals that are not public API. So I recommend locking pip-tools itself with its transitive deps just like you do for your app's deps.

[webknjaz]

Alternatively, if you use one of the latest pip versions, try upgrading pip-tools that can handle them.

[gnilrets]

I tried with pip-tools 7.4.1 and pip 24.0 (latest versions of both) and got the same error message.

I switched to the legacy resolver and got something much more interpretable

There are incompatible versions in the resolved dependencies:
  cryptography==38.0.4 (from -c dagster.txt (line 47))
  cryptography<43,>=41.0.5 (from pyOpenSSL==24.1.0->snowflake-connector-python[secure-local-storage]==3.7.1->-r dbt.in (line 6))
  cryptography<43.0.0,>=3.1.0 (from snowflake-connector-python[secure-local-storage]==3.7.1->-r dbt.in (line 6))
  cryptography>=3.4.0 (from pyjwt[crypto]==2.8.0->-c dagster.txt (line 162))

But I understand this is no longer the recommended resolver?

[webknjaz]

I think the latest pip has removed the legacy resolver already. I know that resolvelib is more correct and can't output conflicting deps like the old one could. If you use the old resolver, it might be worth running pip check post install to verify the integrity. Plus, it's recommended anyway, when you issue multiple pip install commands.

[webknjaz]

Does using just pip install -r dbt.in without pip-tools work?

[gnilrets]

Here's a simplified version of the files that reproduces this issue (still using pip-tools 7.4.1 and pip 24.0):

# dagster.txt

pyOpenSSL==24.1.0
cryptography==38.0.4

# dbt.in

-c dagster.txt

snowflake-connector-python[secure-local-storage]~=3.4

No, pip install -r dbt.in doesn't work, but it does give me a better error message.

ERROR: Cannot install -r dbt.in (line 3) because these package versions have conflicting dependencies.

The conflict is caused by:
    snowflake-connector-python 3.7.0 depends on pyOpenSSL<24.0.0 and >=16.2.0
    snowflake-connector-python 3.6.0 depends on pyOpenSSL<24.0.0 and >=16.2.0
    snowflake-connector-python 3.5.0 depends on pyOpenSSL<24.0.0 and >=16.2.0
    snowflake-connector-python 3.4.1 depends on pyOpenSSL<24.0.0 and >=16.2.0
    snowflake-connector-python 3.4.0 depends on pyOpenSSL<24.0.0 and >=16.2.0
    The user requested (constraint) pyopenssl==24.1.0

Compare this with pip-compile dbt.in:

  ERROR: Cannot install -r dbt.in (line 3) because these package versions have conflicting dependencies.
Traceback (most recent call last):
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/sterling.paramore/miniconda3/envs/eta-dagster/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 439, in resolve
    raise ResolutionImpossible(self.state.backtrack_causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('pyOpenSSL<24.0.0,>=16.2.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/5d/72/67a35dbd9414e07ca99e767733fe0bb0eccc508fb300e104ae2d450e0f73/snowflake_connector_python-3.7.0-cp38-cp38-macosx_11_0_arm64.whl (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('pyOpenSSL<24.0.0,>=16.2.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/b1/c2/5ecaadfc00dc55d8155a1bf44bcc6f060f7f9193f20296ee78776538ebbe/snowflake_connector_python-3.6.0-cp38-cp38-macosx_11_0_arm64.whl (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('pyOpenSSL<24.0.0,>=16.2.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/0a/a2/296f0bf74f1027cd8664bd8defd69f7d676d42be65396ffed3a308ca2035/snowflake_connector_python-3.5.0-cp38-cp38-macosx_11_0_arm64.whl (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('pyOpenSSL<24.0.0,>=16.2.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/1e/da/02591d9f263c00d557c421123cf65685cc7c210e136d6c3c6c65787c6015/snowflake_connector_python-3.4.1-cp38-cp38-macosx_11_0_arm64.whl (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.8)')), RequirementInformation(requirement=SpecifierRequirement('pyOpenSSL<24.0.0,>=16.2.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/6f/de/4d45c29beb1a3f4d430af54ff8184108c3e5d0a73c419bddd2bf6b6929a9/snowflake_connector_python-3.4.0-cp38-cp38-macosx_11_0_arm64.whl (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.8)'))]

It looks like the conflict information I need is somewhere in there, but it's not readable (at least not by a mere human like me).

[webknjaz]

Yeah, this is definitely worth improving. I guess, pip-tools needs to intercept the exception and print out the attached metadata in a more human-oriented format.

[webknjaz]

OTOH, if the error output is coming from a subprocess, we wouldn't be able to do anything about it. Somebody needs to inspect this deeper.

[chrysle]

pip-tools does catch pip's DistributionNotFound error, although one is not able to guess that from the exception. The error is, after resolution is found to be impossible, raised here:

pip-tools/piptools/resolver.py

Lines 662 to 663 in 6f2c9cd

	if cause_existing_ireq is None:
	raise

We could definitely refrain from doing that and instead print out the conflict causes in human-readable format.