You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
My team uses a monorepo that contains a number of different python projects and applications. To simplify dependency management, we use a recursive Makefile strategy to run pip-compile in different directories and potentially with different python versions with a single entrypoint command.
As this process is simple, but can take some time, we've started to invoke make with the -j argument (allowing different targets to be built in parallel. I've recently come to realize that there seems to be some contention or interference between these different parallel invocations of python -m piptools compile.
I realize this is probably fairly uncommon, but I was hopeful that adding more isolation of the dependency resolution process or temporary directories might be a simple solution. Also, please let me know if this is more of a pip issue than a pip-tools issue. I've done some research suggesting that while parallel pip execution isn't a guarantee, it often works. Other than that, the traceback itself was the only thing leading me to raise the issue here.
Environment Versions
OS Type: macOS 14
Python version: multiple 3.8.16, 3.9.11, 3.11.6
pip version: 23.3.1
pip-tools version: 7.3.0
make version: GNU Make 3.81
Steps to replicate
Apologies for the complicated repro. I've attached a "mini monorepo" example that demonstrates the issue.
Use pyenv to install the three versions of python listed in the .python-version files.
Open a shell session to the compile-error directory (from the unzipped example project)
Run make -j
If you don't see the error at first, try a couple more times. The repro has been pretty consistent for me.
Expected result
I hoped that these parallel executions wouldn't interfere with each other, not hit exceptions, and produce a compiled requirements list.
Actual result
Collecting hatchling>1.17.0
Using cached hatchling-1.18.0-py3-none-any.whl (75 kB)
Collecting packaging>=21.3
Using cached packaging-23.2-py3-none-any.whl (53 kB)
Collecting editables>=0.3
Using cached editables-0.5-py3-none-any.whl (5.1 kB)
Collecting pluggy>=1.0.0
Using cached pluggy-1.3.0-py3-none-any.whl (18 kB)
WARNING: Skipping page https://pypi.org/simple/pathspec/ because the GET request got Content-Type: .The only supported Content-Type is text/html
Collecting hatchling>1.17.0
Using cached hatchling-1.17.1-py3-none-any.whl (75 kB)
ERROR: Cannot install -r /var/folders/8b/15h_47_d0_b3fq29xjvzjzgm0000gn/T/build-reqs-lhhf9h4d.txt (line 1) because these package versions have conflicting dependencies.
The conflict is caused by:
hatchling 1.18.0 depends on pathspec>=0.10.1
hatchling 1.17.1 depends on pathspec>=0.10.1
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
WARNING: You are using pip version 22.0.4; however, version 23.3.1 is available.
You should consider upgrading via the '/private/var/folders/8b/15h_47_d0_b3fq29xjvzjzgm0000gn/T/build-env-r0cqsfyf/bin/python -m pip install --upgrade pip' command.
Traceback (most recent call last):
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/piptools/__main__.py", line 19, in <module>
cli()
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/piptools/scripts/compile.py", line 551, in cli
metadata = project_wheel_metadata(
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/build/util.py", line 54, in project_wheel_metadata
env.install(builder.build_system_requires)
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/build/env.py", line 214, in install
_subprocess(cmd)
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/build/env.py", line 79, in _subprocess
raise e
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/build/env.py", line 76, in _subprocess
subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/folders/8b/15h_47_d0_b3fq29xjvzjzgm0000gn/T/build-env-r0cqsfyf/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', '/var/folders/8b/15h_47_d0_b3fq29xjvzjzgm0000gn/T/build-reqs-lhhf9h4d.txt']' returned non-zero exit status 1.
The text was updated successfully, but these errors were encountered:
I don't know why your resolution conflicts occur, but will say that when running pip-compile in parallel you should use the --cache-dir option (or PIP_TOOLS_CACHE_DIR parameter) to use an isolated cache for each process (see #1083).
Thanks, @AndydeCleyre! I tried adding that option to our Makefile setup, but I'm unfortunately still getting the ResolutionImpossible exception. I expect that resolution is not, in fact, impossible, but that something else is causing a failure that's currently indistinguishable from a resolution failure. Perhaps this suggests that the issue is more related to how
Based on the traceback and the likelihood of the ResolutionImpossible exception regardless of --cache-dir, I'm suspicious that two separate, parallel pip-tools processes could end up contending for the same temporary build environment (e.g. /private/var/folders/8b/15h_47_d0_b3fq29xjvzjzgm0000gn/T/build-env-r0cqsfyf).
Do you think that's possible or have any other experience/insights about that potential risk area?
Is there another mechanism to control the location/isolation of the dependency resolution build environment that results from the call to project_wheel_metadata?
My team uses a monorepo that contains a number of different python projects and applications. To simplify dependency management, we use a recursive Makefile strategy to run pip-compile in different directories and potentially with different python versions with a single entrypoint command.
As this process is simple, but can take some time, we've started to invoke
make
with the-j
argument (allowing different targets to be built in parallel. I've recently come to realize that there seems to be some contention or interference between these different parallel invocations ofpython -m piptools compile
.I realize this is probably fairly uncommon, but I was hopeful that adding more isolation of the dependency resolution process or temporary directories might be a simple solution. Also, please let me know if this is more of a
pip
issue than apip-tools
issue. I've done some research suggesting that while parallelpip
execution isn't a guarantee, it often works. Other than that, the traceback itself was the only thing leading me to raise the issue here.Environment Versions
Steps to replicate
Apologies for the complicated repro. I've attached a "mini monorepo" example that demonstrates the issue.
compile-error.zip
pyenv
to install the three versions of python listed in the.python-version
files.compile-error
directory (from the unzipped example project)make -j
Expected result
I hoped that these parallel executions wouldn't interfere with each other, not hit exceptions, and produce a compiled requirements list.
Actual result
The text was updated successfully, but these errors were encountered: