Poetry doesn`t use provided private repo when compiling dependencies from source #3744

ojomio · 2021-03-02T10:26:23Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: Docker image: python:3.7-slim
Poetry version: 1.1.4
Link of a Gist:

We have a private repo, and it seems that while fetching .whl packages from it works fine, when any package on PyPi is published in source (and thus, stored in source in our mirror), building it leads to situation, when that package dependencies are being downloaded directly from the main PyPi.

Here is the log example from out TeamCity server, this time black package triggered the bug, because it only has tar.gz version

[14:48:59]W:	 [Step 1/3] Building test-base
[Step 1/3] Step 1/6 : ARG REPO
[Step 1/3] Step 2/6 : ARG BASE_TAG
[Step 1/3] Step 3/6 : FROM $REPO:base-$BASE_TAG
[Step 1/3]  ---> da6d27550293
[Step 1/3] Step 4/6 : ENV PATH="/opt/venv/bin:$PATH"
[Step 1/3]  ---> Running in fa8731d44d44
[14:50:09]W:	 [Step 1/3] time="2021-03-02T09:50:09.087519321Z" level=info msg="Layer sha256:dec5bbb567d383838da533f429828bcb60aa1bcac2316dff0c95b9a1e0097ade cleaned up"
[Step 1/3] Removing intermediate container fa8731d44d44
[Step 1/3]  ---> 857f149e23d3
[Step 1/3] Step 5/6 : WORKDIR /app
[Step 1/3]  ---> Running in 71b3546fe39b
[Step 1/3] Removing intermediate container 71b3546fe39b
[Step 1/3]  ---> f38c0aa5e084
[Step 1/3] Step 6/6 : RUN . /opt/venv/bin/activate     && poetry install -vvv
[Step 1/3]  ---> Running in b350e46d3fdb
[14:50:09]W:	 [Step 1/3] time="2021-03-02T09:50:09.603014373Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/b350e46d3fdb68d437e262700bbb6a3d5e6363eea9f77fcb554a31be0874f10d/shim.sock" debug=false pid=289 
[Step 1/3] Using virtualenv: /opt/venv
[Step 1/3] Installing dependencies from lock file
[Step 1/3] 
[Step 1/3] Finding the necessary packages for the current system
[Step 1/3] 
[Step 1/3] Package operations: 52 installs, 0 updates, 0 removals, 54 skipped
[Step 1/3] 
[Step 1/3]   • Removing atomicwrites (1.4.0): Skipped for the following reason: Not currently installed
[Step 1/3]   • Removing colorama (0.4.4): Skipped for the following reason: Not currently installed
[Step 1/3]   • Installing six (1.15.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing typing-extensions (3.7.4.3): Skipped for the following reason: Already installed
[Step 1/3]   • Installing zipp (3.4.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing importlib-metadata (3.7.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing jmespath (0.10.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing mccabe (0.6.1)
[Step 1/3]   • Installing pycodestyle (2.6.0)
[Step 1/3]   • Installing pyflakes (2.2.0)
[Step 1/3]   • Installing python-dateutil (2.8.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing urllib3 (1.26.3): Skipped for the following reason: Already installed
[Step 1/3]   • Installing botocore (1.20.15): Skipped for the following reason: Already installed
[Step 1/3]   • Installing flake8 (3.8.4)
[Step 1/3]   • Installing vine (5.0.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing pyparsing (2.4.7)
[Step 1/3]   • Installing wcwidth (0.2.5): Skipped for the following reason: Already installed
[Step 1/3]   • Installing amqp (5.0.5): Skipped for the following reason: Already installed
[Step 1/3]   • Installing attrs (20.3.0)
[Step 1/3]   • Installing chardet (4.0.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing certifi (2020.12.5): Skipped for the following reason: Already installed
[Step 1/3]   • Installing eradicate (2.0.0)
[Step 1/3]   • Installing flake8-polyfill (1.0.2)
[Step 1/3]   • Installing click (7.1.2): Skipped for the following reason: Already installed
[Step 1/3]   • Installing flake8-plugin-utils (1.3.1)
[Step 1/3]   • Installing idna (2.10): Skipped for the following reason: Already installed
[Step 1/3]   • Installing lazy-object-proxy (1.5.2)
[Step 1/3]   • Installing iniconfig (1.1.1)
[Step 1/3]   • Installing isort (5.7.0)
[Step 1/3]   • Installing packaging (20.9)
[Step 1/3]   • Installing pluggy (0.13.1)
[Step 1/3]   • Installing prompt-toolkit (3.0.16): Skipped for the following reason: Already installed
[Step 1/3]   • Installing py (1.10.0)
[Step 1/3]   • Installing s3transfer (0.3.4): Skipped for the following reason: Already installed
[Step 1/3]   • Installing testfixtures (6.17.1)
[Step 1/3]   • Installing toml (0.10.2)
[Step 1/3]   • Installing typed-ast (1.4.2)
[Step 1/3]   • Installing wrapt (1.12.1)
[Step 1/3]   • Installing appdirs (1.4.4): Skipped for the following reason: Already installed
[Step 1/3]   • Installing astroid (2.5)
[Step 1/3]   • Installing boto3 (1.17.15): Skipped for the following reason: Already installed
[Step 1/3]   • Installing click-didyoumean (0.0.3): Skipped for the following reason: Already installed
[Step 1/3]   • Installing click-plugins (1.1.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing filelock (3.0.12): Skipped for the following reason: Already installed
[Step 1/3]   • Installing flake8-breakpoint (1.1.0)
[Step 1/3]   • Installing flake8-comprehensions (3.3.1)
[Step 1/3]   • Installing flake8-eradicate (1.0.0)
[Step 1/3]   • Installing flake8-if-expr (1.0.4)
[Step 1/3]   • Installing click-repl (0.1.6): Skipped for the following reason: Already installed
[Step 1/3]   • Installing flake8-builtins (1.5.3)
[Step 1/3]   • Installing flake8-isort (4.0.0)
[Step 1/3]   • Installing billiard (3.6.3.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing flake8-logging-format (0.6.0)
[Step 1/3]   • Installing flake8-print (4.0.0)
[Step 1/3]   • Installing flake8-pytest (1.3)
[Step 1/3]   • Installing flake8-pytest-style (1.3.0)
[Step 1/3]   • Installing flake8-return (1.1.2)
[Step 1/3]   • Installing freezegun (1.1.0)
[Step 1/3]   • Installing h11 (0.8.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing httptools (0.0.13): Skipped for the following reason: Already installed
[Step 1/3]   • Installing kombu (5.0.2): Skipped for the following reason: Already installed
[Step 1/3]   • Installing lupa (1.9)
[Step 1/3]   • Installing mypy-extensions (0.4.3)
[Step 1/3]   • Installing numpy (1.20.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing overrides (3.1.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing pathspec (0.8.1)
[Step 1/3]   • Installing pep8-naming (0.11.1)
[Step 1/3]   • Installing pydantic (1.7.3): Skipped for the following reason: Already installed
[Step 1/3]   • Installing pytest (6.2.2)
[Step 1/3]   • Installing pytz (2021.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing redis (3.5.3): Skipped for the following reason: Already installed
[Step 1/3]   • Installing regex (2020.11.13)
[Step 1/3]   • Installing requests (2.25.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing scikit-learn (0.24.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing sortedcontainers (2.3.0)
[Step 1/3]   • Installing starlette (0.13.6): Skipped for the following reason: Already installed
[Step 1/3]   • Installing tqdm (4.57.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing untokenize (0.1.1)
[Step 1/3]   • Installing uvloop (0.12.2): Skipped for the following reason: Already installed
[Step 1/3]   • Installing websockets (7.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing allenai-common (1.1.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing black (20.8b1)
[Step 1/3]   • Installing celery (5.0.5): Skipped for the following reason: Already installed
[Step 1/3]   • Installing fakeredis (1.4.5)
[Step 1/3]   • Installing fastapi (0.61.2): Skipped for the following reason: Already installed
[Step 1/3]   • Installing gunicorn (19.9.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing humanize (3.2.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing pylint (2.7.1)
[Step 1/3]   • Installing flake8-awesome (1.2.1)
[Step 1/3]   • Installing mypy (0.812)
[Step 1/3]   • Installing joblib (1.0.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing pytest-custom-exit-code (0.3.0)
[Step 1/3]   • Installing pytest-lazy-fixture (0.6.3)
[Step 1/3]   • Installing pytest-freezegun (0.4.2)
[Step 1/3]   • Installing pytest-mock (3.5.1)
[Step 1/3]   • Installing requests-mock (1.8.0)
[Step 1/3]   • Installing scipy (1.6.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing sentry-sdk (0.14.3): Skipped for the following reason: Already installed
[Step 1/3]   • Installing sklearn (0.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing sqlalchemy (1.3.23): Skipped for the following reason: Already installed
[Step 1/3]   • Installing teamcity-messages (1.28)
[Step 1/3]   • Installing tenacity (6.3.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing threadpoolctl (2.1.0): Skipped for the following reason: Already installed
[Step 1/3]   • Installing tokenizers (0.10.1): Skipped for the following reason: Already installed
[Step 1/3]   • Installing unify (0.5)
[Step 1/3]   • Installing uvicorn (0.8.6): Skipped for the following reason: Already installed
[Step 1/3] 
[Step 1/3]   Stack trace:
[Step 1/3] 
[Step 1/3]   7  /opt/venv/lib/python3.7/site-packages/poetry/installation/executor.py:202 in _execute_operation
[Step 1/3]       200│ 
[Step 1/3]       201│             try:
[Step 1/3]     → 202│                 result = self._do_execute_operation(operation)
[Step 1/3]       203│             except EnvCommandError as e:
[Step 1/3]       204│                 if e.e.returncode == -2:
[Step 1/3] 
[Step 1/3]   6  /opt/venv/lib/python3.7/site-packages/poetry/installation/executor.py:276 in _do_execute_operation
[Step 1/3]       274│             return 0
[Step 1/3]       275│ 
[Step 1/3]     → 276│         result = getattr(self, "_execute_{}".format(method))(operation)
[Step 1/3]       277│ 
[Step 1/3]       278│         if result != 0:
[Step 1/3] 
[Step 1/3]   5  /opt/venv/lib/python3.7/site-packages/poetry/installation/executor.py:411 in _execute_install
[Step 1/3]       409│ 
[Step 1/3]       410│     def _execute_install(self, operation):  # type: (Install) -> None
[Step 1/3]     → 411│         return self._install(operation)
[Step 1/3]       412│ 
[Step 1/3]       413│     def _execute_update(self, operation):  # type: (Update) -> None
[Step 1/3] 
[Step 1/3]   4  /opt/venv/lib/python3.7/site-packages/poetry/installation/executor.py:449 in _install
[Step 1/3]       447│             args.insert(2, "-U")
[Step 1/3]       448│ 
[Step 1/3]     → 449│         return self.run_pip(*args)
[Step 1/3]       450│ 
[Step 1/3]       451│     def _update(self, operation):
[Step 1/3] 
[Step 1/3]   3  /opt/venv/lib/python3.7/site-packages/poetry/installation/executor.py:300 in run_pip
[Step 1/3]       298│     def run_pip(self, *args, **kwargs):  # type: (...) -> int
[Step 1/3]       299│         try:
[Step 1/3]     → 300│             self._env.run_pip(*args, **kwargs)
[Step 1/3]       301│         except EnvCommandError as e:
[Step 1/3]       302│             output = decode(e.e.output)
[Step 1/3] 
[Step 1/3]   2  /opt/venv/lib/python3.7/site-packages/poetry/utils/env.py:1042 in run_pip
[Step 1/3]       1040│         pip = self.get_pip_command()
[Step 1/3]       1041│         cmd = pip + list(args)
[Step 1/3]     → 1042│         return self._run(cmd, **kwargs)
[Step 1/3]       1043│ 
[Step 1/3]       1044│     def _run(self, cmd, **kwargs):
[Step 1/3] 
[Step 1/3]   1  /opt/venv/lib/python3.7/site-packages/poetry/utils/env.py:1332 in _run
[Step 1/3]       1330│             self.unset_env("__PYVENV_LAUNCHER__")
[Step 1/3]       1331│ 
[Step 1/3]     → 1332│             return super(VirtualEnv, self)._run(cmd, **kwargs)
[Step 1/3]       1333│ 
[Step 1/3]       1334│     def execute(self, bin, *args, **kwargs):
[Step 1/3] 
[Step 1/3]   EnvCommandError
[Step 1/3] 
[Step 1/3]   Command ['/opt/venv/bin/pip', 'install', '--no-deps', '/root/.cache/pypoetry/artifacts/77/5b/c2/c460eef2a93c55cee9e14ed5f99ba9e8f9dae6d7203fdc9353db14418a/black-20.8b1.tar.gz'] errored with the following return code 1, and output: 
[Step 1/3]   Processing /root/.cache/pypoetry/artifacts/77/5b/c2/c460eef2a93c55cee9e14ed5f99ba9e8f9dae6d7203fdc9353db14418a/black-20.8b1.tar.gz
[Step 1/3]     Installing build dependencies: started
[Step 1/3]     Installing build dependencies: finished with status 'error'
[Step 1/3]     ERROR: Command errored out with exit status 1:
[Step 1/3]      command: /opt/venv/bin/python /opt/venv/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-jrogzvo4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
[Step 1/3]          cwd: None
[Step 1/3]     Complete output (7 lines):
[Step 1/3]     WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 URLBlocked'))': /simple/setuptools/
[Step 1/3]     WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 URLBlocked'))': /simple/setuptools/
[Step 1/3]     WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 URLBlocked'))': /simple/setuptools/
[Step 1/3]     WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 URLBlocked'))': /simple/setuptools/
[Step 1/3]     WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 URLBlocked'))': /simple/setuptools/
[Step 1/3]     ERROR: Could not find a version that satisfies the requirement setuptools>=41.0
[Step 1/3]     ERROR: No matching distribution found for setuptools>=41.0
[Step 1/3]     ----------------------------------------
[Step 1/3]   WARNING: Discarding file:///root/.cache/pypoetry/artifacts/77/5b/c2/c460eef2a93c55cee9e14ed5f99ba9e8f9dae6d7203fdc9353db14418a/black-20.8b1.tar.gz. Command errored out with exit status 1: /opt/venv/bin/python /opt/venv/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-jrogzvo4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
[Step 1/3]   ERROR: Command errored out with exit status 1: /opt/venv/bin/python /opt/venv/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-jrogzvo4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
[Step 1/3]   
[Step 1/3] 
[Step 1/3]   at /opt/venv/lib/python3.7/site-packages/poetry/utils/env.py:1074 in _run
[Step 1/3]       1070│                 output = subprocess.check_output(
[Step 1/3]       1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
[Step 1/3]       1072│                 )
[Step 1/3]       1073│         except CalledProcessError as e:
[Step 1/3]     → 1074│             raise EnvCommandError(e, input=input_)
[Step 1/3]       1075│ 
[Step 1/3]       1076│         return decode(output)
[Step 1/3]       1077│ 
[Step 1/3]       1078│     def execute(self, bin, *args, **kwargs):
[Step 1/3]

Pay attention to this line

/opt/venv/bin/python /opt/venv/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-jrogzvo4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm

It seems to be the culprit with -i https://pypi.org/simple flag

The text was updated successfully, but these errors were encountered:

ojomio · 2021-03-03T07:31:29Z

We've come to use a temporary workaround for this by setting a pip config option

 pip config set global.index-url ${PYPI_URL}

crrlcx · 2021-03-04T12:50:34Z

You can use this workaround btw:
ENV PIP_INDEX_URL=${PYPI_URL}

aja-intel · 2021-03-29T20:52:35Z

This seems similar to #3249. I will have to try the env var work-around but hoping for a more permanent fix by passing down the default index to pip install commands.

I write to switch system to poetry. Because, `poetry add "git+https://github.com/jvkersch/hsmmlearn"` is failed. poetry don't load system Cython. [Poetry doesn`t use provided private repo when compiling dependencies from source · Issue #3744 · python-poetry/poetry](python-poetry/poetry#3744) ~~~console 2021-06-02T18:46:48 ✖ poetry add "git+https://github.com/jvkersch/hsmmlearn" PackageInfoError Unable to determine package info for path: /tmp/pypoetry-git-hsmmlearn0klkxnrl Fallback egg_info generation failed. Command ['/tmp/tmpqwo321rq/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output: Traceback (most recent call last): File "/tmp/pypoetry-git-hsmmlearn0klkxnrl/setup.py", line 54, in <module> ext_modules=get_extension_modules(), File "/tmp/pypoetry-git-hsmmlearn0klkxnrl/setup.py", line 23, in get_extension_modules from Cython.Build import cythonize ModuleNotFoundError: No module named 'Cython' at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/inspection/info.py:502 in _pep517_metadata 498│ try: 499│ venv.run("python", "setup.py", "egg_info") 500│ return cls.from_metadata(path) 501│ except EnvCommandError as fbe: → 502│ raise PackageInfoError( 503│ path, "Fallback egg_info generation failed.", fbe 504│ ) 505│ finally: 506│ os.chdir(cwd.as_posix()) ~~~ Even if you were to upload it to PyPi. Basically, it is better to keep the information needed for the build closed locally. It is easier to keep the information needed to build locally, even if it is uploaded to PyPi, so I let poetry do the work and made the global library as unnecessary as possible.

neersighted · 2022-10-04T07:53:39Z

Duplicate #3249, indeed.

github-actions · 2024-03-01T07:06:30Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

ojomio added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Mar 2, 2021

ncaq mentioned this issue Jun 2, 2021

changed: switch to poetry jvkersch/hsmmlearn#46

Open

neersighted closed this as not planned Won't fix, can't repro, duplicate, stale Oct 4, 2022

neersighted added status/duplicate Duplicate issues and removed kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Oct 4, 2022

github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Poetry doesn`t use provided private repo when compiling dependencies from source #3744

Poetry doesn`t use provided private repo when compiling dependencies from source #3744

ojomio commented Mar 2, 2021 •

edited

ojomio commented Mar 3, 2021

crrlcx commented Mar 4, 2021 •

edited

aja-intel commented Mar 29, 2021

neersighted commented Oct 4, 2022

github-actions bot commented Mar 1, 2024

Poetry doesn`t use provided private repo when compiling dependencies from source #3744

Poetry doesn`t use provided private repo when compiling dependencies from source #3744

Comments

ojomio commented Mar 2, 2021 • edited

ojomio commented Mar 3, 2021

crrlcx commented Mar 4, 2021 • edited

aja-intel commented Mar 29, 2021

neersighted commented Oct 4, 2022

github-actions bot commented Mar 1, 2024

ojomio commented Mar 2, 2021 •

edited

crrlcx commented Mar 4, 2021 •

edited