tox with dependencies from Artifact Registry (GCP)

[fedexist]

Hi,

I'm currently trying to use tox with a project which has dependencies stored on Artifact Registry, the Google Cloud Platform managed PyPi index, but I can't, due to how PIP_EXTRA_INDEX_URL works for both provisioning of tox and testenv.

This is my tox.ini

[tox]
envlist = py38, py39, flake8, docs
skipsdist = true
minversion = 3.24
isolated_build = True
requires =
    keyrings.google-artifactregistry-auth

[testenv]
usedevelop = True
description = invoke pytest to run automated tests
setenv =
    PYTHONPATH = {toxinidir}/src
    TOXINIDIR = {toxinidir}
    MYPYPATH = {toxinidir}/src
    PIP_EXTRA_INDEX_URL = https://europe-west3-python.pkg.dev/redacted-project-id/pypi/simple/
passenv =
    HOME
    GOOGLE_APPLICATION_CREDENTIALS
deps =
    -r{toxinidir}/requirements_dev.txt
commands =
    # various test commands
extras =
    testing

[testenv:{build,clean}]
description =
    build: Build the package in isolation according to PEP517, see https://github.com/pypa/build
    clean: Remove old distribution files and temporary build artifacts (./build and ./dist)
# https://setuptools.pypa.io/en/stable/build_meta.html#how-to-use-it
skip_install = True
changedir = {toxinidir}
basepython = python3.9
deps =
    setuptools
    setuptools-scm
    wheel
    build: build[virtualenv]
commands =
    clean: python -c 'import shutil; [shutil.rmtree(p, True) for p in ("build", "dist", "docs/_build")]'
    clean: python -c 'import pathlib, shutil; [shutil.rmtree(p, True) for p in pathlib.Path("src").glob("*.egg-info")]'
    build: python -m build {posargs}

[testenv:publish]
description =
    Publish the package you have been developing to a package index server.
    By default, it uses testpypi. If you really want to publish your package
    to be publicly accessible in PyPI, use the `-- --repository pypi` option.
skip_install = True
changedir = {toxinidir}
basepython = python3.9
passenv =
    PYPI_REPO
    GOOGLE_APPLICATION_CREDENTIALS
    HOME
deps =
    twine
commands =
    python3.9 -m twine check dist/*
    python3.9 -m twine upload --skip-existing --verbose --non-interactive {posargs:--repository-url {env:PYPI_REPO:http://default-pypi}} dist/*

As per the documentation provided by Google, I set the keyring as required and use passenv to make sure that the environment variable GOOGLE_APPLICATION_CREDENTIALS is propagated to the testenv.

What happens is the following:

1. Without setting PIP_EXTRA_INDEX_URL, tox -e clean,build,publish works: this means that the keyring plugin gets installed during the provisioning and is actually used for authenticating with Artifact Registry
2. Setting PIP_EXTRA_INDEX_URL, tox can't get past the provisioning phase because it tries to authenticate with the Artifact Registry in order to install keyrings.google-artifactregistry-auth: by default username and password are asked via prompt and tox gets stuck there, because obviously it can't input anything :)

So, what is clear here is that PIP_EXTRA_INDEX_URL is used for both tox provisioning and testenv package installation and I have a few questions:

• Is it even possible to set PIP_EXTRA_INDEX_URL only for the testenv?
• Is it possible for me, for example, to set GOOGLE_APPLICATION_CREDENTIALS during tox provisioning? Using passenv and setenv in that section didn't help.
• Is it possible to install keyrings.google-artifactregistry-auth as a sort of seed package, before everything else? Maybe there's a plugin for that?

I realize that I could put the package dependency in a requirements.txt file with the --extra-index-url flag (that is what's actually shown on the GCP github page linked earlier), but I was wondering if tox could manage this issue in a more elegant way. Or maybe I'm simply missing a way to already do what I want.

I see also that there's a somewhat related issue with #2428, regarding how install_command is used.

Thank you for your help.

[jugmac00]

Hey @fedexist!
Thanks for your detailed questions.

Without using GCP myself, and so without being able to play around with different options, this would require some detailed inspection of the source code, for which, unfortunately, currently I cannot find the time.

I would recommend to apply the suggested way of using tox, as detailed by @di who is a very experienced Python developer, and afaik also a long time tox user, so I imagine he chose the best option already.

Sorry, that is all I have.

[fedexist]

Hi @jugmac00, thanks for the feedback, I ended up using the suggested way by using a requirements file to be installed using deps, but in my opinion it's not really the best way in my case, since the dependencies are listed in setup.cfg.

[jugmac00]

Btw... Is the package you develop an application or a library?

[fedexist]

A library