Separate `tox.ini` into multiple files #2508

ocelotl · 2024-05-08T14:50:26Z

This is still a draft PR because I have not yet refactored all packages so that all of them have their own tox.ini file.

The advantages of this proposal are:

The main tox.ini file will be much smaller. Right now, this file is extremely big.
The tox commands will be much simpler. Instead of running tox -e py38-test-opentelemetry-name-of-the-package we will be able to just cd into the package directory and then run tox -e test-py38.
This PR will add coverage testing for all packages.
Right now we run pytest with the benchmark options active for all tests, even when only one package here has benchmark tests. This introduces a warning in all of our logs. We should only run pytest with the benchmark options when there are benchmark tests, this PR fixes this issue.
Right now, we should not run coverage nor benchmark tests in CI (please read the comments below for an explanation). This PR allows us to not run these tests in CI.
Having a separate tox.ini file per package allows users to simply run tox in the package they changed to run all necessary tests.

ocelotl · 2024-05-08T17:55:06Z

.github/workflows/instrumentations_0.yml

@@ -120,4 +120,4 @@ jobs:
            ~/.cache/pip
          key: v7-build-tox-cache-${{ env.RUN_MATRIX_COMBINATION }}-${{ hashFiles('tox.ini', 'gen-requirements.txt', 'dev-requirements.txt') }}
      - name: run tox
-        run: tox -f ${{ matrix.python-version }}-${{ matrix.package }} -- -ra --benchmark-json=${{ env.RUN_MATRIX_COMBINATION }}-benchmark.json


We should not run the benchmark option for every test run, only for those that include benchmark tests.

ocelotl · 2024-05-08T17:55:28Z

.github/workflows/lint.yml

@@ -68,8 +68,6 @@ jobs:
          - "processor-baggage"
          - "propagator-aws-xray"
          - "propagator-ot-trace"
-          - "resource-detector-container"


I am using these 2 packages as an example.

ocelotl · 2024-05-08T17:56:43Z

.github/workflows/lint_1.yml

+            ~/.cache/pip
+          key: v7-build-tox-cache-${{ env.RUN_MATRIX_COMBINATION }}-${{ hashFiles('tox.ini', 'gen-requirements.txt', 'dev-requirements.txt') }}
+      - name: run tox
+        run: tox -c ${{ matrix.package }}/tox.ini -e lint


Notice how we are running tox using a specific tox.ini file, we use this pattern to avoid having to cd into every package directory and back.

ocelotl · 2024-05-08T17:57:59Z

.gitignore

 coverage.xml
 .coverage
 .nox
 .tox
 .cache
 htmlcov
+benchmark.json


This file is generated when benchmark tests are run in the local computer of a developer. Added in this file to avoid adding it to git unintentionally.

ocelotl · 2024-05-08T17:59:04Z

.isort.cfg

@@ -14,6 +14,6 @@ profile=black
 ; docs: https://github.com/timothycrosley/isort#multi-line-output-modes
 multi_line_output=3
 skip=target
-skip_glob=**/gen/*,.venv*/*,venv*/*,.tox/*
+skip_glob=**/gen/*,.venv*/*,venv*/*,.tox/*,**/.tox/*


Since new tox.ini files are being added, new .tox directories will be generated into every package when tox is executed locally. This tells isort to not check these new .tox directories.

ocelotl · 2024-05-08T18:05:11Z

resource/opentelemetry-resource-detector-container/tox.ini

+  lint: pylint --rcfile={toxinidir}/../../.pylintrc {toxinidir}/src/opentelemetry
+  lint: pylint --rcfile={toxinidir}/../../.pylintrc {toxinidir}/tests
+
+  coverage: coverage erase


Using the coverage commands directly here, so that it is not necessary to check an additional file (scripts/coverage/sh) to understand what coverage tests are doing.

ocelotl · 2024-05-08T18:06:01Z

resource/opentelemetry-resource-detector-container/tox.ini

+  coverage: coverage report --show-missing
+  coverage: coverage xml
+
+  spellcheck: codespell --config {toxinidir}/../../.codespellrc {toxinidir}


spellcheck can now be run only for a certain package instead for the whole repo, making it faster for most changes.

ocelotl · 2024-05-08T18:09:05Z

sdk-extension/opentelemetry-sdk-extension-aws/tox.ini

+
+  spellcheck: codespell --config {toxinidir}/../../.codespellrc {toxinidir}
+
+  benchmark: pytest {toxinidir}/benchmarks --benchmark-json=benchmark.json


Notice how the benchmark tests are now in their own directory, not inside tests. This makes it possible to run them separately and not run them in CI.

ocelotl · 2024-05-08T18:12:46Z

tox.ini

@@ -1169,8 +1126,6 @@ commands =
  lint-exporter-prometheus-remote-write: pylint {toxinidir}/exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry
  lint-exporter-prometheus-remote-write: pylint {toxinidir}/exporter/opentelemetry-exporter-prometheus-remote-write/tests

-  coverage: {toxinidir}/scripts/coverage.sh


Right now, we should not run coverage nor benchmark tests in CI. As they are configured right now, the coverage tests will not fail if a decrease in coverage is detected, the benchmark test will not fail if a decrease in performance is detected. For this reason, they cannot fail CI in a useful manner and it makes no sense to spend resources running them.

For these tests to be usefully executed in CI we need to configure them so that they can fail CI if a decrease in performance (for the benchmark tests) or a decrease in coverage (for the coverage tests) is detected.

ocelotl · 2024-05-08T18:13:34Z

tox.ini

@@ -1120,20 +1091,6 @@ commands =

  test-util-http: pytest {toxinidir}/util/opentelemetry-util-http/tests {posargs}

-  test-sdk-extension-aws: pytest {toxinidir}/sdk-extension/opentelemetry-sdk-extension-aws/tests {posargs}


The size of this tox.ini file will be greatly reduced when the rest of the packages are refactored.

lzchen · 2024-05-24T19:29:31Z

I'm not really super enthusiastic about having so many tox files. Is the largeness of the singular tox file so much that it warrants separating these out?

From the issue:

Right now, tox.ini is extremely long and it is easy to overlook important testing details.

What "important testing details" are you referring to specifically? Maybe if we call them out, there are smaller, less intrusive changes we can make.

ocelotl · 2024-05-28T18:39:57Z

I'm not really super enthusiastic about having so many tox files. Is the largeness of the singular tox file so much that it warrants separating these out?

From the issue:

Right now, tox.ini is extremely long and it is easy to overlook important testing details.

What "important testing details" are you referring to specifically? Maybe if we call them out, there are smaller, less intrusive changes we can make.

When I was introducing the changes that split lint into multiple tests, I noticed that we were missing several missing testing for a few of our components. It was hard to notice among the hundreds of lines in our tox.ini.

Nevertheless, splitting the tox.ini file is just another enhancement that this PR will introduce, please take a look at the advantages listed in this PR description.

aabmass · 2024-06-06T17:10:29Z

My main concern with this approach is that the common parts of the config that all tox.ini share may drift over time and become inconsistent or difficult to maintain. For example, adding/removing a supported Python version will require updating every tox file. More subtle things like changing the flags to a command or adding a new tool would be tricky.

IIRC tox doesn't really have any mechanism for re-use like include/extend?

ocelotl added the Skip Changelog PRs that do not require a CHANGELOG.md entry label May 8, 2024

ocelotl self-assigned this May 8, 2024

github-actions bot assigned NathanielRN May 8, 2024

WIP

7c5ad67

ocelotl force-pushed the issue_2478 branch from bbaad16 to 7c5ad67 Compare May 8, 2024 17:54

ocelotl commented May 8, 2024

View reviewed changes

ocelotl changed the title ~~WIP~~ Separate tox.ini into multiple files May 8, 2024

ocelotl requested review from aabmass and lzchen May 24, 2024 13:41

Merge branch 'main' into issue_2478

e56584b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Separate `tox.ini` into multiple files #2508

Separate `tox.ini` into multiple files #2508

ocelotl commented May 8, 2024 •

edited

ocelotl May 8, 2024

ocelotl May 8, 2024

ocelotl May 8, 2024

ocelotl May 8, 2024

ocelotl May 8, 2024

ocelotl May 8, 2024

ocelotl May 8, 2024

ocelotl May 8, 2024

ocelotl May 8, 2024

ocelotl May 8, 2024

lzchen commented May 24, 2024

ocelotl commented May 28, 2024

aabmass commented Jun 6, 2024


		spellcheck: codespell --config {toxinidir}/../../.codespellrc {toxinidir}

		benchmark: pytest {toxinidir}/benchmarks --benchmark-json=benchmark.json

		@@ -1120,20 +1091,6 @@ commands =

		test-util-http: pytest {toxinidir}/util/opentelemetry-util-http/tests {posargs}

		test-sdk-extension-aws: pytest {toxinidir}/sdk-extension/opentelemetry-sdk-extension-aws/tests {posargs}

Separate tox.ini into multiple files #2508

Are you sure you want to change the base?

Separate tox.ini into multiple files #2508

Conversation

ocelotl commented May 8, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzchen commented May 24, 2024

ocelotl commented May 28, 2024

aabmass commented Jun 6, 2024

Separate `tox.ini` into multiple files #2508

Separate `tox.ini` into multiple files #2508

ocelotl commented May 8, 2024 •

edited