Skip to content
This repository has been archived by the owner on Dec 16, 2022. It is now read-only.

Upload code coverage reports from different jobs, other CI improvements #5257

Merged
merged 28 commits into from Jun 15, 2021

Conversation

epwalsh
Copy link
Member

@epwalsh epwalsh commented Jun 11, 2021

Closes #5253. This gives us a more accurate code coverage report now, showing 91% covered instead of 88%.

image

This also makes a few other improvements to CI. Altogether, the changes proposed are:

  • Run GPU tests directly on the VM instead of inside Docker. I think the reason we didn't do this initially is that we couldn't get the setup-python action to work on the self-hosted runner. But I was able to get it to work now by following this comment.
  • Create a separate job for uploading coverage reports from different jobs. All jobs that generate a coverage report pass their reports on to this job as an artifact, and then all of the coverage reports are uploading together.
  • Generate a code coverage report from the GPU tests.
  • Generate a code coverage report from the models tests.
  • Move flake8 and mypy checks to a separate job called Lint
  • Move black --check . to a separate job called Style.
  • Rename "Check Core" to "CPU Tests" since this job now only runs tests and nothing else.
  • Rename "GPU Checks" to "GPU Tests" to be consistent with "CPU Tests".
  • Rename "Check Models" to "Model Tests"
  • Only run model tests with Python 3.8 to lower the number of concurrent jobs. GitHub Actions puts a limit on concurrency and I think we have enough coverage already since the CI in the models repo still runs with both 3.7 and 3.8.
  • Some other minor cleanup, including removing the .bulldozer.yml config which we don't use anymore.

@epwalsh epwalsh changed the title Upload code coverage reports from different jobs Upload code coverage reports from different jobs, other CI improvements Jun 13, 2021
@epwalsh epwalsh marked this pull request as ready for review June 14, 2021 16:36
@epwalsh epwalsh requested review from dirkgr and AkshitaB June 14, 2021 17:04
Copy link
Contributor

@AkshitaB AkshitaB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just a couple of comments/suggestions.

allennlp/common/testing/__init__.py Outdated Show resolved Hide resolved
mv coverage.xml coverage/

- name: Save coverage report
if: matrix.python == '3.7' && github.repository == 'allenai/allennlp' && (github.event_name == 'push' || github.event_name == 'pull_request')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we fix 3.7 here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, this was needed when these tests ran on multiple versions of Python, because we only needed one coverage report from this job.

pip freeze

- name: Run black
if: '! cancelled()'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this check necessary? Doesn't it always cancel all the checks?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be removed, it was only need when this check was in the other workflow that had a lot of different steps. And then it survived the copy-paste.

Comment on lines 140 to 142
# Could run into issues with the cache if we don't uninstall the editable.
# See https://github.com/pypa/pip/issues/4537.
pip uninstall --yes allennlp
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads like some hard-won wisdom.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed.

Comment on lines 280 to 285
- name: Debug info
run: |
ls -lh coverage
ls -lh coverage/cpu_tests
ls -lh coverage/gpu_tests
ls -lh coverage/model_tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not, I'll take out.

epwalsh and others added 3 commits June 14, 2021 11:22
Co-authored-by: Akshita Bhagia <akshita23bhagia@gmail.com>
@epwalsh epwalsh requested review from dirkgr and AkshitaB June 14, 2021 22:44
@epwalsh epwalsh merged commit 5da5b5b into main Jun 15, 2021
@epwalsh epwalsh deleted the coverage-reports branch June 15, 2021 22:49
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upload coverage report from GPU tests and model tests
3 participants