Coverage cleanups

[mhsmith]

Fixes #1673.

Also revists the source and source_pkgs configuration to make sure that we detect files which are never imported. I couldn't find any such files in core, dummy or winforms, but I tested adding one and it was reported correctly.

PR Checklist:

• All new features have been tested
• All new features have been documented
• I have read the CONTRIBUTING.md file
• I will abide by the code of conduct

[mhsmith]

Comparing the CI coverage output between the main branch and this branch, it appears to be identical, as expected.

[freakboy3742]

A nice set of cleanups, and it definitely seems to be working; however:

1. The ergonomics of the "simple developer case" aren't great. Ideally, either tox -e py in a project would either include a coverage report, or there would be a separate entry point like tox -e coverage to do the "combine and cover" dance. Needing 3 moderately complex commands with very specific arguments to work out whether my new test covers the features I've added isn't a great developer experience - especially when the usual expectation is "pytest && coverage report"
2. Whatever the instructions end up being, they need to be collected in the contribution docs, rather than buried in a config file comment.

Given that tox -e py is being used in each subproject to run the tests, one approach might be to add a top-level tox entry point to run tests & report coverage - tox -e py -- core (or tox -e core?)

[mhsmith]

OK, I've made tox -e py run coverage combine and coverage report automatically, because we want to encourage developers to pay attention to this.

I've also removed the backend coverage merge job, since each backend job is already reporting its own coverage. We might want to restore this if we ever start running the backends on multiple Python versions, but we should still merge each backend separately so we can start enforcing 100% coverage one backend at a time.

[freakboy3742]

All looks good; however the developer docs need to be updated to reflect these changes. The dev instructions still provide instructions for manually running coverage, and don't mention tox at all. We probably want to point people at tox as the "go to" simple case for running the test suite. I'm not sure if we need to preserve the docs about TOGA_BACKEND and manually running coverage in an "expert mode" instructions; technically, that knowledge is wrapped up in the tox and CI scripts if anyone wants to go down that route, and that stops the knowledge bitrotting over time (although hopefully we're nearing the end of the big changes to repo layout).

There's also some potential overlap with #1675 and adding a dev extras requires, so we don't need to manually specify installing pre-commit, coverage, etc. If you want to collect all the doc updates in that PR instead of this one to avoid having edits in 2 places, I'm fine with that. On that basis, I'll mark this as approved, but not merge it; if you want to merge it as-is, consider these comments about developer instructions to be a pre-requisite for my approval on #1675.

[saroad2]

Pay attention that all tox.ini scripts are now using bash, which means they will not work for Windows developers (like me)

[mhsmith]

Pay attention that all tox.ini scripts are now using bash, which means they will not work for Windows developers (like me)

Good point, I'll fix that.

[codecov]

Codecov Report

Merging #1674 (d16fe0d) into main (afa5f29) will increase coverage by 6.20%.
The diff coverage is n/a.

Impacted Files	Coverage Δ
dummy/src/toga_dummy/widgets/detailedlist.py
dummy/src/toga_dummy/widgets/button.py
dummy/src/toga_dummy/fonts.py
dummy/src/toga_dummy/images.py
dummy/src/toga_dummy/icons.py
dummy/src/toga_dummy/colors.py
dummy/src/toga_dummy/documents.py
dummy/src/toga_dummy/widgets/box.py
dummy/src/toga_dummy/dialogs.py
dummy/src/toga_dummy/widgets/table.py
... and 31 more

[mhsmith]

I'm getting tired of editing 7 files whenever I make a change to this, so I've merged all the tox.ini files into the root one, with tox environments named as follows:

$ tox -a
py-android
py-cocoa
py-core
py-gtk
py-iOS
py-web
py-winforms
py37-android
py37-cocoa
...

And I've updated the documentation accordingly.

adding a dev extras requires, so we don't need to manually specify installing pre-commit, coverage, etc.

Coverage commands are now all handled by tox, so there's no need to install coverage in the main virtual environment. As for pre-commit, its scope covers the whole repository, including non-Python stuff like YAML and TOML formatting. So rather than tying that to the installation of any particular package, I think it's better to keep the instructions to install that separately.

[freakboy3742]

The top level tox config works really well for me, and the fallback case of running pytest and coverage manually isn't that more complicated. It's also working for my testing on Windows, so I think we can call this done.

There's some lingering questions about the contribution docs and a dev dependency, but I think we can continue that discussion on #1675.