Support report terminal output in Markdown Table format

[sullivancolin]

The current terminal report is great, but wondered if there would be interest in adding a new output format or flag to support markdown table syntax.

This would enable nice display of coverage reports in rendered html that contains the markdown output. Some examples where this might be nice:

1. Users could used Cog to automate the inclusion of coverage report information in documentation.
2. GitHub actions workflow summaries could have nicely formatted tables of coverage information, example here

Current appearance:

Name                      Stmts   Miss  Cover
---------------------------------------------
my_program.py                20      4    80%
my_module.py                 15      2    86%
my_other_module.py           56      6    89%
---------------------------------------------
TOTAL                        91     12    87%

Sample Markdown syntax:

| Name               | Stmts      | Miss     |    Cover |
| ------------------ | ---------- | -------- | -------- |
| my_program.py      |         20 |        4 |      80% |
| my_module.py       |         15 |        2 |      86% |
| my_other_module.py |         56 |        6 |      89% |
| **TOTAL**          |     **91** |   **12** |  **87%** |

Sample Rendering:

Name	Stmts	Miss	Cover
my_program.py	20	4	80%
my_module.py	15	2	86%
my_other_module.py	56	6	89%
TOTAL	91	12	87%

Describe the solution you'd like
Ideally the a flag to the default report subcommand could indicate to use markdown syntax like:

$ coverage report --markdown
| Name               | Stmts      | Miss     |    Cover |
| ------------------ | ---------- | -------- | -------- |
| my_program.py      |         20 |        4 |      80% |
| my_module.py       |         15 |        2 |      86% |
| my_other_module.py |         56 |        6 |      89% |
| **TOTAL**          |     **91** |   **12** |  **87%** |
$

This would allow the output to be piped to a file in the GitHub action summary, or captured by cog to inject into a markdown file.

Describe alternatives you've considered
If the report subcommand is not suitable place to add this flag then a new subcommand specifically for markdown reporting might be a good alternative with coverage markdown

$ coverage markdown
| Name               | Stmts      | Miss     |    Cover |
| ------------------ | ---------- | -------- | -------- |
| my_program.py      |         20 |        4 |      80% |
| my_module.py       |         15 |        2 |      86% |
| my_other_module.py |         56 |        6 |      89% |
| **TOTAL**          |     **91** |   **12** |  **87%** |
$ 

[nedbat]

This is an interesting idea. Maybe someone wants to make a pull request for it. Some thoughts:

The markdown should use right alignment for the numeric columns:

| Name               | Stmts      | Miss     |    Cover |
| ------------------ | ---------: | -------: | -------: |
| my_program.py      |         20 |        4 |      80% |
| my_module.py       |         15 |        2 |      86% |
| my_other_module.py |         56 |        6 |      89% |
| **TOTAL**          |     **91** |   **12** |  **87%** |

Name	Stmts	Miss	Cover
my_program.py	20	4	80%
my_module.py	15	2	86%
my_other_module.py	56	6	89%
TOTAL	91	12	87%

I think a --format=markdown, with a default of --format=text would be the best UI.

There will probably need to be some significant refactoring in summary.py to make this work well.

Of course, another option is to generate a JSON report, and then use a separate program to produce the markdown :)

[stepeos]

I lke this feature idea, so I'll try to implement this as my first contribution.
Is there a set of coverage files that cover all cases? E.g. I've seen
fmt_skip_covered = "\n%s file%s skipped due to complete coverage."
in the source code.

[nedbat]

Hi, thanks! I'm not sure what you mean by a set of coverage files. The test suite doesn't have stored coverage data files, it generates them as needed. The test suite is large and weird, so it's not always easy to find the right place to tap into it. Let me know if I can add to the contributing docs to help you navigate, or ask here if you need more guidance.

[stepeos]

So far, this is what I've come up with. (https://github.com/stepeos/coveragepy)

| Name                                                                                         |  Stmts |   Miss |  Cover |  Missing |
|--------------------------------------------------------------------------------------------- | -----: | -----: | -----: | -------: |
| /some/path/to/some/veryelaborate/packe123/coverage\_testing/dummy\_package/\_\_init\_\_.py   |      0 |      0 |   100% |          |
| /some/path/to/some/veryelaborate/packe123/coverage\_testing/dummy\_package/a/\_\_init\_\_.py |      0 |      0 |   100% |          |
| /some/path/to/some/veryelaborate/packe123/coverage\_testing/dummy\_package/a/foo.py          |     24 |      9 |    62% |16, 22-29 |
| /some/path/to/some/veryelaborate/packe123/coverage\_testing/dummy\_package/b/\_\_init\_\_.py |      0 |      0 |   100% |          |
| /some/path/to/some/veryelaborate/packe123/coverage\_testing/dummy\_package/b/bar.py          |     13 |      3 |    77% |    18-20 |
|                                                                                    **TOTAL** | **37** | **12** |**68%** |          |

Name	Stmts	Miss	Cover	Missing
/some/path/to/some/veryelaborate/packe123/coverage_testing/dummy_package/__init__.py	0	0	100%
/some/path/to/some/veryelaborate/packe123/coverage_testing/dummy_package/a/__init__.py	0	0	100%
/some/path/to/some/veryelaborate/packe123/coverage_testing/dummy_package/a/foo.py	24	9	62%	16, 22-29
/some/path/to/some/veryelaborate/packe123/coverage_testing/dummy_package/b/__init__.py	0	0	100%
/some/path/to/some/veryelaborate/packe123/coverage_testing/dummy_package/b/bar.py	13	3	77%	18-20
TOTAL	37	12	68%

[stepeos]

@nedbat
Thanks for the offer. Here's my question. I'm trying to run the unit tests of test_summary.py , but all tests using run_command (e.g. https://github.com/stepeos/coveragepy/blob/b2224765db03ceb40435df7585ed33620a18c861/tests/test_summary.py#L632 )fail, since I work in an environment without coverage installed. Obviously I get the output:
$ /bin/sh: 1: coverage: not found
How do I best go about this, without rewriting all tests?

[nedbat]

Hi, thanks so much for taking this on! It would really help me to understand how you discovered what to do. I tried to write instructions for contributions and running tests, but they might be in the wrong place, or hard to find, or out of date. Your new eyes are a valuable resource for improving them. What instructions were you following?

[stepeos]

How I discovered what to do:

I cloned the repor and inserted an ordinary coverage file that I created just for the purpose of this contribution.
I then debugged to understand the process of how to report is printed to stdout.
You already mentioned, that the summary.py is the main mechanism behind the reporting, which was my first clue.

About the testing:

I read here about testing.
I just now noticed, that following the documentation and running tox (tox tests/foo.py::bar) does work, whereaspytest foo::bar does not.
I haven't worked with either before and didn't know the difference, so it was my bad, not poor docu. :)

Running the tests with vscode python testing extension(which uses pytest) doesn't work, which is what I tried to do.
But is there a nice way of debuggin tests in vscode, that's not in the documentation?

Also, I started with a fresh miniconda environment and running tox reported missing modules, I think pytest-xdist and hypothesis, which weren't in the requirements.

[nedbat]

Running the tests with vscode python testing extension (which uses pytest) doesn't work, which is what I tried to do.
But is there a nice way of debugging tests in vscode, that's not in the documentation?

Sorry, I don't have experience with VSCode. Can you show what happened when you try to use its testing extension?

Also, I started with a fresh miniconda environment and running tox reported missing modules, I think pytest-xdist and hypothesis, which weren't in the requirements.

I'm not sure which requirements you used. requirements/pytest.txt mentions pytest-xdist and hypothesis, and it's used by tox.ini, so I'm not sure what's going wrong.

[stepeos]

So with a fresh mniconda environment, after
$ pip install -r requirements/dev.pip
-> zsh: command not found: tox
I can see, that tox is dev.pip, however, this is probably a problem of my configuration?...

Also, when I run tox, I get some errors, but only for py311,pypy3 and py39

py37: commands succeeded
SKIPPED:  py38: InterpreterNotFound: python3.8
ERROR:   py39: commands failed
ERROR:   py310: commands failed
SKIPPED:  py311: InterpreterNotFound: python3.11
SKIPPED:  pypy3: InterpreterNotFound: pypy3
ERROR:   doc: commands failed
ERROR:   lint: could not install deps [-rrequirements/lint.pip]; v = InvocationError('/home/bookboi/Nextcloud/1_Docs/8_Coding/coveragepy/.tox/lint/bin/python -m pip install -U -r 
requirements/lint.pip', 1)

[nedbat]

This was done in #1479, and finished in 01d8677.

[webknjaz]

@nedbat how about dogfooding this in CI?

I was using irongut/CodeCoverageSummary. Here's an integration example: https://github.com/ansible-community/ansible-test-gh-action/blob/ce3e835/action.yml#L493-L511.
This one requires producing XML output, but it does look nice. See: https://github.com/ansible-collections/community.mysql/actions/runs/3437117181#summary-9412662875. (ignore the "Tests passed" block — it's produced by another action, https://github.com/test-summary/action)

[nedbat]

This is now released as part of coverage 7.0.0b1.

[sullivancolin]

Tested this out from the beta and it works great! Using it for github actions summary. Thanks so much for adding this. Wish I had been able to help more.

[webknjaz]

@sullivancolin do you have a demo of your summary output?

[sullivancolin]

@webknjaz I don't have a public project I can share at the moment. It's pretty simple to setup. I add these steps to my GHA CI workflow. Every time it runs, the Summary view will have the markdown formatted table as well as artifacts for the full coverage html report and the raw .coverage data

      - name: Run tests
        run: |
          poetry run coverage run -m pytest
          poetry run coverage report

      - name: Create HTML report
        run: |
          poetry run coverage html
          echo '## Test Coverage Report\n' >> $GITHUB_STEP_SUMMARY
          poetry run coverage report --format=markdown >> $GITHUB_STEP_SUMMARY

      - name: Upload HTML report
        uses: actions/upload-artifact@v3
        with:
          name: coverage_html
          path: htmlcov

      - name: Upload coverage data
        uses: actions/upload-artifact@v3
        with:
          name: coverage
          path: .coverage

[webknjaz]

Thanks, I was mostly curious about the rendering differences.