Skip to content

Commit

Permalink
Restructure docs a bit. Add various new notes.
Browse files Browse the repository at this point in the history 
Ref #139, #176.
  • Loading branch information
@ionelmc
ionelmc committed Nov 24, 2017
1 parent 73e9d66 commit 509f77f
Show file tree
Hide file tree
Showing 10 changed files with 346 additions and 214 deletions.
213 changes: 17 additions & 196 deletions README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,14 @@ Overview

.. end-badges
This plugin produces coverage reports. It supports centralised testing and distributed testing in
both load and each modes. It also supports coverage of subprocesses.
This plugin produces coverage reports. Compared to just using ``coverage run`` this plugin does some extras:

All features offered by the coverage package should be available, either through pytest-cov or
* Subprocess support: you can fork or run stuff in a subprocess and will get covered without any fuss.
* Xdist support: you can use all of pytest-xdist's features and still get coverage.
* Consistent pytest behavior. If you run ``coverage run -m pytest`` you will have slightly different ``sys.path`` (CWD will be
in it, unlike when running ``pytest``).

All features offered by the coverage package should work, either through pytest-cov's command line options or
through coverage's config file.

* Free software: MIT license
Expand All @@ -72,14 +76,14 @@ For distributed testing support install pytest-xdist::

pip install pytest-xdist

Upgrade
=======
Upgrading from ancient pytest-cov
---------------------------------

`pytest-cov 2.0` is using a new ``.pth`` file (``pytest-cov.pth``). You may want to manually remove the older
``init_cov_core.pth`` from site-packages as it's not automatically removed.

Uninstallation
==============
Uninstalling
------------

Uninstall with pip::

Expand All @@ -94,62 +98,13 @@ Under certain scenarios a stray ``.pth`` file may be left around in site-package
Usage
=====

Centralised Testing
-------------------

Centralised testing will report on the combined coverage of the main process and all of its
subprocesses.

Running centralised testing::
::

py.test --cov=myproj tests/

Shows a terminal report::

-------------------- coverage: platform linux2, python 2.6.4-final-0 ---------------------
Name Stmts Miss Cover
----------------------------------------
myproj/__init__ 2 0 100%
myproj/myproj 257 13 94%
myproj/feature4286 94 7 92%
----------------------------------------
TOTAL 353 20 94%


Distributed Testing: Load
-------------------------

Distributed testing with dist mode set to load will report on the combined coverage of all slaves.
The slaves may be spread out over any number of hosts and each slave may be located anywhere on the
file system. Each slave will have its subprocesses measured.

Running distributed testing with dist mode set to load::

py.test --cov=myproj -n 2 tests/

Shows a terminal report::

-------------------- coverage: platform linux2, python 2.6.4-final-0 ---------------------
Name Stmts Miss Cover
----------------------------------------
myproj/__init__ 2 0 100%
myproj/myproj 257 13 94%
myproj/feature4286 94 7 92%
----------------------------------------
TOTAL 353 20 94%


Again but spread over different hosts and different directories::

py.test --cov=myproj --dist load
--tx ssh=memedough@host1//chdir=testenv1
--tx ssh=memedough@host2//chdir=/tmp/testenv2//python=/tmp/env1/bin/python
--rsyncdir myproj --rsyncdir tests --rsync examples
tests/
Would produce a report like::

Shows a terminal report::

-------------------- coverage: platform linux2, python 2.6.4-final-0 ---------------------
-------------------- coverage: ... ---------------------
Name Stmts Miss Cover
----------------------------------------
myproj/__init__ 2 0 100%
Expand All @@ -158,110 +113,15 @@ Shows a terminal report::
----------------------------------------
TOTAL 353 20 94%

Documentation
=============

Distributed Testing: Each
-------------------------

Distributed testing with dist mode set to each will report on the combined coverage of all slaves.
Since each slave is running all tests this allows generating a combined coverage report for multiple
environments.

Running distributed testing with dist mode set to each::

py.test --cov=myproj --dist each
--tx popen//chdir=/tmp/testenv3//python=/usr/local/python27/bin/python
--tx ssh=memedough@host2//chdir=/tmp/testenv4//python=/tmp/env2/bin/python
--rsyncdir myproj --rsyncdir tests --rsync examples
tests/

Shows a terminal report::
http://pytest-cov.rtfd.org/

---------------------------------------- coverage ----------------------------------------
platform linux2, python 2.6.5-final-0
platform linux2, python 2.7.0-final-0
Name Stmts Miss Cover
----------------------------------------
myproj/__init__ 2 0 100%
myproj/myproj 257 13 94%
myproj/feature4286 94 7 92%
----------------------------------------
TOTAL 353 20 94%


Reporting
=========

It is possible to generate any combination of the reports for a single test run.

The available reports are terminal (with or without missing line numbers shown), HTML, XML and
annotated source code.

The terminal report without line numbers (default)::

py.test --cov-report term --cov=myproj tests/

-------------------- coverage: platform linux2, python 2.6.4-final-0 ---------------------
Name Stmts Miss Cover
----------------------------------------
myproj/__init__ 2 0 100%
myproj/myproj 257 13 94%
myproj/feature4286 94 7 92%
----------------------------------------
TOTAL 353 20 94%


The terminal report with line numbers::

py.test --cov-report term-missing --cov=myproj tests/

-------------------- coverage: platform linux2, python 2.6.4-final-0 ---------------------
Name Stmts Miss Cover Missing
--------------------------------------------------
myproj/__init__ 2 0 100%
myproj/myproj 257 13 94% 24-26, 99, 149, 233-236, 297-298, 369-370
myproj/feature4286 94 7 92% 183-188, 197
--------------------------------------------------
TOTAL 353 20 94%

The terminal report with skip covered::

py.test --cov-report term:skip-covered --cov=myproj tests/

-------------------- coverage: platform linux2, python 2.6.4-final-0 ---------------------
Name Stmts Miss Cover
----------------------------------------
myproj/myproj 257 13 94%
myproj/feature4286 94 7 92%
----------------------------------------
TOTAL 353 20 94%

1 files skipped due to complete coverage.

You can use ``skip-covered`` with ``term-missing`` as well. e.g. ``--cov-report term-missing:skip-covered``

These three report options output to files without showing anything on the terminal::

py.test --cov-report html
--cov-report xml
--cov-report annotate
--cov=myproj tests/

The output location for each of these reports can be specified. The output location for the XML
report is a file. Where as the output location for the HTML and annotated source code reports are
directories::

py.test --cov-report html:cov_html
--cov-report xml:cov.xml
--cov-report annotate:cov_annotate
--cov=myproj tests/

The final report option can also suppress printing to the terminal::

py.test --cov-report= --cov=myproj tests/

This mode can be especially useful on continuous integration servers, where a coverage file
is needed for subsequent processing, but no local report needs to be viewed. For example,
tests run on Travis-CI could produce a .coverage file for use with Coveralls.

Coverage Data File
==================
Expand All @@ -273,33 +133,6 @@ this coverage data to coverage data from previous test runs.
The data file is left at the end of testing so that it is possible to use normal coverage tools to
examine it.


Coverage Config File
====================

This plugin provides a clean minimal set of command line options that are added to pytest. For
further control of coverage use a coverage config file.

For example if tests are contained within the directory tree being measured the tests may be
excluded if desired by using a .coveragerc file with the omit option set::

py.test --cov-config .coveragerc
--cov=myproj
myproj/tests/

Where the .coveragerc file contains file globs::

[run]
omit = tests/*

For full details refer to the `coverage config file`_ documentation.

.. _`coverage config file`: https://coverage.readthedocs.io/en/latest/config.html

Note that this plugin controls some options and setting the option in the config file will have no
effect. These include specifying source to be measured (source option) and all data file handling
(data_file and parallel options).

Limitations
===========

Expand All @@ -312,18 +145,6 @@ subprocess. The python used by the subprocess must have pytest-cov installed.
do normal site initialisation so that the environment variables can be detected and coverage
started.

Coverage and debuggers
----------------------

When it comes to TDD one obviously would like to debug tests. Debuggers in Python use mostly the sys.settrace function
to gain access to context. Coverage uses the same technique to get access to the lines executed. Coverage does not play
well with other tracers simultaneously running. This manifests itself in behaviour that PyCharm might not hit a
breakpoint no matter what the user does. Since it is common practice to have coverage configuration in the pytest.ini
file and pytest does not support removeopts or similar the `--no-cov` flag can disable coverage completely.

At the reporting part a warning message will show on screen

Coverage disabled via --no-cov switch!

Acknowledgements
================
Expand Down
37 changes: 37 additions & 0 deletions docs/config.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
=============
Configuration
=============

This plugin provides a clean minimal set of command line options that are added to pytest. For
further control of coverage use a coverage config file.

For example if tests are contained within the directory tree being measured the tests may be
excluded if desired by using a .coveragerc file with the omit option set::

py.test --cov-config .coveragerc
--cov=myproj
myproj/tests/

Where the .coveragerc file contains file globs::

[run]
omit = tests/*

For full details refer to the `coverage config file`_ documentation.

.. _`coverage config file`: https://coverage.readthedocs.io/en/latest/config.html

Note that this plugin controls some options and setting the option in the config file will have no
effect. These include specifying source to be measured (source option) and all data file handling
(data_file and parallel options).

Caveats
=======

A unfortunate consequence of coverage.py's history is that ``.coveragerc`` is a magic name: it's the default file but it also
means "try to also lookup coverage configuration in ``tox.ini`` or ``setup.cfg``".

In practical terms this means that if you have your coverage configuration in ``tox.ini`` or ``setup.cfg`` it is paramount
that you also use ``--cov-config=tox.ini`` or ``--cov-config=setup.cfg``.

You might not be affected but it's unlikely that you won't ever use ``chdir`` in a test.
15 changes: 15 additions & 0 deletions docs/debuggers.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
=====================
Debuggers and PyCharm
=====================

(or other IDEs)

When it comes to TDD one obviously would like to debug tests. Debuggers in Python use mostly the sys.settrace function
to gain access to context. Coverage uses the same technique to get access to the lines executed. Coverage does not play
well with other tracers simultaneously running. This manifests itself in behaviour that PyCharm might not hit a
breakpoint no matter what the user does. Since it is common practice to have coverage configuration in the pytest.ini
file and pytest does not support removeopts or similar the `--no-cov` flag can disable coverage completely.

At the reporting part a warning message will show on screen::

Coverage disabled via --no-cov switch!
19 changes: 12 additions & 7 deletions docs/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,19 @@ Welcome to pytest-cov's documentation!
Contents:

.. toctree::
:maxdepth: 2
:maxdepth: 2

readme
installation
contributing
releasing
authors
changelog
readme
config
reporting
debuggers
xdist
mp
markers-fixtures
changelog
authors
releasing
contributing

Indices and tables
==================
Expand Down
7 changes: 0 additions & 7 deletions docs/installation.rst
View file

This file was deleted.

0 comments on commit 509f77f

Please sign in to comment.