Coverage report generation incompatible with pytest tmpdir feature

[bluenote10]

Given a project structure like:

myproject
├── myproject
│   └── __init__.py
├── tests
│   └── myproject
│       └── test.py
└── setup.cfg

I'm running the command in the toplevel myproject directory:

py.test --cov=myproject --cov-report html

On stdout I can see the log output:

Coverage HTML written to dir htmlcov

However no output has been written, i.e., the myproject directory does not contain htmlcov (and neither of its subdirectories). Even a find / -name "*htmlcov*" reveals nothing.

Notes:

• The output is written if I use --cov-report html:/absolute/path/to/myproject/htmlcov.
• I get the same issue for --cov-report xml.
• My tests chdir a lot -- can this mess up pytest-cov?

Versions:

• pytest: 3.8.1
• pytest-cov: 2.6.0

[blueyed]

My tests chdir a lot -- can this mess up pytest-cov?

Sounds like it. I assume it gets written to some temporary directory maybe (outside of with you run find then).
Try exporting COV_DEBUG (see coverage.py doc for options) to see what is going on.

[blueyed]

Is .coverage created as expected?
I.e. does coverage report work after running the tests?
Then coverage html htmlcov might work also from there.

[bluenote10]

Yes the .coverage file gets created in the current directory and I also get sensible output from the terminal reporters. When running coverage html htmlcov afterwards in the project directory I get the error:

No source for code: '/home/me/workspace/myproject/htmlcov': [Errno 21] Is a directory: u'/home/me/workspace/myproject/htmlcov'

This at least creates the htmlcov folder in the project directory, but it is empty.

I haven't gotten much out of exporting COV_DEBUG so far.

[blueyed]

The command should be coverage html -d htmlcov, of course - sorry.

I haven't gotten much out of exporting COV_DEBUG so far.

What options are you using?

[blueyed]

btw: it is probably better to look at pytest-cov's code, where it handles the html report - i.e. put a pdb.set_trace() there / print os.getcwd().

[bluenote10]

The command should be coverage html -d htmlcov, of course - sorry.

Ah yes, this generates the output just fine.

What options are you using?

What I did was just export COV_DEBUG=true because I couldn't find any documentation on COV_DEBUG. I also don't have a coverage.py.

[blueyed]

coverage.py is https://github.com/nedbat/coveragepy/

The environment variable is COVERAGE_DEBUG (sorry), and you could try COVERAGE_DEBUG=process,config.

[bluenote10]

Debugging in this line showed that

• self.cov_report['html'] is None so it will not try to write to an explicitly given directory
• os.getcwd() returns something like /tmp/pytest-of-username/pytest-66/test_blah0 which is a standard directory using pytests tmpdir feature. This directory will get deleted immediately after running the tests, i.e., the report gets deleted as well.

So the question is: What is the nicest way to recover the working directory? Just store it initially and os.chdir back when writing the output? Or should pytest reset the working directory instead?

[blueyed]

Are you changing the directory yourself?
Normally you would use with tmpdir.as_cwd: …, which restores it automatically.

[blueyed]

I assume pytest-cov could make the path absolute to the session's initial cwd maybe, and/or write the report with changing the dir itself temporarily (to the initial dir).

[blueyed]

pytest itself has problems with this during collection already btw (pytest-dev/pytest#4317).

[bluenote10]

Are you changing the directory yourself? Normally you would use with tmpdir.as_cwd: …, which restores it automatically.

Yes it turned out there was a test which sets the working directory manually.

[ionelmc]

We could try to make sure we dump the report in the original cwd tho (just like we handle config paths).

[blueyed]

There is invocation_dir on the pytest config object that could be used.

[ionelmc]

Check the new examples from #267 - they should clarify how to correctly use pytest-cov in your project.

[tbaledent]

I also ran into the same issue recently where I couldn't chdir into pytest's tmpdir feature without breaking how pytest-cov locates the coverage report at the end of a test suite, so I figured I would leave a comment with a quick solution to help anyone else who might have the same problems in the future.
To keep the line tmpdir.chdir() in the initdir fixture without adversely affecting any downstream tests or the coverage report generation, simply add a constant at the top of your test like OLD_DIR = os.getcwd() and then add the following to your tearDownClass (assuming you're using the pytest fixtures within a unittest TestCase):

    @classmethod
    def tearDownClass(cls) -> None:
        # tear down some other stuff
        os.chdir(OLD_DIR)

This will cleanly revert the current working directory to the previous directory before tmpdir once the tests have completed so the coverage report can successfully be located after the test suite has completed.