Can we use external tests to generate coverage report?

[YJDoc2]

I do not know much about how the code coverage data generation works, so sorry if this is a stupid question, but can we use external programs that run our binary to generate coverage data?

To elaborate, I am contributing to https://github.com/containers/youki/ , where we use cargo-llvm-cov in CI to generate coverage report. As of now, we generate coverage using unit tests, which are Rust tests (marked with #[tests] attribute).

Now to validate that the runtime itself is consistent with OCI spec, OCI has validation tests which run youki binary and test it on various conditions : https://github.com/opencontainers/runtime-tools

My question is : is it possible to generate coverage data from those tests, and use that as well to find total coverage?

Thanks for you time.

[taiki-e]

Currently, there is no easy way to get coverage data from binaries run outside of the cargo process. (When running binaries inside the cargo process, there are a few ways: #68, #1 (comment))

Having said that, I don't think it would be difficult to implement that.

Probably, adding an option to output the environment variables set by set_env function, and combining a command that does that with --no-run should work well. Something like:

cargo llvm-cov show-env > ./env # get the environment variables needed for get coverage data 
source ./env # apply environment variables

cargo build # build rust binaries

# commands using binaries in target/debug/*, including `cargo test`

cargo llvm-cov --no-run --lcov # collect coverage data and generate report

[YJDoc2]

Hey, Thanks for the reply. I will surely try it out by setting the env variables as per the set_env function. If you won't mind, can you keep this issue open for some more time? Thanks again. :)

[davidhewitt]

👍 this looks potentially useful for PyO3 too - we build dynamic libraries for a number of example Python extensions and run pytest against those. I think the same pattern suggested here could apply for measuring coverage of those.

[taiki-e]

show-env subcommand has been added in 0.1.14 (#115, thanks @davidhewitt)

I think the next step here is to actually run external tests and see if the results are reflected in the coverage report. (There is already one case that does not work, reported by #115.)

The current cargo-llvm-cov uses heuristics that are tuned to cargo run/cargo test to collect build artifacts, so it might be missed other build artifacts used in external tests.

EDIT: It is probably difficult to actually use show-env without #121.

[taiki-e]

Closing -- I believe known problems have been fixed in 0.1.15 (#121).