Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test filtering is slow #998

Open
Zylatis opened this issue Apr 28, 2022 · 0 comments
Open

Test filtering is slow #998

Zylatis opened this issue Apr 28, 2022 · 0 comments
Assignees

Comments

@Zylatis
Copy link

Zylatis commented Apr 28, 2022

Describe the bug
I use tarpaulin in a bit of an abusive way (as noted here) - I want a breakdown of coverage per test and so i get cargo to give me a list of tests and call tarpaulin for each of those tests. This works fine but i recently noticed that the process of filtering out the excluded tests is very slow.

Example
I can't provide the code to generate this (though if required i can spend some time making a non-private example) but i can provide an output showing the timestamps posted below.

We do have a weird test setup where we set a bunch of stuff to ignored and then trigger them 'manually' by specifying the test file and using the ignored flag.

The command that was used to generate the data below is
cargo tarpaulin --all --skip-clean --ignore-tests -o json html --ignored --features <feature_flag> -- test_camera

Two things I don't understand: why some appear repeated and why the filtering is slow.

�[2mApr 27 19:06:38.795�[0m �[32m INFO�[0m cargo_tarpaulin::config: Creating config
�[2mApr 27 19:06:38.882�[0m �[32m INFO�[0m cargo_tarpaulin: Running Tarpaulin
�[2mApr 27 19:06:38.882�[0m �[32m INFO�[0m cargo_tarpaulin: Building project
   <COMPILING>
    Finished test [unoptimized + debuginfo] target(s) in 2m 13s
�[2mApr 27 19:08:53.014�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:08:53.014�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/netman-79ed60d51e9a80f4

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 24 filtered out; finished in 0.00s

�[2mApr 27 19:08:53.724�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:08:53.724�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/netman-79ed60d51e9a80f4

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 24 filtered out; finished in 0.00s

�[2mApr 27 19:08:54.293�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:08:54.294�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/camera-a5491540fae07128

running 1 test
test test_camera ... ignored

test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 1 filtered out; finished in 0.00s

�[2mApr 27 19:09:01.124�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:09:01.125�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/camera-a5491540fae07128

running 1 test
test test_camera ... <TEST RUN>
ok
a
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 14.60s

�[2mApr 27 19:09:22.135�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:09:22.135�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/uploader-06582b41884ef3cc

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 8 filtered out; finished in 0.00s

�[2mApr 27 19:09:28.360�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:09:28.360�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/uploader-06582b41884ef3cc

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 8 filtered out; finished in 0.00s

�[2mApr 27 19:09:34.249�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:09:34.249�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/threads-4b930ccc0434f7ad

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

�[2mApr 27 19:09:40.564�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:09:40.564�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/threads-4b930ccc0434f7ad

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

�[2mApr 27 19:09:46.655�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:09:46.655�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/PROJECT-dfbed683b4108eae

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

�[2mApr 27 19:09:52.744�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:09:52.745�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/PROJECT-dfbed683b4108eae

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

�[2mApr 27 19:09:58.704�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:09:58.704�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/serial-101dc2dd4995a0ef

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

�[2mApr 27 19:10:04.880�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:10:04.881�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/serial-101dc2dd4995a0ef

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

�[2mApr 27 19:10:10.859�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:10:10.859�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/migration-fed54f68cc83e80a

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

�[2mApr 27 19:10:16.988�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:10:16.989�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/migration-fed54f68cc83e80a

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

�[2mApr 27 19:10:22.976�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:10:22.976�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/PROJECT-44afaec8f6bab996

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 21 filtered out; finished in 0.00s

�[2mApr 27 19:10:29.032�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:10:29.033�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/PROJECT-44afaec8f6bab996

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 21 filtered out; finished in 0.00s

�[2mApr 27 19:10:35.044�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:10:35.044�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/system-f0d2d1d4b0f682aa

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

�[2mApr 27 19:10:41.060�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:10:41.061�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/system-f0d2d1d4b0f682aa

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

�[2mApr 27 19:10:47.048�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:10:47.048�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/logging-266777b2b1b3f891

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 4 filtered out; finished in 0.00s

�[2mApr 27 19:10:53.064�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:10:53.065�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/logging-266777b2b1b3f891

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 4 filtered out; finished in 0.00s

�[2mApr 27 19:10:59.052�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:10:59.052�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/wifi-bf08ebb1cfdc6145

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

�[2mApr 27 19:11:05.068�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling::linux: Launching test
�[2mApr 27 19:11:05.069�[0m �[32m INFO�[0m cargo_tarpaulin::process_handling: running /CODE_PATH/target/debug/deps/wifi-bf08ebb1cfdc6145

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

�[2mApr 27 19:11:11.118�[0m �[32m INFO�[0m cargo_tarpaulin::report: Coverage Results:
|| Tested/Total Lines:
|| <RESULTS>
|| 
25.82% coverage, 1351/5233 lines covered

Expected behavior
I would have expected, perhaps naively, that the filtering would be quick. In the above example the filtering is comparable in duration to the test itself.

Possible workaround
As noted i know which file each test lives in from getting cargo to list all the tests, is there some way i can feed this information to tarpaulin to tell it to only look in one of the test binaries for each test?

Thanks!
Z

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants