Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify quality of test coverage #131

Open
tomato42 opened this issue Oct 7, 2019 · 4 comments
Open

Verify quality of test coverage #131

tomato42 opened this issue Oct 7, 2019 · 4 comments
Labels
help wanted maintenance issues related to making the project usable or testable

Comments

@tomato42
Copy link
Member

tomato42 commented Oct 7, 2019

Use mutation testing to verify if the existing test coverage is good enough.

Options:

@tomato42 tomato42 added help wanted maintenance issues related to making the project usable or testable labels Oct 7, 2019
@tomato42 tomato42 added this to the v0.14 milestone Oct 7, 2019
@tomato42 tomato42 mentioned this issue Nov 2, 2019
5 tasks
@tomato42
Copy link
Member Author

tomato42 commented Nov 3, 2019

given that we are able to verify something like 20 mutants in 25 minutes in Travis, we would be able to mutation test only the simplest of simple patches – unusable without #127, maybe usable with it

either way, not for 0.14

side note: now, just before 0.14 is released (d6cb288), the test suite has 23.93% survival rate (76.07% mutation score)

@tomato42 tomato42 modified the milestones: v0.14, someday/future Nov 3, 2019
@tomato42
Copy link
Member Author

tomato42 commented Nov 12, 2020

with 0.16 we improved a bit, the survival rate is 21.33%

@tomato42
Copy link
Member Author

tomato42 commented Jul 9, 2022

with 0.18.0 we improved a lot, the survival rate is now 16.86%, though I executed the unchanged test suite (not the one with changes from #156) and cosmic-ray changed significantly too

@tomato42
Copy link
Member Author

And before 0.19 release the survival rate is 21.51% (mutation score 78.49%), but we're able to execute about 33% of all mutations, so the confidence interval for survival rate is quite narrow: 20.80 to 22.22%.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted maintenance issues related to making the project usable or testable
Projects
None yet
Development

No branches or pull requests

1 participant