Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST: add answer testing to bleeding edge CI #3540

Conversation

neutrinoceros
Copy link
Member

PR Summary

matplotlib 3.5 is about to be released, and answer tests are clearly much better at catching integration bugs, so it seems like agood opportunity to add our answer test suite to the "bleeding edge" CI workflow. Hopefully we don't catch anything weird. Otherwise, matplotlib's maintainers are on the lookout so we still have a chance to report stuff before the actual release is out.

@neutrinoceros neutrinoceros added tests: running tests Issues with the test setup infrastructure Related to CI, versioning, websites, organizational issues, etc viz: 2D labels Oct 4, 2021
@neutrinoceros neutrinoceros force-pushed the bleeding_edge_ci_answer_tests branch 2 times, most recently from e34ecb7 to 76d8903 Compare October 4, 2021 18:27
@neutrinoceros
Copy link
Member Author

This work is now finished. I only got two non-null diffs in image comparison tests, which both look like this
Screenshot 2021-10-05 at 07 14 04

So we should be able to bump these answers and then merge this PR.
However #3520 is set to bump the very same images for different reasons, so the present works should wait for that PR to resolve first. I'll keep it as a draft in the mean time.

@neutrinoceros
Copy link
Member Author

We also have good reasons to think that we'll need to bump images using symlog colorbars after matplotlib 3.5 drops, since it should re-enable minor ticks drawing. The present PR should not be considered for merging before that.

@neutrinoceros
Copy link
Member Author

Changed my mind, I don't think this is a good idea anymore: catching subtle changes like the one above well ahead of any matplotlib release will not help much, and I fear that subtle, unimportant changes are far more common than the ones we'd want to see early. In general doing this will increase maintenance cost but the expected gain isn't clear. Closing.

@neutrinoceros neutrinoceros deleted the bleeding_edge_ci_answer_tests branch November 4, 2021 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Related to CI, versioning, websites, organizational issues, etc tests: running tests Issues with the test setup viz: 2D
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant