Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explain lack of reproducibility for @DisabledUntil #645

Merged
merged 4 commits into from May 28, 2022

Conversation

nipafx
Copy link
Member

@nipafx nipafx commented May 28, 2022


PR checklist

The following checklist shall help the PR's author, the reviewers and maintainers to ensure the quality of this project.
It is based on our contributors guidelines, especially the "writing code" section.
It shall help to check for completion of the listed points.
If a point does not apply to the given PR's changes, the corresponding entry can be simply marked as done.

Documentation (general)

  • There is documentation (Javadoc and site documentation; added or updated)
  • There is implementation information to describe why a non-obvious source code / solution got implemented
  • Site documentation has its own .adoc file in the docs folder, e.g. docs/report-entries.adoc
  • Site documentation in .adoc file references demo in src/demo/java instead of containing code blocks as text
  • Only one sentence per line (especially in .adoc files)
  • Javadoc uses formal style, while sites documentation may use informal style

Documentation (new extension)

  • The docs/docs-nav.yml navigation has an entry for the new extension
  • The package-info.java contains information about the new extension

Code

  • Code adheres to code style, naming conventions etc.
  • Successful tests cover all changes
  • There are checks which validate correct / false usage / configuration of a functionality and there are tests to verify those checks
  • Tests use AssertJ or our own PioneerAssert (which are based on AssertJ)

Contributing

  • A prepared commit message exists
  • The list of contributions inside README.md mentions the new contribution (real name optional)

@nipafx nipafx linked an issue May 28, 2022 that may be closed by this pull request
@nipafx nipafx force-pushed the issue/550-reproducibility-disabled-until branch from 7988be0 to 60dcb87 Compare May 28, 2022 05:38
@nipafx
Copy link
Member Author

nipafx commented May 28, 2022

Proposed commit message:

Explain lack of reproducibility for `@DisabledUntil` (#550 / #645)

A test suite that only passes because a failing test is disabled
until a certain date, will fail if executed after that date. That
means the originally successful build can't be reproduced.

We're looking for a proper solution for this (see #478), but until
then, the best we can do is warn users about this in documentation
and with a report entry.

References: #478, #550
PR: 645

@nipafx nipafx added the full-build Triggers full build suite on PR label May 28, 2022
@nipafx nipafx merged commit ef7ff7c into main May 28, 2022
@nipafx nipafx deleted the issue/550-reproducibility-disabled-until branch May 28, 2022 19:15
Bukama pushed a commit to Bukama/junit-pioneer that referenced this pull request Sep 20, 2022
… / junit-pioneer#645)

A test suite that only passes because a failing test is disabled
until a certain date, will fail if executed after that date. That
means the originally successful build can't be reproduced.

We're looking for a proper solution for this (see junit-pioneer#478), but until
then, the best we can do is warn users about this in documentation
and with a report entry.

References: junit-pioneer#478, junit-pioneer#550
PR: 645
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
full-build Triggers full build suite on PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Extensions / Annotations to remind user about unfinished work
2 participants