Explain lack of reproducibility for @DisabledUntil (#550 / #645)

A test suite that only passes because a failing test is disabled
until a certain date, will fail if executed after that date. That
means the originally successful build can't be reproduced.

We're looking for a proper solution for this (see #478), but until
then, the best we can do is warn users about this in documentation
and with a report entry.

References: #478, #550
PR: 645

docs/disabled-until.adoc

@@ -13,6 +13,10 @@ The only problem with this approach is remembering to activate the test again on
 The `@DisabledUntil` annotation may be used to disable a test only for a certain period of time.
 The test will be automatically executed again once the date supplied by the `date` parameter is reached.
 
+WARNING: Applying `@DisabledUntil` can make the test suite non-reproducible.
+If a failing test is disabled during a build that then passes, rerunning that build after the "until" date would fail.
+A report entry is issued for every test that is disabled until a certain date.
+
 == Usage
 
 To mark a test to be temporarily disabled add the `@DisabledUntil` annotation like so:

src/main/java/org/junitpioneer/jupiter/DisabledUntil.java

@@ -28,6 +28,10 @@
  * <p>{@code @DisabledUntil} can be used on the method and class level. It can only be used once per method or class,
  * but is inherited from higher-level containers.</p>
  *
+ * <p><strong>WARNING:</strong> Applying {@code @DisabledUntil} can make the test suite non-reproducible. If a failing
+ * test is disabled during a build that then passes, rerunning that build after the "until" date would fail. A report
+ * entry is issued for every test that is disabled until a certain date.</p>
+ *
  * @since 1.6.0
  * @see org.junit.jupiter.api.Disabled
  */

src/main/java/org/junitpioneer/jupiter/DisabledUntilExtension.java

@@ -62,6 +62,10 @@ private ConditionEvaluationResult evaluateUntilDate(ExtensionContext context, Lo
 		boolean disabled = today.isBefore(untilDate);
 
 		if (disabled) {
+			String reportEntry = format(
+				"This test is disabled until %s. If executing it on this commit would fail, the build can't be reproduced after that date.",
+				untilDate.format(ISO_8601));
+			context.publishReportEntry("DisabledUntil", reportEntry);
 			String message = format("The `date` %s is after the current date %s", untilDate.format(ISO_8601),
 				today.format(ISO_8601));
 			return disabled(message);

src/test/java/org/junitpioneer/jupiter/DisabledUntilExtensionTests.java

@@ -67,7 +67,7 @@ void shouldDisableTestWithUntilDateInTheFuture() {
 				.executeTestMethod(DisabledUntilTestCases.class, "testIsAnnotatedWithDateInTheFuture");
 		assertThat(results).hasNumberOfStartedTests(0);
 		assertThat(results).hasSingleSkippedTest();
-		assertThat(results).hasNoReportEntries();
+		assertThat(results).hasSingleReportEntry().firstValue().contains("2199-01-01", "reproduce");
 	}
 
 	@Test
@@ -78,7 +78,7 @@ void shouldDisableNestedTestWithUntilDateInTheFutureWhenMetaAnnotated() {
 		assertThat(results).hasSingleSkippedContainer(); // NestedDummyTestClass is skipped as container
 		assertThat(results).hasNumberOfStartedTests(0);
 		assertThat(results).hasNumberOfSkippedTests(0);
-		assertThat(results).hasNoReportEntries();
+		assertThat(results).hasSingleReportEntry().firstValue().contains("2199-01-01", "reproduce");
 	}
 
 	static class DisabledUntilTestCases {