CartesianProductTestExtension from JUnit examples

docs/cartesian-product.adoc

@@ -0,0 +1,222 @@
+:page-title: Testing all test parameter combinations (Cartesian product)
+:page-description: Extends JUnit Jupiter with `@CartesianProductTest`, a special `@ParameterizedTest` which tests all combinations of its input
+
+From Wikipedia, the free encyclopedia:
+
+> In mathematics, specifically set theory, the Cartesian product of two sets A and B, denoted A × B, is the set of all ordered pairs (a, b) where a is in A and b is in B.
+> In terms of set-builder notation, that is `A × B = {(a,b) | a ∈ A and b ∈ B}`
+> +[...]+
+> One can similarly define the Cartesian product of n sets, also known as an n-fold Cartesian product, which can be represented by an n-dimensional array, where each element is an n-tuple.
+
+What does all this mean?
+
+The cartesian product of sets is all the possible combinations where you take a single element from each set.
+If you have two sets, `{ 1, 2 }` and `{ 3, 4 }`, their cartesian product is `{ { 1, 3 }, { 1, 4 }, { 2, 3 }, { 2, 4 } }`.
+
+Sometimes its useful to test all possible combinations of parameters set.
+Normally, this results in a lot of written test data parameters.
+For a more comfortable way you may use the `@CartesianProductTest` extension.
+The extension takes the test data parameter values and runs the test for every possible combination of them.
+
+== Basic Use
+
+`@CartesianProductTest` is used _instead_ of `@Test` or other such annotations (e.g. `@RepeatedTest`).
+
+You can supply test parameters to `@CartesianProductTest` in multiple ways.
+The annotation can have a value (see <<Supplying a value to CartesianProductTest>>), the test method can be annotated with `@CartesianValueSource` (see <<Annotating your test method with @CartesianValueSource>>), or the test class can have a static factory method (see <<Writing a static factory method for the parameters>>) providing the arguments.
+
+Just like the mathematical Cartesian product, `@CartesianProductTest` works with sets.
+Duplicate elements get removed automatically.
+If your input is `{ 1, 1, 3 }` and `{ 2, 2 }` the extension will consider their Cartesian product `{ { 1, 2 }, { 3, 2 } }`.
+Otherwise, the test would run with the same parameters multiple times.
+If you need to pass the same parameters multiple times, you might want to look into https://junit.org/junit5/docs/current/user-guide/#writing-tests-repeated-tests[repeated tests].
+
+== Supplying CartesianProductTest with a single set
+
+If all your test parameters are strings, you can supply all input parameters simultaneously by giving a string array value to `@CartesianProductTest`.
+This value is the input for all parameters.
+The test will try every combination of its elements.
+
+```java
+@CartesianProductTest({ "0", "1" })
+void threeBits(String a, String b, String c) {
+    // passing test code
+}
+```
+
+The test `threeBits` is executed exactly eight times, because all three input parameters can have the values "0" or "1".
+`@CartesianProductTest` tests for all input combinations, that's `2 × 2 × 2`, so eight tests in total.
+
+To demonstrate with a table:
+|===
+| # of test | value of `a`   | value of `b`   | value of `c`
+| 1st test  | "0"            | "0"            | "0"
+| 2nd test  | "0"            | "0"            | "1"
+| 3rd test  | "0"            | "1"            | "0"
+| 4th test  | "0"            | "1"            | "1"
+| 5th test  | "1"            | "0"            | "0"
+| 6th test  | "1"            | "0"            | "1"
+| 7th test  | "1"            | "1"            | "0"
+| 8th test  | "1"            | "1"            | "1"
+|===
+
+== Annotating your test method with @CartesianValueSource
+
+If you don't only supply string values to your test method like in the example above, you can annotate your method with `@CartesianValueSource`.
+`@CartesianValueSource` is used to define the possible inputs of a single test parameter.
+You define the parameter values in order (left-to-right/top-to-bottom) and the test will try every combination those values can have.
+
+```java
+@CartesianProductTest
+@CartesianValueSource(ints = { 1, 2, 4 })
+@CartesianValueSource(strings = { "A", "B" })
+void annotated(int number, String character) {
+    // passing test code
+}
+```
+
+This annotation might look familiar - it mimics JUnits `@ValueSource`, except `@CartesianValueSource` is repeatable.
+It also does NOT work with `@ParameterizedTest`.
+
+The test `annotated` is executed exactly six times.
+The first parameter can have any of the three values `1`, `2` or `4`.
+The second parameter can have any of the two values `A` or `B`.
+`@CartesianProductTest` tests for all input combinations, that's `3 × 2`, so six tests in total.
+
+To demonstrate with a table:
+|===
+| # of test | value of `number` | value of `character`
+| 1st test  | 1                 | "A"
+| 2nd test  | 1                 | "B"
+| 3rd test  | 2                 | "A"
+| 4th test  | 2                 | "B"
+| 5th test  | 4                 | "A"
+| 6th test  | 4                 | "B"
+|===
+
+== Writing a static factory method for the parameters
+
+If your tests require special inputs that `@CartesianValueSource` is not able to supply, you can define a static factory method to supply your test parameters.
+By default, this method must have the same name as the test method, but you can specify a different name with the `factory()` annotation parameter.
+This method must return `CartesianProductTest.Sets`.
+`CartesianProductTest.Sets` is a helper class, specifically for creating sets for `@CartesianProductTest`.
+To create the test data, instantiate a `new CartesianProductTest.Sets()` then use the `add()` method to register the values for the parameters.
+
+```java
+@CartesianProductTest
+void nFold(String string, Class<?> clazz, TimeUnit unit) {
+    // passing test code
+}
+
+static CartesianProductTest.Sets nFold() {
+    return new CartesianProductTest.Sets()
+        .add("Alpha", "Omega")
+        .add(Runnable.class, Cloneable.class, Predicate.class)
+        .add(TimeUnit.DAYS, TimeUnit.HOURS);
+}
+```
+
+The test `nFold` is executed exactly twelve times.
+The first parameter can have any of the two values `"Alpha"` or `"Omega"`.
+The second parameter can have any of the three values `Runnable.class`, `Cloneable.class` or `Predicate.class`.
+The third parameter can have any of the two values `TimeUnit.DAYS` or `TimeUnit.HOURS`.
+`@CartesianProductTest` tests for all input combinations, that's `2 × 3 × 2`, so twelve tests in total.
+
+To demonstrate with a table:
+|===
+| # of test  | value of `string` | value of `clazz` | value of `unit`
+| 1st test   | "Alpha"           | Runnable.class   | TimeUnit.DAYS
+| 2nd test   | "Alpha"           | Runnable.class   | TimeUnit.HOURS
+| 3rd test   | "Alpha"           | Cloneable.class  | TimeUnit.DAYS
+| 4th test   | "Alpha"           | Cloneable.class  | TimeUnit.HOURS
+| 5th test   | "Alpha"           | Predicate.class  | TimeUnit.DAYS
+| 6th test   | "Alpha"           | Predicate.class  | TimeUnit.HOURS
+| 7th test   | "Omega"           | Runnable.class   | TimeUnit.DAYS
+| 8th test   | "Omega"           | Runnable.class   | TimeUnit.HOURS
+| 9th test   | "Omega"           | Cloneable.class  | TimeUnit.DAYS
+| 10th test  | "Omega"           | Cloneable.class  | TimeUnit.HOURS
+| 11th test  | "Omega"           | Predicate.class  | TimeUnit.DAYS
+| 12th test  | "Omega"           | Predicate.class  | TimeUnit.HOURS
+|===
+
+Remember, you can reuse the same argument provider method, by explicitly passing its name to `@CartesianProductTest`'s `factory` attribute.
+
+```java
+
+@CartesianProductTest(factory = "provideArguments")
+void testNeedingArguments(String string, int i) {
+    // passing test code
+}
+
+@CartesianProductTest(factory = "provideArguments")
+void testNeedingSameArguments(String string, int i) {
+    // different passing test code
+}
+
+static CartesianProductTest.Sets provideArguments() {
+    return new CartesianProductTest.Sets()
+        .add("Mercury", "Earth", "Venus")
+        .add(1, 12, 144);
+}
+```
+
+=== Conditions for the static factory method
+
+There are multiple conditions the static factory method has to fulfill to qualify:
+
+- must have the same name as the test method (or its name must be specified via `factory()`)
+- must be `static`
+- must have **no** parameters
+- must return `CartesianProductTest.Sets`
+- must register values for every parameter exactly once
+- must register values in order
+
+=== Returning wrong `Sets` in the static factory method
+
+If you register too few, too many, or conflicting parameters, you will get an `org.junit.jupiter.api.extension.ParameterResolutionException`.
+Conflicting parameters meaning your test method has a `TestInfo` or `TestReporter` (or any other parameter that is automatically injected by JUnit), and you register too many sets in your factory method.
+
+Examples of badly configured tests/static factory method:
+```java
+@CartesianProductTest(factory = "resolveParameters")
+void tooFewParameters(String string, int i, boolean b) {
+    // fails because the boolean parameter is not resolved
+}
+
+@CartesianProductTest(factory = "resolveParameters")
+void tooManyParameters(String string) {
+    // fails because we try to supply a non-existent integer parameter
+}
+
+@CartesianProductTest(factory = "resolveParameters")
+void wrongOrderParameters(int i, String string) {
+    // fails because the static factory method declared parameter sets in the wrong order
+}
+
+@CartesianProductTest(factory = "resolveParameters")
+void conflictingParameters(String string, TestInfo info) {
+    // fails because both the factory method and JUnit tries to inject TestInfo
+    // OR
+    // fails because the static factory method tries to inject int instead of TestInfo
+}
+
+static CartesianProductTest.Sets resolveParameters() {
+    return new CartesianProductTest.Sets()
+        .add("A", "B", "C")
+        .add(1, 2, 3);
+}
+```
+
+== Warning: Do not `@CartesianProductTest` with `@Test`
+
+If `@CartesianProductTest` is combined with `@Test` or `TestTemplate`-based mechanisms (like `@RepeatedTest` or `@ParameterizedTest`), the test engine will execute it according to each annotation (i.e. more than once).
+This is most likely unwanted and will probably lead to the following exception/failure message:
+
+> org.junit.jupiter.api.extension.ParameterResolutionException:
+> No ParameterResolver registered for parameter [...]
+
+This is because `@Test` does not know what to do with the parameter(s) of the `@CartesianProductTest`.
+
+== Thread-Safety
+
+This extension is safe to use during https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution[parallel test execution].

docs/docs-nav.yml

@@ -20,6 +20,8 @@
         url: /docs/standard-input-output/
       - title: "Temporary Files and Directories"
         url: /docs/temp-directory/
+      - title: "Testing all test parameter combinations (Cartesian product)"
+        url: /docs/cartesian-product/
       - title: "Vintage @Test"
         url: /docs/vintage-test/
 

src/main/java/org/junitpioneer/jupiter/CartesianProductResolver.java

@@ -0,0 +1,37 @@
+/*
+ * Copyright 2015-2020 the original author or authors.
+ *
+ * All rights reserved. This program and the accompanying materials are
+ * made available under the terms of the Eclipse Public License v2.0 which
+ * accompanies this distribution and is available at
+ *
+ * http://www.eclipse.org/legal/epl-v20.html
+ */
+
+package org.junitpioneer.jupiter;
+
+import java.util.List;
+
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolver;
+
+public class CartesianProductResolver implements ParameterResolver {
+
+	private final List<?> parameters;
+
+	CartesianProductResolver(List<?> parameters) {
+		this.parameters = parameters;
+	}
+
+	@Override
+	public boolean supportsParameter(ParameterContext parameterContext, ExtensionContext extensionContext) {
+		return parameterContext.getIndex() < parameters.size();
+	}
+
+	@Override
+	public Object resolveParameter(ParameterContext parameterContext, ExtensionContext extensionContext) {
+		return parameters.get(parameterContext.getIndex());
+	}
+
+}

src/main/java/org/junitpioneer/jupiter/CartesianProductTest.java

@@ -0,0 +1,64 @@
+/*
+ * Copyright 2015-2020 the original author or authors.
+ *
+ * All rights reserved. This program and the accompanying materials are
+ * made available under the terms of the Eclipse Public License v2.0 which
+ * accompanies this distribution and is available at
+ *
+ * http://www.eclipse.org/legal/epl-v20.html
+ */
+
+package org.junitpioneer.jupiter;
+
+import static java.util.stream.Collectors.toList;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * {@code @CartesianProductTest} is a JUnit Jupiter extension that marks
+ * a test to be executed with all possible input combinations.
+ *
+ * <p>Methods annotated with this annotation should not be annotated with {@code Test}.
+ * </p>
+ */
+@TestTemplate
+@ExtendWith(CartesianProductTestExtension.class)
+@Target(ElementType.METHOD)
+@Retention(RetentionPolicy.RUNTIME)
+public @interface CartesianProductTest {
+
+	/**
+	 * Specifies {@code String} values for all inputs simultaneously.
+	 */
+	String[] value() default {};
+
+	/**
+	 * Specifies the name of the method that supplies the {@code Sets} for the test.
+	 */
+	String factory() default "";
+
+	class Sets {
+
+		private final List<List<?>> sets = new ArrayList<>(); //NOSONAR
+
+		public Sets add(Object... entries) {
+			sets.add(new ArrayList<>(Arrays.stream(entries).distinct().collect(toList())));
+			return this;
+		}
+
+		List<List<?>> getSets() { //NOSONAR
+			return sets;
+		}
+
+	}
+
+}