CartesianProductTestExtension from JUnit examples

docs/cartesian-product.adoc

@@ -0,0 +1,150 @@
+:page-title: Testing all test parameter combinations (Cartesian product)
+:page-description: Extends JUnit Jupiter with `@CartesianProductTest`, a special `@ParameterizedTest` which tests all combinations of its input
+
+Sometimes you want to test not only for specific inputs but also combinations of those inputs.
+`@CartesianProductTest` combines all test inputs and runs a test for each combination.
+
+[#_basic_use]
+== Basic Use
+
+`@CartesianProductTest` is used _instead_ of `@Test` or other such annotations (e.g. `@RepeatedTest`).
+The annotation can have a value (see <<Supplying a value to CartesianProductTest>>), or the test class can have a static factory method (see <<Writing a static factory method for the parameters>>) providing the arguments.
+
+== Supplying a value to CartesianProductTest
+
+If all your test parameters are strings, you can supply all input parameters simultaneously by giving a value to `@CartesianProductTest`.
+This value is the input for all parameters, and the test will try every combination it can have.
+
+```java
+@CartesianProductTest({"0", "1"})
+void threeBits(String a, String b, String c) {
+    int value = Integer.parseUnsignedInt(a + b + c, 2);
+    Assertions.assertThat(value).isBetween(0b000, 0b111);
+}
+```
+
+The test `threeBits` is executed exactly eight times, because all three input parameters can have the value "0" or "1".
+`@CartesianProductTest` tests for all input combinations; that's `2 × 2 × 2`, so eight tests in total.
+
+To demonstrate with a table:
+|===
+| # of test | value of `a` | value of `b` | value of `c`
+| 1st test  | 0            | 0            | 0
+| 2nd test  | 0            | 0            | 1
+| 3rd test  | 0            | 1            | 0
+| 4th test  | 0            | 1            | 1
+| 5th test  | 1            | 0            | 0
+| 6th test  | 1            | 0            | 1
+| 7th test  | 1            | 1            | 0
+| 8th test  | 1            | 1            | 1
+|===
+
+== Writing a static factory method for the parameters
+
+`@CartesianProductTest` can have a static factory method supplying the test parameters.
+By default, this method must have the same name as the test method, but you can specify a different name with the `factory()` annotation parameter.
+This method must return `CartesianProductTest.Sets`.
+`CartesianProductTest.Sets` is a helper class, specifically for creating sets for `@CartesianProductTest`.
+The way to use it: instantiate it (`new CartesianProductTest.Sets()`), then use the `add()` method to register values for the parameters (see <<_static-factory-example, the example>>).
+
+[#_static-factory-example]
+```java
+@CartesianProductTest
+void nFold(String string, Class<?> clazz, TimeUnit unit) {
+    // passing test code
+}
+
+static CartesianProductTest.Sets nFold() {
+    return new CartesianProductTest.Sets()
+        .add("Alpha", "Omega")
+        .add(Runnable.class, Cloneable.class, Predicate.class)
+        .add(TimeUnit.DAYS, TimeUnit.HOURS);
+}
+
+@CartesianProductTest(factory = "provideArguments")
+void aTestMethodThatNeedsArguments(String string, int i) {
+    // passing test code
+}
+
+static CartesianProductTest.Sets provideArguments() {
+    return new CartesianProductTest.Sets()
+        .add("Mercury", "Earth", "Venus")
+        .add(1, 12, 144);
+}
+```
+
+The test `nFold` is executed exactly twelve times.
+The first parameter can have any of the two values `"Alpha"` or `"Omega"`.
+The second parameter can have any of the three values `Runnable.class`, `Cloneable.class` or `Predicate.class`.
+The third parameter can have any of the two values `TimeUnit.DAYS` or `TimeUnit.HOURS`.
+`@CartesianProductTest` tests for all input combinations; that's `2 × 3 × 2`, so twelve tests in total.
+
+To demonstrate with a table:
+|===
+| # of test  | value of `string` | value of `clazz` | value of `unit`
+| 1st test   | "Alpha"           | Runnable.class   | TimeUnit.DAYS
+| 2nd test   | "Alpha"           | Runnable.class   | TimeUnit.HOURS
+| 3rd test   | "Alpha"           | Cloneable.class  | TimeUnit.DAYS
+| 4th test   | "Alpha"           | Cloneable.class  | TimeUnit.HOURS
+| 5th test   | "Alpha"           | Predicate.class  | TimeUnit.DAYS
+| 6th test   | "Alpha"           | Predicate.class  | TimeUnit.HOURS
+| 7th test   | "Omega"           | Runnable.class   | TimeUnit.DAYS
+| 8th test   | "Omega"           | Runnable.class   | TimeUnit.HOURS
+| 9th test   | "Omega"           | Cloneable.class  | TimeUnit.DAYS
+| 10th test  | "Omega"           | Cloneable.class  | TimeUnit.HOURS
+| 11th test  | "Omega"           | Predicate.class  | TimeUnit.DAYS
+| 12th test  | "Omega"           | Predicate.class  | TimeUnit.HOURS
+|===
+
+=== Conditions for the static factory method
+
+There are a couple conditions the static factory method has to fulfill to qualify:
+
+- must have the same name as the test method or its name must be specified via `factory()`
+- must be `static`
+- must have **no** parameters
+- must return `CartesianProductTest.Sets`
+- must register values for every parameter exactly once
+- must register values in order
+
+=== Returning wrong `Sets` in the static factory method
+
+If you register too few, too many, or conflicting parameters, you will get an `org.junit.jupiter.api.extension.ParameterResolutionException`.
+For example, if your test method has a `TestInfo` or `TestReporter` (or any other parameter that is automatically injected by JUnit), and you register too many sets in your factory method:
+
+```java
+@CartesianProductTest(factory = "resolveParameters")
+void tooFewParameters(String string, int i, boolean b) {
+    // fails because the boolean parameter is not resolved
+}
+
+@CartesianProductTest(factory = "resolveParameters")
+void tooManyParameters(String string) {
+    // fails because we try to supply a non-existent integer parameter
+}
+
+@CartesianProductTest(factory = "resolveParameters")
+void conflictingParameters(String string, TestInfo info) {
+    // fails because both the factory method and JUnit tries to inject TestInfo
+}
+
+static CartesianProductTest.Sets resolveParameters() {
+    return new CartesianProductTest.Sets()
+        .add("A", "B", "C")
+        .add(1, 2, 3);
+}
+```
+
+== Combining `@CartesianProductTest` with `@Test`
+
+If `@CartesianProductTest` is combined with `@Test` or `TestTemplate`-based mechanisms (like `@RepeatedTest` or `@ParameterizedTest`), the test engine will execute it according to each annotation (i.e. more than once).
+This is most likely unwanted and will probably lead to the following exception/failure message:
+
+> org.junit.jupiter.api.extension.ParameterResolutionException:
+> No ParameterResolver registered for parameter [...]
+
+This is because `@Test` does not know what to do with the parameter(s) of the `@CartesianProductTest`.
+
+== Thread-Safety
+
+This extension is safe to use during https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution[parallel test execution].

docs/docs-nav.yml

@@ -20,6 +20,8 @@
         url: /docs/standard-input-output/
       - title: "Temporary Files and Directories"
         url: /docs/temp-directory/
+      - title: "Testing all test parameter combinations (Cartesian product)"
+        url: /docs/cartesian-product/
       - title: "Vintage @Test"
         url: /docs/vintage-test/
 

src/main/java/org/junitpioneer/jupiter/CartesianProductResolver.java

@@ -0,0 +1,37 @@
+/*
+ * Copyright 2015-2020 the original author or authors.
+ *
+ * All rights reserved. This program and the accompanying materials are
+ * made available under the terms of the Eclipse Public License v2.0 which
+ * accompanies this distribution and is available at
+ *
+ * http://www.eclipse.org/legal/epl-v20.html
+ */
+
+package org.junitpioneer.jupiter;
+
+import java.util.List;
+
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterContext;
+import org.junit.jupiter.api.extension.ParameterResolver;
+
+public class CartesianProductResolver implements ParameterResolver {
+
+	private final List<?> parameters;
+
+	CartesianProductResolver(List<?> parameters) {
+		this.parameters = parameters;
+	}
+
+	@Override
+	public boolean supportsParameter(ParameterContext parameterContext, ExtensionContext extensionContext) {
+		return parameterContext.getIndex() < parameters.size();
+	}
+
+	@Override
+	public Object resolveParameter(ParameterContext parameterContext, ExtensionContext extensionContext) {
+		return parameters.get(parameterContext.getIndex());
+	}
+
+}

src/main/java/org/junitpioneer/jupiter/CartesianProductTest.java

@@ -0,0 +1,71 @@
+/*
+ * Copyright 2015-2020 the original author or authors.
+ *
+ * All rights reserved. This program and the accompanying materials are
+ * made available under the terms of the Eclipse Public License v2.0 which
+ * accompanies this distribution and is available at
+ *
+ * http://www.eclipse.org/legal/epl-v20.html
+ */
+
+package org.junitpioneer.jupiter;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * {@code @CartesianProductTest} is a JUnit Jupiter extension that marks
+ * a test to be executed with all possible input combinations.
+ *
+ * <p>Methods annotated with this annotation <b>MUST NOT</b> be annotated with {@code Test},
+ * because it will throw an exception.
+ * </p>
+ *
+ * <p>Methods annotated with this annotation are different from {@code ParameterizedTest}s because
+ * they can not have {@code ArgumentsSource}s - those are completely disregarded. Instead a
+ * String array must be provided (for methods with only String parameters) or a static factory method
+ * with the same name as the test method must exist.
+ * </p>
+ *
+ * @since ???
+ */
+@TestTemplate
+@ExtendWith(CartesianProductTestExtension.class)
+@Target(ElementType.METHOD)
+@Retention(RetentionPolicy.RUNTIME)
+public @interface CartesianProductTest {
+
+	/**
+	 * Specifies {@code String} values for all inputs simultaneously.
+	 */
+	String[] value() default {};
+
+	/**
+	 * Specifies the name of the method that supplies the {@code Sets} for the test.
+	 */
+	String factory() default "";
+
+	class Sets {
+
+		private final List<List<?>> sets = new ArrayList<>(); //NOSONAR
+
+		public Sets add(Object... entries) {
+			sets.add(new ArrayList<>(Arrays.asList(entries)));
+			return this;
+		}
+
+		List<List<?>> getSets() { //NOSONAR
+			return sets;
+		}
+
+	}
+
+}

src/main/java/org/junitpioneer/jupiter/CartesianProductTestExtension.java

@@ -0,0 +1,100 @@
+/*
+ * Copyright 2015-2020 the original author or authors.
+ *
+ * All rights reserved. This program and the accompanying materials are
+ * made available under the terms of the Eclipse Public License v2.0 which
+ * accompanies this distribution and is available at
+ *
+ * http://www.eclipse.org/legal/epl-v20.html
+ */
+
+package org.junitpioneer.jupiter;
+
+import static org.junit.platform.commons.support.AnnotationSupport.findAnnotation;
+import static org.junit.platform.commons.support.ReflectionSupport.invokeMethod;
+
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.ParameterResolutionException;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
+import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
+
+class CartesianProductTestExtension implements TestTemplateInvocationContextProvider {
+
+	@Override
+	public boolean supportsTestTemplate(ExtensionContext context) {
+		return findAnnotation(context.getTestMethod(), CartesianProductTest.class).isPresent();
+	}
+
+	@Override
+	public Stream<TestTemplateInvocationContext> provideTestTemplateInvocationContexts(ExtensionContext context) {
+		List<List<?>> sets = computeSets(context.getRequiredTestMethod());
+		return cartesianProduct(sets).stream().map(CartesianProductTestInvocationContext::new);
+	}
+
+	private List<List<?>> computeSets(Method testMethod) {
+		CartesianProductTest annotation = findAnnotation(testMethod, CartesianProductTest.class)
+				.orElseThrow(() -> new AssertionError("@CartesianProductTest not found"));
+		// Compute A ⨯ A ⨯ ... ⨯ A from single source "set"
+		if (annotation.value().length > 0) {
+			List<String> strings = Arrays.asList(annotation.value());
+			List<List<?>> sets = new ArrayList<>();
+			for (int i = 0; i < testMethod.getParameterTypes().length; i++) {
+				sets.add(strings);
+			}
+			return sets;
+		}
+		// No single entry supplied? Try the sets factory method instead...
+		String factoryMethod = annotation.factory().isEmpty() ? testMethod.getName() : annotation.factory();
+
+		return invokeSetsFactory(testMethod, factoryMethod).getSets();
+	}
+
+	private CartesianProductTest.Sets invokeSetsFactory(Method testMethod, String factoryMethodName) {
+		Class<?> declaringClass = testMethod.getDeclaringClass();
+		Method factory = PioneerUtils
+				.findMethodCurrentOrEnclosing(declaringClass, factoryMethodName)
+				.orElseThrow(() -> new AssertionError("Method `CartesianProductTest.Sets " + factoryMethodName
+						+ "()` not found in " + declaringClass + "or any enclosing class"));
+		String method = "Method `" + factory + "`";
+		if (!Modifier.isStatic(factory.getModifiers())) {
+			throw new AssertionError(method + " must be static");
+		}
+		if (!CartesianProductTest.Sets.class.isAssignableFrom(factory.getReturnType())) {
+			throw new AssertionError(method + " must return `CartesianProductTest.Sets`");
+		}
+		CartesianProductTest.Sets sets = (CartesianProductTest.Sets) invokeMethod(factory, null);
+		if (sets.getSets().size() > testMethod.getParameterCount()) {
+			throw new ParameterResolutionException(method + " must register values for each parameter exactly once");
+		}
+		return sets;
+	}
+
+	private static List<List<?>> cartesianProduct(List<List<?>> lists) {
+		List<List<?>> resultLists = new ArrayList<>();
+		if (lists.isEmpty()) {
+			resultLists.add(Collections.emptyList());
+			return resultLists;
+		}
+		List<?> firstList = lists.get(0);
+		// Note the recursion here
+		List<List<?>> remainingLists = cartesianProduct(lists.subList(1, lists.size()));
+		for (Object item : firstList) {
+			for (List<?> remainingList : remainingLists) {
+				ArrayList<Object> resultList = new ArrayList<>();
+				resultList.add(item);
+				resultList.addAll(remainingList);
+				resultLists.add(resultList);
+			}
+		}
+		return resultLists;
+	}
+
+}