Introduce MockitoExtension for JUnit Jupiter (a.k.a. JUnit 5)

[sbrannen]

New Features in JUnit 5

The JUnit Jupiter extension model in JUnit 5 introduces support for constructor and method parameter resolution (i.e., dependency injection).

Specifically, third parties can implement the ParameterResolver extension API to inject dependencies, mocks, etc. into constructors and methods. In addition, the TestInstancePostProcessor extension API can be implemented to post-process a test instance (e.g., to perform field injection).

Status Quo

Mockito supports field injection for mocks via the @Mock annotation. In addition, #438 allows @Mock to be declared on parameters for constructors and methods which makes @Mock support an ideal candidate for both the TestInstancePostProcessor and ParameterResolver extension APIs in JUnit Jupiter. In fact, the JUnit Team has already developed a proof of concept: see the MockitoExtension in the junit5-mockito-extension sample project.

Deliverables

• Introduce an official MockitoExtension for JUnit Jupiter to replace the proof of concept from the JUnit team.

[TimvdLippe]

I think this issue can be merged with #390. Our plan there was to introduce JUnit 5 compatibility in Mockito 3.0 (since 2.0 is on the verge on being released in a couple of weeks).

[sbrannen]

Sorry: I overlooked #390.

Feel free to merge it as you see fit.

thanks!

[TimvdLippe]

Let's track the JUnitExtension in this issue and list the specifications of the MockitoExtension for JUnit 5. The first point is to improve Parameterized compatibility.

[bric3]

@sbrannen Any news on this ?

[sbrannen]

@bric3, the MockitoExtension from the JUnit Team is usable with Mockito 2.0 snapshots and current versions of JUnit Jupiter (JUnit 5).

So, it's really up to the Mockito Team to decide when they want to take it over.

[ChristianSchwarz]

@sbrannen this one -> example/mockito/MockitoExtension ?

[sbrannen]

Yep, that's the one.

[bric3]

Thanks @sbrannen.
I was wondering if any API change would happen on JUnit 5. We will take over in mockito 3 once we are ready to switch branches for mockito 2

[sbrannen]

I was wondering if any API change would happen on JUnit 5.

Well... that's the million dollar question -- isn't it? 😉

All kidding aside, we are currently working on JUnit Jupiter 5.0 M3, but we still have quite a way to go before a GA release. So, although I cannot promise that the Extension APIs won't change any before GA, it appears that the APIs currently implemented by the MockitoExtension should remain rather stable.

[bric3]

OK that's good to know. Anyway we have to release 2.1 before starting 3.0 anyway. And a some API design work on 2.1 too, that may delay JUnit5 integration in the 3.0 beta phase.

[TimvdLippe]

Shall we start with fleshing out the specifications of the extension? Would like to have a working prototype when JUnit 5 is released, ETA Q1 2017.

[bric3]

Yes go for it, I'll create a submodule junit 5 at this time since JUnit 5 is a near complete rewrite, with a lot of binary incompatibilities.

[mockitoguy]

+1

Do we want to completely decouple mockito-core module from JUnit?

[TimvdLippe]

@szczepiq I am not sure, but supporting both JUnit4 and JUnit5 in mockito-core seems problematic. We have seen similar issues with Spring having to support both Mockito 1 and 2. Therefore I think it is better to refactor Mockito 3 such that we publish 2 new artifact: mockito-junit4 and mockito-junit5 which provide the integration layer between Mockito and Junit.

@sbrannen Are you aware of other framework users depending on JUnit which employ this approach or is there a different and better solution?

[sbrannen]

In the spring-test module for the Spring Framework, we provide support for TestNG, JUnit 4, and JUnit Jupiter (what you call JUnit 5) alongside each other, just in different packages.

That's all within a single JAR, and there are no issues since each of those is an optional dependency (in terms of the Maven POM).

Thus, projects that consume spring-test can pick which testing framework they wish to use (or potentially use TestNG, JUnit 4, and JUnit Jupiter all simultaneously, however unlikely that may be).

As long as it's clear what developers need to consume for a given testing framework, it shouldn't be a problem having a single artifact. For example, with Spring, JUnit 4 users use the SpringRunner; whereas, JUnit Jupiter users use the SpringExtension.

[sbrannen]

If you do opt for separate modules (i.e., Maven artifacts), I would recommend against mockito-junit5, since what you are really providing is an extension for JUnit Jupiter -- which is the new programming model in JUnit 5.

The problem with naming such an artifact mockito-junit5 is that there may well be a JUnit 6 or 7 still based on the JUnit Jupiter programming and extension models, and then the artifact name containing a 5 would no longer make sense.

FYI: I have just changed the title of this issue to reflect this fact.

[TimvdLippe]

Okay that seems reasonable, so let's take the package approach! My experience with jar configurations is limited, so in terms of the configuration of users' pom.xml I do not 100% understand how that is going to work. But I think we can figure that one out. Thanks for the explanation @sbrannen !

[mockitoguy]

@sbrannen thank you for describing and suggesting an approach. I like the idea of separation at the Java package level in the same jar. We can go for separate jars only when we have to (e.g. when JUnit versions clash). Do you have integration tests that demonstrate correct behavior with different test frameworks / different versions of test frameworks?

[sbrannen]

Do you have integration tests that demonstrate correct behavior with different test frameworks / different versions of test frameworks?

Is that a rhetorical question? 😉

Yes, of course we have tests for the Spring Framework. I'd be ashamed to let spring-test ship without an automated test suite.

Regarding the package structure, if you look here you'll see the following:

• org.springframework.test.context.junit4
• org.springframework.test.context.junit.jupiter
• org.springframework.test.context.testng

So that's how we split up the functionality within the spring-test JAR.

We naturally have unit and integration tests for our JUnit 4, JUnit Jupiter, and TestNG support which you can find in various subpackages here.

Within the IDE, we execute TestNG tests with the TestNG plugin (e.g., for Eclipse); we execute JUnit 4 tests with the IDE's built-in support; and we execute JUnit Jupiter tests with the JUnitPlatform runner (at least within Eclipse for the time being). For the latter, see SpringJUnitJupiterTestSuite.

Within the Gradle build, we execute TestNG, JUnit 4, and JUnit Jupiter tests via Gradle's standard test task. In other words, in order to keep Spring's build as simple as possible, we don't yet use the JUnit Platform Gradle Plugin to execute JUnit Jupiter tests natively. Instead, we let the Gradle test task pick up the SpringJUnitJupiterTestSuite as a JUnit 4 test (which in turn executes all of our JUnit Jupiter tests as a suite).

I'm not sure if I've now said too much or too little. So if you still have questions, just ask!

Regards,

Sam

[TimvdLippe]

This week I had an interesting discussion with 2 of my student colleagues who are currently analyzing and documenting the architecture of JUnit. In this discussion we talked about the architecture in general as well as the extensions and in particular the MockitoExtension.

In this discussion I expressed my concerns regarding the new approach with injecting Mocks based on parameters. My colleagues told me that JUnit5 focuses more on testing with interfaces, in which the tests are also interfaces. Since interfaces do not have fields, field injection of mocks is not possible. Therefore the JUnit team proposes/chose to inject mocks via parameters.

Maintaining mocks is one of the most precarious tasks when working on a test suite and we at Mockito regularly receive feedback where Mocks are misbehaving. This is the exact reason that we are shipping features such as Strict Stubbing which reduce the amount of confusion and mistakes developers can make. By limiting the developer in making mistakes, we can prevent tests to incorrectly pass while in fact they are relying on misconfigured mocks.

As a result, our general goal for Mockito 3 is to leverage the available tools even more to prevent developers from making mistakes. Most notably I am personally advocating for increasing type-safety of our various features such that developers can rely on the compiler to warn them. Since running tests is a costly task and sometimes takes over an hour before CI finished all tests, the earlier developers can receive feedback regarding misconfiguration, the better.

Having said that, let's go back to the parameter injection currently available with JUnit5. While the status quo is field injection, a compiler can warn developers when they have a typo in the usage of a Mock. E.g. when a user wants to use mock1, while the field is actually named mock2, the compiler errors to notify the developer he/she made a mistake. Parameter injection does not have this behavior, as it is relying merely on the parameter name (or the name supplied to @Mock) which is plain String comparison.

While we are striving to prevent developers of making mistakes by leveraging the compiler, adopting parameter injection feels like a step back to me.

For this reason I am not in favor of going for parameter injection, but given the need for writing tests as interfaces I am not sure what our options are.

Of course we have not started writing an extension, so I have not tried this out. But I do think we should resolve this issue before coming up with a good story for Mockito 3 + JUnit5.

CC @LiamClark @Tarovk

[marcphilipp]

How is parameter injection different from field injection?

Edit: To rephrase my question a bit: How is field injection safer than parameter injection? Is there some Mockito feature I'm not aware of?

[TWiStErRob]

@marcphilipp I think he was talking about this error case:

public class Test {
	@Mock Dependency mock1;
	@Before public void setUp() {
		when(mock1.foo()).thenBar();
	}
	@Test public void test() {
		// mock2 is a compile error
		new SUT(mock2).target();
	}
}

class Test {
	@BeforeEach void setUp(@Mock Dependency mock1) {
		when(mock1.foo()).thenBar();
	}
	@Test void test(@Mock Dependency mock2) { // or @Mock(name="mock2")
		// mock2 is a newly created a mock without .foo() stubbing
		new SUT(mock2).target();
	}
}

The extension can't know if you intentionally want a new mock in your test method or you made a typo, while javac knows that you're referencing a non-existent field.

Re edit: The current extension impl caches the mocks: https://github.com/junit-team/junit5-samples/blob/026a9d9abe06b6173398c1a2518793259cd190f2/junit5-mockito-extension/src/main/java/com/example/mockito/MockitoExtension.java#L57

[TimvdLippe]

Hm, looking at it again, it seems that the breaking change is actually in an internal API, namely TestFinishedEvent, which is created in MockitoSession. As such, I think we should be okay actually. I will do a proper investigation this weekend (hopefully, else next week) and update #1221. I hope you are okay with that @ChristianSchwarz ? Once that is done, I can more definitively say the impact on our API and whether we are risking a breaking change. I have good hopes now we might actually dodge that.

[marcphilipp]

TestFinishedEvent.getTestClassInstance() and TestFinishedEvent.getTestMethodName() are currently only used (twice) like this:

String testName = event.getTestClassInstance().getClass().getSimpleName() 
                  + "." + event.getTestMethodName();

Currently, DefaultMockitoSession always uses null for TestFinishedEvent.getTestMethodName() which strikes me as odd. I think it would be better if TestFinishedEvent only had two methods: getFailure() and getTestName(). Then, MockitoSessionBuilder could get a testName(String) builder method and DefaultMockitoSessionBuilder could pass it to DefaultMockitoSession and so on.

Alternatively, MockitoSessionBuilder could get a testMethodName() builder method and pass that on.

Moreover, I think MockitoSessionBuilder should allow to configure a MockitoLogger. This way, frameworks like JUnit could pass in a custom implementation. For JUnit Jupiter, a MockitoLogger that publishes report entries instead of writing to stdout (which causes problems will parallel execution) comes to mind.

Thoughts?

[ChristianSchwarz]

@marcphilipp
@TimvdLippe
Good idea to continue the discussion about #1232 in that ticket, i copied your the relevant comments over there.

[paulmiddelkoop]

it would be nice if the MockitoExtension works together with @TestFactory based methods. When I use the extension created by the JUnit team, I need to manually reset mocks within the dynamic test method.

@sbrannen you guys already thought about how to integrate that?

[marcphilipp]

@paulmiddelkoop There's no lifecycle support for dynamic tests and we currently have no plans to add any, i.e. there's no hook to reset mocks after a dynamic test. You could use MockitoSession in your @TestFactory method directly. 🤔

[sbrannen]

I agree with @marcphilipp: due to the lack of lifecycle support for dynamic tests (see the Dynamic Test Lifecycle note in the User Guide), you'll have to reset mocks manually.

Note, however, that you'll likely want to reset them within each Executable (i.e., lambda expression or method reference) returned by your @TestFactory method.

[paulmiddelkoop]

Thanks @marcphilipp and @sbrannen for the explanation. Any reason why there is no lifecycle support for dynamic tests?

For now I ended up with a utility function for this:

    fun dynamicMockitoTest(testInstance: Any, displayName: String, executable: () -> Unit): DynamicTest {
        val mockito = Mockito.mockitoSession()
            .initMocks(testInstance)
            .strictness(Strictness.STRICT_STUBS)
            .startMocking()
        try {
            return dynamicTest(displayName, executable)
        } finally {
            mockito.finishMocking()
        }
    }

However this will not work if I also have regular @Test methods and implement the MockitoExtension to also use the MockitoSession instead of MockitoAnnotations.initMocks(testInstance). It will result in a "Previous MockitoSession was not concluded with 'finishMocking()" error which is logical. I could put the ExtendWith(MockitoExtension::class) on every @Test method instead of the class as a solution, I guess.

I think a simpler integration is needed for a good adoption of dynamic tests. It's a regular use case to use mocking inside dynamic tests, right?

[sbrannen]

Any reason why there is no lifecycle support for dynamic tests?

Dynamic tests are intended for simple use cases that do not need lifecycle callback support. If one needs lifecycle callback support, it is then recommended to use parameterized tests.

However this will not work if I also have regular @Test methods...

Actually, that won't work at all, not even for dynamic tests.

Your current code example initializes and resets mocks around the construction of the DynamicTest instance; whereas, it must do that at execution time (i.e., when the Executable is actually invoked by JUnit Jupiter).

You would instead need to create your own wrapper around the invocation of your executable, and that wrapper would need to implement the try-finally logic around the execution of the actual executable.

I think a simpler integration is needed for a good adoption of dynamic tests. It's a regular use case to use mocking inside dynamic tests, right?

We don't know. The dynamic test support is an experimental feature. So that remains to be seen. 😉

[TWiStErRob]

@paulmiddelkoop if you create an extension method you don't need to pass in this@FooTest every time:

fun Any.dynamicMockitoTest(displayName: String, executable: () -> Unit): DynamicTest =
	dynamicTest(displayName) {
		val mockito = Mockito.mockitoSession()
				.initMocks(this@dynamicMockitoTest)
				.strictness(Strictness.STRICT_STUBS)
				.startMocking()
		try {
			executable()
		} finally {
			mockito.finishMocking()
		}
	}

and the usages of dynamicTest and dynamicMockitoTest should look the same. (This version also includes what @sbrannen said to delay the try-finally execution.)

So that remains to be seen.

Well, now you have two votes for mocking in dynamic tests.

[sbrannen]

@paulmiddelkoop & @TWiStErRob, if you want support for lifecycle callbacks for dynamic tests in JUnit Jupiter, the best place to make your wishes heard is junit-team/junit5#378.

[davidkarlsen]

What is the ETA for Mockito 3.x? I'd love to see this feature.

[TimvdLippe]

The intent now is to ship junit5 support in a separate artifact with mockito 2

[davidkarlsen]

@TimvdLippe That is great news - maybe the milestone should be updated to reflect this?
For any others waiting for this https://github.com/JeffreyFalgout/junit5-extensions/tree/master/mockito-extension might be of help in the meantime.

[smoyer64]

A little house-keeping:

The change in #438 was made to allow Mocks to be generated for the parameters to test and support methods.

[TimvdLippe]

@smoyer64 That is correct. I wanted to publish the extension as it was lingering for too long and thus did not want to wait on discussion about "new features" (compared to the 4 runner). Please open a new issue to discuss that change.

[smoyer64]

@TimvdLippe

I haven't had a chance to try out the official MockitoExtension you published yesterday (I'm watching for it in Maven Central but rereading your post perhaps I should be watching Sonatype's OSS-SNAPSHOTS), so I wasn't aware that the version you published was missing this functionality. I just thought it was important to lump these two requests together.

The MockitoExtension prototype that published as a sample by the JUnit 5 team has this feature so I guessed I assumed the official version did too? In any case, I think it's a really useful feature for cases where not every test needs a specific Mock (and the injected Mock has different behaviors if used in more than one test) so I'd be happy to write up a feature request.

I have to admit I was pretty excited to see this released yesterday since it would provide an official way to use Mockito with JUnit 5. Unfortunately we're already using the ParameterResolver from the prototype MockitoExtension so I guess we're in for a bit more of a wait. Let me know if I can help.

[TimvdLippe]

@smoyer64 as I said I am open for adding that feature, but I would like to discuss it first in a separate issue. So please open one with your use-case so we can discuss 😄

[smoyer64]

@TimvdLippe Sorry I wasn't clear in my last comment. I do realize that this issue is closed and went off to write the feature request you asked for (and got side-tracked by an emergency with one of the non-profits I help run). I also looked at all the JUnit 5 related issues that were already posted. Would it be worth having a JUnit 5 label to help organize them?

In any case, I've now added #1348 ... please let me know if additional information is needed.

Thanks for all the hard work you've done to provide official JUnit 5 support in Mockito!

[MarkNBroadhead]

Since this is one of the top Google results for trying to get this working, and I'll likely Google it again, I'd like to add that this is in the mockito-junit-jupiter project. Maven Central link mockito-junit-jupiter