Strictness configurable per mock / stubbing #1272

mockitoguy · 2017-12-11T00:20:52Z

Fixes #792

Problem

Today we have strict stubs feature, a key enabler for cleaner tests and improved debuggability. Strict stubs don't work well with "common stubbing" pattern. They also don't like when we call stubbed methods with unexpected arguments - even though sometimes it is legit. For more, see #792

Solution

@Test public void demo() {
  //1. "lenient()" method on MockSettings interface:
  Foo mock = Mockito.mock(Foo.class, withSettings().lenient());

  //2. "lenient()" method on Mockito class:
  Mockito.lenient().when(mock.foo(1)).thenReturn(1);
  Mockito.lenient().doReturn(1).when(mock).foo(1);
}

For more, see #792

Public API changes

New methods:
- Mockito#lenient()
- MockSettings#lenient()
- MockCreationSettings#isLenient()
- Stubbing#getStrictness()
New interfaces:
- BaseStubber - extracted out from existing "Stubber" interface, so that we can reuse API with "LenientStubber"
- LenientStubber - instance returned by "Mockito.lenient()"
Other changes:
- Made @org.mockito.NotExtensible annotation public so that we can use it in subpackages

codecov-io · 2017-12-11T00:27:53Z

Codecov Report

Merging #1272 into release/2.x will decrease coverage by 0.07%.
The diff coverage is 91.86%.

@@                Coverage Diff                @@
##             release/2.x    #1272      +/-   ##
=================================================
- Coverage          88.62%   88.54%   -0.08%     
- Complexity          2358     2391      +33     
=================================================
  Files                292      296       +4     
  Lines               5950     6005      +55     
  Branches             719      727       +8     
=================================================
+ Hits                5273     5317      +44     
- Misses               497      507      +10     
- Partials             180      181       +1

Impacted Files	Coverage Δ	Complexity Δ
src/main/java/org/mockito/quality/Strictness.java	`100% <ø> (ø)`	`1 <0> (ø)`	⬇️
src/main/java/org/mockito/junit/MockitoJUnit.java	`66.66% <ø> (ø)`	`2 <0> (ø)`	⬇️
.../exceptions/misusing/PotentialStubbingProblem.java	`100% <ø> (ø)`	`1 <0> (ø)`	⬇️
...eptions/misusing/UnnecessaryStubbingException.java	`100% <ø> (ø)`	`1 <0> (ø)`	⬇️
...mockito/internal/invocation/UnusedStubsFinder.java	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
...mockito/internal/stubbing/OngoingStubbingImpl.java	`91.66% <100%> (+1.66%)`	`6 <1> (+1)`	⬆️
...rc/main/java/org/mockito/internal/MockitoCore.java	`97.75% <100%> (+0.05%)`	`35 <2> (+2)`	⬆️
...ckito/internal/stubbing/DoAnswerStyleStubbing.java	`100% <100%> (ø)`	`6 <6> (?)`
.../mockito/internal/junit/UnusedStubbingsFinder.java	`100% <100%> (ø)`	`7 <0> (ø)`	⬇️
.../org/mockito/internal/handler/MockHandlerImpl.java	`100% <100%> (ø)`	`12 <2> (ø)`	⬇️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 940e9ac...0fcc2a3. Read the comment docs.

Stumbled upon it when working on #1272

Informed that 'silent' mode is effectively 'lenient' Strictness. Stumbled on it when working on #1272

TimvdLippe

I like the idea to have an opt-out of the strict stubbing, especially if you just have 1 method. However, the current implementation is very brittle and it seems to introduce quite some duplicate code as well.

For example, the Strictness.LENIENT is scattered throughout the code. This make it very hard to expand later on. Rather, I would add logic to strictness with methods that based on the type determine what they should do. This makes the code a lot more maintainable.

Moreover, I see null used a lot of times, which is unclear to me what the usefulness is.

TLDR: Yay for the feature, but the implementation needs quite some work still.

As a cross-point with JUnit5: I would actually say that we still want method level lenient stubbing, thus these two features can co-exist in my opinion.

mockitoguy · 2018-01-28T16:31:25Z

As a cross-point with JUnit5: I would actually say that we still want method level lenient stubbing, thus these two features can co-exist in my opinion.

Can you provide a use case (test case) that demonstrates the value? You can also reply to the long email thread we had about it on Mockito mailing list :)

mockitoguy · 2018-01-28T16:35:12Z

Moreover, I see null used a lot of times, which is unclear to me what the usefulness is.

Can you be more specific what nulls / are you not clear about? The code changes are light on the javadoc and comments, apologies! Thank you for review!!!

mockitoguy · 2018-01-28T16:41:43Z

For example, the Strictness.LENIENT is scattered throughout the code. This make it very hard to expand later on. Rather, I would add logic to strictness with methods that based on the type determine what they should do. This makes the code a lot more maintainable.

Strictness is a public type and we want to be very judicious adding methods to it. I'm very curious about your idea, though! Perhaps we can put those methods on some new internal Strictness object? Let me try to work on this and get back. Great feedback!

mockitoguy · 2018-02-05T01:40:35Z

Hey @mockito/developers, any feedback?

@TimvdLippe, I've addressed portion of your feedback. Can you clarify the other portion (see my questions to your questions :).

TimvdLippe

Mostly the usage of StubbedInvocationMatcher with null remains very brittle. I think we should handle all fallbacks of strictness in a type-safe manner. Preferably by defining a default strictness somewhere and update existing code to use this logic instead.

This is from a quick overview, I do not have the time to do a full code review sadly

TimvdLippe · 2018-02-05T13:49:29Z

src/main/java/org/mockito/internal/stubbing/InvocationContainerImpl.java


    private MatchableInvocation invocationForStubbing;

    public InvocationContainerImpl(MockCreationSettings mockSettings) {
        this.registeredInvocations = createRegisteredInvocations(mockSettings);
+        this.mockStrictness = mockSettings.isLenient() ? Strictness.LENIENT : null;


We assign null here and use it throughout the code. I still think this is brittle, as it is really easy to miss. Moreover, in addConsecutiveAnswer null is passed in and in addAnswer again null is propagated. Rather, we should define a default somewhere and use this instead. That way, this field is never null and we always know what "not specifying a strictness for this method" actually means (e.g. fallback to mockStrictness)

Thank you for feedback! I absolutely agree with you on the nulls.

When implementing this feature, I tried to have a default Strictness. This would require us to add new public value to the enum, for example: Strictness.UNSPECIFIED. I decided against it because it's adding a new awkward value to the public API.

Do you think we should add Strictness.UNSPECIFIED? What would that mean to the user? (e.g. how would we document it?).

I would add a constant static attribute in Strictness like public static final Strictness DEFAULT = Strictness.LENIENT; (or whatever value we decide on).

For mock objects and stubbings, the default Strictness is "unspecified", e.g. it will be inherited from MockitoSession or JUnit rule. The question is whether we want to add Strictness.UNSPECIFIED to the public API?

I think we should extract this discussion into an issue, as it seems that this is a more fundamental issue with Strictness. Since we are using it more and more and it is configurable in multiple places, maybe we should extract all this logic into a separate class? In this way, you would make a chain of configurations, where we take the "most local" definition. Anyways, let's create an issue with the requirements so that we can tinker.

There is already an issue: #792 - please write comment there with your idea! What do you mean about extracting it into a separate class?

@mockitoguy Filed #1305 of what I meant and to hopefully address this issue.

I have been trying to incorporate #1305 in this PR, but sadly Strictness is used in many places in the codebase. I feel like the Strictness feature as a whole has had more impact on our overall architecture than I would have liked. For now I am okay with these changes, but I think that we should revisit the whole Strictness story in a later stadium to see if we can reduce duplication and boilerplate code (such as the LenientStubber)

I agree. Thank you for giving it a shot!

I'll fix the conflicts so that we're ready to merge.

TimvdLippe

I am okay with this change. Please address the left-over TODO's and add a new section in the Mockito.java javadoc. Then you can merge this PR 👍

TimvdLippe · 2018-03-02T16:00:20Z

src/main/java/org/mockito/internal/junit/UnusedStubbings.java


-            Reporter.unncessaryStubbingException(invocations);
+        List<Invocation> invocations = new LinkedList<Invocation>();
+        for (Stubbing stubbing : unused) {


Java 8 will make this so much nicer, but this is already an improvement 👍

TimvdLippe · 2018-03-02T16:01:01Z

src/main/java/org/mockito/internal/stubbing/DefaultLenientStubber.java

+import org.mockito.stubbing.OngoingStubbing;
+import org.mockito.stubbing.Stubber;
+
+public class DefaultLenientStubber implements LenientStubber {


Unfortunate that we require all this boilerplate here. Would be great to have a future improvement to get rid of that. Not blocking though.

TimvdLippe · 2018-03-02T16:02:10Z

src/main/java/org/mockito/mock/MockCreationSettings.java

@@ -109,4 +109,7 @@
     */
    @Incubating
    Object getOuterClassInstance();
+
+    @Incubating//TODO x javadoc


Please fix this TODO

TimvdLippe · 2018-03-02T16:02:52Z

src/main/java/org/mockito/stubbing/Stubbing.java

@@ -50,4 +52,7 @@
     * @since 2.2.3
     */
    boolean wasUsed();
+
+    @Incubating//TODO x javadoc


Please fix this TODO

TimvdLippe · 2018-03-02T16:03:27Z

src/test/java/org/mockito/internal/stubbing/InvocationContainerImplStubbingTest.java

@@ -46,7 +46,7 @@ public void setup() {
    public void should_finish_stubbing_when_wrong_throwable_is_set() throws Exception {
        state.stubbingStarted();
        try {
-            invocationContainerImpl.addAnswer(new ThrowsException(new Exception()));
+            invocationContainerImpl.addAnswer(new ThrowsException(new Exception()), null);


Would be great to overload addAnswer, to not having to modify all tests.

TimvdLippe · 2018-03-02T16:04:54Z

src/test/java/org/mockitousage/strictness/StrictnessPerMockTest.java

+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.withSettings;
+
+//TODO 792 also move other Strictness tests to this package (unless they already have good package)


I am not really sure what you mean with this TODO. Is this something we need to address before or after merging this PR?

mockitoguy · 2018-03-05T15:32:14Z

Please address the left-over TODO's and add a new section in the Mockito.java javadoc. Then you can merge this PR 👍

Absolutely! Thank you for review!

mpkempson · 2018-06-26T10:25:21Z

Hey guys, any ideas when this will be ready to merge in and released? Im just poking because we're having the exact issue this would resolve at our work.

Thank you in advance!

mockitoguy · 2018-06-26T21:58:44Z

@mpkempson, thank you for upvoting - that's what I needed! I'll work on this shortly.

Implemented basic support for setting strictness per stubbing. Refactorings and more test coverage is pending. Some features not supported yet.

We don't need Strictness per mock just yet. Let's offer our users simple 'lenient()' setting on the mock.

Based on code review feedback.

Avoided unnecessary call to 'mockingDetails' to avoid reflective calls. Added new argument to the listener - it's ok because we're planning to make the listener public - doing so requires to move all parameters to an 'event' object.

marchpig · 2018-07-24T15:41:16Z

src/main/java/org/mockito/internal/stubbing/DoAnswerStyleStubbing.java

+        this.answers.addAll(answers);
+    }
+
+    boolean isSet() {


Is there any special meaning of this method name?
I think isEmpty() is clear.

mockitoguy mentioned this pull request Dec 11, 2017

New lenient() strictness setting available at mock and at stubbing level #792

Closed

mockitoguy added a commit that referenced this pull request Jan 4, 2018

Added an extra test case

7f40c34

Stumbled upon it when working on #1272

mockitoguy mentioned this pull request Jan 4, 2018

Extra test case, javadoc update #1282

Merged

mockitoguy added a commit that referenced this pull request Jan 4, 2018

Added an extra test case

1e5ab2a

Stumbled upon it when working on #1272

mockitoguy added a commit that referenced this pull request Jan 5, 2018

Updated documentation

83aea05

Informed that 'silent' mode is effectively 'lenient' Strictness. Stumbled on it when working on #1272

mockitoguy force-pushed the fine-strictness branch from 411865d to 6d54766 Compare January 8, 2018 08:27

mockitoguy mentioned this pull request Jan 28, 2018

MockitoExtension for JUnit5 #1221

Merged

TimvdLippe requested changes Jan 28, 2018

View reviewed changes

TimvdLippe reviewed Feb 5, 2018

View reviewed changes

TimvdLippe mentioned this pull request Feb 6, 2018

Implement reusable Strictness ordering #1305

Open

mockitoguy mentioned this pull request Feb 18, 2018

Fix several static analysis warnings as reported by IntelliJ #1316

Closed

mockitoguy force-pushed the fine-strictness branch from 546c32b to 00efcf1 Compare February 25, 2018 05:43

TimvdLippe requested changes Mar 2, 2018

View reviewed changes

TimvdLippe mentioned this pull request Apr 12, 2018

Strict Stubbing and Overloaded Methods #1353

Closed

mockitoguy self-assigned this Jul 4, 2018

mockitoguy force-pushed the fine-strictness branch 2 times, most recently from 1c25d1a to dec1194 Compare July 21, 2018 13:41

mockitoguy added 5 commits July 23, 2018 21:11

Experimenting with Strictness per mock

18456b7

Progress on strictness per mock/stubbing

ab2e7eb

Implemented basic support for setting strictness per stubbing. Refactorings and more test coverage is pending. Some features not supported yet.

Reduced duplication in test

876d458

Improved the test case

f00f140

Added handy test utility

218296a

mockitoguy added 24 commits July 23, 2018 21:11

Added TODO

d7cf98e

Added coverage

ec24ce1

Refactoring and tidy up

b9bf63e

Removed a TODO after verification, added documentation

61403ea

Added a TODO

2edf0ce

Added more sanity coverage

d4d12ab

Refactoring: placed stubbing evaluation code in the same spot

8150867

Added more coverage for edge cases

8e23747

Simplified public API for now

08a9b9c

We don't need Strictness per mock just yet. Let's offer our users simple 'lenient()' setting on the mock.

Checkstyle

7bcc548

Added TODOs to help organization

da0481f

Reduced conditional duplication

d777650

Based on code review feedback.

Added Javadoc

d60108d

Updated strictness documentation

99f6610

Documented new Mockito.lenient() method

8f44b3b

Updated since tags and documentation

5bebc8f

Updated MockitoRule documentation

a837411

Documented feature in the main manual in Mockito Javadoc

cf434ab

Performance tweak

2787696

Avoided unnecessary call to 'mockingDetails' to avoid reflective calls. Added new argument to the listener - it's ok because we're planning to make the listener public - doing so requires to move all parameters to an 'event' object.

Fixed problem with the stubbings order

bef4d86

Rename job for consistency

f30ec35

Finalized the documentation

cf62c73

Bumped version

e51245a

Made documentation consistent.

0fcc2a3

mockitoguy force-pushed the fine-strictness branch from 79d3346 to 0fcc2a3 Compare July 24, 2018 04:11

mockitoguy merged commit d2145dd into release/2.x Jul 24, 2018

marchpig reviewed Jul 24, 2018

View reviewed changes

TimvdLippe deleted the fine-strictness branch October 9, 2018 15:51

dalewking mentioned this pull request Dec 3, 2018

Add way to suppress MockitoHint messages for a particular stub call #1527

Open

thevoiceless mentioned this pull request Feb 20, 2019

Add support for per-mock strictness mockito/mockito-kotlin#325

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strictness configurable per mock / stubbing #1272

Strictness configurable per mock / stubbing #1272

mockitoguy commented Dec 11, 2017 •

edited

codecov-io commented Dec 11, 2017 •

edited

TimvdLippe left a comment

mockitoguy commented Jan 28, 2018

mockitoguy commented Jan 28, 2018

mockitoguy commented Jan 28, 2018

mockitoguy commented Feb 5, 2018

TimvdLippe left a comment

TimvdLippe Feb 5, 2018

mockitoguy Feb 5, 2018

TimvdLippe Feb 5, 2018

mockitoguy Feb 5, 2018

TimvdLippe Feb 5, 2018

mockitoguy Feb 5, 2018

TimvdLippe Feb 6, 2018

TimvdLippe Feb 17, 2018

mockitoguy Feb 18, 2018

TimvdLippe left a comment

TimvdLippe Mar 2, 2018

TimvdLippe Mar 2, 2018

TimvdLippe Mar 2, 2018

TimvdLippe Mar 2, 2018

TimvdLippe Mar 2, 2018

TimvdLippe Mar 2, 2018

mockitoguy commented Mar 5, 2018

mpkempson commented Jun 26, 2018

mockitoguy commented Jun 26, 2018

marchpig Jul 24, 2018

Strictness configurable per mock / stubbing #1272

Strictness configurable per mock / stubbing #1272

Conversation

mockitoguy commented Dec 11, 2017 • edited

Problem

Solution

Public API changes

codecov-io commented Dec 11, 2017 • edited

Codecov Report

TimvdLippe left a comment

Choose a reason for hiding this comment

mockitoguy commented Jan 28, 2018

mockitoguy commented Jan 28, 2018

mockitoguy commented Jan 28, 2018

mockitoguy commented Feb 5, 2018

TimvdLippe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TimvdLippe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mockitoguy commented Mar 5, 2018

mpkempson commented Jun 26, 2018

mockitoguy commented Jun 26, 2018

Choose a reason for hiding this comment

mockitoguy commented Dec 11, 2017 •

edited

codecov-io commented Dec 11, 2017 •

edited