Mockito vs EasyMock performance

[sergey-morenets]

Hi

I tried to compare the performance of latest versions of Mockito (5.10) and EasyMock (5.2). They use the same technology stack (ByteBuddy, ASM, Objenesis) so the results should be similar. I used the simplest interface for mocking - functional (Callable) and three different mock makers for Mockito (inline, sub-classing and proxy).

Here's configuration for testing:

1. JMH version: 1.36
2. VM version: JDK 21.0.2, Java HotSpot(TM) 64-Bit Server VM, 21.0.2+13-LTS-58
3. 32 GB

Mockito benchmark:

	private Map<String, Callable> map = new HashMap<>();

	@Setup
	public void setup() throws Exception {
		Callable<?> inline = mock(Callable.class, withSettings().mockMaker(MockMakers.INLINE));
		map.put("inline", inline);
		Callable<?> proxy = mock(Callable.class, withSettings().mockMaker(MockMakers.PROXY));
		map.put("proxy", proxy);
		Callable<?> subClassing = mock(Callable.class, withSettings().mockMaker(MockMakers.SUBCLASS));
		map.put("sub-classing", subClassing);
		map.values().forEach(callable -> {
			try {
				when(callable.call()).thenReturn("1");
			} catch (Exception e) {
				e.printStackTrace();
			}
		});

	}

	@Param({"inline", "proxy", "sub-classing"})
	private String mockMaker;

	@Benchmark
	public Object testMockito() throws Exception {
		return map.get(mockMaker).call();
	}

Results:

Benchmark                         (mockMaker)  Mode  Cnt     Score     Error  Units
MockitoBenchmarking.testMockito        inline  avgt    5  4598.860 ± 690.677  ns/op
MockitoBenchmarking.testMockito         proxy  avgt    5  4303.254 ± 257.643  ns/op
MockitoBenchmarking.testMockito  sub-classing  avgt    5  4377.738 ± 476.623  ns/op

EasyMock benchmark:

	Callable mock;

	@Setup
	public void setup() throws Exception {
        	mock = EasyMock.mock(Callable.class);

		EasyMock.expect(mock.call()).andReturn("1").times(1, Integer.MAX_VALUE);
		EasyMock.replay(mock);
	}

	@Benchmark
	public Object testEastMock() throws Exception {
		return mock.call();
	}

Results:

Benchmark                         Mode  Cnt   Score   Error  Units
MockitoBenchmarking.testEastMock  avgt    5  46.058 ± 0.443  ns/op

So you see that EasyMock shows 100x less average execution time. And that's very strange. Can you please check my benchmarks? Have I done something wrong? Or how can you explain that?

[TimvdLippe]

We have had performance questions before, but those were focused on ByteBuddy. We didn't discuss the similar stack thus far. @raphw any immediate ideas as to where this discrepancy would come from?

[raphw]

Have you tried removing the map and write three wxplicit benchmarks?

[sergey-morenets]

Hi @raphw

Yes, I created new benchmark that tested INLINE mock maker only:

Callable mock;

	@Setup
	public void setup() throws Exception {
		 mock = mock(Callable.class, withSettings().mockMaker(MockMakers.INLINE));
		 when(mock.call()).thenReturn("1");
	}
	
	@Benchmark
	public Object testMockitoInline() throws Exception {
		return mock.call();
	}

Results:

Benchmark                              Mode  Cnt     Score     Error  Units
MockitoBenchmarking.testMockitoInline  avgt    5  4428.439 ± 356.722  ns/op

[raphw]

One would need to dig down here, but I'd assume there's a volatile field involved or the like. I don't know the easy mock implenentation, but is it subclass based?