Continuous Delivery Details

Mockito champions true continuous delivery in the Open Source.

• Introduction
• Continuous Delivery
• Versioning
• Why 2.1.0 and not 2.0.0?
• Lessons learned
• Too many beta versions!

Introduction

For motivation see Continuous Delivery Overview page.

Mockito implements continuous delivery since 2014. In early 2017 the team decided to change the release model, for motivation and the design spec see Continuous Delivery Pipeline 2.0.

Continuous Delivery allows interested parties to access to the latest versions including betas, thus allowing a drastically shortened feedback cycle. Examples of quick feedback cycles are issues like #360, #335 or #237. All these issues pointed to specific beta versions which allowed us to track down very quickly which pull request or commit was the culprit.

Continuous Delivery

Mockito Continous Delivery Pipeline is implemented using Shipkit library and configured in "shipkit.gradle" file. The details are described in README.md file.

Versioning

Before the introduction of the continuous delivery drone, the team intuitively followed the semver scheme. Upon the the introduction of the introduction of the continuous delivery, the version scheme was the same. The 1.10.x line, could follow semver x being the build number :

1.10.<build number>

That leads to the following qestion...

Why artifact 2.0.0 does not exist and 2.1.0 is the first release of Mockito 2.x ?

tldr: the first release of Mockito 2.X is 2.1.0 and you can download this artifact from maven central.

Origin of the problem

To answer this question, we need to go back in December 2014. With this commit the version scheme didn't change, the beta version was just marked with the beta marker (label) to indicate Mockito 2.x beta builds had started.

This lead to versions like :

2.0.<build number>-beta

Indeed the official semver definition also states that

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

While this notion is seemingly innocent, we were now unable to publish Mockito 2.0.0: maven and gradle with version ranges would fetch the very latest beta release instead of our official release! Moreover, maven-central indicated that, while 2.0.0 was published later, it was superseded in various patch updates. e.g.

1.10.18 < 1.10.19 < 2.0.0-beta < 2.0.0 < 2.0.1-beta < 2.0.111-beta

It took us quite some time to address that problem during the 2.x beta phase. But we did :

Resolution

There were multiple workarounds to this problem (see #595).

• The first workaround considered was unpublishing the beta releases of Mockito. However, the recent problems of such an invasive action with left-pad in the NPM ecosystem (read more) made clear that this was not a reasonable solution. Removing beta releases would break a significant number of projects, which is unacceptable.

• The second solution considered was to publish 2.0.0 anyways and describe the version caveat clearly in the release notes and in the upgrade guide. While this is certainly possible, the issue of maven central indicating the incorrect latest version remains. Moreover, end-users might not read the notes or guide and download the incorrect beta version thinking of it as next verion after 2.0.0.

• Lastly we considered skipping releasing Mockito 2.0.0 and instead jump to 2.1.0 immediately.

  This is the solution Mockito team chose.

  While this is certainly confusing at first, it does not break applications of end-users and fixes the issue with maven central.

The chosen solution also tweaked the version to append the build number after the beta label. e.g.

1.10.19 < 2.0.0-beta < 2.0.1-beta < 2.0.111-beta < 2.1.0-beta.112 < 2.1.0 < 2.2.0.beta.1 < 2.2.0 < ...

Lessons learned

• If you are employing continuous delivery and publishing beta builds or wishes to set it up on your project, make sure to not increment the patch version. Instead, append the increment after the tag: 2.0.0-beta.15. If you realized at this point you are using a problematic version scheme, do not remove the published artifacts. Look for a solution that is the least invasive to the end-users. This might differ from package-manager ecosystem and individual libraries/frameworks.

• When you get the version scheme and continuous delivery correct, the benefits are very valuable. The feedback cycle is greatly shortened and debugging an issue can be done a lot quicker. End-users or contributors can easily bisect an issue by switching artifacts rather easily, instead of building source and using git bisect. This lowers the bar for end-users resulting in more and precise feedback.

Too many beta versions!

Publishing every single beta version caused confusion. Due to life priorities the active work on Mockito 2 was unfortunately intermittent. This caused prolonged beta cycle with hundreds of beta versions in Maven Central. This caused confusion and issue #595 discusses this problem at length. The issue will addressed by #618.