Core Principles
Core principles describe the general philosophy we use when considering adding features or making changes to JUnit.
This is a work in progress. There are likely things here that people may disagree with, and we have violated these principles in the past. Still, it's useful having a place to bounce around ideas of how we decided to evolve JUnit.
JUnit is a simple framework for writing repeatable tests of Java code using Java.
- by kcooney
It's better to enable new functionality by creating or augmenting an extension point rather than adding the functionality as a core feature.
- JUnit has never tried to be a swiss army knife
- Third party developers move more quickly than we do
- Once we create an API, we often cannot easily modify it. Third party libraries can make mistakes and fix them because they have fewer users.
We could have added built in support for things like test names and temporary folders. Instead, we created extension points (@Rule and later @ClassRule) and provided the functionality via implementations of those extension points.
This is both an example of what to do and a counter example. We provided parameterized tests via our own Runner. On the one hand, third parties have been able to re-use the Runner interface to provide their own API for specifying test parameters. On the other hand, a test class has one runner, so you can't use Parameterized with one of the Runners provided by Spring.
Instead, we could provide extension points to allow third-party developers to specify a strategy for parameterized tests
- by saff
"If you release it, you're going to maintain it."
Parameterized was never intended to be a first-class citizen of the JUnit interface; I wrote it late in the JUnit 4.0 development cycle as a demonstration that since Runner could be used to enable parameters in tests, we didn't need a built-in, irreplaceable parameter-parsing functionality. I knew from the first that it wasn't full-featured, nor flexible enough for power users.
However, by releasing it in the main JUnit source tree, with docs in the main JUnit javadoc, it was naturally seen by users that "this is how JUnit developers want us to write parameterized tests", and so there's been a constant flow of feature requests to beef up this quick one-off demonstration code, and a surprising number of people see using a third-party package like JUnitParams to be somehow going against the philosophy of JUnit, rather than the real intent all along.
The Theories runner had very similar problems.
I'd strongly recommend that in JUnit 5, the team be very careful with what they release with the package, and to publish a fair amount of even seemingly "core" functionality as written examples to inspire third-party experimentation.
- by saff
Let an extension point be good at what it's good at, and don't be afraid to introduce new extension points to handle weak points in existing ones.
- It's tempting to enable new functionality by extending the interface of an existing extension point, or deprecating an old interface to add a new one.
- However, it can be difficult to retrofit the new interface for happy implementors of the old extension point
- And as an extension point grows more and more flexibility, the ways its methods combine can get quite complicated.
I've been told on multiple occasions that Runner is a failure as an extension point because two Runners cannot be combined. But Runner's primary goal was always that it allowed complete replacement of the functionality of the built-in behavior; there's no well-defined composition of complete replacement.
Instead, we worked to add extension points to many of the already-defined Runners, like Rules (mentioned above), to make it easier to write extensions that want to do "what the built-in behavior does, with one small change", which has more easily defined semantics of composition.
This has an additional advantage, in that custom runners other than the built-in ones can also, if they wish, look for and execute Rules, making Rules re-usable across many Runners.
- by kcooney
Tests often fail. In fact, it's their raison d'être to fail when there are problems. If a test fails when run in one mode (a build tool) but not in another (IDEs) it makes it hard to debug problems and easy to introduce new failures.
- Running tests in an IDE vs a build tool
- Running all classes in a package vs a test suite vs a single class vs a single method
It should be possible to understand how JUnit will treat a class based on reading the test class (and base class) and looking at the annotations.
To quote David Saff:
I have sometimes been in the situation in which someone used a test framework in an extremely clever way that I couldn't find without serious digging. Because of this cleverness, tests failed in ways that looked like they were finding bugs in the production code, or (much worse!) tests passed that should have failed, because it wasn't clear that there was a convention that needed to be followed for the tests to have any meaning at all.
- The quote from David was in a thread about supporting meta annotations
- This has come up when people have proposed adding behavior via package-level annotations
- by saff
JUnit should very, very rarely (if ever!) add dependencies to third-party libraries. Instead, JUnit should, if needed, include extension points that make it easy for those libraries, or (fourth-party?) integrators, to write the glue code
- JUnit has always been a small development group.
- Third-party libraries can be upgraded in breaking ways.
- Keeping up with these changes can suck all of the development group's time.
JUnit added an explicit dependency on Hamcrest in order to effectively offload the development of an assertion library to a different party. Not long after, Hamcrest 1.2 was released, which introduced compile-breaking changes in Matcher typing. Some users were tied to Hamcrest 1.1 because they were using it in their production code. Soon, others were similarly tied to Hamcrest 1.2, for similar reasons. It suddenly became impossible to release a single version of JUnit that would compile for both sets of users.
Sadly, this tight integration was never really necessary; it would have been easy to have simply suggested that users use both packages together, and static import the assertThat method from hamcrest into their JUnit tests.
We have also tended to be very conservative in requiring new JDK versions; there are often good reasons why people need to be able to test and maintain code that requires old JDK versions, and to break that just because there's a new release, or there's a small API improvement that promises cleaner code, is not worth it.
This will be especially tricky as the JUnit Lambda team looks at Java 8 features. On the one hand, Java 8 is a language change potentially as big as Java 5, which prompted the birth of the totally new JUnit 4 API. On the other hand, adoption of Java 8 may be even slower than Java 5, especially on platforms like Android, so unnecessarily requiring Java 8 for features that don't strictly need it will limit the new API's reach, and it may be necessary to eschew nice Java-8-enabled idioms within the JUnit code base even while supporting developers who want to use Java-8-enabled features in their client test code.
User Guide | Javadoc | Issues