Create annotation that disables all other tests

[nipafx]

#1 contained this idea:

disable all tests in a class except for one (junit-team/junit5#346, proof of concept)

I'm no longer sure why I started working on @DisableOthers, @DisableRest, or @EnableOnly (whatever you want to call it): there seems to be little use for it since IDEs make it easy to run a single tests. Maybe for building from the command line? What other use cases could there be?

But we're here for experimentation, so any half-way believable use case is fine with me. Who wants to do this?

[smoyer64]

See my POC at https://github.com/selesy/junit5-extensions/blob/develop/junit5-extensions-disable-rest-extension/src/main/java/com/selesy/testing/junit5/extensions/disable/rest/DisableRestExtension.java - is there anything of worth in there?

[Bukama]

By @nicolaiparlog in #163, because I think it might be helpful here too:

We just updated to Jupiter 5.5 (#278) to use the new InvocationInterceptor API. If that API gets called for individual parameterized test invocations, it would be a very powerful way to implement disabling parameterized tests. The provided Invocation allows skipping tests and the ReflectiveInvocationContext seems to give access to the actual arguments. Someone should look into that and possibly create an issue if the approach seems promising. 😊

[Michael1993]

InvocationInterceptor absolutely makes this possible, but it seems really tedious.
You intercept every test method call, to check if the test class has a @DisableRest annotation and, if yes, is the current method annotated with @DisableRest (disregarding multiple tests being annotated with @DisabledRest).

This seems really unwieldy (slow?) and there is also the problem of registering this extension. I'd guess you would not really want to register it globally because then if you don't use the annotation in a test class it is just bogs it down.

You mentioned that building from the command line could use this feature - AFAIK, maven's Surefire plugin let's you specify which test class and even which test methods you want to run. Not sure about gradle.
I think overall this extension is not worth the time and effort we would put into it because there are plenty of alternatives.

Worst case scenario, someone is going to comment out every "unnecessary" test in their class.

[Bukama]

Thank you @Michael1993 for investigating.
I don't see the usage for this extension by myself (as IDE and Maven already allows you to just execute one specific method), but it was requested in the JUnit project (they closed their issue, because of this one here) and in our #1.

I also see the issue, that the extension will slow down the test executions, but I think if someone wants to use such a ("strange") feature, he/she has to accept that. Of course it should be mentioned in the docs.

@Michael1993 Can you make a guess about how much effort it would take to implement it? I think if it it's not too much we, as an extension provider, should think about providing it, because the feature requests seems to exists (like RepeatFailingTests), even if we personally don't see the case. But if we do, it must be an manually opt-in, so that only those use it, who wants to use it explicit, but not every project which uses our extension library, because of another extension e.g. the common used Set/ClearSystemProperty

[nipafx]

So here's how I see that...

Use case

IDEs and build tools allow you to run a single test, but that requires developers to use such tools (as opposed to... just a text editor like VS Code?) and understand how to do that. That can be mostly assumed, but there might still be a bit of a use case left where it's just easier/faster to apply the annotation.

I'm afraid, it won't be that easy though - read on.

Implementation

It's straight forward (albeit tedious) to create the extension such that it disables all tests that it extends except the one with the annotation. The interesting bit is that it extends - the extension needs to be called for all tests that it needs to disable, but how can that be done?

It would be possible to, by default, globally register this extension, but then the (possibly expensive) operation to scan all tests for presence of the annotation has to be paid by every Pioneer user - that's a no-go. So what we're left with is to activate the extension globally with some command line option or configuration.

But would this really be easier than to learn how to make the IDE/build tool/whatever run just a single test? I don't think it would be much of a difference.

Verdict

As I see it, the combination of "small use case" and "non-trivial configuration" damns this feature, so I will close this issue.

If you want this feature and think I'm off on one of those two points or missed others, feel free to comment and we can revisit the request.