Disabled Extensions

[nipafx]

There are many possible ideas floating around the @Disabled annotation:

• disable tests by evaluating a script (e.g., JavaScript, Groovy, etc)
  • See @EnabledIf and @DisabledIf in Jupiter 5.1
• disable tests on selected OS (DisabledOnOs #2, Introduce predefined @Disabled*/@Enabled* variants junit-team/junit5#219, proof of concept)
  • See @EnabledOnOs and @DisabledOnOs in Jupiter 5.1
• disable tests on selected Java versions
  • See @EnabledOnJre and @DisabledOnJre in Jupiter 5.1
• system property matches a pattern (comment in #219)
  • See @EnabledIfSystemProperty and @DisabledIfSystemProperty in Jupiter 5.1
• environment variable matches a pattern (comment in #219)
  • See @EnabledIfEnvironmentVariable and @DisabledIfEnvironmentVariable in Jupiter 5.1
• disable all tests in a class except for one (Add possibility to disable all tests in a class except for one junit-team/junit5#346, proof of concept)
  • not yet implemented, but tracked by Create annotation that disables all other tests #71
• current date/time matches some criteria, e.g. @DisabledUntil (comment in #219) ~> Create @DisabledUntil #366
  • See also: @EnabledOnDayOfWeek and @DisabledOnDayOfWeek ~> it's a fun idea, but unlikely to be actually useful
• until a Github/Gitlab (extensible) issue has been closed ~> Disable tests until issue is resolved #367

It is conceivable to create a universal @DisabledIf annotation that covers all of these cases but instead of going there from the outset, it might make more sense to tackle individual use cases first and look for room for generalization later.

[sbrannen]

it might make more sense to tackle individual use cases first and look for room for generalization later.

👍

In case you're not aware of it, Spring 5 milestones already support @EnabledIf and @DisabledIf variants that accept SpEL expressions which make the implementation of custom conditions like @DisabledOnWindows extremely simple.

[nipafx]

Thanks for the hint (links: EnabledIf, DisabledIf), didn't know about that. I'd like to keep dependencies to a minimum but I'll think about it.

Regarding generalizations, I was exploring ideas to get the compiler to check conditions. But I'll start with the simple cases...

[sbrannen]

Thanks for the hint (links: EnabledIf, DisabledIf), didn't know about that.

You're welcome.

I'd like to keep dependencies to a minimum but I'll think about it.

Sure. I wasn't implying that you should re-implement or copy them. Rather, I meant for them to serve as possible inspiration. Although supporting an expression language of any sort in core JUnit would naturally require the EL implementation to be an optional dependency. Anyway, just food for thought.

Regarding generalizations, I was exploring ideas to get the compiler to check conditions.

Can you expound on that?

[nipafx]

Spawned #2 and added as a task to the issue description.

[nipafx]

Spawned #3 to continue discussion about generalized conditions there.

[sormuras]

A DisabledOn("java.version", "^9.*") would be convenient to have here: https://github.com/junit-team/junit5/pull/598/files -- with first parameter pointing to a System property key and the second parameter is a regex pattern to match.

[nipafx]

Good point about the Java version, I added an item to the checkbox list in the issue description.

[jbduncan]

My knowledge of annotations is pretty limited, so I've no idea if what I'm going to suggest is even feasible, but here goes!

How about a @DisabledIf(BooleanSupplier) annotation that disables if the supplier evaluates to true, but doesn't if it evaluates to false?

[nipafx]

@jbduncan: I considered something like that and was halfway done implementing it when I realized that annotations can only use constants as fields. Now I'm toying with the though to name a static field that us a BooleanSupplier and gets evaluated.

[nipafx]

@smoyer64 [wrote]:

We should probably have a discussion about the disabling extensions as I have some that overlap. I started with @enableif and @DisableIf with a predicate as the value and another String as a message template. Then I built everything else on that. I have @EnableIfOs (and disable) in the middle and specific annotations at the end.

This code is not done but I like the generic nature of the extension and how you add new annotations with only their associated classes. I'll get what I have checked into the old selesy project so you can assess it.

That is excellent! As you can see above (particularly here) I started on the specific end but planned to go that very direction. I did some experiments and also tried to support predicates. I failed to find a way around the fact that you can not define a lambda as an annotation argument and found no other solution I immediately liked. I'm curious to see which way you went.

[nipafx]

Just had this idea: An enabled/disabled extension with which you can form Boolean formulas over existing extensions: @DisabledIf(and(EnabledOnLinux.class, DisabledOnJava9.class)).

[smoyer64]

You can't put the code for a lambda in an annotation but you can put the name of a class that implements predicate ... your example is a bit harder as a repeatable annotation would effectively be an or (but you're still using classes to specify the predicates. On Jul 2, 2017 10:04 AM, "Nicolai Parlog" <notifications@github.com> wrote: Just had this idea: An enabled/disabled extension with which you can form Boolean formulas over existing extensions: @DisabledIf(and(EnabledOnLinux.class, DisabledOnJava9.class)). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUCje20kzybW6W1lGSni8Sxqdeojn8Vks5sJ6NcgaJpZM4K3Xjs> .

[smoyer64]

@nicolaiparlog I've committed the POC code I had for @DisabledIf along with a predicate for OS and a meta-annotation (@DisabledIfOperatingSystemIsLinux). This code isn't anywhere near done and it certainly needs some tests written if we decide to go this way but it does allow the user to write new predicates that determine enabling or disabling. As I noted in the commit, @EnableIf is simply the logical negation of the @DisabledIfExtension.

See https://github.com/selesy/junit5-extensions/blob/develop/junit5-extensions-disable-rest-extension/src/main/java/com/selesy/testing/junit5/extensions/disable/rest/os/DisabledIfOperatingSystemIsLinux.java

[ttddyy]

Hi @smoyer64

As @sbrannen mentioned, spring already uses @DisabledIf annotation.
From user's perspective, it would be nice that annotation names don't collide.
If same names are used, users need to check the import statement for which @DisabledIf is used or need to use full package name especially both @DisabledIf annotations are used in single file.

So, just IMO for junit-pioneer, naming @DisabledIf to @DisabledWhen or similar might be a tiny good thing for users.

[sbrannen]

I'm afraid that multiple projects in this world will come up with an annotation named @DisabledIf.

TBH, I tried to come up with a better name for Spring, but in the end I figured @DisabledIf was better than something like @DisabledIfSpel.

So I think the conflicts are unavoidable.

In any case, it's definitely something to keep in mind.

[sbrannen]

@nicolaiparlog, FYI: there's been a very simplistic PoC for a system property condition here for quite some time. That also serves as a PoC for an environment variable condition. Of course, I'd come up with a better name than @SystemProperty if I were to publish it.

[smoyer64]

@ttddyy - I think the name is apropos of its function and don't mind that it collides. There's a large population of JUnit users that doesn't use Spring and some words are just right - even early versions of the JDK had java.util.Date and java.sql.Date.

I've also got a set of temporal conditions that are tentatively using @DisabledBefore, @DisabledAfter and @DisabledBetween.

Unfortunately, naming is hard - I do like the fact that the Spring version accepts EL and that was something I was considering after Java 9 (we should be able to exec code then). I think the real question is whether we can create something that's worth importing into a Spring project!

[sbrannen]

I think the real question is whether we can create something that's worth importing into a Spring project!

Yep.... that will be the real challenge!

Because... with SpEL support in Spring's @EnabledIf, you can basically implement 90% of the conditions proposed in this issue as one-liners, like @EnabledOnMac, which basically just boils down to the following one-liner (if you don't want a custom annotation).

@EnabledIf(expression = "#{systemProperties['os.name'].toLowerCase().contains('mac')}", reason = "Enabled on Mac OS")

[mkobit]

I wonder if something similar to how @ParameterizedTest is structured would be useful here? Then different implementations of enable if and disable if could be implemented by consumers with not too much effort.

Incomplete code below, but would allow for the solution up above (I think) as well as other possible implementations (System property, environment variable, etc.):

public class DisabledIfConditionalExecution implements ExecutionCondition {

  @Override
  ConditionEvaluationResult evaluateExecutionCondition(ExtensionContext context) {
    // ...
  }
}

public @interface DisabledIf {
}

public @interface DisabledIfSource {
  Class<? extends DisabledIfProvider> value();
}

public interface DisabledIfProvider {
  boolean isDisabled(ExtensionContext context);
}

@DisabledIfSource(DisabledIfSpelProvider.class)
@interface DisabledIfSpel {
  String expression();
  String reason();
}

class DisabledIfSpelProvider implements DisabledIfProvider {
  boolean isDisabled(ExtensionContext context) {
    // ...
  }
}

class MyTest {
  @DisabledIfSpel(expression = "#{systemProperties['os.name'].toLowerCase().contains('mac')}", reason = "Enabled on Mac OS")
  void myTest() {
  }
}

[mkobit]

Related to junit-team/junit5#219 as well.

[sormuras]

For the record and some-what related to this initial pioneer issue: there's a new label for "pioneer" ideas at https://github.com/junit-team/junit5/issues?q=is%3Aissue+is%3Aopen+label%3A%223rd-party%3A+pioneer%22 -- tagging issues that may be interesting for @junit-pioneer

[nipafx]

@sormuras Can you take a look at my initial bullet point list and tell us which ideas Jupiter already stole implemented. 😉

[sbrannen]

I'm not @sormuras, but I'll update this issue's description to check off what's already covered. 😉

[sbrannen]

done

[nipafx]

@sbrannen: Thank you for the input. 😺

@ALL: From the three unchecked ones, I consider each fair game, particularly @DisabledUntil. If anyone wants to tackle these, feel free to do so.

[smoyer64]

@nicolaiparlog The conditional disabling of tests was something we were both working in before joining forces - there's a @DisableRest in the portion of selesy/junit-extension I had asked you to review a while ago (https://github.com/selesy/junit5-extensions/tree/develop/junit5-extensions-disable-rest-extension/src/main/java/com/selesy/testing/junit5/extensions/disable/rest). Will this satisfy the "disable all but one" criteria? It's a bit more than that because if you annotate (e.g.) two of ten tests they'll both run.

I should also note that I don't really like the name - Rest is not REST but Roy Fielding has effectively defined the meaning of that word for developers. Perhaps @DisableOthers?

[marcphilipp]

In the Javascript frameworks I‘ve used that‘s usually called only.

[nipafx]

Opened #71 for @DisableOthers/@DisableRest/@EnableOnly/whatever.

[nipafx]

At least the last two items from the opening post still sound interesting to me. If anybody else shares that, PRs are welcome. :)

[nipafx]

@mkobit I never replied to your proposal above. I liked the idea a lot and was about to create an issue for it when I was pointed at Jupiter's custom conditions, which provide all the features your proposal does. Main difference: Jupiter's solution is a bit simpler, but yours is type safe and survives refactorings. I don't think that's enough to warrant our own implementation, so I won't create that issue.

[nipafx]

As I see it, we've covered all ideas in this issue, so we can finally close it. 🥳