How To Write a TCK

[t1]

I have three artifacts:

• api contains some interfaces and a ServiceLoader to load the implementations of the api interfaces.
• ri containing a reference implementation of the api; includes the services files for the ServiceLoader.
• tck a test suite (Test Compatibility Kit) that only has dependencies to the api.

It's easy to run the tests in the tck by adding a runtime dependency on ri. But the tck should be usable by any other implementation as well: I want the ri to load the tck and run it, not the other way around. The tck should be a library of tests that the ri (and any other implementation) includes as a test dependency. The setup required in the ri should be minimal.

I didn't find specific instructions in the User Guide, so I tried TestTemplate and DynamicTests, I searched the internet and looked specifically on StackOverflow, but nothing seemed to be just what I want. Other people use TestNG for such things, but I'd prefer to stay with JUnit.

Before I try the JUnit Platform Launcher API (which was made for a different use-case), I'd like to ask here. If there is a good way that I just can't see, I'd be willing to add a section to the User Guide.

I'd be thankful for any hint.

[jbduncan]

I'm not familiar with more runtime-related mechanisms like ServiceLoader, but a pattern I've used before is the test interface pattern. If you have the ability to write a little extra code in your ri artifact's tests, then you may be able to create a test interface in your TCK artifact like:

package my.tck.package;

public interface MyTck<T extends InterfaceOfRi> {

  T createValue();

  @Test 
  void tckTest1() {
    ...
    assertEquals(..., createValue());
    ...
  }

  @TestFactory
  Stream<DynamicTest> tckTest2() {
    ...
  }

}

And then your ri's tests would implement the interface like this:

class MyRiTests implements MyTck<MyRi> {

  @Override 
  public MyRi createValue() {
    return new MyRi();
  }

}

I've not actually put this code through a compiler, so I don't know if it actually works. And if I wasn't clear enough, perhaps the link will get the point across better.

I hope this helps!

[t1]

I gave it a try, but there are severe drawbacks:

1. As mentioned above, I have to manually aggregate all interfaces; there is no test discovery.
2. The structure of the test interfaces is lost, i.e. all tests are running at the same level.
3. I can't have @Nested classes within test interfaces.

As you can imagine, this makes large test suites quiet unusable. There has to be a better way.

[jbduncan]

Wow. What an interesting problem to have!

I don't suppose your project is open source or is otherwise available to look at, is it? Alternatively, would you be able to give a minimal, reproducible example (MRE) of what your setup looks like? I ask because it's hard to help you out any further without a concrete example to go by.

Regardless, another idea's come to mind - see below. But having written it, it seems awfully convoluted. So if you could provide an MRE of your setup, or the original setup itself, then that would be great.

---

The (terrible?) idea

JUnit 5 has something called @TestTemplate, which I understand is a generic, low-level way of creating tests at runtime.

Ignoring the fact that you use ServiceLoader for a moment, it might be possible to create a test class in your ri artifact that:

1. Imports your tck,
2. Has a test method annotated with @TestTemplate and a "test template context provider" that you provide...
3. ...where your context provider uses ClassGraph to load your TCK's test classes and find the methods indirectly annotated with @Testable (which is something that @Test, @TestTemplate and @TestFactory are themselves annotated with),
4. And then get your context provider to run your RI implementation against these methods.

But this is assuming:

1. Your TCK test classes allow an RI impl to be injected with a constructor or method (in other words, the good old Dependency Injection pattern).
2. Your TCK doesn't have any methods annotated with @BeforeEach et al.

[t1]

I actually do need it for an open source project: https://github.com/t1/smallrye-graphql It's about the TCK in the client. The dependencies are currently the wrong way around, i.e. tck→impl instead of impl→tck.

But I also had built an MRE and just shared it: https://github.com/t1/junit-tck-mre Maybe this is easier to play around with.

To be honest, I'm just glad that I'm not too stupid. Maybe we need a new feature in JUnit, which already scans for test methods. Should I create a corresponding issue?

[jbduncan]

Thanks for showing me the MRE! I can see now that this is more complicated than I thought.

So if I've understood correctly, you want to run an entire test suite N times against an Iterable of objects - one test suite run for each and every object in this Iterable.

I just found these two issues: #871 and #878. They cover "container templates" or "class-level @ParameterizedTests", which are a bit like what you're after, except you're more after "test suite-level @ParameterizedTests".

So I think the best way forward is to raise a new issue, as you suggested.

In the meantime, it sounds like TestNG is the way forward for your TCK tests. It's not unheard of for people to use more than 1 testing framework (I use both JUnit 4 and Kotest at work), so it's not the end of the world. But sadly I can't help you any further there, because I'm not familiar with TestNG. :(

[t1]

I've opened #2594