API for registering custom plugins

[tymfear]

Initially asked it in #2583 and also discussed in #2641
So how would the community like to have an API for registering custom plugins with hooks called on test lifecycle events. E.g.

class Reporter {
  onBeforeStart(t: Context);
  onBeforeEnd(t: Context);

  onBeforeEachStart(t: Context);
  onBeforeEachEnd(t: Context);

  onAssertionFailure(t: Context);
  
  onTestStart(t: Context);
  onTestEnd(t: Context);

  onAfterEachStart(t: Context);
  onAfterEachEnd(t: Context);

  onAfterStart(t: Context);
  onAfterEnd(t: Context);
}

I believe that would make it easy to build custom logic around AVA, especially, creating custom reports, like Allure, or generating screenshot for failed tests (or even assertions), and so on.

[novemberborn]

I think it'd be great for AVA to be compatible with things like Allure, yes.

afterEach.always() hooks can be used to generate screenshots, but I suspect you're also hoping to reference them from the reports?

Could you expand on how you'd use this "reporter" plugin? How would it behave? How would it integrate with AVA's output?

[novemberborn]

I've been a bit busy, still meaning to get back to you on this @tymfear!

[novemberborn]

How do you extract the output from that?

If you could instead decorate a test() chain so you get those callbacks, how would that work for you? That's what I started to experiment with in #2435.

[novemberborn]

Also regarding screenshots, if you remember, I fixed the passed field in the teardown hooks, specifically for the purpose of taking screenshots (I believe it's more convenient that way)

Ah yes!

[tymfear]

How do you extract the output from that?

If you could instead decorate a test() chain so you get those callbacks, how would that work for you? That's what I started to experiment with in #2435.

The output is generated by the allure itself. So there is everything ready to use, just would be nice to have a single place of definitions for those callbacks. Probably, I can do that even now by introducing a set of macros that will be wrapping tests and other lifecycle hooks. But that's not quite re-usable in sense of introducing a library that could be just installed and plugged in. At least in my current understanding - got to experiment on that.

Regarding the new test interface - do I understand it correctly, that it would be pretty much they same approach that https://github.com/microsoft/folio provides?

[novemberborn]

The output is generated by the allure itself.

Yes, but does it end up on stderr? A file?

Regarding the new test interface - do I understand it correctly, that it would be pretty much they same approach that microsoft/folio provides?

Conceptually yes. Thanks I hadn't seen that before. Though I think we'd be a bit lower level in terms of API. I suppose you could build something like Folio on top of it.